Voice Live API support clarification for RAG + Avatar

Implement RAG as a backend tool invoked from the active Voice Live session, return retrieved passages and source metadata to Voice Live, and configure Voice Live’s integrated avatar output for synchronized audio and video.

Native capabilities versus application responsibilities

Voice Live does not itself become the enterprise retrieval index. In the documented RAG flow, the application retrieves content, augments the model input, and generates the grounded answer; indexes can store citation fields such as title, URL, and file name.^1^

Recommended request flow

1. Register a function such as search_enterprise_content in the Voice Live session.
2. When Voice Live requests that function, your backend queries Azure AI Search/vector search or calls your existing RAG API.
3. Return compact JSON containing passages plus stable source identifiers.
4. Send the function result back to the same session and request model response generation.
5. Persist the returned source metadata against the response/turn ID for UI citations.
6. Configure avatar in the Voice Live session and establish the avatar WebRTC connection.

Voice Live function calling is specifically designed for invoking external backends during a conversation. The application handles the function-call event, executes the function, sends its output back, and triggers the response.^2^

A representative tool result should preserve citation data rather than returning only flattened text:

{
  "answer_context": [
    {
      "text": "Retrieved passage...",
      "source_id": "policy-123",
      "title": "Travel Policy",
      "url": "https://example.contoso/policy/123",
      "last_updated": "2026-11-01"
    }
  ]
}

Because the function executes when requested, subsequent questions can query the latest searchable index state—even within the same Voice Live session. Whether an update is visible immediately depends on completion of your ingestion/indexing process, not conversation state.

For custom tool results, maintain the authoritative mapping between response, passage, and source in your application. The API also defines response.audio_transcript.annotation.added for annotations such as citations produced by web or file search, but its schema depends on the annotation source; this should not be assumed to create citations automatically for arbitrary custom-tool JSON.^3^

Avatar and audio transport

Voice Live natively supports real-time avatars synchronized with audio. Configure the avatar parameter, perform SDP negotiation using session.avatar.connect, and render the returned stream. The integrated path returns synchronized avatar and audio; your application does not need to stream generated Voice Live audio into a separate Avatar API.^4^

Avatar streaming is WebRTC-based by default, with video, animation, and blendshape support.^5^ For ordinary non-avatar browser/mobile audio, WebRTC is also recommended over WebSocket for latency and network resilience.^6^

Models, resources, regions, and quotas

• Use a Microsoft Foundry resource for full functionality. An Azure Speech in Foundry Tools resource does not support Foundry Agent Service integration or BYOM.
• Voice Live supports GPT-Realtime, GPT-5, GPT-4.1, GPT-4o, Phi, and additional listed models. Availability differs by region and processing scope; select from the current model-region matrix rather than assuming every model is available in every Voice Live region.^7^
• Choose a region satisfying all three requirements: Voice Live speech features, the selected LLM, and real-time avatar. Their regional matrices are separate.^8^
• Standard S0 Voice Live limits include 100 new connections per minute and a maximum 60-minute session. Voice Live avatars also inherit real-time avatar limits, including two new connections per minute, 30 minutes speaking, and five minutes idle. Adjustable quota requests are available for specified S0 limits.^9^

Do not select a model solely by model family: verify that the chosen model, API version, region, and tool-calling mode support the required combination.

Identity and private networking

Voice Live supports Microsoft Entra ID for a Microsoft Foundry resource. Assign Cognitive Services User and Foundry User to the calling user or managed identity, request a token for https://ai.azure.com/.default (or the documented legacy scope), and pass it as a Bearer [Token]^4^

For the retrieval tier, Azure AI Search supports managed identity, RBAC, firewalls, and private endpoints.^10^ Place the RAG backend in a network that can resolve and reach the Search private endpoint.

The cited Voice Live documentation establishes Entra ID and managed-identity authentication but does not establish end-to-end Voice Live Private Endpoint/VNet support. Confirm that requirement for the exact resource type, endpoint mode, region, and API version before treating Voice Live itself as privately reachable. Also account for WebSocket/WebRTC, ICE/STUN/TURN, DNS, and outbound identity endpoints in the network design.

Samples and reference starting points

There is no single documented sample here combining Voice Live, Azure AI Search RAG, citations, and Avatar end to end. Use these two official starting points:

• aisearch-openai-rag-audio: voice-to-RAG with Azure AI Search and full-duplex real-time audio.^11^
• The Voice Live JavaScript avatar sample referenced by the Voice Live avatar guide.^4^

Combine them at the orchestration boundary: retain the RAG backend/tool contract and citation store from the first, and replace its output presentation with the integrated Voice Live avatar session.

References

1. Retrieval augmented generation (RAG) and indexes
2. How to use function calling in a Voice Live session (programming-language-python)
3. Voice Live 2026-04-10 API Reference
4. How to use the Voice Live API
5. Voice Live 2025-10-01 API Reference
6. Voice Live API with WebRTC
7. Voice Live API for real-time voice agents
8. Supported regions for Azure Speech
9. Quotas and limits for Azure Speech
10. Configure your Azure AI Search service in the Azure portal
11. Python samples for Azure AI Search