Selecting the number of chunks to use as context when doing RAG?

Bao, Jeremy (Cognizant) 110 Reputation points
2024-03-18T21:51:14.91+00:00

When you are using the Python SDK for Azure OpenAI to generate text based on user input and RAG, how do you specify the number of chunks of text to use as context?

I read that it involves some parameter called "topNDocuments," but where does that go? It is not recognized as an argument to client.chat.completions.create().

Azure OpenAI in Foundry Models
Answer accepted by question author
Charlie Wei 3,335 Reputation points
2024-03-19T05:29:55.88+00:00

Hello Bao, Jeremy (Cognizant),

Regarding the topNDocuments you mentioned, please refer to this document and try the following example code.

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint=...,
    api_key=...,
    api_version="2024-02-01",
)

completion = client.chat.completions.create(
    model=...,
    messages=[...],
    extra_body={
        "data_sources": [
            {
                "type": "azure_search",
                "parameters": {
                    "endpoint": "<search_endpoint>",
                    "index_name": "<search_index>",
                    "authentication": {
                        "type": "system_assigned_managed_identity"
                    },
                    "top_n_documents": 5
                }
            }
        ]
    }
)

Best regards,
Charlie

If you find my response helpful, please consider accepting this answer and voting 'yes' to support the community. Thank you!

Was this answer helpful?

1 person found this answer helpful.

0 additional answers

Sort by: Most helpful
Most helpful Newest Oldest

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.