How to add a vectorizer for a search index in C#

Question

How to add a vectorizer for a search index in C#

Thibault Verlinde 120

Hiya

I'm currently working on a project that indexes a certain amount of data. This goes flawlessly when I create the index, with vectors, inside the portal.
Whenever I create such index with a C# program, it succeeds, but without the vectorizer. There isn't also any info on how/where you add a Vectorizer inside the new index with that code.

Thanks in advance!

EDIT: I found the answer myself, but for some reason i can't accept my own answer. See the answer posted by me to find the solution.

My code:

//Initialisation of new index
var indexName = _srchClient.IndexName;
var vectorSearchConfigName = "vectorconfig-" + _options.ProjectName;
var vectorSearchProfileName = "vectorprofile-" + _options.ProjectName;
//var semanticConfigName = "semanticconfig-" + _options.ProjectName;
var index = new SearchIndex(indexName)
{
    VectorSearch = new()
    {
        Algorithms =
        {
            new HnswAlgorithmConfiguration(vectorSearchConfigName)
        },
        Profiles =
        {
            new VectorSearchProfile(vectorSearchProfileName, vectorSearchConfigName)
        }
    },
    Fields =
    {
        new SimpleField("id", SearchFieldDataType.String) { IsKey = true },
        new SearchableField("content") { AnalyzerName = LexicalAnalyzerName.StandardLucene },
        new SimpleField("sourcePage", SearchFieldDataType.String) {IsFacetable = true},
        new SimpleField("sourceFile", SearchFieldDataType.String) {IsFacetable = true},
        new SearchField("embedding", SearchFieldDataType.Collection(SearchFieldDataType.Single))
        {
            VectorSearchDimensions = 1536,
            IsSearchable = true,
            VectorSearchProfileName = vectorSearchProfileName
        }
    },
    //When using free tier, this can't be used!
    //SemanticSearch = new()
    //{
    //    Configurations =
    //    {
    //        new SemanticConfiguration(semanticConfigName, new()
    //        {
    //            ContentFields =
    //            {
    //                new SemanticField("content")
    //            }
    //        })
    //    }
    //}
};
// Create a new search index
await _srchIndexClient.CreateIndexAsync(index);

0 comments

Answer accepted by question author

Azar 31,725 MVP Volunteer Moderator

Hi again Thibault Verlinde

Thats awesome finding the solution, so let me paste your solution below as you cant accept your own answer, accept the answer and this helps the other community members who face the same problem. Thanks very much for the contrubution.

This is the updated code:

//Initialisation of new index
var indexName = _srchClient.IndexName;
var vectorSearchConfigName = "vectorconfig-" + _options.ProjectName;
var vectorSearchProfileName = "vectorprofile-" + _options.ProjectName;
var vectorizerName = "vectorizer-" + _options.ProjectName;
//var semanticConfigName = "semanticconfig-" + _options.ProjectName;
var index = new SearchIndex(indexName)
{
    VectorSearch = new()
    {
        Algorithms =
        {
            new HnswAlgorithmConfiguration(vectorSearchConfigName)
        },
        Profiles =
        {
            new VectorSearchProfile(vectorSearchProfileName, vectorSearchConfigName)
            {
                Vectorizer = vectorizerName
            }
        },
        Vectorizers =
        {
            new AzureOpenAIVectorizer(vectorizerName)
            {
                AzureOpenAIParameters = new AzureOpenAIParameters()
                {
                    ResourceUri = new Uri(_options.OAIEndpoint),
                    ApiKey = _options.OAIKey,
                    DeploymentId = _embeddingConfigs.EmbeddingModel
                }
            }
        }
    },
    Fields =
    {
        new SimpleField("id", SearchFieldDataType.String) { IsKey = true },
        new SearchableField("content") { AnalyzerName = LexicalAnalyzerName.StandardLucene },
        new SimpleField("sourcePage", SearchFieldDataType.String) {IsFacetable = true},
        new SimpleField("sourceFile", SearchFieldDataType.String) {IsFacetable = true},
        new SearchField("embedding", SearchFieldDataType.Collection(SearchFieldDataType.Single))
        {
            VectorSearchDimensions = _embeddingConfigs.VectorSearchDimensions,
            IsSearchable = true,
            VectorSearchProfileName = vectorSearchProfileName
        }
    },
    //When using free tier, this can't be used!
    //SemanticSearch = new()
    //{
    //    Configurations =
    //    {
    //        new SemanticConfiguration(semanticConfigName, new()
    //        {
    //            ContentFields =
    //            {
    //                new SemanticField("content")
    //            }
    //        })
    //    }
    //}
};

0 comments

1 additional answer

Your answer

Answer 1

Hey there Thibault Verlinde

Thats a good question thanks for using QandA platform

So i guess you need to define and configure the vectorizer within the SearchIndex object before creating the index.

Lemme modify your code to include a vectorizer, below

var indexName = _srchClient.IndexName;
var vectorSearchConfigName = "vectorconfig-" + _options.ProjectName;
var vectorSearchProfileName = "vectorprofile-" + _options.ProjectName;

var vectorizer = new SearchIndexVectorizer
{
    Name = vectorSearchConfigName,
    Source = "/document/content", // Specify the source field for vectorization
    Tokenizer = new LexicalTokenizer(LexicalTokenizerName.StandardLucene),
    OdataType = "#Microsoft.Azure.Search.StandardAnalyzer"
};
var index = new SearchIndex(indexName)
{
    VectorSearch = new()
    {
        Algorithms =
        {
            new HnswAlgorithmConfiguration(vectorSearchConfigName)
        },
        Profiles =
        {
            new VectorSearchProfile(vectorSearchProfileName, vectorSearchConfigName)
        }
    },
    Fields =
    {
        new SimpleField("id", SearchFieldDataType.String) { IsKey = true },
        new SearchableField("content") { AnalyzerName = LexicalAnalyzerName.StandardLucene },
        new SimpleField("sourcePage", SearchFieldDataType.String) {IsFacetable = true},
        new SimpleField("sourceFile", SearchFieldDataType.String) {IsFacetable = true},
        new SearchField("embedding", SearchFieldDataType.Collection(SearchFieldDataType.Single))
        {
            VectorSearchDimensions = 1536,
            IsSearchable = true,
            VectorSearchProfileName = vectorSearchProfileName
        }
    },
    Vectorizers = { vectorizer } // Add the vectorizer to the index
};
// Now ccreate a new search index
await _srchIndexClient.CreateIndexAsync(index);

If this helps kindly accept the answer thanks much.

Thibault Verlinde 120 Reputation points

2024-03-07T15:19:21.3933333+00:00

Hiya @Azar

Thanks for your quick answer!
I'm currently working on the nuget package of Azure Search Documents, version 11.6.0-beta.3, but I don't have any implementation of SearchIndexVectorizer... My only option is a AzureOpenAIVectorizer, but with only a name, what is also weird.
FYI: I'm not using a skillset or anything.

I read that it has to do something with AzureOpenAIParameters, but I'm not sure if that's the case/how to implement it.
Deleted

This comment has been deleted due to a violation of our Code of Conduct. The comment was manually reported or identified through automated detection before action was taken. Please refer to our Code of Conduct for more information.
Ajay Kumar N 28,291 Reputation points Microsoft Employee Moderator

2024-03-11T11:17:17.5233333+00:00

Thibault Verlinde, Glad to know that you had figured out the issue. Thanks for sharing the solution with the community.

As Azar mentioned, since the Microsoft Q&A community has a policy that the question author cannot accept their own answer. They can only accept answers by others, please consider to Accept the answer summarized, it will benefit the community find the answers quickly. Much appreciate your feedback and update.