文字分割技能

Text Split 技能能將文字拆分成文字塊。 你可以指定要將文字拆成句子或特定長度的頁面。 位置元資料如偏移與序數位置也可作為輸出提供。 若下游技能有最大文字長度要求，例如將資料區塊傳入 Azure OpenAI 及其他模型提供者的嵌入技能，這項技能非常有用。 欲了解更多此情境資訊，請參閱 向量搜尋的區塊文件。

@odata.type

Microsoft.Skills.Text.SplitSkill

技能參數

參數依大小寫區分。

技能輸入

技能產出

範例定義

{
    "name": "SplitSkill", 
    "@odata.type": "#Microsoft.Skills.Text.SplitSkill", 
    "description": "A skill that splits text into chunks", 
    "context": "/document", 
    "defaultLanguageCode": "en", 
    "textSplitMode": "pages", 
    "unit": "azureOpenAITokens", 
    "azureOpenAITokenizerParameters":{ 
        "encoderModelName":"cl100k_base", 
        "allowedSpecialTokens": [ 
            "[START]", 
            "[END]" 
        ] 
    },
    "maximumPageLength": 512,
    "inputs": [
        {
            "name": "text",
            "source": "/document/text"
        },
        {
            "name": "languageCode",
            "source": "/document/language"
        }
    ],
    "outputs": [
        {
            "name": "textItems",
            "targetName": "pages"
        }
    ]
}

範例輸入

{
    "values": [
        {
            "recordId": "1",
            "data": {
                "text": "This is the loan application for Joe Romero, a Microsoft employee who was born in Chile and who then moved to Australia...",
                "languageCode": "en"
            }
        },
        {
            "recordId": "2",
            "data": {
                "text": "This is the second document, which will be broken into several pages...",
                "languageCode": "en"
            }
        }
    ]
}

範例輸出

{
    "values": [
        {
            "recordId": "1",
            "data": {
                "pages": [
                    "This is the loan...",
                    "In the next section, we continue..."
                ],
                "offsets": [
                    {
                        "utf8": 0,
                        "utf16": 0,
                        "codePoint": 0
                    },
                    {
                        "utf8": 146,
                        "utf16": 146,
                        "codePoint": 146
                    }
                ],
                "lengths": [
                    {
                        "utf8": 146,
                        "utf16": 146,
                        "codePoint": 146
                    },
                    {
                        "utf8": 211,
                        "utf16": 211,
                        "codePoint": 211
                    }
                ],
                "ordinalPositions" : [
                    1,
                    2
                ]
            }
        },
        {
            "recordId": "2",
            "data": {
                "pages": [
                    "This is the second document...",
                    "In the next section of the second doc..."
                ],
                "offsets": [
                    {
                        "utf8": 0,
                        "utf16": 0,
                        "codePoint": 0
                    },
                    {
                        "utf8": 115,
                        "utf16": 115,
                        "codePoint": 115
                    }
                ],
                "lengths": [
                    {
                        "utf8": 115,
                        "utf16": 115,
                        "codePoint": 115
                    },
                    {
                        "utf8": 209,
                        "utf16": 209,
                        "codePoint": 209
                    }
                ],
                 "ordinalPositions" : [
                    1,
                    2
                ]
            }
        }
    ]
}

分塊與向量化的範例

此範例用於整合向量化。

• pageOverlapLength：重疊文字在 資料分塊 情境中非常有用，因為它能保留同一文件產生區塊間的連續性。

• maximumPagesToTake：頁數限制在 向量化 情境中非常有用，因為它幫助你控制在提供向量化的嵌入模型的最大輸入限制以下。

範例定義

此定義加入 pageOverlapLength 了 100 個字元和 maximumPagesToTake 1 個字元。

假設 maximumPageLength 是 5,000 字元（預設），接著 "maximumPagesToTake": 1 處理每個來源文件的前 5,000 字元。

此範例設定 textItems 為 至 myPagestargetName。 因為 targetName 是設定的， myPages 你應該用來選擇文本分割技能輸出的數值。 在下游技能中，索引器輸出/document/myPages/*欄位映射、知識庫投影與索引投影。

{
    "@odata.type": "#Microsoft.Skills.Text.SplitSkill",
    "textSplitMode" : "pages", 
    "maximumPageLength": 1000,
    "pageOverlapLength": 100,
    "maximumPagesToTake": 1,
    "defaultLanguageCode": "en",
    "inputs": [
        {
            "name": "text",
            "source": "/document/content"
        },
        {
            "name": "languageCode",
            "source": "/document/language"
        }
    ],
    "outputs": [
        {
            "name": "textItems",
            "targetName": "myPages"
        }
    ]
}

取樣輸入（與前述範例相同）

{
    "values": [
        {
            "recordId": "1",
            "data": {
                "text": "This is the loan application for Joe Romero, a Microsoft employee who was born in Chile and who then moved to Australia...",
                "languageCode": "en"
            }
        },
        {
            "recordId": "2",
            "data": {
                "text": "This is the second document, which will be broken into several sections...",
                "languageCode": "en"
            }
        }
    ]
}

取樣輸出（注意重疊）

在每個「textItems」陣列中，第一個項目的後續文字會被複製到第二個項目的開頭。

{
    "values": [
        {
            "recordId": "1",
            "data": {
                "myPages": [
                    "This is the loan...Here is the overlap part",
                    "Here is the overlap part...In the next section, we continue..."
                ]
            }
        },
        {
            "recordId": "2",
            "data": {
                "myPages": [
                    "This is the second document...Here is the overlap part...",
                    "Here is the overlap part...In the next section of the second doc..."
                ]
            }
        }
    ]
}

錯誤案例

若某語言不支援，會產生警告。

另請參閱

• 內建技能
• 如何定義技能組合