Content Understanding (2025-11-01): Knowledge labelled samples ignored, even on the labelled document

Akhilesh Mishra 0 Reputation points
2026-11-10T06:44:31.0466667+00:00

We use Azure Content Understanding (GA API 2025-11-01) with a custom document analyzer (baseAnalyzerId prebuilt-document, models gpt-5.2 and text-embedding-3-large) to extract data from purchase order acknowledgements received from our suppliers. We added a labelled sample in Content Understanding Studio, but the corrected labels have no effect, even when analysing the exact document that was labelled.

What we tested (same 3-page document):

  1. Line-level UnitPrice (method extract): corrected on all 5-line items to the printed net unit prices. The analyzer still returns the original (pre-discount) prices.
  2. Diagnostic test on a header field: SupplierOrderNumber (method extract, top-level, not nested) was deliberately corrected to a different value that is clearly printed and labelled on the page. The analyzer still returns the original value.

What we already ruled out:

  • Analyzer was rebuilt after labeling (and recreated from scratch).
  • All affected fields use extract; also tested with no generate fields at all.
  • Not OCR: the labeled values are read correctly in the result markdown.
  • Label files are saved as "corrected", active, and in the analyzer's knowledgeSources folder.
  • Storage access is fine (managed identity has Blob Data Contributor; network open to all).
  • Analyzer is "ready" with no warnings; embedding tokens show retrieval runs.

Questions:

  1. Is there anything else required for labelled samples to take effect in the GA API that we may be missing?
  2. Is there a way to verify which labelled samples were retrieved and applied during a specific analysis?
  3. Is this a known issue, and would the 2026-06-01-preview training approach behave differently?

We can share the analyzer definition, label files and analyze response (with operation ID) privately if needed.

Azure Content Understanding in Foundry Tools
0 comments

1 answer

Sort by: Oldest
Most helpful Newest Oldest

  1. AI answer

    2026-11-10T06:45:52.0766667+00:00

    Your 2025-11-01 configuration already satisfies the documented labeled-sample workflow, so capture GA diagnostics for the failing operation and open an Azure support case with the analyzer definition, label files, build details, and operation ID.

    No additional GA activation step is documented

    For a custom document analyzer, the documented requirements are:

    • Use document inputs.
    • Use labeled samples for fields that do not use generate.
    • Save corrected labels.
    • Rebuild the analyzer after adding the samples.

    Both 2025-11-01 GA and 2026-06-01-preview support the same Studio labeling workflow. Corrected labels are examples from which the analyzer generalizes; they are not documented as deterministic overrides, even when the same document is analyzed again.^1^

    Your tests already cover those requirements, including a top-level extract field, rebuild/recreation, valid OCR text, active corrected files, and storage access. There is no documented extra “activate labels” operation to perform.

    Capture the available GA diagnostics

    For 2025-11-01, add x-ms-diagnostics: true to every polling GET request made to the Operation-Location URL. Adding it only to the initial analysis POST does not enable diagnostics in the result.^2^

    curl -i -X GET "{operation-location}" \
  -H "Ocp-Apim-Subscription-Key: {key}" \
  -H "x-ms-diagnostics: true"

    Retain the complete final response, including:

    • result.analyzerId
    • result.warnings
    • result.infos
    • Field values, confidence, and source grounding
    • HTTP response headers and operation ID
    • UTC submission and completion times

    The currently documented infos diagnostic is LLMStats, which summarizes completion/embedding calls and latency. It does not document the identities of retrieved labeled samples. Therefore, embedding-token activity confirms model/retrieval work occurred but does not prove that a particular corrected sample or field label was selected and applied.^2^

    For GA, the built analyzer retains labeled training data and uses it at analysis time.^3^ Public documentation does not expose a per-analysis retrieval trace containing sample filenames, sample IDs, retrieved labels, or ranking scores.

    Is this a known issue?

    The available documentation does not identify a confirmed 2025-11-01 GA defect in which valid corrected extract labels are universally ignored. The preview release notes describe training-data problems that can affect individual samples—invalid labels, schema mismatches, text-recognition problems, and label-geometry problems—but that does not establish which one, if any, affected this GA build.^4^

    Given that the deliberately changed top-level field is also unaffected, this is suitable for service-side investigation rather than further schema tuning. Include privately:

    1. Analyzer definition returned by GET /analyzers/{analyzerId}.
    2. Original document and corresponding label files.
    3. Build operation/result and timestamps, if retained.
    4. Failing analysis operation ID and diagnostic response.
    5. Resource region and analyzer ID.
    6. Screenshots showing the saved corrected labels.
    7. Expected versus actual values for both SupplierOrderNumber and each UnitPrice.

    Avoid posting documents, keys, SAS URLs, or supplier data publicly.

    How 2026-06-01-preview differs

    The preview uses the same labeling UI but changes the training architecture:

    • Training distills patterns and domain context into the built analyzer.
    • The built analyzer does not retain or retrieve labeled documents at analysis time.
    • Runtime token consumption is reduced relative to GA.^1^
    • Training reports provide sample-level warnings for invalid labels, schema mismatches, text-recognition issues, and label-geometry issues; usable labels are preserved while training continues with valid samples.^4^

    That makes preview materially better for diagnosing whether a sample was rejected or only partially usable. It does not guarantee that the corrected value will be reproduced, and it does not provide runtime retrieval identities because preview analyzers do not retrieve labeled documents during analysis.

    Run a parallel preview analyzer against a copy of the dataset and inspect its training report; do not replace the production analyzer solely for this test. Microsoft recommends 2025-11-01 for production and 2026-06-01-preview only for evaluating preview features because preview has no SLA.^5^

    References

    1. Validate document analyzer quality with confidence, grounding, and labeled samples
    2. Retrieve diagnostics from Azure Content Understanding in Foundry Tools
    3. Data, privacy, and security for Content Understanding
    4. What's new in Azure Content Understanding in Foundry Tools?
    5. What is Azure Content Understanding in Foundry Tools?
    AI-generated content may be incorrect. Read our transparency notes for more information.

    Was this answer helpful?

Your answer

Answers can be marked as 'Accepted' by the question author and 'Recommended' by moderators, which helps users know the answer solved the author's problem.