Effective information retrieval - use of Markdown files (.md)

Use Markdown with YAML front matter as an intermediate publishing format, then connect the third-party DMS through either a Microsoft 365 Copilot connector or an authenticated Copilot Studio retrieval API that enforces the DMS permissions on every query.

Recommended architecture

Third-party DMS
  ├─ Export/API: document, stable ID, revision, URL, ACL
  ├─ Conversion: DOCX → structured Markdown
  ├─ Validation: parse YAML front matter
  └─ Retrieval target:
       A. Microsoft 365 Copilot connector
       B. Azure AI Search + permission-aware retrieval API + Copilot Studio
       C. Federated connector/API when content must remain in the DMS

This is an enterprise integration project rather than a capability supplied by Windows Copilot or a Copilot+ PC alone. It requires DMS API/export access, appropriate Copilot licensing, service credentials, and involvement from the DMS and Microsoft 365/Power Platform administrators.

A. Prefer Microsoft 365 Copilot connectors for Microsoft 365 retrieval

Use a synced connector when the DMS permits content to be indexed in Microsoft Graph:

1. Register an Entra application.
2. Create an external connection.
3. define a schema for fields such as title, content, url, revision, and classification.
4. Transform each DMS document into an external item.
5. Attach the source document’s access-control list (ACL) to that item.
6. Create, update, or delete the item whenever the corresponding DMS record changes.

The connector API explicitly supports ingesting external content together with information about who may access each item.^1^ Apply the title and url semantic labels; the URL is required for citations back to the source system.^2^

If data cannot leave the DMS, investigate a federated MCP-based connector. Federated connectors fetch information at query time without synchronizing it into Microsoft Graph; availability depends on the target experience.^3^

B. Use Copilot Studio and Azure AI Search for a custom agent

Use this route when you need your own agent, search ranking, or transformation pipeline:

1. Export approved DMS content to Azure Blob Storage, Azure Files, or OneLake.
2. Configure Azure AI Search with Markdown parsing.
3. Use oneToMany parsing so headings become separate searchable sections. Use oneToOne only when each complete file should remain one search record. Azure AI Search splits Markdown on headings; lists, tables, and code blocks remain content.^4^
4. Add the search index to Copilot Studio through Agent → Knowledge → Add knowledge → Featured → Azure AI Search.^5^
5. Include the canonical DMS URL in each indexed result so Copilot Studio can produce citations.

The direct Copilot Studio–Azure AI Search connection is appropriate only when all indexed records have a common permitted audience, or when a supported permission-trimming design has been validated.

For item-level DMS permissions, create an OnKnowledgeRequested custom knowledge topic instead. Its retrieval API must:

• Authenticate or otherwise resolve the requesting user.
• Map that identity and group memberships to the DMS ACL model.
• Apply an authorization filter before returning search results.
• Return only authorized Content, Title, and ContentLocation values.

Copilot Studio’s OnKnowledgeRequested topic is configured in YAML code view and can call a custom search endpoint, then assign authorized results to System.SearchResults.^6^

Merely indexing fields such as allowed_groups does not enforce permissions. The retrieval API must apply them on every request. Permission to open a citation is also separate: the user must be authorized both to retrieve the snippet and to open its DMS URL.

Markdown and YAML design

Use short, descriptive headings because Markdown parsing uses the heading hierarchy to create sections:

---
id: "POL-0042"
title: "Customer data retention"
revision: "7"
modified: "2026-10-15"
classification: "Internal"
source_url: "canonical DMS URL"
owner: "Records Management"
tags: ["retention", "customer-data"]
---

# Customer data retention

## Standard retention period
Authoritative policy text.

## Exceptions
Approved exceptions and escalation procedure.

YAML is only metadata storage until your pipeline parses, validates, and maps it into index or connector fields. The documented Markdown parser does not establish that arbitrary front matter is automatically promoted into Azure AI Search fields.

Also keep these two YAML uses distinct:

• Document YAML: front matter containing title, revision, classification, and source identity.
• Copilot Studio YAML: executable topic configuration for OnKnowledgeRequested.

A minimal front-matter parser could be:

from pathlib import Path
import hashlib, yaml

text = Path("document.md").read_text(encoding="utf-8")
_, front_matter, body = text.split("---", 2)
metadata = yaml.safe_load(front_matter)

required = {"id", "title", "revision", "source_url"}
missing = required - metadata.keys()
if missing:
    raise ValueError(f"Missing metadata: {sorted(missing)}")

metadata["checksum"] = hashlib.sha256(body.encode()).hexdigest()
metadata["content"] = body.strip()

Synchronization and pilot

Use the immutable DMS ID as the parent key and deterministic section keys such as POL-0042:0001. Track the source revision or checksum, update only changed documents, and delete every child section when its source document is removed.

Start with 20 representative documents:

1. Export and convert them to Markdown.
2. Validate required YAML fields.
3. Test a fixed set of exact, conceptual, and follow-up questions.
4. Verify citations open the original DMS records.
5. Test updates and deletions.
6. Test two users with different permissions and confirm that restricted text is neither retrieved nor used in an answer.
7. Compare retrieval quality against the original DOC/DOCX collection before scaling.

References

1. Use the Copilot connectors API
2. Manage connector connections
3. Microsoft 365 Copilot connectors overview
4. Index Markdown blobs and files in Azure AI Search
5. Add Azure AI Search as a knowledge source
6. Connect to custom knowledge sources