Evaluate a Workflow in Foundry

Hello PG F,

Thank you for sharing your scenario and the details of your multi-agent workflow.

We understand that you need to evaluate complete workflow conversations, including failed or incomplete runs that do not reach the final agent. You also mentioned that the Evaluate tab in Azure AI Foundry does not allow you to select traces or conversations for evaluation.

1. Native evaluation capabilities in Microsoft Foundry

Microsoft Foundry supports evaluating AI applications and multi-turn conversations using its evaluation capabilities. The Foundry evaluation service supports built-in evaluators such as relevance, coherence, and task adherence, with additional evaluators available for scenarios such as safety, tool usage, and agent behavior, depending on the evaluation configuration.

You can use the evaluation service to assess conversation datasets and review evaluation results in the Foundry portal.

For more information, see the official documentation:

• Microsoft Foundry evaluation approach
• Evaluate generative AI applications

2. Evaluating complete workflow conversations

Your requirement is to evaluate all conversations, including those that fail before reaching the final agent. To achieve this, the evaluation dataset must contain the complete conversation history and relevant interaction data for both successful and unsuccessful workflow executions.

The available documentation does not confirm a native workflow-portal capability to automatically generate a dataset containing every complete workflow conversation, including failed or incomplete runs. It also does not establish the specific reason why trace or conversation selection is disabled in your workflow's Evaluate tab.

Therefore, we cannot confirm that the portal currently supports selecting all workflow traces and generating a complete conversation dataset directly.

3. Recommended approach

Your current approach of extracting complete conversation histories from Cosmos DB and preparing a dataset for evaluation remains a practical workaround, provided Cosmos DB contains the required records for all workflow executions.

You can then submit the prepared dataset to the Foundry evaluation service or use your custom evaluator to assess the entire workflow conversation rather than only the final agent's response.

For dataset preparation and evaluation guidance, refer to the official documentation above.

4. Regarding workflow deprecation

Since you mentioned that Workflows are expected to be deprecated, we recommend reviewing the applicable migration guidance before investing further in a workflow-specific evaluation process. The documentation referenced above describes general Foundry evaluation capabilities; it does not confirm that the current workflow Evaluate tab will support your complete conversation dataset requirement.

I Hope this helps. Do let me know if you have any further queries.

Thank you!