Summary
- Microsoft Copilot Studio is improving the Evaluations experience for agent makers to better assess agent quality and behavior.
- New capabilities include richer outcome explanations, reasoning traces, cited knowledge sources, and comparison of evaluation runs.
- Dataset creation and validation are being enhanced, including support for up to 100 test cases, customizable question/dataset generation, single-response datasets, and dataset generation from configured knowledge sources.
- Additional pre-run validation, connection checks, credit-limit enforcement, and clearer error messages are intended to reduce configuration issues.
- This is an informational update for Copilot Studio makers; no admin action is required.
Primary Service: Power Platform
Admin Impact: Low
User Impact: Medium
Release Start: 21 Oct 2026
Release End: 21 Oct 2026
Services: M365 Copilot, Power Platform
Category: Stay informed
Tags: New Feature
History
10/21/2026 Item Added to Message Center
Microsoft Message
We are enhancing the Evaluations experience to help makers better understand agent quality and behavior in Microsoft Copilot Studio. This feature is now available.
How does this affect me?
The improvements will bring new capabilities that help makers identify quality issues more quickly, understand why evaluations succeed or fail, compare results across runs, and create higher quality evaluation datasets with less manual effort. Additional validation and guidance will help makers resolve configuration issues before running evaluations. New capabilities include the following:
- Richer explanations of evaluation outcomes will help makers understand why evaluations succeed or fail.
- Reasoning traces and cited knowledge sources will help makers understand agent behavior and grounding.
- Evaluation run comparison will help identify improvements and regressions across runs.
- Evaluations will support datasets with up to 100 test cases.
- Makers will be able to use customizable dataset and question generation.
- Makers will be able to generate datasets from configured knowledge sources.
- Single-response datasets will be supported.
- Improved validation, connection prerequisite checks, credit-limit enforcement, and actionable error messages will help makers resolve configuration issues before running evaluations.
These enhancements are designed to reduce manual effort, improve troubleshooting, and help makers evaluate and optimize agents more effectively.
What action do I need to take?
This message is for awareness, and no action is required.