Microsoft Releases ASSERT Framework to Automate AI Agent Evaluation
Microsoft has open-sourced ASSERT, a new framework designed to transform natural-language AI behavior specifications into executable evaluations, streamlining the testing and refinement of AI systems.
Microsoft has introduced ASSERT (Adaptive Spec-driven Scoring for Evaluation and Regression Testing), an open-source framework aimed at bridging the gap between intended AI functionality and practical, application-specific behavioral testing. The framework automates the conversion of natural-language behavior specifications into executable evaluation pipelines, generating test scenarios, datasets, metrics, and scorecards.
High-quality behavioral evaluations are crucial for ensuring AI systems operate as intended. However, creating these evaluations is often a slow, complex, and error-prone process. Existing evaluation methods can quickly become outdated as product requirements, policies, or underlying models evolve. ASSERT addresses these challenges by treating behavior specifications as a primary input for the evaluation process, ensuring that tests directly reflect the AI's intended context and operational boundaries.
The ASSERT pipeline consists of four key stages. First, it systematizes broad behavior specifications into explicit concept specifications. This is followed by a taxonomization stage, where these concepts are transformed into a granular, editable taxonomy of permissible and impermissible behaviors, complete with supporting artifacts. Developers and policy experts can review and refine this taxonomy before proceeding.
In the third stage, ASSERT generates stratified test cases based on the defined taxonomy and developer-specified dimensions, such as task type, persona, or tool availability. These cases can range from single-turn prompts to complex multi-turn scenarios, including adversarial probes designed to test edge cases. The framework ensures that behavior is tested across a diverse set of conditions, rather than a narrow subset of easy examples.
The fourth stage involves running these generated test cases against the target AI system—be it a model, an agent, or an application workflow. Through its instrumentation layer, ASSERT captures not only the final output but also critical intermediate data, such as tool calls, retrieved context, and routing decisions, providing a comprehensive trace for later analysis.
Finally, in the scoring stage, ASSERT evaluates each trace against the behavior taxonomy and its associated policy stance. The output includes not just a pass/fail label but also a detailed rationale, a policy citation referencing the specific rule violated, and the exact turn or action that led to the verdict. This detailed feedback loop allows developers to precisely identify and address behavioral failures.
Microsoft conducted internal validation studies to assess ASSERT's effectiveness. A coverage study compared ASSERT's behavior-specific evaluations against a more direct generation approach, while another study evaluated the accuracy of ASSERT's AI judges against human review. The framework is designed to support a wide range of AI applications, from customer support agents to research assistants, by enabling rigorous testing of their specific, context-dependent behaviors.
By automating and systematizing the evaluation process, ASSERT aims to significantly reduce the effort required to build and maintain robust AI systems. This open-source release empowers developers to systematically test and refine AI behaviors against defined requirements, ultimately leading to more reliable and trustworthy AI products.