How to make AI coding agents better at using your technology
Microsoft’s AX Practitioner Playbook provides a structured method to assess and improve how AI coding agents interact with specific technologies, addressing recurring errors in SDK selection, authentication, and setup patterns.
The AX Practitioner Playbook offers a systematic approach to evaluate AI coding agents’ performance with specific technologies, identifying and diagnosing errors such as incorrect SDK versions or deprecated authentication patterns that agents may reproduce from training data. The method focuses on adjusting the sources agents rely on—documentation, tools, plugins, and APIs—rather than waiting for model improvements, as knowledge cutoffs limit a model’s awareness of current product details. Since fall 2025, Microsoft’s Developer Relations team has tested agents using real developer prompts across Azure, Cosmos DB, SharePoint Framework, and Microsoft 365 Copilot extensions, compiling findings into the Agent Experience series before consolidating them into the playbook.
The playbook outlines an end-to-end evaluation process, from setup to implementing fixes, emphasizing the need for rigorous criteria to avoid misleading results, such as perfect scores for non-functional code or false positives in platform usage checks. It guides users on writing measurable, repeatable criteria, calibrating them, and versioning them as products evolve, while warning against letting models generate criteria that may introduce bias or inaccuracies. The evaluation model also helps distinguish between different types of failures, such as extensions that fail to load, load but aren’t called, or are called incorrectly, each requiring targeted fixes.
Evaluations only drive change when the root causes of agent behavior are addressed, and the playbook instructs teams on testing proposed fixes as hypotheses before deployment, presenting evidence to the relevant teams—whether internal, external collaborators, or open-source contributors. The method is grounded in real-world scenarios tested on technologies like Cosmos DB and SharePoint Framework, leading to dozens of improvements, including 46 fixes to the Azure Cosmos DB Agent Kit and a project upgrade for SPFx documented in the playbook.
Designed for developers who build technologies or extensions used by AI coding agents, the playbook includes a quality checklist for running evaluations and introduces the AX Practitioner skill, which integrates into coding agents to provide guidance on applying the playbook’s methods. The skill, trained exclusively on the playbook, answers questions about criteria, agent behavior, and scenario reviews with 95% accuracy across 330 test questions, helping users apply the method while working on their own evaluations.