How Databricks Uses AI to Accelerate Incident Investigation
Databricks has introduced AI SRE, an AI-powered debugging agent that accelerates incident resolution by automatically gathering evidence and guiding engineers through root cause analysis across multiple services and regions.
Databricks describes how its engineers use AI to operate hundreds of microservices across more than 1,500 Kubernetes clusters spanning 70 regions and three cloud providers. When an incident occurs, the on-call engineer’s first task is to determine what changed. AI SRE, an AI-powered debugging agent, initiates an investigation immediately upon incident detection, correlating signals from across the stack and guiding engineers through root cause analysis. The system is designed to reduce the time engineers spend manually connecting disparate signals during high-pressure situations.
The development of AI SRE began with extensive observation of on-call engineers’ debugging workflows. Through interviews and analysis of postmortems, Databricks identified that debugging is a sequence of repeatable investigative steps followed by expert judgment. The company concluded that AI agents could accelerate this process, but required a shared platform to handle common tasks such as gathering context, executing tools, and correlating evidence. AI SRE supports both automatic triage, which begins when an incident fires, and interactive investigation, allowing engineers to explore hypotheses and request additional evidence.
When an incident triggers AI SRE, it launches three parallel investigation tracks: platform health checks to assess the environment, service-level analysis to pull relevant logs and metrics, and runbook execution to perform team-specific debugging procedures. The system identifies anomalies relative to baseline behavior and executes steps encoded by teams, such as checking thresholds or performing mitigations. By the time the engineer reviews the incident, AI SRE has assembled a diagnostic summary including what broke, what changed, and recommended next steps.
AI SRE also provides an interactive debugging environment where engineers can ask follow-up questions in natural language, request additional signals, or drill into specific components. The system is built as a layered platform with clear responsibilities: primitives for raw operational data, an API layer for uniform access to underlying systems, and a core engine for orchestration and LLM-powered synthesis. This architecture ensures that debugging tools remain functional even if underlying systems are replaced.