How Snowflake Built an Agent to Fix Broken Data Pipelines

For years, Snowflake’s data team manually resolved pipeline failures through a repetitive process of paging, log analysis, root-cause tracing, and pull requests. This approach became unsustainable as the number of pipelines grew exponentially, outpacing team growth and increasing failure volumes despite stable pipeline quality. The team sought an automated solution capable of not only detecting issues but also proposing fixes, shifting from reactive alerts to proactive resolution. The agent now handles the full cycle from incident detection to a ready-to-review pull request in under 16 minutes on average.

The agent’s architecture evolved from a single-server REST API to a scalable, parallelized system using Snowpark Container Services. This design eliminated bottlenecks by allowing multiple agents to run concurrently without shared compute constraints. The team plans to further simplify operations by adopting Snowflake’s new EXECUTE AGENT TASK from CoCo automations, which enables SQL-triggered agent runs without server management or compute pooling overhead.

Workflows are defined in JSON, specifying instructions, accessible repositories, model selection, and notification channels. The agent operates within a turn budget and can execute simple or complex workflows, including those requiring human feedback. This flexibility allows the agent to handle both routine repairs and multi-step processes, reducing manual intervention for recurring pipeline issues.

In production for five months, the agent has identified and addressed compute waste from underperforming pipelines, a problem often overlooked by teams due to the effort required for manual detection. The initiative has also informed Snowflake’s product development, with plans to bring similar autonomous capabilities to customers as part of its vision for autonomous data engineering.