Built for Agents: The Case for Spark Declarative Pipelines

Databricks analyzed 528 agent-created data pipelines, comparing procedural and declarative authoring methods. Declarative pipelines, which describe desired datasets while leaving execution details to Spark, proved more efficient. Procedural pipelines require step-by-step instructions, making errors detectable only during runtime, whereas declarative pipelines allow Spark to validate structural issues before execution. This reduces wasted compute and speeds up development cycles.

The benchmark tested 22 real-world data engineering tasks, each with 12 varied input datasets, producing 528 pipeline builds. Agents used both procedural PySpark and Spark Declarative Pipelines to solve identical problems. Declarative pipelines completed 258 of 264 tasks, while procedural pipelines completed 255. However, the key difference lay in error handling and resource usage, with declarative pipelines catching structural errors early and reducing compute time by 34 times.

Declarative pipelines averaged 64 lines of code compared to 131 for procedural pipelines, offering shorter, more consistent implementations. Procedural pipelines showed greater variability, with 19 builds resorting to in-memory processing, a method prone to failure with larger datasets. Declarative pipelines avoided such shortcuts entirely, ensuring more reliable and maintainable code for teams.

At production scale, declarative pipelines trade slightly higher LLM token costs for significant savings by preventing wasted cluster compute on failing pipelines. This approach becomes increasingly cost-effective as data volumes grow, since token costs remain fixed while compute waste scales proportionally. The findings suggest declarative authoring is the preferable method for agent-built pipelines in large-scale environments.