How a major freight railroad scaled pipeline creation with Genie Code
A major Canadian railway automated the modernization of hundreds of legacy data pipelines using Databricks Genie Code, reducing new table ingestion from days to minutes and achieving over 90% automation.
A leading Canadian freight railroad modernized its decades-old data estate by automating pipeline creation with Databricks Genie Code, Unity Catalog, and custom Agent Skills. The company, operating 20,000 route miles and handling C$250 billion in goods annually, faced challenges scaling manual modernization efforts across hundreds of pipelines. The solution transformed pipeline development into a repeatable factory process, enabling rapid, consistent ingestion code generation aligned with enterprise standards.
Before automation, building a single pipeline required multi-day efforts, including schema inspection, business logic definition, and manual creation of historical, streaming, and incremental merge logic. Teams relied on Source-to-Target Mapping spreadsheets and extensive testing for schema changes and soft deletes. The manual process was unsustainable for hundreds of pipelines, prompting the need for a standardized, scalable approach to pipeline construction and modernization.
The automated workflow combines Genie Code with custom Agent Skills to generate production-ready PySpark and Spark SQL pipelines, using Delta Lake and Lakeflow Jobs. A compact YAML prompt initiates the process, with Genie Code parsing inputs, discovering schemas via Unity Catalog, and generating artifacts such as DDL, historical loads, streaming ingestion, and automated test suites. The system embeds enterprise conventions like audit columns, deduplication, and change-sequence-aware merges directly into the generation process.
A custom Agent Skill centralizes the company’s ingestion standards and pipeline patterns, versioned like a codebase and uploaded to the workspace. The skill includes pattern files for catalog discovery, raw ingestion, historical loads, and incremental merges, ensuring consistent artifact generation. Developers initiate code generation with minimal YAML prompts, while Unity Catalog’s real-time metadata inspection eliminates the need for separate discovery layers, enabling context-aware mapping and validation before code emission.