The future of orchestration: Pine59’s journey to Airflow 3 on Google Cloud
Pine59 migrated to Google Cloud’s Managed Airflow (Gen 3) with Airflow 3 to optimize data pipelines handling millions of daily data points, improving speed, reliability, and developer workflows.
Pine59, a location intelligence provider, operates data pipelines processing millions of data points daily to generate metrics like Daily Foot Traffic for up to 14 million locations. The company relies on Google Cloud’s BigQuery for heavy computation and Managed Service for Apache Airflow to orchestrate these pipelines. As workloads grew, Pine59 sought to modernize its monorepo containing hundreds of directed acyclic graphs (DAGs) to enhance reliability and speed.
To achieve this, Pine59 stress-tested its production workloads against the newly available Managed Airflow (Gen 3) architecture running Airflow 3. The transition delivered immediate improvements in processing speed, task scheduling, and stability. The company then fully migrated, recognizing the potential for significant performance gains across its data and machine learning pipelines.
A key focus of the migration was optimizing the orchestration of machine learning inference workloads. Previously using standard Kubernetes operators, Pine59 leveraged Managed Airflow (Gen 3)’s optimized infrastructure to set up a dedicated Google Kubernetes Engine (GKE) cluster for model inference. This separation of orchestration and compute improved efficiency, demonstrating Managed Airflow’s scalability for enterprise MLOps.
Beyond infrastructure, Pine59 capitalized on Airflow 3’s improved developer workflow and user interface. The company built custom plugins, including BigQuery Auto-linkify and DAG Run Configuration Search, to enhance observability and reduce troubleshooting time. A compatibility shim layer was also deployed to streamline operator migration across Airflow versions, further accelerating development velocity.