Announcing General Availability of Real-Time Mode for Spark Declarative Pipelines on Databricks

Real-Time Mode (RTM) for Spark Declarative Pipelines (SDP) is now generally available, expanding SDP’s capabilities to support operational, low-latency streaming workloads. Previously, customers relied on separate engines like Flink for real-time processing, but RTM integrates this functionality directly into SDP. This eliminates the need for duplicate codebases and fragmented governance models while maintaining SDP’s automatic dependency resolution, incremental processing, and failure recovery.

RTM achieves end-to-end latencies in the millisecond range by streaming data continuously and executing all query stages simultaneously. It passes data between stages as soon as it is produced, enabling applications such as fraud detection, real-time alerting, and live personalization. Benchmarks show RTM is up to 92% faster than specialized engines like Flink for feature computation workloads, supporting use cases like fraud scoring and real-time recommendation systems.

Spark Declarative Pipelines Serverless with RTM removes the need to configure compute or cluster sizing, delivering sub-second latency while charging only for cores used. It also inherits SDP features like versionless upgrades and low-downtime maintenance, reducing operational overhead. The Databricks Feature Store leverages this architecture to achieve p99 latency of 200ms from Kafka to the online feature store.

To enable RTM, pipelines must run in continuous mode with spark.databricks.streaming.realTimeMode.enabled set to true. Users define an update flow with pipelines.trigger set to "RealTime" and target a sink, such as Lakebase or Redis, using a ForEach sink. This allows features to land in operational stores within milliseconds, enabling ultra-low-latency queries for applications like fraud detection models.