Databricks Network Configuration delivery to Tens of Millions of Serverless VMs
Databricks re-architected its serverless platform to deliver network configurations to tens of millions of VMs daily with 97.5% lower latency and 99.99% availability by shifting from synchronous to event-driven pre-computation.
Databricks’ serverless compute platform launches tens of millions of virtual machines daily across AWS, Azure, and GCP to power products such as SQL warehouses and ML serving endpoints. Before any workload executes, each VM requires its network configuration, which historically was assembled synchronously from multiple upstream services during cluster creation. This approach became unsustainable as usage grew, driving expensive, duplicated computations across workspaces and increasing load proportional to tenant resources.
The company redesigned the system to separate management and serving paths. Upstream services now emit change events to a message queue, processed asynchronously to update per-workspace configurations stored in a pre-computed snapshot store. A periodic reconciler ensures eventual consistency by re-syncing workspaces in the background. When a serverless cluster starts, the network configuration service retrieves the pre-computed snapshot with a single storage read, eliminating upstream calls and reducing latency from 5,000ms to 125ms.
The event-driven architecture handles common cases efficiently by pushing updates to affected workspaces, while the reconciler addresses missed events. This modular design allows new upstream data sources to be added with minimal changes, ensuring scalability as Databricks’ product offerings expand. The system now serves billions of network configuration requests daily with 99.99% availability.
The re-architecture decouples critical paths by moving expensive aggregation to the background, transforming a multi-service dependency chain into a single storage read. This shift ensures the network configuration delivery scales alongside Databricks’ rapidly growing serverless compute platform, maintaining performance and reliability at global scale.