Object Storage + WAL: Lakebase Postgres for the agentic era
Databricks introduces Lakebase Postgres, a transaction-centric Postgres deployment that uses object storage and the write-ahead log (WAL) to make agent workloads more efficient and cost-effective.
Agents interacting with traditional OLTP databases often face bottlenecks at the storage layer due to data movement during deployments, copies, restores, and replicas. Object storage, such as Amazon S3, offers a scalable and cost-effective alternative, but integrating it with a transactional database requires rethinking the storage model. Lakebase Postgres addresses this by placing the write-ahead log (WAL) at the core, treating the database as a journal of transactions rather than a snapshot of the present. This shift enables agents to query transaction history directly, eliminating the need for slow and expensive data copies or backups.
Postgres’ WAL records every database modification before it reaches the data files, originally designed for crash recovery. Each modification is assigned a unique log sequence number (LSN), creating an addressable timeline of changes. This timeline can be interpreted as a complete, ordered account of every page the database has ever changed, making it possible to derive the database state at any point in time. Lakebase Postgres leverages this by making the WAL the primary storage layer, with data files serving as a derived, cached representation. This inversion allows the full transaction history to be preserved without moving data, enabling cheap and rapid deployments, restores, and replicas.
In a conventional Postgres deployment, the WAL is a temporary safeguard, trimmed once its records are applied to data files. The database’s identity is tied to the machine running it, with storage limited to attached disks. Lakebase Postgres inverts this model by making the WAL the source of truth, stored in object storage, while the compute layer runs standard Postgres. The compute nodes handle query execution, MVCC, and indexing but no longer bear responsibility for data durability. This separation of concerns allows compute nodes to scale, start, or stop without risking data integrity, as the storage layer ensures correctness, durability, and history.
The design introduces a new operation, GetPage@LSN, where compute nodes request specific pages at a given LSN from the storage layer. This replaces synchronous replication’s network hop, as externalizing the WAL involves a similar network round trip. Read requests are served from cached pages in RAM or NVMe, ensuring performance remains high. Agents can query historical states by requesting pages at past LSNs, making the full transaction timeline addressable. This architecture reduces operational overhead and cost, aligning with the demands of agent workloads that prioritize transaction history over current state snapshots.