Accelerating the borderless Lakehouse: Announcing preview of cross-cloud caching

Google Cloud announced enhancements to its borderless Lakehouse, enabling data engineers, scientists, and AI agents to query governed data across clouds without moving it. The service now supports direct federation to catalogs like Databricks Unity Catalog and AWS Glue via the Apache Iceberg REST catalog specification, reducing reliance on brittle ETL pipelines. Partner Cross-Cloud Interconnect provides private, high-bandwidth links to other clouds, lowering per-gigabyte transfer costs compared to public internet routes. These capabilities aim to unify distributed datasets spanning Amazon S3, Azure Data Lake Storage, and operational databases while maintaining enterprise-grade governance and security.

The new cross-cloud caching feature, available in preview, accelerates queries in BigQuery by caching frequently accessed data locally within Google Cloud. It operates at sub-file block granularity for columnar formats like Apache Parquet, transferring only the specific column chunks and dictionary pages required by a query. Cached data blocks are encrypted at rest by default using Google-managed encryption keys, and cache entries are partitioned by project and catalog boundaries to prevent cross-tenant data exposure. Regional isolation ensures compliance with data residency requirements during cross-cloud queries.

BigQuery cross-cloud connections, also in preview, allow direct querying of non-Iceberg data in Amazon S3 and Azure Storage without requiring an Iceberg catalog. This enables organizations to create external tables referencing remote bucket paths for formats like CSV, JSON, and ad-hoc Parquet files. The feature simplifies access to raw data across clouds while leveraging BigQuery’s query engine for performance. It complements cross-cloud caching by providing an additional method to unify data access without duplicating storage or relying on complex pipelines.

In a practical example, an e-commerce team querying a 10 TiB Iceberg sales table in Amazon S3 saw significant reductions in data transfer during peak analysis. Initial queries processed 214.5 GiB of logical data but only transferred 24.1 GiB due to Iceberg’s columnar compression. Subsequent queries with modified dimensions achieved a 94.8% cache hit rate, transferring just 1.33 GiB for new columns. These optimizations, combined with Partner Cross-Cloud Interconnect, reduce network transfer to under 3% of total data processed, lowering total cost of ownership for cross-cloud analytics and AI workloads.