Building live intelligence: Databricks AI Search supports Streaming Tables, Materialized Views, and Managed tables using Iceberg
Databricks AI Search now directly integrates Streaming Tables, Materialized Views, and Iceberg v3+ tables, eliminating separate pipelines for live data synchronization in AI applications.
AI Search now natively supports Streaming Tables, Materialized Views, and Managed tables using Iceberg v3+ in addition to Delta tables. This integration allows live events, continuously maintained state, and open-format data to flow directly into AI Search applications without requiring separate conversion or synchronization pipelines. The feature reduces operational overhead by enabling AI Search to index new records as they arrive, keeping search results current for applications such as real-time restaurant discovery or trend-based recommendations.
Streaming Tables enable append-only data flows where new records are indexed immediately, supporting use cases like inventory updates or user activity tracking. Materialized Views aggregate and join data from multiple sources, maintaining incremental updates that AI Search propagates to the index. This allows for personalized experiences such as session-aware recommendations or content discovery tailored to recent user behavior, with changes reflected in real time.
Managed Iceberg tables can now serve as direct sources for AI Search indexes without conversion to Delta format, preserving open-format data access. This enables AI applications to search across disparate systems, unify business data stored in Iceberg tables, or build experiences over distributed datasets without relocating the underlying data. The open-format support ensures compatibility with existing data environments while simplifying integration for AI-driven search.
Previously, customers had to create a separate Delta representation of Streaming Tables, Materialized Views, or Iceberg tables and build custom pipelines to synchronize them with AI Search. The new native support eliminates this intermediate layer and operational overhead by allowing AI Search to connect directly to these sources. Users can now create an index using the same workflow as Delta tables, with AI Search handling synchronization automatically behind the scenes.