Ingest semi-structured data faster and more efficiently with Variant

For years, data teams faced a tradeoff when handling semi-structured data such as JSON or XML: either schematize it for fast queries at the cost of flexibility, or store it as strings for adaptability but endure slow performance. Databricks introduced the Variant data type to resolve this, collaborating with Delta, Spark, Parquet, and Iceberg communities to standardize semi-structured data handling in lakehouses. Variant is now generally available in Databricks, including Variant Shredding, which uses Predictive Optimization to automatically enhance query performance on Variant data. Over 5,000 teams already use Variant to ingest streaming events, API payloads, and schemaless database data, particularly where upstream schema changes complicate downstream pipelines.

Variant eliminates the need for upfront schematization, allowing teams to ingest semi-structured data first and determine its utility later. This reduces the time data engineers spend justifying pipeline investments. Databricks reports users execute over 500 million Variant queries monthly across 160 terabytes of Variant data, with reading performance comparable to structured data on managed tables. Shredding stores common fields as columns in Parquet files, while Predictive Optimization identifies critical fields and collects statistics to improve file skipping, scanning only necessary data for queries.

Variant Shredding delivers nearly four times faster reads than unshredded Variant and 30 times faster reads than storing JSON as a string, enabling efficient querying of complex, nested JSON structures even at petabyte scale. The data type integrates across Delta and Iceberg formats, allowing interoperability within the lakehouse ecosystem. Teams can use Genie Code in Lakeflow Pipelines Editor to generate Auto Loader ingestion pipelines via natural language, simplifying setup and enabling direct consumption of Variant data in dashboards and reporting.

Databricks plans to expand Variant support with features like Liquid Clustering by Variant fields, additional SQL functions, and further integrations. The company emphasizes that Variant removes the traditional tradeoff between flexibility and performance for semi-structured data, leveraging Predictive Optimization to automatically optimize Variant data across its products. Users can begin using Variant immediately through Databricks' platform.