Streaming Data into Apache Iceberg with Snowflake

Snowflake has published a high-performance architecture demonstrating Snowpipe Streaming ingesting more than one million transactions per second into Apache Iceberg tables entirely within Snowflake infrastructure. The demo uses Snowpark Container Services to generate synthetic load via the Streaming SDK, which lands rows in a Snowflake-managed Iceberg table that is immediately queryable with RBAC, lineage and masking applied. The setup requires no external S3 buckets, EKS clusters, Kafka topics or IAM policies, allowing teams to evaluate streaming throughput in a single afternoon without additional provisioning or platform-team sign-offs.

The demonstration leverages a GitHub repository with Python code that can be adapted to Java, Node or REST SDKs, enabling engineers to validate performance claims using their own stack. Inside the test environment, multiple Docker containers running in Snowpark Container Services produce synthetic events at the target TPS rate, stream them through the SDK and commit to an Iceberg table managed by Snowflake. Authentication is handled via short-lived OAuth tokens injected by SPCS, eliminating secret management. The same code can later target an external S3 volume or remain on managed storage, with no changes required to the streaming logic.

Snowflake notes that Snowpipe Streaming does not require SPCS for operation, but running the SDK inside SPCS accelerates evaluation by removing the need to provision external infrastructure. Rows are buffered server-side and committed in batches, avoiding per-row file creation, while Snowflake automatically compacts small Parquet files and manifests in the background for managed Iceberg tables. Despite these optimizations, frequent commits can still create metadata overhead in Iceberg’s current design, as each commit writes new metadata JSON, manifest list and manifest files even for small batches.

To address this, the Iceberg v4 proposal introduces an Adaptive Metadata Tree that inlines small commits into the root manifest, replacing repeated metadata writes with a single Parquet write and an atomic pointer swap. This reduces metadata I/O per commit to a constant level, improving streaming latency for high-frequency workloads. Snowflake participates in the v4 specification process and highlights streaming as a key scenario for the new metadata design. The company invites teams to clone the repository, deploy to Snowpark Container Services and validate streaming performance without external infrastructure.