Amazon MSK Express brokers now deliver data to streaming tables for Apache Iceberg
Amazon MSK Express brokers now stream data directly to Apache Iceberg tables on Amazon S3, cutting ingestion costs by up to 60% and reducing downstream query costs by up to 30%.
Amazon Web Services (AWS) announced that Amazon MSK Express brokers can now continuously materialize Apache Kafka topics as Apache Iceberg tables on Amazon S3 Tables. This capability reduces the cost of ingesting and delivering Kafka data into Amazon S3 Tables by up to 60% compared to self-managed deployments, while also lowering downstream query costs by up to 30% versus self-managed Kafka setups.
Customers using Apache Kafka for real-time data ingestion—such as fraud detection or personalization—often face challenges integrating it with Apache Iceberg tables for near real-time analytics. Previously, this required complex custom pipelines, format conversions, and managing the small-file problem, where high-volume ingestion creates many small Parquet files that slow queries and increase costs.
The new feature addresses these issues by providing intelligent inline compaction to eliminate the performance impact of small files while maintaining data freshness. It also resolves concurrent writer conflicts across high-throughput consumers. Amazon MSK supports throughput of up to 10 GB/s for delivery to Apache Iceberg on Amazon S3 Tables, with no additional broker egress throughput costs.
To enable the capability, customers can use the Amazon MSK console, APIs, or MCP server. The feature is available today in all AWS Regions where Amazon MSK Express brokers are offered. For pricing details, customers should refer to the AWS pricing page, while additional information is available in the Amazon MSK Developer Guide and Amazon MSK AI skills.