AWS Lambda now supports queues for Apache Kafka in self-managed Kafka event sources

AWS Lambda now supports queue-based event source mappings for self-managed Apache Kafka clusters, including managed services like Confluent Cloud. This feature allows customers to scale Kafka workload processing beyond topic partition limits, improving throughput during traffic surges and enhancing resilience. Previously, Lambda relied on Kafka’s consumer group protocol, which restricted parallelism to partition counts and required additional infrastructure for higher scalability.

Queue support is built on Kafka share groups, available in Apache Kafka 4.2 and later. Each event poller can now handle up to 10 concurrent invocations per partition, decoupling parallelism from partition count. This eliminates the need to over-partition topics or integrate separate queuing systems, reducing operational overhead while maintaining compatibility with existing producers and topic configurations.

The new capability improves processing resilience by preventing slow or failed events from blocking other events in the same partition. However, queue mode does not preserve partition ordering, making it suitable for workloads where strict ordering is not required, such as image processing or telemetry ingestion. Customers must enable share groups on their Kafka cluster to use this feature.

To enable queue mode, customers can configure a self-managed Kafka event source mapping with ConsumptionMode set to Queue via the AWS Management Console, API, CLI, CloudFormation, SAM, or CDK. Standard Lambda pricing applies, with additional charges for Event Poller Unit usage in provisioned mode. Detailed guidance is available in the AWS Lambda Developer Guide and pricing documentation.