Rewinding the exponential: How we keep hardware usage down
Cloudflare’s Data team reduced hardware growth pressures by rearchitecting its data pipeline, cutting transit traffic by up to 40% through schema changes, compression shifts, and decentralized sampling since 2022.
Cloudflare’s Data team manages a global pipeline that processes over two billion events per second for services including Cloudflare Logs and internal analytics. The pipeline spans tens of thousands of servers, where exponential traffic growth has historically outpaced linear hardware expansion. To control costs, the team implemented a series of architectural changes over five years, including breaking up a monolithic schema in 2022, switching compression algorithms in 2024, and redesigning pipeline replication and sampling in 2026.
In 2022, Cloudflare dismantled its single monolithic data schema, which forced internal consumers to ingest all fields even when irrelevant to their needs. By creating product-specific schemas, the team reduced backbone bandwidth usage by 35–40%, effectively offsetting roughly a year’s worth of compressed traffic growth without duplicating data unnecessarily.
In 2024, Cloudflare replaced LZ4 compression with zstd across its fleet after testing showed minimal CPU impact. The change delivered nearly a 40% reduction in compressed traffic throughput while supporting 1.5 times more traffic through the same infrastructure, deferring hardware expansion by an estimated 10 months.
In 2026, Cloudflare introduced three strategies to reduce uncompressed bytes: adopting a shared ingest pool for analytics streams, implementing late replication from centralized regions, and expanding sampling earlier in the pipeline at the Logfwdr level. These changes cut wide-area network traffic by 36 GB/s for migrated datasets like HTTP analytics and Ready-Analytics, while maintaining disaster resilience.