Enabling Cloud Storage end-to-end checksums for improved data integrity and durability

Cloud Storage has long stored checksums for every object, but developers previously had to manually provide checksums for uploads or verify them on downloads, leaving gaps in protection. The latest SDK versions now automatically compute and pass checksums for uploads if not provided by the application, and verify them during downloads, closing these gaps. This change ensures data integrity from the application layer through to disk storage without requiring additional developer effort.

Bit flips, though rare, can occur during data processing at scale, such as during encryption where plaintext is copied to a new memory buffer for ciphertext. To prevent corruption, Cloud Storage reverses the process: it calculates a checksum after encryption, decrypts the data, and verifies the plaintext matches the original before proceeding. This safeguard, though CPU-intensive, is routinely applied at Google’s scale to maintain data integrity.

Data in Cloud Storage is split into chunks with individual checksums, then grouped into shard files for efficient storage. These shards are further divided into blocks protected by Reed Solomon encodings across Colossus, Google’s storage system. Chain-of-custody is preserved using CRC properties, such as concatenation, to compute shard-level checksums from chunk-level CRCs without re-checksumming the data.

On reads, data is verified at multiple layers: Colossus verifies against inline checksums, and the Cloud Storage frontend checks each chunk before sending it to the client. Range reads via gRPC leverage built-in end-to-end checksums, allowing SDKs to verify data integrity without losing custody. These layered verifications ensure data remains uncorrupted from upload to final download.