Hyperdisk:  Doubling throughput for half the price 

In Customer.io, we operate an ever-growing number of MySQL instances which holds petabytes of data. This fleet contains the core OLTP for all message processing and underpinning the whole operation is Google Cloud’s block storage.

“At the right time” has a hidden meaning: low latency. Our MySQL fleet keeps data in the hot path to provide the lowest possible latency, helping you send the right messages to your audiences at exactly the right time. How did we double our peak throughput while reducing costs?

Balanced Persistent Disks (PD-Balanced) are network-attached block storage backed by Solid State Disks (SSDs). The performance of these disks are limited by 2 important numbers: IOPS and Throughput: the maximum achievable of which are 80,000 IOPS and 1,200 MiB/s of independent reads/writes. This means that PD-Balanced is only ever able to write 1,200 MiB/s under the most optimal conditions.

When we reach those performance limits, GCP throttles the disk causing disk I/O latency to increase significantly. Google previously provided a metric to quantify the throttling, instance/disk/throttled_write_bytes_count , but has since removed it. Instead we must infer throttling by a spike in node_disk_write_time_seconds_total.

Google made Hyperdisk generally available in 2024. This storage type shares many characteristics with Persistent Disk; however, Hyperdisk’s higher performance limits are shared across reads and writes. Hyperdisk Balanced (HDB) supports up to 2,400 MiB/s of combined read/write throughput.

Measure twice, cut once. To ensure compatibility with our fleet, we needed to validate that HDB provides at least the same performance as PD-Balanced. Google publishes a cloud benchmarking tool, PerfKitBenchmarker, which compares VM and disk performance metrics. Benchmark testing is important because the VM type and CPU count determines Hyperdisk’s performance limits. Running --benchmarks=fio,fio_max_iops,fio_latency_sla , we saw HDB reach 2,400 MiB/s Read-Write throughput giving us confidence in Hyperdisk.

MySQL is well known for having a triple write penalty. For every 16 KB of data written, you can expect to use roughly 3x that amount in disk throughput. This makes MySQL write-heavy, often saturating write throughput while overall read throughput can remain low. That pattern is inefficient for PD-Balanced, whose independent throughput limits are lower than HDB’s combined throughput.

Hyperdisk Balanced’s higher peak write throughput is a better fit for this workload. With HDB effectively providing twice the throughput through a combined limit, MySQL can handle its write demand with a more stable disk I/O latency. By migrating to Hyperdisk, our fleet has gone from PD-Balanced’s 1,200 MiB/s to HDB’s 2,400 MiB/s of combined throughput.

Achievement Unlocked: Double throughput!

HDB also offers another major benefit: it halves the price of block storage by utilizing Advanced Hyperdisk Pools. Advanced Hyperdisk Pools are logical, thinly provisioned groupings of disks in which the volumes are compressed and deduplicated in a way that is transparent to your workload. There is no performance difference between pooled and unpooled Hyperdisks.

While the absolute price of Advanced Hyperdisk Pools is higher, we only pay for the bytes utilized. Google does not publish information on how the Hyperdisk Pools’ compression and deduplication works, but for our MySQL workloads, we see a 65% data reduction. Workloads using innodb table compression may not see such a huge benefit. After adjusting for the cost of provisioned IOPS and throughput, Advanced Hyperdisk Pools is half the cost we paid for Persistent Disks.

Achievement Unlocked: Half the price!

To keep our workloads efficient, we rebalance customer workspaces between MySQL instances. When particularly large workspaces leave an instance, that instance is left with a large amount of free space. With Hyperdisk or Persistent Disks you pay for the capacity of the disk regardless of free space which is a huge waste to costs.

Because neither Hyperdisks nor Persistent Disks can be shrunk, you must copy the data to a smaller disk. Data transfer time is non-negligible for large disks and the transfer between Availability Zones can be costly depending on how frequently a shrinking operation must be done. However with Hyperdisk Storage Pools we only pay for what is provisioned meaning we don’t have to pay for wasted free space. Advanced Hyperdisk Storage Pools have eliminated our need to shrink our disks, saving us engineering time and lowering our total cost of ownership.

Achievement Unlocked: Reduced operations burden!

Part of the 3x write penalty in MySQL is innodb_doublewrite also known as torn-page protection. This essentially means that half of our write throughput is dedicated to writing pages twice, which is how MySQL protects against torn pages caused by disk power failures. Here is an excellent talk from LSFMM 2024 summit about Torn Pages for anyone interested in the subject.

Historically disabling innodb_doublewrite would need specialized hardware with battery backup disks. However, Hyperdisks support 128 KB atomic writes natively! As of writing, Google’s emulated NVMe device Atomic Write Unit Power Fail advertisement isn’t consistent for all machine types; making native calls to pwritev2 with RWF_ATOMIC appear to fail. Using the default MySQL page size and the right bigalloc disk format with Hyperdisk is enough to guarantee device-level torn-write protection. Google publishes recommended MySQL settings that are required for torn-write protection. We can see immediate improvements from Hyperdisk’s atomic write safety.

Eliminating doublewrite not only has improved our peak disk throughput, we observed improvements in our foreground flush latency as well. When a MySQL transaction requires LRU or sync flushing, the fsync associated with innodb_doublewrite increased transaction commit latency. By relying on Hyperdisk’s Atomic Write Unit Power Fail guarantees, we have turned off innodb_doublewrite for our MySQL fleet and observed improved latency across the board.

Achievement Unlocked: Triple throughput!

Operating MySQL at our scale has interesting challenges in how we approach balancing storage while creating the fastest possible message latency. Hyperdisks represent a huge performance boost, and Advanced Hyperdisk Pools represent significant operational and cost savings. At Customer.io, our Platform team is always looking for ways to improve our ability to send your messages to the right person at light speed. This was a necessary first step toward unlocking our next architectural changes. Stay tuned for more exciting petabyte-scale MySQL endeavors!