Maximizing Apache Spark availability: Mitigating compute stockouts with flexible VMs and other best practices
Google’s Managed Service for Apache Spark introduces flexible VMs to mitigate compute stockouts, allowing clusters to dynamically adapt to regional capacity shortages and maintain pipeline availability.
The surge in AI development has intensified global demand for compute capacity, often leading to availability constraints for Apache Spark workloads. Whether using self-managed or managed Spark services, teams face risks such as cluster creation delays or failed executions due to rigid single-VM requirements. Google’s Managed Service for Apache Spark addresses this by offering flexible VMs, which enable clusters to request compute resources across multiple machine families rather than being bound to a single instance type. This approach reduces the likelihood of provisioning failures during regional or zonal capacity stockouts.
Flexible VMs allow teams to define an ordered list of acceptable machine families for master, primary worker, and secondary worker nodes, supporting multi-family blending across generations. For example, configurations can mix Gen2 families like N2 or N2D with Gen4 families such as N4 or C4. The feature also broadens storage options by dynamically adapting to disk types supported by the underlying host family. By applying flexible rules to all node types, including preemptible or spot workers, clusters can be provisioned end-to-end even under capacity constraints.
To maximize availability, Google recommends specifying at least two machine families in the highest priority tier of the flexible VM list. For pipelines using legacy shapes like n1-standard-16, transitioning to newer architectures such as n2d-standard-16 can improve resilience. Modern storage architectures like Hyperdisk Balanced are also recommended, as newer instance families rely on it for predictable performance across variable VM sizes.
Beyond flexible VMs, additional strategies include implementing AutoZone routing to select optimal zones, using smaller machine shapes to avoid high-demand configurations, and deploying cluster autoscaling with reasonable maxInstances. Configuring partial cluster creation and establishing regional fallbacks further enhances resilience. Managed Spark clusters benefit from built-in platform resilience and these architectural approaches to protect critical pipelines from hardware shortages.