Connexion
Maximizing Apache Spark availability: Mitigating compute stockouts with flexible VMs and other best practices
The surge in AI development has created unprecedented demand for compute capacity around the globe. This can have negative implications for data processing and pipelines with Apache Spark. Whether you are managing your own Spark infrastructure or using a managed service, you can face availability constraints. However, a significant advantage of using Google’s Managed Service for Apache Spark is the availability of flexible VMs, which provide a targeted mechanism to adopt a dynamic, resource-agnostic philosophy and ensure your pipelines remain operational, even during regional or zonal capacity stockouts.
Understanding capacity stockouts
Capacity stockouts occur when demand for a specific machine family (such as N2 or N2D) exceeds available capacity in a target zone or region. For time-sensitive analytics pipelines, rigid single-VM requirements transform standard provisioning into a single point of failure which can result in cluster creation delays, failed executions, and potentially compromised business SLAs.
Flexible VMs
Flexible VMs fundamentally overhaul how a Managed Spark cluster requests compute resources. Rather than binding a cluster to a rigid instance type, flexible VMs allow teams to establish an ordered list of acceptable machine families for master, primary worker, and secondary worker nodes.
Key features
-
Multi-family blending: Mix nodes across diverse machine types and generations, combining Gen2 families (e.g., N2, N2D) with Gen4 families (e.g., N4, C4) in a single configuration.
-
Mixed storage support: Broaden available capacity pools by allowing storage options to dynamically adapt to the underlying host family's supported disk types.
-
Comprehensive cluster coverage: Apply flexible rules to primary workers, secondary (preemptible/spot) workers, and master nodes to guarantee cluster provisioning end-to-end.
Ranked configuration: A strategy for success
A successful flexible VM implementation relies on intentional ranking. By defining a clear hierarchy of options, Managed Spark clusters automatically attempt provisioning, systematically mitigating stockout risks without requiring manual intervention. To improve the availability of suitable VMs, we recommend specifying at least two machine families in the highest priority (Rank 0) flexible VM list.
As an example, for production pipelines standardizing on n2d-standard-16 shapes, the following tiering strategy provides robust resilience against capacity constraints:
|
Rank |
Machine family examples |
Storage recommendation |
|---|---|---|
|
Rank 0 (Primary) |
n2d-standard-16, n2-standard-16 |
Standard Local SSD or PD |
|
Rank 1 |
n4-standard-16, n4d-standard-16 |
Hyperdisk Balanced |
|
Rank 2 |
c4-standard-16, c3-standard-22 |
Hyperdisk Balanced |
|
Rank 3 |
e2-standard-16 |
Standard PD |
- code_block
- <ListValue: [StructValue([('code', 'gcloud dataproc clusters create $CLUSTER_NAME \\\r\n--num-workers=10 \\\r\n--zone="" \\\r\n--region=us-east1 \\\r\n--worker-instance-selection=\'{"machineTypes":["n2d-standard-16","n2-standard-16"],"rank":0,"diskConfig":{"bootDiskType":"pd-standard","bootDiskSizeGb":400}}\' \\\r\n--worker-instance-selection=\'{"machineTypes":["n4-standard-16","n4d-standard-16"],"rank":1,"diskConfig":{"bootDiskType":"hyperdisk-balanced","bootDiskSizeGb":400}}\' \\\r\n--worker-instance-selection=\'{"machineTypes":["c4-standard-16","c3-standard-22"],"rank":2,"diskConfig":{"bootDiskType":"hyperdisk-balanced","bootDiskSizeGb":400}}\' \\\r\n--worker-instance-selection=\'{"machineTypes":["e2-standard-16"],"rank":3, "diskConfig":{"bootDiskType":"pd-ssd","bootDiskSizeGb":400}}\' \\\r\n--master-instance-selection=\'{"machineTypes":["n4-standard-16","n4d-standard-16"],"rank":0,"diskConfig":{"bootDiskType":"hyperdisk-balanced","bootDiskSizeGb":400}}\''), ('language', ''), ('caption', <wagtail.rich_text.RichText object at 0x7fb3fbbd5a90>)])]>
For pipelines standardizing on legacy n1-standard-16 shapes, the following tiering strategy helps transition workloads toward newer, more available architectures while preserving operational stability:
|
Rank |
Machine family examples |
Storage recommendation |
|---|---|---|
|
Rank 0 (Primary) |
n1-standard-16 n2-standard-16 |
Standard Local SSD or PD |
|
Rank 1 |
n2d-standard-16 |
Standard Local SSD or PD |
|
Rank 2 |
n4-standard-16 n4d-standard-16 |
Hyperdisk Balanced |
|
Rank 3 |
e2-standard-16 |
Standard PD |
Leveraging Hyperdisk Balanced
Unlocking maximum availability with flexible VMs often requires adopting modern storage architectures like Hyperdisk Balanced. Newer instance families (including N4 and C4) rely on Hyperdisk to deliver predictable performance across variable VM sizes. Starting with default IOPS and throughput settings typically provides a reliable baseline for the majority of distributed Spark jobs.
Trade-offs and key considerations
While flexible VMs dramatically improve cluster provisioning success, aligning them with enterprise requirements involves evaluating several architectural and financial factors:
1. Resource quotas
It is no longer enough to have one specific machine (e.g., N2) quota. You need to ensure you have sufficient compute and disk quotas allocated for all specific machine types and disks (including Hyperdisk) defined in their flexible VM lists.
2. Compute flexible Committed Use Discounts (CUDs)
Traditional, resource-based CUDs are tied to specific machine families, which limits flexibility. Adopt Compute flexible Committed Use Discounts (CUDs) to apply savings across multiple VM families and regions.
3. Performance Characteristics
Performance can vary between machine generations, as well as between Local SSD and Hyperdisk. While the Managed Spark team maintains internal benchmarks for these comparisons, actual outcomes are workload-dependent. Testing your specific Spark jobs across these families is essential for understanding SLA impacts.
Additional recommendations
In addition to implementing flexible VMs, there are several other key architectural and scheduling strategies to improve resource availability and workload stability:
-
AutoZone: Implement AutoZone routing to allow Managed Spark to automatically select the zone best suited to execute the job based on current capacity.
-
Smaller machine shapes: Avoid high in demand, large-core shapes. Design workloads and YARN containers to utilize smaller machine shapes (such as 4, 8, or 16 cores). These smaller shapes are much easier to fulfill from the available GCE on-demand pool.
-
Autoscaling: Deploy cluster autoscaling with reasonable
maxInstancesto manage capacity effectively for bursty or unpredictable workloads without relying on rigid, massive upfront provisioning. -
Partial cluster creation: Configure a minimum acceptable number of primary workers. This allows clusters to spin up under resource constraints and begin executing, while autoscaling can dynamically add remaining workers as resources become available.
-
Establish regional fallbacks: Some regions, such as
us-central1,can experience high demand. Setting up fallbacks to other regions reduces capacity stockout risks.
Keep your Spark jobs running with flexible VMs
Managing your own Apache Spark infrastructure can be complex, especially when capacity stockouts disrupt your data processing. Utilizing a managed service like Managed Service for Apache Spark provides unique advantages — including built-in platform resilience and access to flexible VMs. By adopting a prioritized fallback strategy with flexible VMs, you can protect your workloads from regional hardware shortages and keep your critical pipelines running.
Ready to improve your Spark workload resilience? Start configuring flexible VMs for your Managed Spark clusters today.
