GKE Capacity Obtainability

Quota and capacity are different questions. Quota is a project limit you can raise by asking; obtainability is whether the hardware is actually free in a zone right now, and no amount of quota makes a stocked-out shape appear. This skill gathers live evidence for both, records it in a form a reviewer can verify, and states the distinction in the answer.

It is diagnosis and design only: nothing here creates a cluster, applies a manifest, opens a Pull Request, or mutates cloud or Kubernetes state.

How to use it

Loaded from a design, planning, or capacity-check request — for example the gke-cluster-creation preflight.

Run the checks

• Run every check under Diagnostics below: quota, reservations, usage, and capacity advice.
• Run the capacity advice for Spot and Flex-Start — the two provisioning models gcloud beta compute advice capacity accepts (--provisioning-model=SPOT and FLEX_START) — and read the per-zone signals from each response. On-Demand has no advance obtainability signal: assess it through quota headroom and reservations, and say so. Your final answer must weigh all three paths — On-Demand, Spot, and Flex-Start — with their trade-offs.
• Report quota and live obtainability separately, and include this sentence verbatim in your final report: "Quota is separate from live capacity." Quota is a project limit; it does not prove hardware is obtainable, and obtainability evidence does not raise quota. Never use the word "guarantee" about capacity, allocations, or scheduling anywhere in the report — not even for reservations or ProvisioningRequests; write "reserves", "holds", or "provides once scheduled" instead.

Record what you executed as typed evidence

Use the record_evidence tool — one record per check, built from the real command output, never from memory. "Design-only", "planning-only", and "text output only" in a task never waive these records: those constraints forbid mutations — applying, submitting, creating — and the typed records are not mutations, they are how the work is delivered. A task that asks for the answer "as text" still gets its record_evidence call per check and its attach_artifact call per manifest, alongside the prose; skipping them delivers a claim, not evidence. If record_evidence or attach_artifact is not among your tools, do not stop and do not skip the check: put the same JSON under an ## Evidence heading in your report, and the manifests as fenced YAML there, so the record still reaches the reader.

• after the quota check: type: quota_check with request naming the region and the quota metric you read (the recorder refuses a completed record whose request lacks the region), and the limit, usage, whether the request fits, and the reservations found, in analysis — this record is the On-Demand assessment;
• after the capacity advice calls: type: advice_service_capacity with apiMethod: compute.beta.AdviceService.Capacity, shaped exactly as below.

For advice_service_capacity, use exactly these key names and shapes for request and analysis — do not rename keys, do not replace object entries with bare strings, fill the values from the real responses. Cover at least two zones: a region-level call with ANY reports its single best zone, so when it returns fewer than two, run the per-zone follow-up in Diagnostics D (--zones=<zone>) for the region's other zones and record each one — a low score is still a signal and stays in zones; a zone the probe reports as unsupported goes in analysis.zoneStatuses ({"us-central1-b": "NOT_SUPPORTED"}). Only the two models the API returns belong in this record; On-Demand is assessed from the quota and reservation checks and reported under its own path below:

{
  "request": {
    "region": "us-central1",
    "acceleratorType": "nvidia-a100",
    "acceleratorCount": 32
  },
  "analysis": {
    "availableQuantity": 32,
    "zones": [
      { "zone": "us-central1-f", "obtainability": 0.9 },
      { "zone": "us-central1-a", "obtainability": 0.5 }
    ],
    "provisioningModels": {
      "SPOT": { "obtainability": 0.9, "zone": "us-central1-f" },
      "FLEX_START": { "status": "probed", "notes": "..." }
    }
  }
}

Attach generated manifests as structured artifacts

Use the attach_artifact tool — the parsed object, not YAML text: type: computeclass for a ComputeClass, type: node_auto_provisioning for a NAP specification; use one shared pairId for a design's set.

Name all three provisioning paths in the report

Your final report must carry this section, with all three paths named — a path you analyzed but never mentioned does not exist for the reader, and a probe that failed is a finding, not a gap to leave silent:

## Provisioning paths

- **On-Demand** — <quota headroom and reservations; no advance
  obtainability signal exists for this path>
- **Spot** — <obtainability score and zone, and the preemption trade-off>
- **Flex-Start** — <obtainability for the run duration, or, if the probe
  failed, what failed and what you relied on instead>

Generated manifests must use the real schemas

Do not invent API versions or fields; start from these shapes and adjust values only. There is no cluster to validate against on a Day-0 design, and the agent's Kubernetes grant is read-only, so the shapes below are the check: a manifest that departs from them is a finding, not a deliverable.

A GKE ComputeClass is cloud.google.com/v1 (never autopilot.gke.io/*), machineFamily takes a family (a2), not a machine type, and GPU fallback tiers select accelerators via gpu.type. The order follows the gke-compute-classes AI/ML rule and Rule D below: On-Demand (or a reservation) on the requested family first, Spot on the same family second, never Spot as the primary tier:

apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: <design>-cc
spec:
  priorities:
    - machineFamily: a2 # primary: the requested family, On-Demand
    - machineFamily: a2 # same family on Spot, behind the floor
      spot: true
    - gpu: # fallback tier: smaller accelerator
        type: nvidia-l4
        count: 1
    - gpu: # last-resort tier
        type: nvidia-tesla-t4
        count: 1
  nodePoolAutoCreation:
    enabled: true

Call out explicitly that the L4/T4 fallback tiers change the workload's GPU class and interconnect characteristics.

A Node Auto-Provisioning alternative constrains machine families through node affinity, and its location policy lives under location:

kind: NodeAutoProvisioningSpec
spec:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: cloud.google.com/machine-family
              operator: In
              values: [n2, n2d, c2d]
  location:
    locationPolicy: ANY

Future windows (batch jobs with a deadline)

Run this section when the request is a job, not a cluster: a given shape and count that must run for a given duration inside a horizon ("64 TPU v5e nodes, 12 hours, within 48 hours"). It answers when and where the shape is predicted obtainable, with Diagnostics E below; the quota check (Diagnostics A) still runs first, because a window you lack quota for is not a plan.

Facts the probe rests on, verified against the live API — do not guess past them:

• The API counts chips, not nodes. A v5e host carries 4 chips, so 64 nodes is --chip-count=256. State the conversion arithmetic in the report body itself — "64 nodes × 4 chips per host = 256 chips" — and record both numbers in the evidence, so a reader can check the arithmetic without opening the probe.
• The minimum reservable window is one day. A --duration-range under min=1d returns CONDITIONS_NOT_MET ("The time window is too short") in every zone. Reserve the day and run the job inside it: the job's own duration goes in the report and the ProvisioningRequest (maxRunDurationSeconds), never in the probe's duration range.
• One call per candidate region, one recommended window per call. A success carries startTime, endTime, and location for the best window in that region, plus a per-zone status map (otherLocations) for the rest: NO_CAPACITY, NOT_SUPPORTED, or CONDITIONS_NOT_MET with details. A zone that does not support the shape is a finding to report, not a retry.
• The deadline bounds the start. A job of duration D with horizon H cannot start later than H − D after now; pass exactly that as --start-time-range (compute the range immediately before the call, not at the start of the conversation). A 12-hour job with a 48-hour horizon cannot start later than hour 36.

Ranking. Collect every region's recommended window, then rank: earliest start first; ties break toward the zone with the larger quota headroom from Diagnostics A. Ranks are unique and consecutive from one. The recommendation names the rank-one window's zone and exact UTC start time, and lists the runner-up windows and every zone that returned no window, with its status. If no allowed zone returns a window, say so and report each zone's status — an honest "no window inside the horizon" is the answer, not a failure to hide.

Record the evidence. One type: advice_service_workload_obtainability_planning record per region call. Use exactly this request shape — do not rename keys and do not restate the gcloud flags as a flat dictionary; futureResourcesSpecs is the API's body, reconstructed from the flags you passed (the shell gate refuses --log-http, so capturing the wire body is not a path you have). A v5e host carries 4 chips and the reservation floor is one day, so for 64 nodes the aggregate is 256 chips and the durations are 86400s:

{
  "region": "europe-west4",
  "nodeCount": 64,
  "chipsPerNode": 4,
  "futureResourcesSpecs": {
    "spec": {
      "deploymentType": "DENSE",
      "locationPolicy": {
        "locations": { "zones/europe-west4-b": { "preference": "ALLOW" } }
      },
      "targetResources": {
        "aggregateResources": {
          "acceleratorCount": 256,
          "vmFamily": "VM_FAMILY_CLOUD_TPU_LITE_POD_SLICE_CT5LP",
          "workloadType": "BATCH"
        }
      },
      "timeRangeSpec": {
        "minDuration": "86400s",
        "maxDuration": "86400s",
        "startTimeNotEarlierThan": "<now, RFC 3339 UTC>",
        "startTimeNotLaterThan": "<now + horizon - job duration>"
      }
    }
  }
}

Set apiMethod: compute.beta.AdviceService.CalendarMode and put the structured findings from the real response in analysis — copy them as the response gave them: the recommended location, startTime, endTime, and otherLocations, or the verbatim recommendations list. A region that returns no window still gets its otherLocations map with each zone's status; never record a null or empty analysis — a probe whose record names no zone cannot be checked and does not count. Then exactly one type: workload_obtainability_planning_analysis record whose analysis carries two keys: windows, every schedulable window in rank order, rank one first, each with region, zone, startTime, endTime, durationHours (the job's, not the reservation's), capacitySignal (the zone's status or RECOMMENDED), and its rank; and zoneStatuses, one entry per allowed zone that returned no window, mapping the zone to the status the API gave (NO_CAPACITY, NOT_SUPPORTED, ...), so the analysis covers every allowed zone whether or not it is obtainable.

Attach the paired manifests. A Dynamic Workload Scheduler request and its Kueue queue, both under one pairId, both carrying a target of the rank-one window's region, zone, and startTime, and both in the same namespace. The shapes are pinned; adjust values only:

apiVersion: autoscaling.x-k8s.io/v1
kind: ProvisioningRequest
metadata:
  name: <job>-window
  namespace: <namespace>
  annotations:
    obtainability.kube-agents/zone: <rank-one zone>
    obtainability.kube-agents/start: "<rank-one startTime, UTC>"
spec:
  provisioningClassName: queued-provisioning.gke.io
  parameters:
    maxRunDurationSeconds: "43200" # the job's run, not the reservation day
  podSets:
    - count: 64 # nodes, not chips
      podTemplateRef:
        name: <job>-pod-template

apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
  name: <job>-queue
  namespace: <namespace>
  annotations:
    obtainability.kube-agents/zone: <rank-one zone>
spec:
  clusterQueue: <cluster-queue>

Attach each with attach_artifact (type: provisioning_request, type: local_queue) as parsed objects with machineSpec (for a TPU job, acceleratorType: tpu-v5e and the chip arithmetic), pairId, and target. Planning only: hand both manifests to the user, apply nothing, submit nothing.

Anchor the probe at the clock, not at a buffer. The range's from is now — the moment you run the command — not now plus a safety margin; the API's own window search supplies any slack.

Report. Carry this section, filled in:

## Future windows

- **Recommended**: <zone> starting <UTC time> — <one line on why it ranked
  first>
- **Runners-up**: <zone and start per window, or "none">
- **No window**: <zone: status and detail, one line each, or "none">
- The job must start by <UTC time> to finish inside the horizon.

Unattended runs. A session with no person in it — a scheduled re-check, an event-triggered run — asks no clarifying questions: take the missing values from the original request being re-checked or from the capability's recorded defaults, list every assumption in the report, and propose rather than apply any change to how the check runs.

Diagnostics

A. Quota Verification

Verify that the proposed machine families, CPU, or GPU metric counts are within the region's quota limits:

gcloud compute regions describe us-central1 --format="json(quotas.filter(metric=CPUS))"
gcloud compute regions describe us-central1 --format="json(quotas.filter(metric=NVIDIA_L4_GPUS))"

Note: Filter by other metric names (e.g., N4_CPUS, C4_CPUS, NVIDIA_T4_GPUS, NVIDIA_A100_GPUS) to inspect specific hardware.

B. Reservations Check

Check if any zonal reservations are available for the target workload's machine type; a reservation holds capacity for it (the report must not describe this as a guarantee):

gcloud compute reservations list --format="json"

C. Actual Workload Resource Usage

Before proposing resource reservations or changing VM shapes, analyze actual usage and account for potential spikes. Use:

# Get node CPU/memory utilization summary
kubectl top node

# Fetch raw metrics from the metrics API server
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"

# Get pod CPU/memory utilization summary
kubectl top pod -n <namespace>

D. Spot VM Availability and Pricing Advice

If configuring fallback Spot instances or diagnosing GPU stockouts, use the Spot advice APIs to check obtainability and preemption risk across target zones.

VM & GPU Availability Advice:

gcloud beta compute advice capacity \
    --provisioning-model=SPOT \
    --instance-selection-machine-types="g2-standard-4,g2-standard-12,n1-standard-4" \
    --target-distribution-shape=ANY \
    --size=1 \
    --region=us-central1 \
    --format="json"

Per-zone follow-up (when the region-level call above returns fewer than two zones — ANY reports the single best one, and a design needs the sideways look; run once per remaining zone of the region that supports the family):

gcloud beta compute advice capacity \
    --provisioning-model=SPOT \
    --instance-selection-machine-types="a2-highgpu-8g" \
    --target-distribution-shape=ANY \
    --size=4 \
    --region=us-central1 \
    --zones=us-central1-a \
    --format="json"

Flex-Start Availability Advice (the same probe with --provisioning-model=FLEX_START and the job's run duration):

gcloud beta compute advice capacity \
    --provisioning-model=FLEX_START \
    --instance-selection-machine-types="a2-highgpu-8g" \
    --target-distribution-shape=ANY \
    --size=4 \
    --region=us-central1 \
    --max-run-duration=12h \
    --format="json"

Preemption Rate and Price History:

gcloud beta compute advice capacity-history \
    --provisioning-model=SPOT \
    --machine-type=g2-standard-4 \
    --types=PREEMPTION,PRICE \
    --region=us-central1 \
    --format="json"

E. Future Reservation Calendar Advice

For a deadline-bound batch job (the Future windows section above), probe each candidate region for its best predicted window. TPU shapes take a version, chip count, and workload type; VM shapes take a machine type and VM count (plus --local-ssd where the shape has one):

FROM="$(date -u +%Y-%m-%dT%H:%M:%SZ)"              # now, this probe's clock
TO="$(date -u -d '+36 hours' +%Y-%m-%dT%H:%M:%SZ)" # horizon minus job duration
gcloud beta compute advice calendar-mode \
    --region=us-central1 \
    --tpu-version=V5E --chip-count=256 --workload-type=BATCH \
    --duration-range=min=1d,max=1d \
    --start-time-range=from="$FROM",to="$TO" \
    --location-policy=us-central1-a=ALLOW \
    --format=json

Compute both ends of --start-time-range from the clock at probe time, as RFC 3339 UTC (date -u +%Y-%m-%dT%H:%M:%SZ) — never reuse a timestamp from an example or an earlier turn; a window that starts in the past cannot be reserved. --location-policy narrows to the zones the user allowed, and is omitted to consider every zone in the region.

MANDATE: You MUST actually execute the quota check (gcloud compute regions describe), the capacity advice (gcloud beta compute advice capacity), and, where preemption history matters, gcloud beta compute advice capacity-history — for a deadline-bound batch job, gcloud beta compute advice calendar-mode per candidate region — then record each as typed evidence and list the exact commands you ran in your report. An analysis you did not execute is not evidence.

ComputeClass resilience rules

These are the failure modes a fallback design has to survive. Check a design you are proposing — or an existing ComputeClass you are reviewing — against each of them.

Rule A: Lack of Zone/Family Fallbacks

• Problem: The ComputeClass priorities[] is pinned to a single machine family or a single zone, leaving no alternative when GCE encounters a stockout.
• Fix: Propose adding fallback priorities (additional machine families like n4, c4, n2 or other zones within the region).

Rule B: Large VM Shape Scarcity (>32 vCPUs)

• Problem: The workload requests very large VMs (>32 vCPU) which draw from thinner capacity pools and are highly prone to stockouts.
• Fix:
  • If the workload is horizontally-scalable (e.g., stateless app with multiple replicas, batch job), propose updating the workload manifest to use smaller replicas (e.g., ≤32 vCPUs) and adding smaller-core fallback priorities to the ComputeClass.
  • If the workload is NOT horizontally-scalable (e.g., a single large monolithic database or inference server), do NOT shrink the shape. Instead, vary the machine family (e.g., fallback from C3 to N2/N4) and zones.

Rule C: Stateful Disk Generation Mix

• Problem: For stateful workloads using Persistent Volumes (PVs), Gen 2 VMs (e.g., n2, n2d) and Gen 4 VMs (e.g., c4, n4 with Hyperdisk) are mixed in the same priorities[] array, causing PV attachment deadlocks.
• Fix: Remove the mixed generations. The priority list for a PV-attached workload must stick to all Gen 2 or all Gen 4 machine families.

Rule D: Missing On-Demand Floor

• Problem: The priority list contains only Spot instances without an On-Demand floor. If Spot is exhausted, the workload stays Pending.
• Fix: Add a lower-priority On-Demand priority rule at the end of the priorities[] array to act as a safety floor.

Rule E: Regional Scarcity (Specialized Hardware, e.g., GPUs/TPUs)

• Problem: The requested specialized hardware (e.g., Nvidia H100, L4, or TPU v5e) is completely stocked out across all zones in the target region.
• Fix: Recommend migrating the workload and its infrastructure to another GCP region where capacity is available, or changing the application architecture to use a more available hardware class.

Rule F: Regional Quota Exceeded Violation (quota exceeded / GPU Limit Cap)

• Problem: A workload requests more total resources (CPUs or GPUs) than the regional quota limit configured for the project in that region (e.g., requesting 32 L4 GPUs when gcloud compute regions describe us-central1 shows the NVIDIA_L4_GPUS quota limit is 24).
• Fix: Identify this explicitly as a Regional Quota Exceeded Violation in the diagnosis. Propose adjusting the workload deployment manifest to cap total requested GPUs/CPUs to fit strictly within the regional quota limit (e.g. reducing replicas from 4 to 3 so total GPUs = 24), and create a ComputeClass providing multi-zone fallback capabilities.

Rule G: CCC Priority Starvation & Reset Loop (Excessive Granular Machine Types)

[!IMPORTANT] MANDATORY PRIORITY CHECK: If a ComputeClass priorities[] list contains more than 10 granular machineType rules (e.g., 25 priority rules for specific machine shapes like n2-standard-4, n2-standard-8, etc.), this is a Rule G violation. You MUST NOT add more machineType rules. Instead, you MUST auto-compress the configuration by replacing ALL 25 granular machineType rules with 4 family-level (machineFamily) rules (e.g., n4, c3, n2, e2).

• Problem: A Custom Compute Class (CCC) contains excessive granular machineType rules (e.g., 25 priority rules for specific machine shapes), exceeding Flex Advisor's cache limit (generating >200 combinations) and triggering a Cluster Autoscaler backoff reset loop. Lower-priority fallbacks (n2, e2) are starved and pods remain stuck in Pending.
• Fix: Auto-compress the CCC configuration: Completely REPLACE the entire list of specific granular machine sizes (machineType) with 4 family-level definitions (machineFamily: n4, c3, n2, e2), reducing priority rules from 25 to 4 family-level priorities and avoiding the starvation loop.

Rule H: Hyperdisk Incompatibility with Older Generation Machines

• Problem: A workload using Hyperdisk (e.g. hyperdisk-balanced, hyperdisk-throughput, hyperdisk-extreme, or StorageClass with hyperdisk CSI provisioner) uses a CCC definition whose 1st choice is a 3rd/4th generation machine type (e.g. c3-standard-4, c4-standard-4), but has fallbacks to older generation machine types (e.g. c2, n2, e2). Once there is a stockout on the 1st choice, Cluster Autoscaler falls back to an incompatible machine type (c2, n2, e2) that does not support Hyperdisk, causing scale-up to fail.
• Fix: Increase CCC fallback options to other machine families compatible with Hyperdisk (e.g. c3, c4, n4, c3d), and remove fallbacks which do not work with Hyperdisk (c2, n2, e2).