Introduction to GKE for Developers
Google Kubernetes Engine (GKE) is Google Cloud's managed Kubernetes service. For the PCD exam, GKE is the answer whenever a workload needs sticky pod identity, fine-grained traffic shaping at the service mesh layer, stateful storage attached to specific pods, or a Kubernetes-native operator. The exam will not test you on raw kubelet tuning — it tests whether you pick the right mode (Autopilot vs Standard), wire identity correctly (Workload Identity instead of static keys), configure pods to actually survive a restart (probes), and choose the right autoscaler (HPA vs VPA vs Cluster Autoscaler) for the failure mode in front of you.
This guide walks every developer-facing surface you need: cluster mode selection, Workload Identity, liveness/readiness/startup probes, the three autoscalers and how they layer, ConfigMap/Secret/Secret Manager CSI patterns, Ingress vs Gateway API, StatefulSet behavior, Backup for GKE, Multi-Cluster Ingress, GKE Sandbox (gVisor), Dataplane V2 (Cilium eBPF), regional vs zonal clusters, and image streaming.
白話文解釋（Plain English Explanation）
Three analogies make GKE click before you wade into the API surface.
Analogy 1: Autopilot vs Standard Is a Hotel vs a House
GKE Standard is buying a house: you own the land (nodes), you pick the appliances (machine types), you decide when to repaint (node upgrades), and you pay the property tax even when you sleep elsewhere (idle nodes still bill). GKE Autopilot is a hotel suite: you tell the front desk how many beds you need (pod resource requests) and the staff handles plumbing, heating, and security. You pay per pod per second — exactly what you sleep in. You cannot rewire the building, and you cannot turn a hotel suite into a house mid-stay. Likewise, you cannot convert an existing cluster between Autopilot and Standard in place — you must create a new cluster of the new mode and migrate workloads.
Analogy 2: Workload Identity Is a Visitor Badge, Not a Master Key
Hardcoded service account JSON keys are like photocopying the building's master key and taping it to every laptop that walks in. Lose a laptop and the building is compromised. Workload Identity is the visitor-badge system: each pod's Kubernetes ServiceAccount (KSA) is bound to a Google ServiceAccount (GSA) via an annotation, and the GKE metadata server hands the pod short-lived tokens scoped to that GSA. The pod never sees a private key. Revoking access is a single IAM policy edit, not a fleet-wide key rotation.
Analogy 3: Probes Are Three Different Health Checks at an Airport
The startup probe is the airline's "have you finished boarding the plane?" question — it runs once during the long initial spin-up and silences the other probes until it passes. The readiness probe is the gate agent asking "is this aircraft ready to take passengers right now?" — fail it and the pod is removed from the Service load balancer but not killed, because maybe a GC pause is happening. The liveness probe is the flight attendant asking "is anyone alive in the cockpit?" — fail it and the kubelet kills the container and restarts it, because the process is unrecoverable. Mixing them up is the most common GKE config bug on the exam: setting a liveness probe that fails during long startup will cause an endless crash loop.
Autopilot vs Standard
GKE ships in two operational modes. Picking correctly is the first decision on any new project and the most likely exam item.
What Autopilot Manages
Autopilot manages every node: provisioning, machine type selection, OS, security patches, upgrades, and right-sizing of node pools. You only declare pods (with explicit resources.requests), and Google bills per pod vCPU/memory/ephemeral-storage second. There are no node SSH access points, no DaemonSets on system namespaces, and no privileged containers — Autopilot enforces a hardened security posture by default (Shielded GKE Nodes, Workload Identity required, no kubelet certificate rotation surface to mismanage).
What Standard Gives You
Standard exposes the node pool API. You pick machine types (e2-standard-4, n2-highmem-8, GPU-attached a2-highgpu-1g), set node count or autoscaling bounds, choose between Container-Optimized OS and Ubuntu, attach Local SSD, allow privileged containers, run system DaemonSets (Datadog agents, custom CNI helpers), and tune kubelet flags. You also pay for every node hour regardless of pod density.
The Conversion Trap
You cannot toggle an existing cluster between Autopilot and Standard. Both gcloud container clusters create-auto and gcloud container clusters create produce immutable mode clusters. Migrating means creating a new cluster of the target mode, applying workloads via GitOps or kubectl apply, and cutting traffic over with Multi-Cluster Ingress or DNS. Plan the choice up front — exam questions about "how do I switch" expect "create a new cluster and migrate" as the answer.
Picking Between Them
Pick Autopilot for greenfield microservices, web APIs, batch jobs, and anything where you want the smallest operational surface. Pick Standard when you need GPUs, TPUs, Local SSD, Windows Server nodes, privileged DaemonSets (Istio sidecar injectors used to be one — now native via Istio Operator on Autopilot), custom node taints for licensed software, or strict per-node bin-packing for memory-heavy single-tenant workloads.
Workload Identity
Workload Identity is the only sanctioned way for pods to call Google Cloud APIs from GKE. It replaces the legacy "service account JSON key in a Secret" pattern entirely.
The KSA → GSA Binding
The mechanism: a Kubernetes ServiceAccount (KSA) in namespace prod annotated with iam.gke.io/gcp-service-account: [email protected] is bound, via an IAM policy binding on the GSA, to the principal serviceAccount:my-project.svc.id.goog[prod/my-ksa]. When a pod uses that KSA, the GKE metadata server intercepts calls to metadata.google.internal and returns short-lived OAuth tokens minted from the linked GSA. No key file ever lives in the cluster.
Enabling It
On Autopilot, Workload Identity is on by default. On Standard, enable it at cluster create with --workload-pool=PROJECT_ID.svc.id.goog and per node pool with --workload-metadata=GKE_METADATA. The metadata server runs as a DaemonSet that returns tokens only when the calling pod's KSA is properly bound.
The IAM Binding Command
gcloud iam service-accounts add-iam-policy-binding \
[email protected] \
--role=roles/iam.workloadIdentityUser \
--member="serviceAccount:my-project.svc.id.goog[prod/my-ksa]"
The KSA annotation alone is insufficient — both sides must agree.
Workload Identity is the only secure way to grant GKE pods access to Google Cloud APIs (BigQuery, GCS, Pub/Sub, Spanner). Static service account keys mounted as Secrets are an exam-trap anti-pattern: they cannot be rotated without redeploying, they leak via container introspection, and they outlive pod lifecycle. The annotation key is exactly iam.gke.io/gcp-service-account; case and dashes matter.
Pod Probes: Liveness, Readiness, Startup
Kubernetes ships three probes. Confusing them is the most common cause of crash-looping pods that nobody can diagnose.
Liveness Probe
The liveness probe runs continuously after startup. If it fails failureThreshold times in a row, the kubelet kills the container and restarts it. Use it only for unrecoverable conditions: deadlocked threads, exhausted connection pools that the app cannot self-heal, infinite GC loops. A liveness probe that returns 500 because a downstream is briefly slow will cascade outages — the pod kills itself, the replacement also fails, and you've manufactured an outage.
Readiness Probe
The readiness probe controls whether the pod's IP appears in the Service Endpoints object. When it fails, traffic stops; when it passes, traffic resumes. The pod is not killed. Use readiness for transient conditions: cache warming after a deploy, a downstream rate-limit you're backing off from, an in-progress GC pause. Readiness should be the probe most apps rely on; liveness should be conservative.
Startup Probe
The startup probe runs once at pod start and disables liveness and readiness while it runs. This solves the JVM-warmup problem: an app that takes 90 seconds to boot will fail any aggressive liveness probe configured for steady-state latency. With a startup probe configured for failureThreshold: 30, periodSeconds: 10 (5 minutes of grace), the kubelet waits for the app to come up before applying tight liveness rules.
Probe Types
All three accept the same probe handlers: httpGet (path, port, scheme, optional headers), tcpSocket (port), exec (command + args inside the container), and the newer grpc (port + service). Use httpGet for HTTP services, grpc for gRPC (since Kubernetes 1.24 it's first-class), and exec sparingly because it forks a process every probe.
Set startupProbe.failureThreshold * periodSeconds to at least 2× your observed P99 cold-start time. A pod that warms in 45 seconds needs roughly 90 seconds of startup grace. Pair with terminationGracePeriodSeconds: 30 so graceful shutdown has time to drain in-flight requests.
HPA, VPA, and Cluster Autoscaler
GKE has three independent autoscalers that solve different problems. They compose, but only if you know which dimension each one moves.
Horizontal Pod Autoscaler (HPA)
The HPA changes pod replica count based on a metric. The default metric is CPU utilization vs resources.requests.cpu — at 50% target with requests.cpu=500m, the HPA tries to keep average CPU at 250m per pod and adds replicas if it's higher. HPA also supports memory, custom metrics from Cloud Monitoring (via the Custom Metrics Adapter), and external metrics (Pub/Sub queue depth, Cloud Tasks backlog). Configure via the HorizontalPodAutoscaler resource with minReplicas, maxReplicas, and metrics.
Vertical Pod Autoscaler (VPA)
The VPA changes a pod's CPU and memory requests based on observed usage. It runs in three modes: Off (recommend only), Initial (set requests at admission, then leave alone), and Auto (evict and re-create pods with new requests when observations drift). VPA is most useful for workloads whose ideal request size you don't know up front — long-tail batch jobs, ML inference with unpredictable input shapes. Do not combine VPA Auto with HPA on CPU/memory: both will fight. The supported pattern is HPA on a custom metric (queue depth) + VPA Auto on CPU/memory.
Cluster Autoscaler
The Cluster Autoscaler changes the node count in a node pool. When a pod is Pending because no node has room, the autoscaler provisions a new node. When nodes have been underutilized for --scale-down-unneeded-time (default 10 minutes), it drains and removes them. On Autopilot this is automatic; on Standard you enable per node pool with --enable-autoscaling --min-nodes=N --max-nodes=M.
The three autoscalers move three different things: HPA = more pods, VPA = bigger pods, Cluster Autoscaler = more nodes. On Autopilot you only think about HPA and VPA because nodes are invisible. On Standard, all three layer: HPA adds pods, those pods go Pending, Cluster Autoscaler adds nodes. Combine VPA Auto with HPA only on non-resource metrics.
ConfigMap, Secret, and Secret Manager CSI Driver
GKE supports three ways to inject configuration into pods. The Secret Manager CSI driver is the modern recommended path for sensitive values.
ConfigMap
ConfigMaps hold non-sensitive key/value data, mounted as env vars or files. Updates to a ConfigMap propagate to mounted files within roughly 60 seconds without restarting the pod — useful for tunable parameters. They are not encrypted at rest beyond etcd's default encryption, and kubectl get configmap -o yaml exposes everything.
Native Secret
The Kubernetes Secret resource is base64-encoded, not encrypted. By default it's stored in etcd, where GKE encrypts it at rest using Google-managed keys. With Application-layer Secrets Encryption enabled, GKE encrypts Secrets with a Cloud KMS key you control. Rotation requires kubectl-applying a new Secret object and either restarting pods or relying on the (eventually consistent) projected volume update.
Secret Manager CSI Driver
The recommended pattern: store secrets in Secret Manager, mount them into the pod via the Secret Manager CSI provider for the Secrets Store CSI Driver. The pod gets a file at /secrets/my-key; Secret Manager owns rotation, audit logging, IAM, and version pinning. The CSI driver authenticates to Secret Manager via Workload Identity — no static keys, no Kubernetes Secret object, no etcd exposure.
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: db-creds
spec:
provider: gke
parameters:
secrets: |
- resourceName: "projects/PROJECT/secrets/db-password/versions/latest"
path: "db-password"
Prefer Secret Manager CSI driver + Workload Identity over native Kubernetes Secrets for any production credential. This combination gives you Secret Manager's version history, IAM-driven access control, Cloud KMS-backed encryption, and rotation without redeploying pods. Native Secrets are acceptable only for opaque short-lived tokens that don't justify the CSI overhead.
Ingress vs Gateway API
GKE supports two north-south traffic APIs. Gateway API is the modern replacement; Ingress remains valid but capped in feature surface.
GKE Ingress
GKE Ingress is the original Kubernetes Ingress implementation backed by Google Cloud Load Balancing. Annotations like kubernetes.io/ingress.class: gce provision an external HTTPS Load Balancer, gce-internal provisions an internal one. It supports path-based and host-based routing, Google-managed SSL certificates (networking.gke.io/managed-certificates), Cloud Armor policies, IAP integration, and Cloud CDN. It is single-cluster, with the multi-cluster variant available via Multi-Cluster Ingress.
Gateway API
Gateway API is the upstream Kubernetes successor to Ingress: a richer model with three resource layers — GatewayClass (provider-level), Gateway (a single load balancer instance), and HTTPRoute/TCPRoute/TLSRoute (the routes attached to it). GKE Gateway controller ships with several GatewayClass options: gke-l7-global-external-managed, gke-l7-regional-external-managed, gke-l7-rilb (internal), and gke-l7-gxlb (multi-cluster). Gateway API supports advanced traffic management — weighted routing, header-based routing, mirror traffic — that Ingress requires custom annotations to fake.
When to Pick Which
Pick Ingress for simple HTTPS routing on a single cluster with managed certs. Pick Gateway API for production multi-cluster topologies, traffic-splitting deploys (canary), or anywhere you'd reach for an Istio VirtualService when you don't actually need a full service mesh.
StatefulSet for Databases and Persistent Workloads
Most GKE workloads are stateless Deployments. The exception is StatefulSet, which is the right primitive for databases, queues, and clustered systems that need sticky identity.
What StatefulSet Guarantees
Each pod in a StatefulSet has a stable metadata.name (mysql-0, mysql-1, mysql-2) and a stable DNS name via the headless Service. Pods are created and deleted in order (0, then 1, then 2; reversed on delete). Each pod gets its own PersistentVolumeClaim provisioned from a volumeClaimTemplate — bound to that pod for life, surviving pod recreations.
Storage Backing
Use Persistent Disk (pd-balanced, pd-ssd) for general workloads, Hyperdisk for IOPS-intensive databases, or Filestore CSI for shared NFS access. On GKE Standard, attach Local SSD for high-throughput temporary scratch — but Local SSD is ephemeral and disappears on node deletion, so never use it as the primary database disk.
Picking a Managed Service First
Before reaching for a StatefulSet of MySQL or PostgreSQL, ask whether Cloud SQL, Spanner, Memorystore, or AlloyDB fits. Self-hosting a database on GKE means owning backups, patching, replication, and failover. On the PCD exam, if the question allows a managed database, prefer the managed option; StatefulSet is the answer when the workload genuinely requires a database you cannot get as a managed service (Cassandra, ScyllaDB, custom Elasticsearch).
Backup for GKE
Backup for GKE is the supported way to back up cluster state and persistent volume data. It's billed separately and supports cross-region restore.
What It Backs Up
A BackupPlan captures namespaced and cluster-scoped Kubernetes objects (Deployments, Services, ConfigMaps, Secrets, CRDs) plus the contents of PersistentVolumes backed by pd.csi.storage.gke.io. Backups can be scheduled (cron syntax), retention configured, and encrypted with Cloud KMS keys.
Restore Scopes
A RestorePlan defines what to restore: full cluster, a namespace, or a label-selected subset. The companion Restore resource executes one restore. Cross-region restore lets you restore a us-central1 backup into a europe-west1 cluster — the answer to a DR question on the exam.
A Backup for GKE resource that defines what to back up (which namespaces, whether to include volume data), how often, and how long to retain. Each BackupPlan produces a series of immutable Backup resources. Source: About Backup for GKE.
Multi-Cluster Ingress and Fleet Management
When a workload runs across multiple GKE clusters in different regions, Multi-Cluster Ingress (MCI) is the single global load balancer that fans out to backend clusters.
How MCI Works
You register clusters into a fleet (formerly Anthos environ), nominate one cluster as the config cluster, and apply a MultiClusterIngress plus MultiClusterService resource there. The MCI controller provisions a global external HTTPS Load Balancer with backend buckets pointing at each member cluster's Network Endpoint Group (NEG). Traffic routes to the closest healthy cluster automatically via Google's premium tier network.
When to Use It
Use MCI for global low-latency user-facing services that must survive a region outage. Use single-cluster Ingress if your service is regional and you can live with cold-start failover times.
Fleet Identity
Fleet membership also enables Fleet Workload Identity, which extends the Workload Identity model across the fleet — a workload running in us-cluster and eu-cluster can both impersonate the same GSA using the fleet's identity pool, simplifying multi-region IAM.
GKE Sandbox (gVisor)
GKE Sandbox runs untrusted code in an additional isolation layer.
What gVisor Does
gVisor is a user-space kernel that intercepts syscalls from the container and re-implements them in Go, refusing dangerous ones. The host kernel never sees raw syscalls from the sandboxed container, dramatically shrinking the kernel attack surface. The cost is a 5-30% performance overhead depending on syscall mix.
Enabling It
On Standard, create a node pool with --sandbox=type=gvisor. Pods opt in via spec.runtimeClassName: gvisor. Autopilot supports Sandbox via the same runtimeClassName annotation; Google places the pod on a gVisor-capable node automatically.
When to Use It
Use Sandbox for multi-tenant SaaS where you run customer-supplied code, CI/CD job runners executing arbitrary build steps, or compiling user-provided source on the platform. Do not use it for trusted first-party workloads — the overhead is real and the benefit is zero.
Dataplane V2 (Cilium eBPF)
Dataplane V2 replaces the legacy kube-proxy + iptables data plane with Cilium backed by eBPF.
What Changes
Service IP lookup, NetworkPolicy enforcement, and inter-pod connection tracking all move from iptables rules (which scale poorly past ~5000 services) to eBPF programs attached at the kernel network stack. The result is lower latency at high service counts, real visibility via hubble (flow logs), and richer NetworkPolicy support including FQDN-based egress policies that iptables cannot express.
How to Enable It
On new clusters, pass --enable-dataplane-v2. Once enabled, you cannot revert. Autopilot clusters use Dataplane V2 by default in new versions. The exam payoff: Dataplane V2 is the answer for "I have 10,000 services and kube-proxy is melting" and for "I need FQDN-based egress policies in GKE."
Dataplane V2 also enables Network Policy logging and Hubble UI for flow visibility — features you'd otherwise reach for Istio to get. If a question asks how to observe pod-to-pod traffic in GKE without a service mesh, the answer is Dataplane V2 + Hubble.
Regional vs Zonal Clusters
GKE clusters come in two availability shapes that affect both control plane and data plane resilience.
Zonal Cluster
A zonal cluster has a single control plane replica in one zone. If that zone goes down, the API server is unavailable; existing pods continue running but you cannot deploy, scale, or self-heal. Nodes can be spread across multiple zones in the same region (--node-locations), but the control plane is the single point of failure. Zonal is cheaper and acceptable for dev/test.
Regional Cluster
A regional cluster runs three control plane replicas across three zones in the region, fronted by a regional ILB. Losing one zone keeps the API responsive. Node pools in a regional cluster default to spreading across all three zones, so a zone failure removes at most a third of the data plane. Production workloads should default to regional.
Cost Trade-off
Regional clusters charge for three control plane replicas where zonal charges for one. The first zonal cluster per billing account is free; subsequent control planes (zonal or regional) bill at the per-cluster-hour rate. For production, regional is non-negotiable — the SLA only covers regional clusters at 99.95%.
Image Streaming
Pulling a large container image can take minutes. Image streaming lets pods start before the full image is downloaded.
How It Works
When image streaming is enabled, GKE mounts the container's root filesystem as a remote FUSE filesystem backed by Artifact Registry. Files are fetched on demand the first time the container reads them. A 4 GB image that only touches 200 MB of files at startup can boot in seconds instead of minutes.
Enabling and Constraints
Enable per node pool on Standard with --enable-image-streaming. Autopilot uses image streaming automatically. The image must live in Artifact Registry (gcr.io legacy registries do not support it). For images that don't benefit (small images, images where startup touches every file), Image streaming adds no cost but no benefit either — Google enables it broadly because the worst case is parity with traditional pulls.
For batch jobs that boot a fresh pod per task, image streaming can cut cold-start time from 90 seconds to under 10. Pair with Artifact Registry virtual repositories to deduplicate base layers across teams, and Vulnerability Scanning to enforce a clean SBOM before the image streams into production.
Frequently Asked Questions (FAQs)
Q1: Can I convert an existing Standard cluster to Autopilot in place?
A1: No. Cluster mode is set at creation and is immutable. The supported migration path is: create a new Autopilot cluster, apply your manifests via GitOps or kubectl apply, validate, then cut traffic over using Multi-Cluster Ingress or a DNS swap. Backup for GKE plus a restore into the new cluster is the fastest way to migrate persistent volume data.
Q2: When should I use VPA Auto vs HPA?
A2: Use HPA when load is bursty and replica count should react — most web APIs. Use VPA Auto when load is steady but your initial resources.requests guesses are wrong, and the workload can tolerate occasional pod restarts as VPA re-sizes them. Do not combine VPA Auto with HPA on the same resource (both on CPU): they fight. The supported combo is HPA on a custom metric (queue depth) + VPA Auto on CPU/memory.
Q3: What's the difference between a liveness probe and a readiness probe in practice?
A3: A failed liveness probe kills the container. A failed readiness probe removes the pod from the Service load balancer but lets it live. Make readiness probes aggressive (fail fast on slow downstreams) and liveness probes conservative (only fail on true deadlocks). Use a startup probe with a generous failureThreshold for slow-booting apps so liveness doesn't kill a healthy-but-warming pod.
Q4: How do I rotate a database credential without redeploying pods?
A4: Store the credential in Secret Manager, mount it via the Secret Manager CSI driver, and configure the SecretProviderClass with auto-rotation: true (via the projected volume sync interval). Pod processes that re-read the credential file pick up the new value automatically. If your app caches the credential in memory at boot, you'll need a restart — in that case, deploy a rolling restart, which is still cheaper than re-issuing a Kubernetes Secret and reapplying manifests.
Q5: Why is my pod stuck in CrashLoopBackOff right after a deploy?
A5: The four most common causes: (1) liveness probe is too aggressive and kills the container before the app finishes warming up — add a startup probe; (2) the image pulled but the container exits because $PORT isn't bound — check the container logs; (3) a missing IAM binding on the GSA so Workload Identity can't mint a token — check gcloud iam service-accounts get-iam-policy; (4) resource requests are too low and the OOM killer fires — check kubectl describe pod for OOMKilled.
Q6: Should I use GKE Ingress or Gateway API for a new project?
A6: For greenfield workloads, prefer Gateway API — it's the Kubernetes-blessed successor, supports weighted routing and traffic splitting natively, and has a richer multi-cluster story via gke-l7-gxlb. Stick with Ingress only if you're maintaining existing manifests, need a feature that's only on Ingress (a specific annotation), or your tooling hasn't caught up to Gateway API yet.
Q7: What does Backup for GKE actually back up?
A7: A BackupPlan captures Kubernetes API objects (namespace-scoped and cluster-scoped resources you select) plus the contents of PersistentVolumes that use the pd.csi.storage.gke.io driver. It does not back up data that lives outside the cluster (Cloud SQL, GCS, Spanner) — use those services' own backup tooling. Restore can target the same cluster, a different cluster in the same region, or a cluster in a different region for DR.