Load Testing and Capacity Planning

Introduction to Load Testing and Capacity Planning

Predicting the future is impossible, but preparing for it is mandatory. Capacity Planning is the process of estimating the resources (CPU, Memory, Storage, Network) your system will need to meet future demand. Load Testing is the practical verification of those estimates by subjecting the system to simulated real-world traffic.

For a Professional Cloud Architect, this is about ensuring that a marketing campaign (like a Super Bowl ad) doesn't result in a "404 Site Not Found" error for your customers.

Plain-Language Explanation: Load Testing and Capacity Planning

Analogy 1 — The Restaurant's Friday Night

Capacity Planning is the restaurant owner deciding how many tables to set up and how many chefs to hire before a busy Friday night. Load Testing is like hiring 100 students to show up all at once and order complex meals to see if the kitchen can handle the rush or if the waiter starts dropping plates (System Failure).

Analogy 2 — The Bridge Weight Limit

A bridge has a "maximum capacity." Engineers perform Load Testing by driving heavy trucks onto the bridge and measuring how much it bends. If the bridge starts to crack before the target weight (The expected peak load), you either need to strengthen the bridge (Vertical Scaling) or build a second bridge (Horizontal Scaling).

Analogy 3 — The Water Pipe System

Load Testing is like turning on every faucet in a house at the same time to see if the water pressure drops. If the pressure is too low, your "Capacity" is insufficient. In the cloud, this might mean your database connection pool is too small or your network bandwidth is saturated.

The Load Testing Lifecycle

1. Define Objectives: What is the target? (e.g., "10,000 concurrent users with <1s response time").
2. Choose Tools:
   • Locust: Python-based, easy to script, great for distributed testing on GKE.
   • JMeter: Java-based, highly feature-rich, industry standard.
3. Create Scenarios: Don't just ping the home page. Simulate real user journeys (Login -> Search -> Add to Cart -> Checkout).
4. Execute and Monitor: Watch metrics like CPU, Memory, and Database latency in Cloud Monitoring.
5. Analyze and Optimize: Find the bottleneck. Is it the code? The database? The network?

Capacity Planning Best Practices

• Monitor Historical Trends: Use Cloud Monitoring to see how traffic has grown over the last 6 months to forecast the next 6.
• Plan for Peak: Don't plan for the "average" day; plan for the "highest" day (e.g., Black Friday).
• Manage Quotas Proactively: Check your GCP quotas weeks before a major event. Request increases early, as some requests require manual approval from Google.
• Validate Auto-scaling: During your load test, ensure that your Managed Instance Groups (MIGs) and GKE clusters actually scale up as expected.

Architect's Insight: On the PCA exam, if a company is planning a "massive global launch" and wants to ensure success, the best answer involves performing a distributed load test and reviewing resource quotas across all targeted regions. ::

Load Generators on GCE and GKE — k6, Locust, JMeter

The choice of load generator platform shapes both the realism of the test and the cost. Running generators inside Google Cloud (next to or remote from the system-under-test) lets you saturate VPC bandwidth without ISP throttling, and IAM lets workers reach internal Load Balancers directly.

Tool comparison

• k6 (Go): Best for engineers who want JavaScript test scripts, native Prometheus output, and a single static binary. Trivial to run as a Cloud Run Job or a GKE Job with a fixed number of parallel pods. Default output is JSON / OpenTelemetry, which streams cleanly into Cloud Monitoring via the OTLP collector.
• Locust (Python): First-class distributed mode with a master pod and N worker pods — the Google reference architecture cloud.google.com/architecture/distributed-load-testing-using-gke deploys exactly this on GKE Autopilot.
• JMeter (Java): Heaviest of the three but unmatched for protocol breadth (JMS, JDBC against Cloud SQL, gRPC via plugin). Run JMeter slaves on a Managed Instance Group of c3-standard VMs when you need 50k+ concurrent threads.

Distributed pattern on GKE

            ┌────────────────┐
            │ Locust master  │  (1 pod, web UI on :8089)
            └────────┬───────┘
                     │ gRPC
        ┌────────────┼────────────┐
        ▼            ▼            ▼
   Worker pod    Worker pod    Worker pod   ... N workers across nodes
        │            │            │
        └────────────┴────────────┘
                     │ HTTPS
                     ▼
           System-Under-Test (HTTPS LB → Cloud Run / GKE)

Each worker pod is sized small (e.g., 500m CPU / 512Mi memory) so the Cluster Autoscaler can pack workers densely. Spread workers across topologyKey: kubernetes.io/hostname to avoid one node becoming the bottleneck. For multi-region tests, deploy a Locust master per region and aggregate metrics in Cloud Monitoring using a target_pool label per region.

Capacity Baselining, Headroom, and Latency Percentiles

A baseline is not a single number — it is a steady-state profile captured at the current production traffic level. Without it, headroom math is guesswork.

The baseline checklist

1. Pick a representative 30-minute window from the last 7 days using Cloud Monitoring MQL (e.g., fetch k8s_container | metric 'kubernetes.io/container/cpu/core_usage_time').
2. Record p50, p95, p99 latency from the Cloud Load Balancer's https/total_latencies metric.
3. Record CPU utilization, memory working set, network egress, and database active connections at the same instant.
4. Compute requests per second per pod / per vCPU — this is your unit-of-work cost.

Headroom math

For a target peak P requests per second and a baseline of B rps per vCPU at p99 ≤ SLO:

Required vCPUs = (P / B) × HeadroomFactor

HeadroomFactor: 1.3 for predictable workloads (batch, internal APIs)
                1.5 for spiky consumer traffic
                2.0 for Black-Friday-class launches

The headroom is not waste — it is the buffer that absorbs the time between autoscaler signal and pod-ready.

Why p99 matters more than average

A 100 ms average can hide a 4-second p99 caused by GC pauses or cold connections. PCA exam answers that mention "the system meets SLO on average" are usually wrong — the correct answer ties capacity to a percentile-based SLO (e.g., "p99 < 300 ms at 5× peak"). Configure SLO burn-rate alerts in Cloud Monitoring against the Service object, not against raw metrics.

Cloud Monitoring Instrumentation and Autoscaling Response Curves

A load test without observability is just a denial-of-service attack against yourself. Before the first synthetic request, wire up the following:

Required dashboards

• Golden signals row: Request rate, p50/p99 latency, error rate, saturation (CPU, memory, connections) — one chart each, time-aligned to the test start.
• Autoscaler row: For GKE — kubernetes.io/autoscaler/desired_replicas vs current_replicas. For MIGs — compute.googleapis.com/instance_group_manager/desired_size. For Cloud Run — run.googleapis.com/container/instance_count.
• Dependency row: Cloud SQL database/cpu/utilization, Spanner instance/cpu/utilization_by_priority, Memorystore redis.googleapis.com/stats/cpu_usage_time.

Reading the autoscaling response curve

Plot replica count over time and overlay request rate. A healthy curve has:

1. Lag phase (0-60 s): Replicas flat while traffic ramps. Latency rises temporarily.
2. Scale-up phase (60-180 s): Replicas climb linearly; new pods reach Ready and absorb load. p99 should drop within 2-3 minutes.
3. Steady-state plateau: Replicas stabilize at target_cpu_utilization (default 60-70 % for HPA).
4. Scale-down phase: After traffic stops, GKE HPA waits 5 minutes (--horizontal-pod-autoscaler-downscale-stabilization) before removing pods — this prevents flapping.

Common bottleneck signatures

• Flat replicas, rising latency: Quota hit, or the HPA metric is wrong (CPU when the real bottleneck is RPS).
• Replicas oscillate every 5 minutes: Downscale stabilization too aggressive — increase --horizontal-pod-autoscaler-downscale-stabilization.
• Latency p99 jumps at fixed intervals: Cloud SQL connection storms or JVM GC — switch to Cloud SQL connector with pooling or a sidecar like PgBouncer.

Database Load Test Patterns — Spanner and BigQuery

Stateless tiers scale linearly. Stateful tiers do not — and they are usually the first thing to fail during a load test.

Cloud Spanner — node sizing and hotspots

Spanner is provisioned by nodes (or fractional 100/1000 PU units). Each node sustains roughly 10,000 QPS of point reads or 2,000 QPS of writes at p99 ≤ 10 ms. Load testing strategy:

• Run a mixed read/write workload at 65 % CPU; this is Google's recommended steady-state target. Spanner will auto-split, but splits take minutes to stabilize.
• Use the Key Visualizer during the test to spot hotspots. A bright vertical band = monotonic primary key (timestamp, sequence) = rewrite to use a hash prefix or UUIDv4.
• Validate that session pool size = 100 × nodes in your client. Under-provisioned pools manifest as client-side queueing, not Spanner CPU.

BigQuery — slots and concurrency

BigQuery capacity comes from slots. Two purchasing modes behave very differently under load:

For a launch dashboard backing 1,000 concurrent analysts, run the same query mix at 1×, 2×, and 5× expected concurrency and watch bigquery.googleapis.com/slots/total_allocated_for_reservation and query/execution_times. If queries queue beyond your latency budget, buy a dedicated reservation with autoscaler or pre-materialize results into BigQuery BI Engine.

Compute Autoscaler Sizing — GKE HPA, Cluster Autoscaler, Cloud Run Concurrency

The autoscaler is only as good as its configuration. Three knobs dominate.

GKE — HPA plus Cluster Autoscaler

The Horizontal Pod Autoscaler scales pods; the Cluster Autoscaler scales nodes. They must be tuned together:

# HPA scales pods on CPU + custom metric
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
spec:
  minReplicas: 3
  maxReplicas: 200
  metrics:
    - type: Resource
      resource: { name: cpu, target: { type: Utilization, averageUtilization: 60 } }
    - type: Pods
      pods: { metric: { name: http_requests_per_second }, target: { type: AverageValue, averageValue: "100" } }

For the Cluster Autoscaler, set --scale-down-unneeded-time=10m and use node auto-provisioning so the scaler can create new node pools of the right machine shape mid-test. With GKE Autopilot, this is automatic — but the pod-level resource requests must be accurate, or Autopilot over-provisions and inflates the bill.

Cloud Run — concurrency tuning

Cloud Run scales on active container instances, not CPU. The decision variable is --concurrency (requests per instance, default 80, max 1000):

• CPU-bound JSON API: --concurrency=20 to 40 — more requests starve each other for CPU.
• I/O-bound proxy or LLM front-end: --concurrency=200 to 500 — most threads are waiting on the backend.
• Cold-start sensitive: set --min-instances=N where N covers the baseline traffic so the first user does not pay the cold-start tax.

Load testing surfaces the optimal value: plot p99 latency vs --concurrency at fixed RPS, then pick the highest concurrency that keeps p99 under SLO. This often halves your Cloud Run bill compared to the default.

FAQ — Load Testing and Capacity Planning

Q1. Should I load test against my production environment?

Ideally, no. You should use a production-like staging environment that has the same configuration and data scale. If you must test in production, do it during off-peak hours and have a strict "Stop" protocol.

Q2. What is the difference between "Load Testing" and "Stress Testing"?

Load Testing checks if the system can handle the expected peak. Stress Testing pushes the system until it breaks to find the absolute limit and see how it fails (e.g., does it crash or just get slow?).

Q3. How do I handle database bottlenecks during a load test?

If the database is the bottleneck, consider Read Replicas for read-heavy workloads, upgrading to a larger instance (Vertical Scaling), or switching to a globally scalable database like Cloud Spanner.

Q4. Can I use Google Cloud to generate the test load?

Yes. A common pattern is to deploy a cluster of Locust workers on GKE. This allows you to generate massive amounts of traffic from within the Google network (or from multiple regions) to test your application.

Q5. What is "Warm-up" in the context of capacity?

Some services, like Cloud Functions or Cloud Run, might experience "Cold Starts." Also, database caches need to be "warmed up" with data before they reach peak performance. Ensure your load test includes a warm-up phase to get realistic results.