Incident Response and Management

Introduction to Incident Response

An incident is an unplanned interruption to or reduction in the quality of a service. No matter how well you architect your system, incidents will happen. Incident Response Management is the process of handling these events to restore service as quickly as possible while minimizing the impact on the business.

For the GCP Professional Cloud Architect, the focus is on building a repeatable process that relies on roles, communication, and learning rather than individual heroism.

Plain-Language Explanation: Incident Response

Analogy 1 — The Fire Department

When a fire breaks out, the fire department doesn't just run into the building randomly. They have an Incident Commander (the chief) who stays outside to coordinate. They have specialized roles (hose, ladder, medical). After the fire is out, they investigate the cause (Post-mortem) to prevent future fires, rather than just blaming the person who left the stove on.

Analogy 2 — The ER Trauma Team

In a hospital ER, when a "Code Red" is called, a specific team assembles. One person leads the resuscitation (The lead doctor), while others handle specific tasks (vitals, medication). They communicate clearly using "Closed-loop communication" (e.g., "Giving 5mg of Adrenaline" -> "5mg of Adrenaline given"). This prevents mistakes during the chaos of the incident.

Analogy 3 — The Airline Pilot's Emergency Checklist

When an engine fails, pilots don't panic. They pull out a Quick Reference Handbook (QRH). This is like an Incident Playbook. It contains step-by-step instructions for known problems, allowing them to remain calm and follow a proven path to safety.

The Incident Response Lifecycle

1. Detection: The incident is identified via alerts (Cloud Monitoring) or user reports.
2. Triage: Determining the Severity (Impact) and Priority (Urgency).
3. Containment/Mitigation: Taking immediate action to stop the bleeding (e.g., rolling back a deployment, adding more capacity). This is not the time for a permanent fix.
4. Resolution: The permanent fix is implemented.
5. Post-mortem/Retrospective: Analyzing the root cause and identifying actions to prevent recurrence.

Key Roles in Incident Response

• Incident Commander (IC): The person in charge of the response. They coordinate the teams, make high-level decisions, and ensure everyone has what they need. The IC does not write code or fix bugs.
• Operations Lead: The technical expert responsible for the hands-on mitigation and resolution.
• Communications Lead: The person responsible for updating stakeholders (internal executives and external customers).

Blameless Post-mortems

The goal of a post-mortem is to learn, not to punish.

• Blameless: We assume that everyone acted with the best intentions and information available at the time.
• Root Cause Analysis (RCA): We look for systemic failures (e.g., "The automated test suite didn't catch this case") rather than human errors (e.g., "John typed the wrong command").
• Action Items: Every post-mortem must result in concrete tasks to improve the system or process.

Architect's Insight: On the PCA exam, if a question asks how to improve a company's "Culture of Reliability," the answer is almost always to implement blameless post-mortems. This encourages honesty and continuous improvement rather than fear and hiding mistakes. ::

Severity Classification (SEV1–SEV4)

A consistent severity scale removes ambiguity from triage and tells the PagerDuty / Cloud Monitoring routing engine who to wake up. Google SRE shops typically standardise on four levels:

SEV1 — Critical Customer Impact

Full or majority outage of a user-facing GCP workload (e.g., Cloud Run revision returns 5xx for >50% of requests, GKE Ingress unreachable, Cloud SQL primary failed without replica promotion). Page the on-call immediately, open a war room (Google Meet bridge), and notify the Communications Lead within 5 minutes. Update the status page every 15 minutes until mitigated.

SEV2 — Significant Degradation

Latency or error rate breaches the fast burn-rate alert (e.g., HTTP(S) Load Balancer P95 latency > SLO target for >10 minutes), but the service is still partially usable. One region degraded while others healthy also fits here. Page on-call, no all-hands war room required.

SEV3 — Minor / Partial Issue

Non-customer-facing breakage: failed Cloud Scheduler job, Dataflow batch pipeline stuck, BigQuery scheduled query failed. File a ticket, fix during business hours. No page outside working hours.

SEV4 — Cosmetic / Informational

Typo on dashboard, deprecated log message, missing label. Track in backlog.

# Example severity → notification channel mapping
sev1:
  channels: [pagerduty-page, slack-incident-room, status-page-update]
  ack_sla_minutes: 5
sev2:
  channels: [pagerduty-page, slack-incident-room]
  ack_sla_minutes: 15
sev3:
  channels: [slack-alerts]
  ack_sla_minutes: 240

Document the severity matrix in your runbook repo so any engineer can self-assign correctly at 3am.

On-Call Rotation Design

A healthy on-call schedule prevents burnout while guaranteeing 24/7 coverage. Common patterns in GCP shops:

• Follow-the-sun: APAC / EMEA / AMER pods each cover ~8 hours of their working day. Requires three regional teams; eliminates pagers at 3am for anyone.
• Primary + Secondary: One engineer is paged first; if no ack within 5 minutes, PagerDuty escalates to a secondary. Both rotate weekly.
• Tiered Escalation: Tier 1 (on-call SRE) → Tier 2 (service owner) → Tier 3 (Engineering Manager / VP). Each tier has its own ack SLA.

Schedule Hygiene

• Max 25% of working time on-call, per Google SRE guidance.
• Compensation: time-in-lieu or on-call pay; never treat it as "extra duty."
• Hand-off ritual: outgoing on-call posts a Slack summary of open incidents, suppressed alerts, and known-flaky services to the incoming on-call.
• Practice rotations: new joiners shadow for 2-3 weeks before holding the primary pager.

Runbook Design and Playbooks

A runbook (or playbook) is the QRH for your service — step-by-step mitigation for a known failure mode. Good runbooks turn a SEV1 from a 90-minute scramble into a 10-minute checklist.

What Every Runbook Must Contain

1. Alert signature — which Cloud Monitoring alert fires and what its labels mean.
2. Likely causes — top 3 root causes for this alert, ranked by frequency.
3. Diagnostic commands — copy-pasteable gcloud / kubectl / bq commands. Example:

   # Check GKE pod status
   kubectl get pods -n prod -l app=checkout --field-selector=status.phase!=Running
   # Inspect Cloud SQL replication lag
   gcloud sql operations list --instance=prod-db --filter="status=RUNNING"
   

4. Mitigation steps — explicit rollback, scale-out, failover commands. Each step says "expected output."
5. Escalation criteria — when to page the service owner or declare SEV1.
6. Owner & last-reviewed date — runbooks rot; review quarterly.

Storage and Discovery

• Keep runbooks in Git alongside service code (single source of truth, code-reviewed changes).
• Link the runbook URL directly in the Cloud Monitoring alert policy documentation.content field so the page contains the link, not a hunt.
• Render runbooks via a static site (Hugo, MkDocs) deployed to Firebase Hosting so they're reachable even if internal wiki is down.

Incident Command Structure (IC, SME, Comms)

Google's Incident Management at Google (IMAG) framework formalises three core roles. For a SEV1, fill all three with distinct humans:

Incident Commander (IC)

• Owns the response, not the fix. Sets priorities, delegates, declares severity changes.
• Runs the Meet bridge: "What's the current hypothesis? What's the next step? Who's doing it?"
• Decides when to declare resolved and when to hand off to the next IC (during long incidents, rotate every 4 hours).

Subject Matter Expert (SME) / Operations Lead

• The hands on the keyboard. Executes gcloud, reads Cloud Logging, drives mitigation.
• Reports findings up to IC; does not make scope decisions ("should we fail over to us-east1?" → IC decides).
• Can be multiple SMEs in parallel (one per affected service: GKE SME, Spanner SME, networking SME).

Communications Lead (Comms)

• Drafts and publishes status page updates, internal Slack announcements, exec emails.
• Translates SME jargon ("Spanner leader election timing out") into customer language ("API requests may be slower than usual in europe-west4").
• Owns the post-incident customer email and any regulator notifications.

Why Separate Roles Matter

A single engineer cannot simultaneously debug a Spanner query, update the status page, and brief the VP. Splitting roles reduces context-switching and is enforced by the IC declaring roles at the start: "I'm IC, Alice is SME for checkout, Bob is Comms."

Cloud Logging Incident Timeline

A reconstructable timeline is the spine of every post-mortem. Cloud Logging is your authoritative source, but only if you set it up before the incident.

Capture Strategy

• Aggregated log sink at the Organization level routing to a dedicated BigQuery dataset (logs_incidents) with 400-day retention — long enough to investigate slow-burn incidents and satisfy most audit windows.
• Required log buckets: Cloud Audit Logs (Admin Activity + Data Access for sensitive services), VPC Flow Logs (sampled), and application logs with structured jsonPayload.
• Standardise correlation IDs: every request carries a trace_id that propagates through GKE, Cloud Run, Pub/Sub, and Cloud SQL proxy logs so you can reconstruct a single user's journey.

Building the Timeline During an Incident

1. Pin a Log Analytics saved query in the incident channel that filters by service + severity≥ERROR.
2. Use Logs Explorer time-range slider to find the first anomalous log entry — that's t0.
3. Cross-reference with Cloud Monitoring dashboards (MQL or PromQL) to confirm when the SLI broke.
4. Export the timeline to a Google Doc using gcloud:

   gcloud logging read 'resource.type="k8s_container" severity>=ERROR
     timestamp>="2026-05-12T10:00:00Z" timestamp<="2026-05-12T11:30:00Z"' \
     --format="csv(timestamp,resource.labels.pod_name,jsonPayload.message)" \
     --limit=500 > incident-timeline.csv
   

Common Pitfalls

• Forgetting to enable Data Access audit logs for Cloud SQL / BigQuery before the incident — you cannot retroactively turn them on.
• Sampling VPC Flow Logs too aggressively (0.1%) — network incidents become unreproducible.
• Dropping logs at the agent due to Ops Agent buffer overflow under load; pre-provision higher buffer sizes on tier-1 instances.

PagerDuty + GCP Integration

PagerDuty is the most common pager in GCP shops; the integration is bidirectional via Pub/Sub and webhook.

Wiring Cloud Monitoring → PagerDuty

1. In Cloud Monitoring, create a Notification Channel of type "Webhook" pointing at PagerDuty's Events API v2 endpoint, or use the prebuilt PagerDuty channel that needs only an integration key.
2. Attach the channel to alert policies. PagerDuty events carry incident_id + condition_name so the on-call sees the alert title in the page.
3. Map alert severity labels (severity=sev1) to PagerDuty Urgency so SEV1 pages even if the on-call has "low urgency notifications" off.

Wiring PagerDuty → GCP for Automation

• Use PagerDuty webhooks → Cloud Functions to auto-execute safe mitigation (e.g., on SEV1 for service=payment, automatically scale the GKE node pool by +50% and post the action to the incident channel).
• Use PagerDuty Event Orchestration to deduplicate noisy alerts from the same resource.labels.instance_id so a flapping VM doesn't generate 50 pages.

Configuration Pitfalls

• Routing keys per service, not per team — lets PagerDuty correctly map to schedules even if team boundaries shift.
• Auto-resolve Cloud Monitoring alerts when the condition clears, so PagerDuty incidents close automatically instead of staying open for days.
• Maintenance windows: silence pages during planned Cloud SQL maintenance or GKE upgrades using PagerDuty maintenance windows, not by disabling the alert (you'll forget to re-enable).

Postmortem Template

A consistent template makes post-mortems searchable and comparable. The canonical Google SRE structure:

1. Header

• Incident ID: INC-2026-0512-001
• Severity: SEV1
• Date/Duration: 2026-05-12, 10:00–11:42 UTC (1h 42m)
• Authors / IC / SMEs: named individuals
• Status: Draft / In Review / Published

2. Summary (2-3 sentences, exec-readable)

What broke, who was impacted, how it was mitigated. No jargon.

3. Impact

• Customer impact: e.g., "12% of checkout API requests returned 503 for 1h 42m."
• SLO impact: "Consumed 38% of monthly error budget for checkout-api."
• Revenue / contractual: "Estimated $X in failed transactions; SLA credits owed to 4 enterprise accounts."

4. Timeline

Chronological, with timestamps in UTC. Each row: time, who, what they did or observed. Pull from Cloud Logging exports.

5. Root Cause (the "5 Whys")

Drill into systemic causes. Example: a bad deployment → why did it pass review? → why didn't tests catch it? → why was the canary stage skipped?

6. What Went Well / What Didn't

Acknowledge what saved time (good runbook, fast rollback) and what cost time (stale dashboard, missing alert).

7. Action Items (the most important section)

Every action item has an owner, due date, and JIRA link — otherwise it never ships.

8. Lessons Learned

Brief reflections to feed back into team-wide tech talks and onboarding material.

Error Budget Burn During Incidents

The error budget (1 - SLO) connects incident response to product velocity. During an incident, you're actively spending budget — track it.

Fast Burn vs Slow Burn Alerts

Google SRE recommends a multi-window, multi-burn-rate alerting policy on every tier-1 SLO:

• Fast burn (14.4x, 1h window): spending 2% of monthly budget per hour → page immediately (SEV1/SEV2 candidate).
• Slow burn (3x, 6h window): spending 1% per hour over 6 hours → ticket, investigate during business hours.

In Cloud Monitoring, create an SLO on the service, then attach a SloBurnRateCondition alerting policy:

conditions:
  - displayName: "Fast burn — checkout availability"
    conditionThreshold:
      filter: 'select_slo_burn_rate("projects/PROJECT/services/checkout/serviceLevelObjectives/availability-99-9", "3600s")'
      comparison: COMPARISON_GT
      thresholdValue: 14.4

Decision Framework

• >50% of monthly budget consumed in one incident → freeze non-critical feature launches for that service until budget recovers.
• Budget exhausted twice in a quarter → mandatory reliability sprint; product team agrees beforehand via the SLO contract.
• Budget consistently untouched (<10% used) → SLO is too loose; tighten it so engineering effort goes elsewhere.

Tooling

• Cloud Monitoring SLO dashboards show real-time budget remaining, displayed prominently in the incident war room so the IC can see the cost of every minute the incident continues.
• Export SLO metrics to BigQuery via Cloud Monitoring → Pub/Sub → Dataflow for long-term trend analysis and quarterly reliability reviews.

Cloud Monitoring Outage Response

When the outage IS in Cloud Monitoring (or upstream Google infrastructure), your normal observability stack may itself be degraded.

Defensive Patterns

• Cross-region dashboards: host critical dashboards in two regions; if us-central1 Cloud Monitoring console is slow, switch to europe-west4.
• Out-of-band paging: keep a secondary alerting path (e.g., Datadog, Grafana Cloud) for the top 5 SLIs so you're not blind during a Cloud Monitoring incident.
• Synthetic checks from outside GCP: use PagerDuty synthetic monitoring or a third-party prober hosted on AWS / Azure to detect "GCP itself is down" cases.

Google Cloud Service Health

• Subscribe the on-call rotation to Google Cloud Service Health RSS feeds and Personalized Service Health alerts (configurable via Cloud Monitoring notification channels).
• When Service Health declares a regional issue, confirm before assuming your own bug — saves hours of misdirected debugging.
• Cross-reference with status.cloud.google.com for the global outage view.

Communication During Provider Outages

If the root cause is a Google Cloud regional outage:

1. Acknowledge to customers on your status page that the upstream issue is GCP-side, with a link to Google's incident report.
2. Activate your DR runbook if the outage exceeds your RTO (e.g., fail over from us-central1 to us-east1 for Spanner multi-region, switch active region for Cloud Run behind Global External HTTP(S) LB).
3. Capture Google's eventual public RCA in your post-mortem; even though you didn't cause it, your customers experienced impact and your action items are about defensive architecture (multi-region, circuit breakers, graceful degradation).

Customer Status Page Strategy

A public status page is part of the incident response, not an afterthought. Customers judge your reliability by how you communicate during incidents as much as by uptime numbers.

Choosing a Platform

• Statuspage (Atlassian), Better Stack, Instatus, Cachet — hosted, ready in an hour.
• For GCP-native: host a static page on Cloud Storage + Cloud CDN + Cloud Load Balancing, fed by a Cloud Function triggered from PagerDuty webhooks. Crucial: host the status page in a different cloud or at minimum a different region from your primary workload, otherwise the same outage takes both down.

Update Cadence

Writing Updates That Build Trust

• State facts, not theories. "We are investigating elevated 5xx errors on the checkout API" — not "It might be a database issue."
• Acknowledge impact specifically. "Approximately 8% of EU customers are unable to complete purchases" beats "some users may experience issues."
• Promise the next update time. "Next update at 14:30 UTC" — and stick to it even if you have no news ("Still investigating, next update at 15:00").
• Publish the post-mortem within 5 business days of SEV1, linked from the status page. This is the single biggest trust-builder.

Subscriber Channels

Allow customers to subscribe via email, SMS, RSS, Slack webhook, or Microsoft Teams connector. Use Cloud Tasks or Pub/Sub fan-out to deliver updates so a 10,000-subscriber status page update doesn't take 20 minutes to ship.

FAQ — Incident Response Management

Q1. What is the difference between "Mitigation" and "Resolution"?

Mitigation is a temporary fix to restore service (e.g., restarting a VM). Resolution is the permanent fix that addresses the root cause (e.g., fixing the memory leak in the code).

Q2. How often should we update stakeholders during an incident?

It depends on the severity. For a P0 (Critical) incident, updates every 15-30 minutes are common. For a P2, every 2-4 hours might be sufficient. Consistency is more important than frequency.

Q3. What is "Closed-loop Communication"?

It's a technique where the receiver repeats the instruction back to the sender to ensure it was understood correctly. This is vital during high-stress incidents to prevent technical mistakes.

Q4. Should we always do a post-mortem for every incident?

Ideally, yes for any incident that violated an SLO or impacted customers significantly. For minor incidents, a "micro-post-mortem" or a simple log entry might suffice to save time.

Q5. How can we simulate incidents without breaking production?

Use Chaos Engineering (e.g., Fault Injection) or conduct Game Days, where the team goes through a simulated scenario in a staging environment to practice their response and test their playbooks.