Secret Manager and KMS

Introduction: Two Services, Two Different Jobs

For a Professional Cloud Developer (PCD) candidate, the first mental model to lock in is that Secret Manager and Cloud KMS solve different problems even though both live under "security." Secret Manager is a value store for opaque sensitive strings — database passwords, third-party API keys, OAuth client secrets, TLS private keys — up to 64 KiB per secret version. Cloud KMS is a key management service that never returns the underlying key material to your code; instead you call Encrypt, Decrypt, AsymmetricSign, or MacVerify and KMS performs the cryptographic operation server-side using a key that stays inside a Google-managed boundary (or an HSM, with protectionLevel: HSM).

The exam loves to test the boundary between these two. Storing a Cloud KMS key inside Secret Manager is wrong (you cannot export software key material at all, and HSM material absolutely cannot leave the FIPS boundary). Encrypting a 5 GB BigQuery export by calling kms.encrypt 50,000 times is also wrong — that violates the 64 KiB per request payload limit and you should use envelope encryption with a generated DEK. This study note walks every angle the PCD blueprint covers: versions and state machine, replication policies, CMEK on Secret Manager itself, the IAM model anchored on roles/secretmanager.secretAccessor, Cloud Run --set-secrets vs --set-env-vars, GKE CSI driver mounts, the four-level KMS resource hierarchy, CMEK on BigQuery / GCS / Compute Engine, symmetric vs asymmetric purposes, HSM-backed keys with FIPS 140-2 Level 3 validation, the envelope encryption pattern, and automatic key rotation.

Secret Manager Version State Machine: ENABLED, DISABLED, DESTROYED

Every Secret Manager secret is a container; the actual payload lives in versions. Each version has a monotonically increasing integer ID (1, 2, 3, …) and a state field. The three states form a one-way state machine that the PCD exam tests directly.

ENABLED — the only state that returns payload

A version in ENABLED state will return its payload from gcloud secrets versions access 3 --secret=db-password or the AccessSecretVersion REST/gRPC call. This is the default state when a version is first created via gcloud secrets versions add db-password --data-file=./pw.txt.

DISABLED — temporary lockout, payload retained

Calling gcloud secrets versions disable 3 --secret=db-password flips the state to DISABLED. Any AccessSecretVersion request now returns FAILED_PRECONDITION. Crucially, the payload bytes are retained — you can re-enable the version with gcloud secrets versions enable 3 and access resumes. This is the right tool for "I think key 3 might be compromised; freeze it while I investigate" scenarios.

DESTROYED — terminal, payload purged

Calling gcloud secrets versions destroy 3 --secret=db-password performs an irreversible delete of the payload bytes. The version metadata (creation time, destroy time) remains visible for audit, but the secret value is gone. There is no undestroy. Pair this with the secret_manager.googleapis.com/SecretVersionDestroyed Cloud Audit Log to track destruction events for compliance.

The latest alias

The string literal latest resolves to the highest version number that is in ENABLED state. If you destroy version 3, latest falls back to version 2. This is why pinning to latest in production is a footgun for rollback — prefer pinning to an explicit integer.

Replication Policies: Automatic vs User-Managed

Secrets are global-namespaced but physically replicated. You pick the policy at create time and cannot switch later — this is one of the most common exam traps.

Automatic replication (--replication-policy=automatic)

Google picks regions automatically and replicates the secret across multiple regions inside the chosen multi-region footprint. You get the highest availability with zero ops. The catch: you cannot pair automatic with a CMEK key, because automatic replication needs to choose regions Google controls.

User-managed replication (--replication-policy=user-managed)

You explicitly enumerate replica regions: --locations=us-east1,europe-west1,asia-southeast1. This is required when you need data residency (regulator says "data must stay in EU"), and it is also required for CMEK on Secret Manager because each replica region needs its own KMS key in that region.

# Automatic: Google chooses, no CMEK allowed
gcloud secrets create app-api-key \
  --replication-policy=automatic \
  --data-file=./key.txt

# User-managed: you control regions, CMEK supported
gcloud secrets create app-api-key-cmek \
  --replication-policy=user-managed \
  --locations=us-east1,europe-west1 \
  --data-file=./key.txt

CMEK on Secret Manager

By default Secret Manager encrypts payloads with Google-managed keys. To meet regulatory requirements where you must hold the root of trust, you can enable CMEK on a user-managed-replication secret. Each replica region needs its own KMS key (or key ring) in that same region — you cannot use a global key, and you cannot use a key from a different region.

gcloud secrets create regulated-secret \
  --replication-policy=user-managed \
  --locations=us-east1,europe-west1 \
  --kms-key-name=projects/p/locations/us-east1/keyRings/sm/cryptoKeys/k1 \
  --kms-key-name=projects/p/locations/europe-west1/keyRings/sm/cryptoKeys/k2

The Secret Manager service agent (service-PROJECT_NUMBER@gcp-sa-secretmanager.iam.gserviceaccount.com) must have roles/cloudkms.cryptoKeyEncrypterDecrypter on each CMEK key. If you revoke that role, the secret becomes unreadable — this is the cryptographic shred pattern for emergency lock-down.

IAM: roles/secretmanager.secretAccessor at the Right Resource Level

Secret Manager IAM has a clean three-role model that maps to the principle of least privilege.

The three primary roles

• roles/secretmanager.secretAccessor — read payload only. This is what runtime applications need.
• roles/secretmanager.secretVersionManager — create, enable, disable, destroy versions. CI/CD pipelines need this.
• roles/secretmanager.admin — full control including IAM modifications. Humans on the security team only.

Resource-level bindings beat project-level bindings

The classic mistake is granting secretAccessor at the project level — that gives the service account read access to every secret in the project. The correct pattern is to bind the role at the secret level:

gcloud secrets add-iam-policy-binding db-password \
  --member="serviceAccount:[email protected]" \
  --role="roles/secretmanager.secretAccessor"

This way the Cloud Run service account can read db-password but not stripe-key or oauth-client-secret, even though all three live in the same project. The exam will give you scenarios with multiple secrets per project and a single workload; resource-level binding is almost always the correct answer.

Cloud Run Secret Patterns: --set-secrets vs --set-env-vars

Cloud Run (both fully managed and Cloud Run for Anthos) has first-class Secret Manager integration so application code never needs to call the Secret Manager API directly.

Mounted as environment variable

gcloud run deploy api \
  --image=gcr.io/p/api:v3 \
  --set-secrets=DB_PASSWORD=db-password:latest,STRIPE_KEY=stripe-key:7

The container sees DB_PASSWORD and STRIPE_KEY as ordinary env vars. The secret value is resolved at container start for latest, or pinned to version 7 for stripe-key. Pinning is recommended for production because a latest reference picks up new versions only on the next cold start, which can cause drift across revisions.

Mounted as a file (volume)

gcloud run deploy api \
  --set-secrets=/var/secrets/tls.key=tls-private-key:latest

This mounts the secret payload as a file at /var/secrets/tls.key. Use this for TLS keys, JSON service account keys, or any payload larger than a typical env var (env var size limits vary by OS and shell — files are safer for binary or large content).

--set-secrets vs --set-env-vars

• --set-env-vars=DB_HOST=10.0.0.5 puts the literal string 10.0.0.5 into the env var; it lives in plaintext in the Cloud Run revision config and shows up in audit logs of the revision YAML. Use only for non-sensitive config.
• --set-secrets=DB_PASSWORD=db-password:latest stores a reference in the revision config; the actual value is fetched from Secret Manager at start using the Cloud Run service identity. The plaintext never appears in any Cloud Run configuration field.

GKE Secret Mount Patterns

On GKE the recommended pattern is the Secret Manager CSI driver (secrets-store-csi-driver plus the secrets-store-csi-driver-provider-gcp provider). It mounts Secret Manager payloads as files inside the pod, with Workload Identity authenticating the pod's KSA to a GSA that holds secretAccessor.

A typical SecretProviderClass

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: app-secrets
spec:
  provider: gcp
  parameters:
    secrets: |
      - resourceName: "projects/p/secrets/db-password/versions/latest"
        path: "db-password.txt"

The pod then mounts a CSI volume referencing app-secrets, and /mnt/secrets/db-password.txt contains the payload. Combine with Workload Identity so no JSON key file exists on the node.

Why not native Kubernetes Secret objects?

Native Secret objects are base64-encoded (not encrypted) in etcd by default. You can enable application-layer secrets encryption with Cloud KMS to wrap etcd entries, but the Secret Manager CSI driver is preferred because (a) the source of truth is centralized across services and (b) you get Cloud Audit Logs on every access.

Cloud KMS Resource Hierarchy: Project → Location → Key Ring → Key → Version

KMS resources form a strict four-level hierarchy. Internalize this — exam questions test exact resource paths.

Project

KMS keys live inside a Google Cloud project. Billing and quota attach at the project level.

Location

Every key has a location (global, regional like us-east1, multi-regional like us, or dual-regional like nam4). Location is immutable. CMEK keys must be in the same location as the resource they protect — a us-east1 Cloud Storage bucket needs a key in us-east1 (or the matching multi-region us).

Key Ring

A key ring is a logical grouping of keys at one location. Common pattern: one key ring per environment per location (e.g. prod-keys-us-east1). Key rings exist forever — you cannot delete them.

Key (CryptoKey)

The named cryptographic key. Has a purpose (ENCRYPT_DECRYPT, ASYMMETRIC_SIGN, ASYMMETRIC_DECRYPT, MAC, or RAW_ENCRYPT_DECRYPT), an algorithm (GOOGLE_SYMMETRIC_ENCRYPTION, RSA_SIGN_PKCS1_2048_SHA256, etc.), and a protectionLevel (SOFTWARE, HSM, or EXTERNAL).

Key Version

Actual key material. A key has many versions; the primary version is used for new encrypt operations. Disabling a version blocks operations using it but you can re-enable. Destroying a version starts a 24-hour scheduled destruction (configurable 1–120 days via destroyScheduledDuration) during which you can restore.

A full resource name: projects/p/locations/us-east1/keyRings/prod/cryptoKeys/data-key/cryptoKeyVersions/4.

CMEK Across BigQuery, Cloud Storage, and Compute Engine

CMEK lets you swap the default Google-managed encryption key for one you control. The pattern is identical across services: grant the service's service agent the cryptoKeyEncrypterDecrypter role on your key, then attach the key.

Cloud Storage

gcloud storage buckets update gs://my-bucket \
  --default-encryption-key=projects/p/locations/us-east1/keyRings/r/cryptoKeys/k

The bucket must be regional in us-east1 or in a compatible multi/dual region. Service agent: service-PROJECT_NUMBER@gs-project-accounts.iam.gserviceaccount.com.

BigQuery

Set at dataset or table level:

bq mk --table \
  --destination_kms_key=projects/p/locations/us/keyRings/r/cryptoKeys/k \
  mydataset.mytable

Service agent: [email protected]. The key location must match the dataset location.

Compute Engine (Persistent Disks)

gcloud compute disks create data-disk \
  --kms-key=projects/p/locations/us-east1/keyRings/r/cryptoKeys/k \
  --zone=us-east1-b

Service agent: [email protected]. Snapshots created from a CMEK disk inherit CMEK; you can optionally re-encrypt with a different CMEK key.

Symmetric vs Asymmetric Keys

KMS supports both flavors, and the exam will hand you a scenario that maps cleanly to one or the other.

Symmetric (purpose: ENCRYPT_DECRYPT)

Algorithm GOOGLE_SYMMETRIC_ENCRYPTION (AES-256-GCM). The same key both encrypts and decrypts. Payload limit: 64 KiB per Encrypt request. This is what you use as a KEK in envelope encryption, and what CMEK on GCS/BQ/Compute uses internally.

Asymmetric (purpose: ASYMMETRIC_SIGN or ASYMMETRIC_DECRYPT)

Algorithms include RSA_SIGN_PSS_2048_SHA256, EC_SIGN_P256_SHA256, RSA_DECRYPT_OAEP_2048_SHA256. The public key is downloadable via cryptoKeyVersions.getPublicKey and can be distributed freely; the private key never leaves KMS. Use cases:

• ASYMMETRIC_SIGN — signing JWTs, container image signatures (Binary Authorization), code signing
• ASYMMETRIC_DECRYPT — receiving payloads encrypted under your public key by external parties who must not see your private key

MAC and RAW_ENCRYPT_DECRYPT

MAC keys (HMAC-SHA256) verify message integrity. RAW_ENCRYPT_DECRYPT (AES-256-GCM raw, AES-256-CBC) is for compatibility with external systems that demand a specific scheme. These are niche but appear occasionally on the exam.

HSM-Backed Keys: FIPS 140-2 Level 3

Setting protectionLevel: HSM on a key version stores material inside a physical Hardware Security Module validated to FIPS 140-2 Level 3. The HSM is tamper-evident and tamper-resistant; key material cannot be extracted, period. Cryptographic operations are performed inside the HSM and the result is returned over an authenticated channel.

gcloud kms keys create payments-signing \
  --keyring=prod --location=us-east1 \
  --purpose=asymmetric-signing \
  --default-algorithm=ec-sign-p256-sha256 \
  --protection-level=hsm

HSM keys cost more per operation than software keys (check current Cloud KMS pricing — typically a small flat multiple) and have slightly higher latency, but they are mandatory for PCI-DSS card data scenarios, certain government workloads, and any regulator that explicitly requires FIPS 140-2 Level 3.

Envelope Encryption Pattern

The hard constraint on KMS symmetric Encrypt is the 64 KiB payload limit per request. To encrypt large objects (multi-GB blobs, video files, database backups) you use envelope encryption:

1. Generate a one-time Data Encryption Key (DEK) locally — e.g. a 256-bit random value from crypto/rand.
2. Encrypt the data with the DEK using AES-256-GCM in your application.
3. Call KMS Encrypt on the DEK bytes (well under 64 KiB) using your KEK. The result is a small wrapped DEK.
4. Store the ciphertext plus the wrapped DEK together (e.g. as a sidecar field or prepended to the file).
5. Discard the plaintext DEK from memory.

To decrypt: read the wrapped DEK, call KMS Decrypt to unwrap it, then decrypt the bulk data locally with the recovered DEK. CMEK on GCS/BQ/Compute uses this pattern internally — your KMS key is the KEK, and each object/block/row has its own DEK that the service manages.

# Pseudocode
dek = os.urandom(32)
ciphertext = aes_gcm_encrypt(dek, plaintext)
wrapped_dek = kms_client.encrypt(name=KEK, plaintext=dek).ciphertext
store(ciphertext, wrapped_dek)

Key Rotation

Both Secret Manager and Cloud KMS support rotation, but the mechanics differ.

Cloud KMS automatic rotation

For symmetric ENCRYPT_DECRYPT keys, set --rotation-period=7776000s (90 days) and --next-rotation-time=.... KMS creates a new primary version on schedule. Existing ciphertext stays decryptable because every ciphertext records the version that encrypted it. There is no "re-encrypt all old data" step — that is part of the design. To force re-encryption of an object, call gcloud storage objects update --encryption-key=... (CMEK), which rewrites with the current primary.

Asymmetric keys do not support automatic rotation because rotating a signing key invalidates downstream signature verification trust until consumers refresh public keys. You rotate asymmetric keys manually with gcloud kms keys versions create.

Secret Manager rotation

Secret Manager does not rotate secret values itself (it does not know what valid replacement values look like). Instead it provides rotation notifications: configure --next-rotation-time and a --topics Pub/Sub topic, and Secret Manager publishes a SECRET_ROTATE event when the timer fires. You subscribe with a Cloud Function or Cloud Run service that calls the upstream system (e.g. issues a new database password) and calls secrets versions add with the new value. Then destroy the old version after a grace period.

白話文解釋（Plain English Explanation）

Analogy 1: The Safe vs. The Locksmith

Secret Manager is a bank safe — you put the actual valuables (passwords) inside and the bank handles storage, audit logs, and access control. Cloud KMS is the locksmith — they make and operate the keys but they never hand you the raw key blank. You ask the locksmith "please lock this box for me" and they do it; the key stays in the locksmith's vault (the HSM if you upgraded). Mixing the two — putting the locksmith's keys inside the bank safe — defeats both: you lose KMS's tamper resistance and you bloat Secret Manager with material it was not designed for.

Analogy 2: Sealed Envelope Inside a Bigger Envelope

Envelope encryption is like sending a confidential letter through the postal service. You write the letter and seal it inside an inner envelope (the DEK encrypts the data). Then you put that inner envelope inside an outer envelope addressed to the recipient (the KEK in KMS wraps the DEK). Postal sorting machines (your storage system) only see the outer envelope, which is small and standard-sized — so the 64 KiB KMS limit never matters even if the letter is a 1000-page manuscript.

Analogy 3: The Hotel Master Key vs. Room Keys

A hotel has one master key (the KMS KEK in HSM) locked in the manager's office. Each guest gets a disposable room key (the DEK) that opens just their room. If a room key is lost the hotel re-keys that one door — they do not re-issue the master. CMEK on Cloud Storage works exactly like this: your KMS key is the master, every object has its own internal DEK, and rotating the master does not require rewriting every object. Re-encrypting the DEK is fast; re-encrypting petabytes of data would not be.

Putting It Together: A Realistic Cloud Run Workload

Picture a Cloud Run service that talks to Cloud SQL and Stripe:

1. The Cloud Run service runs as service account [email protected].
2. Three secrets exist with user-managed replication to us-east1 and europe-west1, CMEK enabled, keys in prod-secrets-us-east1 and prod-secrets-europe-west1 key rings:
   • cloudsql-password (rotated every 30 days via Pub/Sub-triggered Cloud Function)
   • stripe-secret-key (rotated manually quarterly)
   • tls-private-key (rotated annually, asymmetric)
3. IAM bindings are per-secret with roles/secretmanager.secretAccessor granted to api-runtime@.
4. Deployment uses --set-secrets=DB_PASSWORD=cloudsql-password:latest,STRIPE_KEY=stripe-secret-key:12,/tls/key=tls-private-key:latest.
5. The TLS key is mounted as a file (binary safe); database and Stripe come through as env vars with Stripe pinned to an explicit version so a bad rotation cannot bring down the API on the next cold start.
6. Cloud SQL itself uses CMEK with a separate KMS key in us-east1, so even a compromised Secret Manager would not yield decrypted database storage.

This single design exercises versions, user-managed replication, CMEK on Secret Manager, resource-level secretAccessor, both Cloud Run mount styles, and CMEK on Cloud SQL — the full PCD blueprint for this topic in one workload.

Frequently Asked Questions (FAQ)

Q1: Can I switch a secret from automatic to user-managed replication later?

A1: No. Replication policy is immutable after creation. Create a new secret with the desired policy, copy the payload via gcloud secrets versions add ... --data-file=..., update the application to reference the new secret, then destroy the old versions.

Q2: What is the right way to decommission a key suspected of compromise?

A2: For Secret Manager, immediately disable the suspect version (reversible — gives you investigation time), add a new version with rotated material, switch the application to the new version, then destroy the old version after the grace period. For KMS, disable the key version to block new operations; existing ciphertext encrypted under that version becomes undecryptable while disabled, so coordinate with downstream consumers before disabling.

Q3: Why can a single Cloud Run revision read Secret Manager but pods on GKE cannot?

A3: Cloud Run uses the service identity (runtime service account) attached to the revision. GKE pods use the node service account by default unless you enable Workload Identity, which maps a Kubernetes Service Account to a Google Service Account. Without Workload Identity the pod's effective identity is the node, which usually does not have secretAccessor on your secret.

Q4: When must I use HSM-backed keys instead of software keys?

A4: When the regulator requires FIPS 140-2 Level 3 hardware (common for PCI-DSS card data, certain government workloads, some banking regulations) or when your threat model says key material must be impossible to extract even by a Google insider. For most workloads SOFTWARE protection (FIPS 140-2 Level 1) is sufficient and cheaper.

Q5: Does CMEK on BigQuery encrypt the query results too?

A5: CMEK protects the stored data at rest (tables, materialized views, persisted query cache). Query results returned to the client are encrypted in transit via TLS and not persisted unless you save them to a destination table, in which case the destination table's CMEK applies. Temporary results held in the query cache inherit the table's CMEK.

Q6: Why is my Secret Manager CSI mount on GKE returning "permission denied" even though secretAccessor is granted?

A6: Almost always a Workload Identity misconfig. Check three things: (1) the KSA has an iam.gke.io/gcp-service-account annotation pointing at the GSA, (2) the GSA has roles/iam.workloadIdentityUser granting the KSA, and (3) the GSA itself has roles/secretmanager.secretAccessor on the specific secret resource. All three must align.

Q7: Can I use a global location KMS key to encrypt a regional GCS bucket?

A7: No. CMEK requires the key location to match the resource location. A regional bucket in us-east1 must use a key in us-east1 (or in the us multi-region for a multi-region bucket). Cross-location CMEK is not allowed because it would violate data-residency guarantees.