Data Storage Security and IAM

Introduction to Storage Security and IAM Best Practices

Storage Security and IAM Best Practices on Google Cloud is the discipline of locking down Cloud Storage buckets and objects so that only the right principal touches the right byte at the right time. The PDE exam treats this as table stakes: every data lake, every Dataflow job, every BigQuery external table eventually leans on a Cloud Storage bucket, and one wrong allUsers grant can leak millions of records overnight.

This note walks through Storage Security and IAM Best Practices from the ground up — IAM roles, uniform bucket-level access, signed URLs, encryption with CMEK and CSEK, VPC Service Controls, retention, audit, and secure data exchange via Analytics Hub. Read it once, then come back to the FAQ before exam day.

白話文解釋（Plain English Explanation）

Before diving into role names and JSON policies, it helps to picture Cloud Storage as a physical thing. Three different analogies work because Storage Security and IAM Best Practices touches three different worlds: who is allowed in, what they can carry out, and how long the building stands.

Think of a Cloud Storage Bucket as an Office Building With Door Codes

Imagine a six-floor office. The lobby guard is IAM. Every visitor swipes a badge. Some badges only open the lobby (Storage Object Viewer — read), some open the lobby plus the supply closet (Storage Object Creator — write but no read), and some open every door including the fire-alarm panel (Storage Admin). Old buildings let each floor manager hand out their own keys for individual rooms — that is object ACLs. New buildings forbid floor-level keys and route every request through the lobby card reader — that is uniform bucket-level access. Public Access Prevention is the rule that says "even if a floor manager hands a stranger a master key, the lobby door still refuses anyone without a registered badge." Storage Security and IAM Best Practices is essentially: switch on the lobby card reader, throw away the floor keys, and turn on the no-public-keys rule.

Think of a Signed URL as a Single-Use Hotel Key Card

Hotels do not give every food-delivery driver an employee badge. The front desk encodes a key card that opens room 412, expires at midnight, and works exactly once. Signed URLs do the same: the bucket owner signs a request that grants a stranger time-boxed access to a specific object without ever creating an IAM identity. Signed policy documents are the variant for browser uploads — they let a web form POST a file straight into the bucket while constraining size, content type, and target prefix. Both let you punch a controlled hole through Storage Security and IAM Best Practices without permanently widening the door.

Think of Bucket Lock as a Bank Safety-Deposit Box With a Notarised Lease

When a bank rents out a safety-deposit box for ten years, you sign a notarised contract. Even the bank president cannot pull your gold out before the lease ends. Bucket Lock works the same: once you lock a retention policy of, say, seven years, no one — not the project owner, not the org admin, not Google support — can delete an object before its retention clock expires. This is what makes Cloud Storage acceptable to FINRA, SEC 17a-4, and HIPAA auditors. Object Versioning is the parallel "shred bin" rule: every overwrite or delete leaves a noncurrent version sitting in the basement until lifecycle rules sweep it out.

Think of CMEK and CSEK as Whose Keys Open the Vault

Default encryption (Google-managed keys, GMEK) is like the bank using its own master keys — convenient but you trust the bank fully. Customer-Managed Encryption Keys (CMEK) means you own the key inside Cloud KMS; the bank still operates the vault but cannot open it without asking your key. Customer-Supplied Encryption Keys (CSEK) means you bring your own physical key from home every visit and the bank never keeps a copy — strongest separation, highest operational pain. Storage Security and IAM Best Practices for regulated industries usually lands on CMEK with key rotation and Cloud HSM backing.

Core Concepts of Storage Security and IAM Best Practices

The PDE exam tests whether you can match a scenario to the right control. The toolbox below covers about 90% of what you will see.

IAM principals are the who: user accounts, Google groups, service accounts, workforce identities, and the sentinel allUsers / allAuthenticatedUsers. Storage Security and IAM Best Practices starts with naming the principal correctly — most leaks happen because a developer typed allUsers instead of a group address.

IAM roles for Cloud Storage define the what. The four you must memorise:

• roles/storage.objectViewer — read object data and metadata, list objects.
• roles/storage.objectCreator — write new objects only; no read, no overwrite.
• roles/storage.objectUser — read, write, delete objects (modern replacement for the old objectAdmin in many cases).
• roles/storage.objectAdmin — full object control plus list and delete.
• roles/storage.admin — everything above plus bucket create, delete, IAM policy change, and lifecycle config.

Resources are the where: organisation, folder, project, bucket, or individual object. Cloud Storage IAM bindings live at project or bucket level. Object-level ACLs are legacy and incompatible with uniform bucket-level access.

Conditions narrow the when and on-what. IAM Conditions let you say "this binding only applies to objects with prefix /finance/ between 09:00 and 18:00 from request.auth.access_levels matching corp-trusted." Conditions move Storage Security and IAM Best Practices from coarse role grants to fine-grained policies without spawning a thousand custom roles.

Architecture and Design Patterns for Storage Security and IAM Best Practices

A production Cloud Storage layout usually splits buckets by trust boundary, not by application. The pattern that scores points on the PDE exam:

prj-data-landing       (raw external upload)
  bkt-landing-eu       UBLA on, PAP enforced, retention 30d, CMEK key-A
prj-data-curated       (post-validation)
  bkt-curated-eu       UBLA on, CMEK key-B, VPC-SC perimeter "data-prod"
  bkt-archive-eu       UBLA on, Bucket Lock 7y, Object Versioning on
prj-data-consumer      (BI / ML readers)
  no buckets — only IAM grants on curated buckets

The landing bucket sits outside the perimeter so external partners can drop files via signed URLs. A Cloud Function triggered by Object Finalize validates schema, scans with DLP, and copies clean data into the curated bucket inside the VPC-SC perimeter. The archive bucket lives behind Bucket Lock for compliance. Consumers never see the landing zone — they read from curated only, and their service accounts get roles/storage.objectViewer with an IAM Condition restricting them to specific prefixes.

This pattern hits four Storage Security and IAM Best Practices targets at once: blast radius isolation, encryption key separation, perimeter enforcement, and immutable archive.

GCP Service Deep Dive

Bucket-level IAM versus Object ACLs

Two access systems coexist for historical reasons. Bucket-level IAM grants apply uniformly to every object in the bucket. Object ACLs let you set per-object permissions — a relic from the original Google Storage XML API. The exam wants you to know that mixing both creates an unauditable mess: the effective permission on an object is the union of bucket IAM and object ACL, so "least privilege" calculations require reading two systems.

The fix is uniform bucket-level access. Toggle it on, ACLs disappear from the request path, and every audit query is a single IAM lookup. UBLA has a 90-day "lock" timer — once locked you cannot turn it off, which is exactly what auditors want.

Public Access Prevention (PAP)

PAP is an org-policy-style switch on each bucket that refuses any IAM binding to allUsers or allAuthenticatedUsers, even if a developer tries. You can set it at organisation level via storage.publicAccessPrevention constraint. With PAP enforced, a single allUsers grant attempt returns an error instead of silently exposing the bucket. Storage Security and IAM Best Practices says: enforce PAP at org level, period. The only exception is buckets backing public static websites, which should live in a separate isolated project.

Signed URLs and Signed Policy Documents

Signed URLs solve "I need to share file X with a non-Google identity for 15 minutes." The bucket owner signs a V4 request using a service account's private key (or via the iamcredentials.signBlob API to avoid distributing keys). The signed URL embeds the HTTP verb, object path, expiry, and signature. Anyone with the URL can perform that exact request until it expires.

Signed policy documents (also called POST policies) are the variant for browser-direct uploads. The server hands the browser a signed JSON policy that constrains: target bucket, key prefix, content-type allowlist, max content-length, and expiry. The browser POSTs the file straight to Cloud Storage; your backend never proxies the bytes. This is how SaaS apps let users upload 5 GB files without burning Cloud Run egress.

VPC Service Controls for Cloud Storage

VPC-SC builds a perimeter around Google APIs. Once you put storage.googleapis.com inside a perimeter, requests from outside — even with valid IAM — get a context error. This stops the classic exfiltration scenario where a stolen service-account key is used from an attacker's laptop.

Key concepts: a perimeter contains projects; ingress and egress rules define which identities and methods may cross the boundary; access levels (built from Access Context Manager) gate based on IP range, device posture, or workforce identity. For Cloud Storage specifically, VPC-SC blocks gsutil cp from outside the perimeter, blocks BigQuery external queries against perimeter buckets unless BigQuery's service agent is allow-listed, and blocks signed URL access by default (you must add an ingress rule for unauthenticated signed URL access if you want to keep that pattern).

Encryption: GMEK, CMEK, and CSEK

Every object in Cloud Storage is encrypted at rest. The choice is whose key.

GMEK (Google-managed) is the default — zero config, zero ops, Google rotates keys for you. Acceptable for most non-regulated workloads.

CMEK (Customer-managed via Cloud KMS) means you create a key in KMS, grant the Cloud Storage service agent (service-PROJECT_NUMBER@gs-project-accounts.iam.gserviceaccount.com) the roles/cloudkms.cryptoKeyEncrypterDecrypter role on it, and assign the key as the bucket default. Every object write triggers a KMS encrypt; every read triggers a KMS decrypt. You get key rotation control, audit logs of every KMS use, and the ability to destroy the key (which crypto-shreds all objects). CMEK supports software keys, HSM-backed keys, and external keys (EKM) for the highest separation tier.

CSEK (Customer-supplied) means you pass a raw 256-bit AES key on every request via the x-goog-encryption-key header. Google never stores the key — losing it means losing the data. CSEK is rare in modern architectures because CMEK with EKM gives you the same separation with far less operational risk.

Object Versioning and Object Lock

Object Versioning tracks every overwrite and delete by stashing the prior bytes as a noncurrent version. Combined with a lifecycle rule like "delete noncurrent versions older than 30 days" you get a built-in undo with bounded cost. Versioning protects against accidental gsutil rm -r gs://bucket/, ransomware that overwrites files, and bad ETL jobs that truncate good data.

Object Lock (per-object retention) and bucket-level Retention Policies enforce minimum age before deletion. A retention policy on the bucket says "no object may be deleted until N seconds after creation." Bucket Lock then makes that policy itself immutable — even reducing the retention period requires waiting it out. Together with Versioning they form WORM (write-once-read-many) storage that satisfies SEC 17a-4(f), CFTC 1.31, and FINRA 4511.

Audit Logs for Cloud Storage

Cloud Storage emits two relevant log streams:

• Admin Activity logs — always on, free, capture IAM changes and bucket configuration changes.
• Data Access logs — opt-in (per project, per service), capture every object read, write, and metadata get. Volume can be enormous on a busy data lake; budget for it.

For Storage Security and IAM Best Practices, the minimum is enabling DATA_READ and DATA_WRITE on critical buckets and routing logs to a separate logging project that the data team cannot delete. Pair with a sink to BigQuery for forensic queries and a sink to Cloud Storage with Bucket Lock for the audit log retention itself.

Secure Data Exchange via Analytics Hub

Analytics Hub (now part of BigQuery sharing) lets a data publisher expose datasets to subscribers across organisations without copying data. For Cloud Storage data, the pattern is: register the bucket-backed BigQuery external table, publish it as a listing in an exchange, and subscribers create linked datasets in their own projects. The publisher controls who can subscribe via IAM; subscribers query in their own billing without ever touching the source bucket directly.

This matters for Storage Security and IAM Best Practices because it replaces the old anti-pattern of "give partner X a service account with bucket read access" — which leaves you with N service accounts to rotate and N egress paths to audit — with a single declarative listing whose ACL lives in one place.

Common Pitfalls and Trade-offs

The exam loves scenarios where a well-meaning engineer creates a problem. Watch for these.

Granting roles/storage.admin at project level "for convenience." This silently lets the principal change bucket IAM, disable Bucket Lock during the unlock window, and delete buckets. Use roles/storage.objectAdmin at bucket level instead.

Mixing UBLA-on and UBLA-off buckets in the same project. Tooling and runbooks diverge — engineers forget which bucket needs ACL syntax. Enforce UBLA via org policy storage.uniformBucketLevelAccess so new buckets get it by default.

CMEK key in the same project as the bucket. Defeats the separation-of-duties goal because anyone with project owner gets both KMS and Storage admin. Put KMS keys in a dedicated prj-kms project owned by the security team.

Signed URLs with 7-day expiry. The V4 signing spec allows up to 7 days, but anything beyond a few hours is essentially a permanent credential leak waiting to happen. Cap signed URL lifetime at 1 hour for human use, 15 minutes for browser uploads.

Forgetting the Cloud Storage service agent on CMEK rotation. When you rotate a key, the bucket keeps using the old version until you update the bucket default. New writes get encrypted with the new version automatically; existing objects still need the old version to decrypt. Never disable an old key version while objects encrypted with it still exist.

Bucket Lock as a panic button. Bucket Lock is irreversible. Lock a 100-year retention policy by accident and you own that storage bill until 2126. Always test retention on a throwaway bucket first, then apply to production with a reviewed runbook.

Best Practices for Storage Security and IAM

A condensed checklist drawn from Google's own security blueprint and CIS benchmark:

• Enforce uniform bucket-level access on every bucket; lock it after the 90-day grace.
• Enforce Public Access Prevention via the org policy storage.publicAccessPrevention.
• Grant IAM at bucket level using groups, never individual user accounts. Audit group membership monthly.
• Use IAM Conditions to scope access by object prefix, request time, or access level rather than spawning custom roles.
• Place sensitive buckets inside a VPC Service Controls perimeter; start in dry-run for at least two weeks before enforcement.
• Use CMEK with HSM-backed keys for regulated data; keep KMS keys in a separate project owned by the security team.
• Turn on Object Versioning plus a lifecycle rule that expires noncurrent versions in 30 to 90 days.
• Apply a retention policy and Bucket Lock to any bucket subject to compliance retention (financial, healthcare, legal hold).
• Enable Data Access audit logs on critical buckets; sink to a separate logging project.
• Use signed URLs (V4) with short expiry instead of widening IAM for one-off external sharing.

Real-World Use Case

A mid-sized European insurer (about 800 engineers, EUR 300M annual revenue) runs claims data through Cloud Storage as the backbone of its data lake. The compliance team requires GDPR-compliant retention, Solvency II audit trails, and the ability to honour customer deletion requests within 30 days.

The architecture they landed on after six months of iteration applies Storage Security and IAM Best Practices end-to-end:

• Three layers of buckets: bkt-claims-landing-eu, bkt-claims-curated-eu, bkt-claims-archive-eu, each in a separate project with its own service-account boundary.
• All buckets enforce UBLA, PAP, and live inside a single VPC-SC perimeter that also wraps BigQuery and Pub/Sub.
• CMEK with HSM-backed keys in a prj-eu-kms-prod project; rotation every 90 days; bucket service agents granted decrypter role explicitly.
• Landing bucket: 30-day retention policy (unlocked) so the team can adjust during onboarding of new partners; signed URLs with 15-minute expiry for partner uploads.
• Curated bucket: Object Versioning on, lifecycle deletes noncurrent after 60 days, IAM Conditions restrict the analytics group to prefix:/non-pii/.
• Archive bucket: 10-year retention policy locked via Bucket Lock; Versioning on; objects move to Coldline after 30 days, Archive after 365.
• Customer deletion requests run through a Cloud Workflows process that uses a privileged service account (auditable, behind break-glass IAM Conditions) to issue the rare authorised delete.
• Data Access logs sink to a logging project owned by the security team; the data team has zero access to that project.
• Partner data sharing happens via Analytics Hub listings on top of BigLake tables that read from the curated bucket — no partner ever holds direct Cloud Storage IAM.

The result: zero public-bucket findings in two consecutive third-party audits, GDPR deletion SLA met at 7-day median, and a single Cloud KMS dashboard the security team uses to prove key custody.

Exam Tips for Storage Security and IAM Best Practices

The PDE exam phrases storage security questions in a few predictable ways. Practise pattern matching.

"A regulator requires data to be immutable for seven years." Answer: bucket retention policy plus Bucket Lock. Object Versioning alone is not enough — versions can still be deleted.

"The security team must control encryption keys and revoke access by destroying the key." Answer: CMEK. CSEK works too but is operationally heavy and rarely the best answer.

"A partner must upload files but should not see other tenants' files." Answer: signed policy document scoped to a tenant-specific prefix, short expiry. Avoid creating IAM identities for external partners.

"Prevent data exfiltration if a service account key is leaked." Answer: VPC Service Controls perimeter around storage.googleapis.com. IAM alone does not stop a valid credential from being used outside Google's network.

"Make sure no developer can ever expose a bucket publicly." Answer: org policy storage.publicAccessPrevention set to enforced. PAP at the bucket level helps but org policy is the durable control.

"The same data must be shared with three external companies with auditable access." Answer: Analytics Hub (BigQuery sharing) on top of a BigLake table over the bucket. Direct Cloud Storage IAM grants do not scale and are hard to audit.

Distinguish ObjectViewer vs ObjectUser vs ObjectAdmin. Viewer reads. ObjectUser reads, writes, and deletes objects (the modern unified role). ObjectAdmin adds metadata and ACL control. Storage Admin is bucket-level (create, delete, IAM).

UBLA is reversible for 90 days, irreversible after lock. Bucket Lock on retention policies is irreversible immediately. Memorise these timers.

Frequently Asked Questions (FAQ)

What is the difference between uniform bucket-level access and object ACLs?

Object ACLs are the legacy per-object permission system inherited from the original Google Storage XML API. Each object can have its own ACL listing principals and roles, completely independent of bucket IAM. Uniform bucket-level access (UBLA) disables ACLs entirely and forces every request through bucket-level IAM, which is far easier to audit and reason about. UBLA is now the default for new buckets created via the console and is required by the CIS GCP benchmark. Once you enable UBLA you have a 90-day window to disable it; after lock it is permanent. For Storage Security and IAM Best Practices, treat UBLA as non-negotiable on every new bucket.

When should I use a signed URL instead of granting IAM access?

Use a signed URL whenever the recipient is a non-Google identity, the access need is short-lived, or the recipient should never appear in your IAM policy. Common cases: a customer downloading their exported report, a partner uploading a daily file, a browser POSTing a user-uploaded image. Signed URLs avoid creating service accounts you would otherwise have to track and rotate. Keep expiry short (15 minutes for uploads, 1 hour for downloads), sign via the iamcredentials.signBlob API to avoid distributing private keys, and audit signing through Cloud Audit Logs on the signing service account.

What is the practical difference between CMEK and CSEK on Cloud Storage?

CMEK keeps the encryption key inside Cloud KMS where Google manages the key material but you control the key lifecycle — rotation, version destruction, IAM on the key, and audit of every encrypt/decrypt operation. You can back CMEK with software, HSM, or external keys (EKM). CSEK requires you to send the raw 256-bit AES key on every request; Google never stores it, so losing the key means crypto-shredding all objects encrypted under it. CSEK gives slightly stronger separation but costs you operational pain (every client must hold the key) and most regulatory frameworks accept CMEK with HSM backing as equivalent. In modern PDE-era architectures CMEK is the default answer; CSEK appears mainly in legacy or extremely regulated edge cases.

How do I prevent a service account key leak from leading to data exfiltration?

Layer two controls. First, eliminate long-lived service account keys wherever possible: use Workload Identity Federation for workloads outside GCP, attached service accounts for compute inside GCP, and short-lived tokens issued via the iamcredentials API. Second, wrap your Cloud Storage projects in a VPC Service Controls perimeter so even a valid stolen credential cannot pull data from outside trusted networks. Add IAM Conditions tied to access levels (IP, device, identity) for additional defence in depth. Audit logs on the signing service account give you the forensic trail if something does slip.

Can I make a Cloud Storage object truly immutable?

Yes — apply a retention policy at the bucket level and lock it with Bucket Lock. Once locked, no principal at any IAM level can shorten the retention period or delete objects before their retention clock expires. Combine with Object Versioning so overwrites also leave an immutable noncurrent version. This combination satisfies SEC 17a-4(f), CFTC 1.31, FINRA 4511, and most healthcare retention rules. Be cautious: Bucket Lock is irreversible, so test retention durations on a disposable bucket before locking production data for a decade.

How does VPC Service Controls actually block a stolen credential?

VPC-SC enforces a context check on every API call to services inside the perimeter. The check validates the request's network origin (which Google project, which VPC, which on-prem connection via Private Google Access) and any attached access level. A stolen service-account key used from an attacker's laptop fails the network-origin check because the request originates outside the trusted perimeter, and it returns a context-violation error regardless of how valid the credential is. The credential still works fine for principals inside the perimeter, so legitimate workflows continue. The catch: every external integration (third-party SaaS, partner pipelines, on-prem ETL) must be explicitly added via ingress rules or it will break.

What audit logs should I enable on a sensitive Cloud Storage bucket?

At minimum enable Data Access audit logs for storage.googleapis.com covering DATA_READ and DATA_WRITE. Admin Activity logs are always on and capture IAM and bucket configuration changes for free. Route both streams through an aggregated sink to a dedicated logging project owned by the security team, and from there to BigQuery for forensic queries and to a Bucket-Locked Cloud Storage bucket for long-term retention. Be aware that Data Access logs on a high-traffic data lake can produce terabytes per month — budget for the log ingestion cost and use exclusion filters on truly low-value reads (for example, public asset downloads on a CDN-fronted bucket).

Should I use Analytics Hub or signed URLs for sharing data with partners?

It depends on the consumption pattern. Signed URLs work for one-off file downloads where the partner just needs the bytes. Analytics Hub fits when partners want to query the data in BigQuery, join it with their own datasets, and consume continuously. Analytics Hub on top of a BigLake table over your Cloud Storage bucket gives partners SQL access in their own projects (their billing, not yours) without ever issuing them Cloud Storage IAM. Subscriptions are auditable in one place and revocable in one click. For modern data-sharing scenarios in the PDE exam, Analytics Hub is usually the intended answer when the question mentions "multi-tenant," "external organisations," or "data marketplace."

Related Topics

• Cloud Storage Data Lake Design — how to lay out buckets, prefixes, and lifecycle for analytics workloads.
• PII De-identification with DLP — pair DLP scans with bucket security to prevent sensitive data landing unprotected.
• Data Sovereignty and Compliance Design — region selection, CMEK, and VPC-SC patterns for GDPR, HIPAA, and Solvency II.

Further Reading

• Cloud Storage access control overview — official documentation covering IAM, ACLs, signed URLs, and access policy interactions.
• Uniform bucket-level access — enabling, locking, and migrating buckets off legacy ACLs.
• Cloud Storage encryption options — comparison of GMEK, CMEK, CSEK, and HSM/EKM backing.
• VPC Service Controls overview — perimeter design, dry-run mode, and ingress/egress rules.
• Bucket Lock and retention policies — WORM compliance patterns and irreversibility considerations.