examlab .net The most efficient path to the most valuable certifications.
Support
In this note ≈ 18 min

Cloud Audit Logs and Log Sinks

3,500 words · ≈ 18 min read ·

Master observability and compliance in Google Cloud. Learn about the different types of audit logs, configuring log sinks for centralized security analysis, and managing log retention.

Do 20 practice questions → Free · No signup · PSE

Introduction to Cloud Audit Logs

In security, if it isn't logged, it didn't happen. Cloud Audit Logs provide the "Who, What, Where, and When" for every action taken in your Google Cloud environment.

For the PSE, understanding the nuances of log types, how to route them for centralized analysis, and how to maintain them for compliance is essential for incident response and forensic investigations.

白話文解釋（Plain English Explanation）

Analogy 1: The Office Building's Security Cameras

Think of your GCP organization as a large office building. Admin Activity Logs are the cameras at the front lobby and IT closet doors — they record every time someone changes the building (creates a VM, edits an IAM policy). They're always on, free, and the tape is kept for 400 days. Data Access Logs are the cameras inside the executive filing rooms — they record who opens which folder (reads a GCS object, queries a BigQuery table). Because these cameras record so much footage, they're off by default (except BigQuery), cost money to run, and the tape is only kept for 30 days unless you upgrade the storage. System Event Logs are the building maintenance log — Google's own staff fixing the elevator (live migrating your VM). You don't pay for these and they're kept for 400 days.

Analogy 2: The Mail Sorting Room (Log Router)

The Log Router is the mail sorting room of your GCP environment. Every log entry is a letter. A Log Sink is a forwarding rule: "every letter matching this filter goes to this destination." You can forward letters to four mailboxes — Cloud Storage (the cheap archive warehouse, ideal for 7-year compliance with Bucket Lock), BigQuery (the analyst's desk with SQL), Pub/Sub (the express courier feeding Splunk/Chronicle SIEM), or Log Buckets (an internal P.O. box inside Google Cloud Logging). An Aggregate Sink at the Organization level is like installing one forwarding rule on the building's central mail room — every floor (project) gets covered automatically.

Analogy 3: The Bank Vault Time-Lock

GCS Bucket Lock retention policies are the bank vault's time-lock mechanism. Once you set a 7-year retention policy on a logging bucket and lock it, nobody — not even a project owner with full IAM — can delete those logs before the timer expires. This is what auditors want to see for SOX, HIPAA, and PCI-DSS evidence. Pair this with an Aggregate Sink at the Organization level routing to a separate "Compliance Project," and you've built a tamper-evident vault that even a compromised admin in a child project cannot erase. The vault key (project IAM) is separated from the cameras (log producers) — that separation of duties is the whole point.

The Three Types of Audit Logs

Google Cloud categorizes audit logs into three main types, each with different retention and cost profiles.

1. Admin Activity Logs

  • Content: Log entries for API calls or other actions that modify the configuration or metadata of resources (e.g., creating a VM, changing an IAM policy).
  • Cost: Always enabled and free of charge.
  • Retention: 400 days.

2. Data Access Logs

  • Content: Log entries for API calls that create, modify, or read user-provided data (e.g., reading an object in GCS, querying a table in BigQuery).
  • Cost: Disabled by default (except for BigQuery) because they can generate high volumes of data and incur costs.
  • Retention: 30 days (default), but can be extended.

3. System Event Logs

  • Content: Log entries for Google Cloud administrative actions that modify resource configurations (e.g., a Google-initiated VM migration).
  • Cost: Always enabled and free of charge.
  • Retention: 400 days.

Because Data Access Logs are disabled by default, you must explicitly enable them in the IAM & Admin console for the services you want to monitor. This is a common "gotcha" in security audits.

Routing Logs with Log Sinks

The Log Router determines where your logs go. By creating Log Sinks, you can export logs to external destinations for long-term storage or analysis.

Sink Destinations:

  • Cloud Storage (GCS): For cost-effective, long-term compliance storage.
  • BigQuery: For complex security analysis using SQL.
  • Pub/Sub: For real-time streaming to a SIEM (like Splunk or Chronicle) or triggering automated remediation via Cloud Functions.
  • Log Buckets: For centralized logging within Google Cloud.

A Log Sink is a configuration that includes a Filter (which logs to export) and a Destination (where to send them).

Centralized Logging Architectures

For enterprise environments, logs should be centralized in a dedicated "Logging Project."

  • Aggregate Sinks: Created at the Organization or Folder level to automatically collect logs from all child projects.
  • Log Buckets: You can create custom log buckets in the logging project and route logs from other projects into them.

For long-term compliance archives (e.g. 7-year HIPAA / SOX / PCI-DSS retention), do not rely on the default _Default log bucket (max 3650 days, project-scoped, project-owner can delete). Instead, create an Aggregate Sink at the Organization node with a destination of a GCS bucket in a dedicated Compliance Project, then enable Bucket Lock with a Retention Policy. Bucket Lock makes the retention period irrevocable — even an Organization Admin cannot shorten or delete locked objects before expiry, which is exactly what auditors require for tamper-evident WORM storage.

Log Analytics and SQL

Log Analytics is a feature that allows you to query your logs using standard SQL within the Logging console.

  • Linked Datasets: You can link a Log Bucket to BigQuery, allowing you to join log data with other business data for deeper insights.
  • Security Use Case: "Find all users who accessed a specific KMS key in the last 24 hours and identify their IP addresses."

Cost Management: Exclusion Filters

Logging can become expensive if not managed properly.

  • Exclusion Filters: Allow you to discard logs that you don't need (e.g., "Exclude all successful get requests from a high-traffic GCS bucket").
  • Sampling: You can choose to only log a percentage of certain events to reduce volume while still maintaining visibility.

Always prioritize logging write operations and IAM changes. Read operations (Data Access) are high-volume; only enable them for high-value resources.

A common exam trap: Exclusion Filters configured on the _Required sink will be silently ignored — Admin Activity, System Event, and Access Transparency logs are mandatory and cannot be excluded or disabled at the sink level. Exclusions only apply to the _Default sink and user-created sinks. A second trap: when you create an Aggregate Sink at the Org with include_children=true, the sink's writer service account (p[PROJECT_NUMBER]-[ID]@gcp-sa-logging.iam.gserviceaccount.com) must be granted roles/bigquery.dataEditor or roles/storage.objectCreator on the destination — sinks fail open and silently drop logs if the writer identity lacks write permission, and you only discover this during an incident when the evidence isn't there.

Auditing the Auditors

Logs themselves are sensitive data. You must protect them from unauthorized access or deletion.

  • IAM for Logging: Use the logging.viewer and logging.privateLogViewer roles. The latter is required to see Data Access logs.
  • Access Transparency: Logs that show when Google personnel access your data (supplementary to Audit Logs).

Investigating Security Events

Log Explorer

The primary tool for ad-hoc troubleshooting and investigation.

  • Use Power Queries to filter by resource.type, methodName, and principalEmail.
  • Save queries for common investigations (e.g., "Unauthorized Firewall Changes").

VPC Flow Logs and Firewall Insights

  • VPC Flow Logs: Record network traffic (source/destination IP, port, protocol). Crucial for detecting lateral movement.
  • Firewall Insights: Identifies shadowed or over-permissive firewall rules.

PSE Exam Scenarios

Scenario 1: Retaining Logs for 7 Years

"A company is required by law to retain all Admin Activity and Data Access logs for a period of 7 years. How can this be achieved in the most cost-effective way?" Answer: Create an Aggregate Log Sink at the Organization level with a destination of a Cloud Storage bucket in a separate "Compliance Project." Use a Retention Policy on the GCS bucket to enforce the 7-year requirement.

Scenario 2: Real-time Alerting for IAM Changes

"The Security Operations Center (SOC) needs to be alerted within seconds whenever a new user is granted the 'Owner' role in any project. What is the best architecture?" Answer: Create a Log Sink with a filter for protoPayload.methodName="SetIamPolicy" and route it to Pub/Sub. Have a Cloud Function triggered by the Pub/Sub topic to parse the log and send an alert to the SOC.

Memorize: Retention, Log Types, and Sink Mechanics

Retention defaults (commit to memory):

  • Admin Activity logs: 400 days, free, always-on, cannot disable.
  • System Event logs: 400 days, free, always-on, cannot disable.
  • Data Access logs: 30 days, paid, off by default (exception: BigQuery DATA_READ is on). Sub-types: ADMIN_READ, DATA_READ, DATA_WRITE.
  • Policy Denied logs: 30 days, free, on by default when VPC Service Controls denies a request; route to SIEM for perimeter-breach alerting.
  • Access Transparency logs: 400 days, captures Google personnel access (requires Premium/Enterprise support tier).

Log Bucket retention: _Required = 400 days (immutable, cannot change); _Default = 30 days (configurable up to 3650 days); custom buckets = 1 to 3650 days.

Log-based metrics (two types):

  • System-defined metrics: pre-built (e.g. logging.googleapis.com/byte_count).
  • User-defined metrics: Counter (events matching a filter) or Distribution (extracts a numeric value, e.g. response latency). Counter metrics feed Cloud Monitoring alerting policies — the canonical pattern is protoPayload.methodName="SetIamPolicy" counter → alert when count > 0 in 60s.

Sink destination decision matrix:

  • BigQuery → ad-hoc SQL forensics, joins with business data, partitioned tables.
  • GCS (with Bucket Lock) → multi-year WORM compliance archive.
  • Pub/Sub → real-time fan-out to Splunk / Chronicle / Cloud Function remediation.
  • Log Bucket → in-GCP retention + Log Analytics SQL without leaving Logging.

privateLogViewer role is required to read Data Access and Access Transparency logs; plain logging.viewer cannot.

Summary Checklist

  • Differentiate between Admin Activity, Data Access, and System Event logs.
  • Explain why Data Access logs are disabled by default.
  • List the four primary destinations for a Log Sink.
  • Describe the benefit of using an Aggregate Sink at the Folder/Org level.
  • Identify the role required to view Data Access logs (privateLogViewer).
  • Explain how to use Log Analytics to query logs with SQL.

Official sources

More PSE topics