Firestore and Cloud Bigtable

Introduction to Firestore and Bigtable

For applications that require flexible schemas or massive throughput, Google Cloud provides two complementary NoSQL options that frequently appear together on the PCD exam: Firestore, a serverless document database tuned for mobile, web, and IoT clients that need real-time sync and offline support; and Cloud Bigtable, a wide-column, sparsely populated table service tuned for petabyte-scale operational and analytical workloads with sub-10 ms latency. Both are NoSQL, but they target very different access patterns. Firestore is optimized for small, semi-structured documents fetched by ID or by indexed query, with strong consistency and ACID transactions. Bigtable is optimized for huge, append-heavy time-series or telemetry data, where a single primary key (the row key) determines almost everything about performance, and there are no secondary indexes, joins, or multi-row transactions. The PCD blueprint expects you to choose between them based on data shape, query pattern, latency targets, durability, and replication needs — and to know the design rules (mode at creation time, row-key anti-patterns, Key Visualizer, Autoscaling) that frequently appear as distractors.

Firestore Native vs Datastore Mode

Firestore is a single product with two modes — Firestore Native and Datastore mode — and the mode is chosen at database creation time and cannot be changed later. This irreversibility is the most common exam trap. Pick the wrong mode and the only remediation is to create a second database and migrate, often through Dataflow or scheduled export/import.

Firestore Native Mode

Native mode is the default and the recommended choice for almost all new workloads. It offers real-time listeners (onSnapshot), offline persistence for the Web, iOS, and Android SDKs, mobile/web Security Rules, strong consistency for all queries, and ACID transactions across up to 500 documents. Read and write fan-out is optimized for mobile clients hitting the database directly without a backend server. Throughput is virtually unbounded but is rate-limited at the document level — roughly 1 sustained write per second per document — to give predictable strong consistency.

Datastore Mode

Datastore mode is the heir to the original App Engine Datastore. It exposes the legacy Datastore API, supports very high write throughput per entity group, and is intended for server-side workloads where you do not need mobile SDKs, real-time listeners, or Security Rules. It is the right pick when you are migrating an existing App Engine application or running batch back-ends that need eventually consistent global queries with high QPS.

Choosing Between Modes

If your workload includes mobile clients, real-time UI updates, or client-side Security Rules, you need Native mode. If you are running stateless App Engine, Cloud Run, or GKE back-ends behind a service account and never expose the database to end clients, Datastore mode is acceptable. Either way, the decision is permanent for that database.

Firestore Multi-Region: nam5 and eur3

When you provision a Firestore database you must also choose a location — and this is also fixed for the life of the database. There are three location types: regional (single region such as us-central1), multi-region, and eventual multi-region. The two multi-region locations you must know for PCD are nam5 and eur3.

nam5 (United States)

nam5 is a North-America multi-region location that replicates data synchronously across multiple US regions, giving Firestore a 99.999% availability SLA and protection from a regional outage. Writes are committed using Paxos across the constituent regions, which adds a few tens of milliseconds of latency compared to a regional database, but the database survives the loss of an entire region with zero data loss.

eur3 (Europe)

eur3 is the European counterpart, replicating across European regions. It is the right answer when the question mentions GDPR, EU data residency, or European user base needing multi-region durability. The same 99.999% SLA applies.

Regional vs Multi-Region Trade-offs

Regional Firestore has a 99.99% availability SLA, lower write latency, and lower cost. Multi-region (nam5, eur3) costs more but adds the extra "9" of availability and survives regional failures. The location, like the mode, cannot be changed after creation, so the PCD answer for "highest durability across regions" is to provision in nam5 or eur3 up front.

Firestore Security Rules

Firestore Security Rules are a domain-specific language that runs inside Google's edge and authorizes every client read and write before it reaches the database. They are what allows mobile and web clients to talk to Firestore directly without a Cloud Run/App Engine middleware tier.

Authoring Rules

Rules are deployed via firebase deploy --only firestore:rules and live in a firestore.rules file. The basic shape is match /collection/{docId} { allow read, write: if <condition>; }. Conditions can reference request.auth.uid (the Firebase Auth-authenticated user), request.resource.data (the proposed document state), resource.data (the existing state), and even get() / exists() lookups against other documents.

Common Patterns

The canonical pattern is per-user data: allow read, write: if request.auth != null && request.auth.uid == userId;. Role-based access uses get(/databases/$(database)/documents/users/$(request.auth.uid)).data.role == 'admin'. Rules can validate field types, enforce schema with request.resource.data.keys().hasAll(['title','body']), and prevent unauthorized field changes.

Rules vs IAM

A common PCD trap is mixing up Security Rules and IAM. IAM grants service accounts and developers access to the Firestore API itself; Security Rules govern what end-user clients can read or write through the SDK. Both are evaluated, but Security Rules are the layer that matters for mobile/web access.

Firestore vs Realtime Database

Firebase Realtime Database (RTDB) is the predecessor product to Firestore Native, and the PCD exam still expects you to choose between them. RTDB stores data as one giant JSON tree with WebSocket-pushed updates, simple "shallow" queries, and a single-region location. It scales to roughly 100,000 concurrent connections per database instance and is cheap for small, latency-sensitive presence/chat workloads.

When to Pick Firestore

Pick Firestore when you need: structured collections and documents instead of a JSON tree; rich queries with sorting, filtering, and pagination on multiple fields; ACID transactions across multiple documents; multi-region (nam5/eur3) replication for 99.999% availability; or growth beyond a single-instance connection limit. Firestore is also the strategic platform Google is investing in.

When RTDB Still Wins

RTDB still wins for hyper-low-latency presence systems where you need sub-100 ms fan-out of a single small payload to many connected clients — leaderboards, multiplayer game state, or typing indicators — and you are happy with a flat data model.

Firestore for Offline-Capable Mobile Apps

Offline support is one of Firestore Native's headline features and a frequent PCD scenario. The Firestore SDKs for iOS, Android, and Web maintain a local cache of all documents the app has read or listened to. When the device loses connectivity, reads transparently return cached data and writes are queued in a mutation queue.

Optimistic Local Writes

When an app calls db.collection('orders').add({...}) offline, the SDK assigns the document a local ID, writes it to the cache, fires local listeners immediately so the UI updates, and queues the mutation. When connectivity returns, the SDK replays the mutation queue in order and resolves any conflicts using last-write-wins semantics.

Real-Time Listeners with onSnapshot

Listeners fire twice — once from cache, once from the server — letting the UI show stale data instantly and refresh when the server confirms. The metadata.fromCache flag tells the app whether a snapshot is local or authoritative.

Cache Size and Tabs

Default cache size is 100 MB on mobile and 40 MB on the Web. You can raise mobile cache to unlimited (CACHE_SIZE_UNLIMITED). Multi-tab persistence on the Web requires enableMultiTabIndexedDbPersistence.

Bigtable Row Key Design

Row-key design is the single most important decision you make in Bigtable, and the PCD exam tests it directly. Bigtable stores rows sorted lexicographically by row key, distributes contiguous key ranges across tablets, and assigns tablets to nodes. If your row keys cluster in time or value, one node serves all the writes and you create a hotspot.

Anti-Pattern: Monotonically Increasing Keys

Timestamps (1715616000000), auto-increment IDs, or sequence numbers as the leading component of the row key are the textbook anti-pattern. Every new write lands on the last tablet, which lives on a single node, so a 30-node cluster gives you single-node throughput.

Good Patterns

• Field promotion: lead with a high-cardinality identifier such as deviceId and append the reversed timestamp: device12345#20260513.
• Salting: prepend a hash-based prefix (hash(deviceId) % 100) to spread writes evenly. This breaks scans by time but eliminates hotspots.
• Reverse timestamps: store Long.MAX_VALUE - timestamp so the newest rows sort first when you do want time scans.
• Composite keys: combine a tenant ID with a domain identifier — tenant42#user88#20260513 — so multi-tenant scans stay efficient.

Keep Row Keys Short

Row keys are stored on every cell, so a 50-byte key on a billion rows is 50 GB of overhead. Aim for compact, fixed-width encodings.

Bigtable Column Families

A Bigtable row is divided into column families declared at table-creation time. Within a family you can have an unbounded, schema-less set of column qualifiers. Families exist for two operational reasons.

Storage and Compression Locality

Each column family is stored together on disk and shares a compression dictionary. Putting columns that are read together in the same family (for example, all telemetry samples in t:, all metadata in m:) keeps scans I/O-efficient.

Per-Family Garbage Collection

GC policies — max versions (maxversions=5), max age (maxage=30d), or intersection/union rules — are configured per column family. A typical pattern is one family raw: keeping seven days of high-resolution data and another family agg: keeping a year of aggregates, on the same row.

Limits

A table can have up to 100 column families, but staying under ~10 is best practice. Family names should be short — they are stored with every cell — and column qualifiers within a family can run into the thousands without issue.

Bigtable Tall vs Wide Tables

Bigtable schemas come in two shapes and PCD scenarios will hint at the right one.

Tall (Time-Series) Tables

In a tall table each event is a new row with the row key encoding entity plus timestamp: deviceId#20260513T120000. Reads scan a key range; writes are append-only. Cells per row are small. This is the right model for IoT telemetry, financial ticks, ad-impression logs, and most append-heavy workloads. Tall tables play nicely with garbage collection by age.

Wide (Entity) Tables

In a wide table each entity is one row and history is stored as columns or cell versions across that row. The row key is deviceId, and columns hold the latest N samples or per-attribute history. Wide rows are bounded — Bigtable enforces a 256 MB hard limit per row — so they only work when history per entity is small. Choose wide tables for user-profile-like data where you usually fetch one entity at a time and want a single read to return all its attributes.

Trade-Offs

Tall tables scan well across time ranges. Wide tables fetch one entity faster but degrade if any row exceeds the 256 MB ceiling. PCD scenarios describing "millions of events per second per device, kept for 90 days" almost always point to tall.

Bigtable Key Visualizer

Key Visualizer is Bigtable's diagnostic tool for hotspots. It renders a heatmap with row-key ranges on the Y-axis and time on the X-axis; brightness encodes operations per second (or other metrics like CPU per range).

Reading the Heatmap

A horizontal bright line means one row-key range receives a disproportionate share of traffic — the textbook signature of a monotonic key hotspot. Vertical bright stripes mean a moment of total cluster load and are usually fine. Diagonal patterns can indicate slow scans crawling across the keyspace.

Required Scale

Key Visualizer needs at least 30 GB of data and enough recent traffic before it generates a scan. It is free, runs automatically on production-sized tables, and is the primary tool the PCD exam expects you to name when a question asks "how do you find a Bigtable hotspot?".

Bigtable Replication

Bigtable replication adds one or more clusters to an instance. Each cluster lives in a single zone, holds a full copy of the data, and serves traffic independently. Replication is asynchronous and eventually consistent across clusters.

High Availability

A single-cluster instance has a 99.9% SLA. Adding a second cluster in the same region (multi-cluster routing, MULTI_CLUSTER_ROUTING_USE_ANY) raises the SLA to 99.99%. Adding clusters in two regions raises availability further and survives a regional outage. Replication can also place data near readers — a US cluster plus an EU cluster lets EU users read at low latency.

App Profiles and Routing

App profiles decide how a client connects: single-cluster routing pins a workload to one cluster (useful for read-after-write or workload isolation between OLTP and batch), while multi-cluster routing sends each request to the nearest healthy cluster and automatically fails over.

Consistency Caveats

Because replication is eventually consistent, a write that lands on cluster A may take seconds (typically sub-second, but worst case longer under load) to appear on cluster B. PCD scenarios needing strong read-after-write across regions are not a Bigtable fit — choose Cloud Spanner instead.

Bigtable Autoscaling

Bigtable's compute units are nodes, and historically you provisioned them manually. Autoscaling (GA since 2022) lets a cluster grow and shrink automatically based on CPU utilization and storage utilization targets.

Configuration

You configure: --autoscaling-min-nodes, --autoscaling-max-nodes, --autoscaling-cpu-target (default 50%, range 10–80%), and --autoscaling-storage-target (typically 2560 GB per SSD node, 8192 GB per HDD node). Autoscaling reacts in roughly one-minute intervals but Bigtable will not scale down faster than one node every 10 minutes to protect tablet redistribution.

gcloud bigtable clusters update my-cluster \
  --instance=my-instance \
  --autoscaling-min-nodes=3 \
  --autoscaling-max-nodes=30 \
  --autoscaling-cpu-target=60

When Manual Beats Autoscaling

Workloads with sharp, predictable spikes (Black Friday, market open) sometimes benefit from a scheduled scale-up via Cloud Scheduler + Cloud Functions, because autoscaling lags the spike by a minute. For steady or organically growing workloads, Autoscaling with a 50–70% CPU target is the right default.

白話文解釋（Plain English Explanation）

Analogy 1: The Filing Cabinet vs. The Massive Spreadsheet

Firestore is like a smart filing cabinet. Each drawer (collection) contains folders (documents), each folder can hold a different mix of papers (fields), and a librarian (Security Rules) checks every visitor's badge before letting them touch a folder. Bigtable is like a warehouse full of infinitely long spreadsheets where every row has the same shape: one giant key on the left and as many sparse cells as you want on the right. The forklift driver (the tablet server) can only carry one shelf at a time, so if everyone wants the newest row, one driver gets buried while the others stand idle — that is the hotspot you avoid with row-key design.

Analogy 2: A Swiss Army Knife vs. A Power Drill

Firestore is a Swiss Army knife: real-time sync, offline mode, transactions, queries, Security Rules — it does many things adequately and a few things very well, perfect for mobile apps. Bigtable is a 30-horsepower industrial drill: one job (massive sequential reads and writes at single-digit-millisecond latency), but it does that job at a scale nothing else on GCP can match. You do not bring a drill to fix a watch, and you do not bring a Swiss Army knife to drill through concrete.

Analogy 3: A Mall Directory vs. A Toll-Road Sensor Network

Firestore is the mall directory at the entrance: small map, well-organized, anyone can walk up, find what they need, and even use it on their phone offline. Bigtable is the highway sensor network: every car triggers a row of telemetry, billions per day, no human ever reads a single row in isolation — they scan time ranges to compute traffic patterns. If you tried to put highway sensor data in the mall directory it would topple over; if you tried to put the mall directory in the sensor network you would lose every nice feature you bought it for.

Frequently Asked Questions (FAQs)

Q1: Can I change a Firestore database from Datastore mode to Native mode later?

A1: No. Mode is fixed at database creation. To switch you must create a second Firestore database in the desired mode, export data from the source with gcloud firestore export to a Cloud Storage bucket, and import into the new database with gcloud firestore import. Pointers in application code must be updated to the new database ID.

Q2: What is the difference between nam5 and us-central1 for Firestore?

A2: us-central1 is a regional location — one region, 99.99% SLA, lower write latency, lower cost. nam5 is a multi-region location spanning several US regions with synchronous replication, 99.999% SLA, and survival of a full regional outage at the price of higher write latency and cost. Location, like mode, cannot be changed after creation.

Q3: How do I diagnose a Bigtable hotspot?

A3: Open Key Visualizer in the Bigtable console. A horizontal bright band on the row-key axis indicates one key range is taking disproportionate load — usually a monotonically increasing row key. Redesign the row key with field promotion (high-cardinality ID first), salting, or reverse timestamps. Key Visualizer requires roughly 30 GB of data to generate scans.

Q4: Does Firestore support offline reads and writes?

A4: Yes. The iOS, Android, and Web SDKs maintain a local cache (100 MB default on mobile, 40 MB on Web; configurable up to unlimited). Reads return cached data when offline; writes go into a mutation queue and replay when connectivity returns. Real-time listeners fire from cache first and again from the server when reachable.

Q5: How does Bigtable replication relate to Bigtable Autoscaling?

A5: They are independent. Replication adds clusters (each in a single zone) to an instance for higher availability (99.99% multi-cluster SLA) and lower-latency regional reads via app profiles. Autoscaling adjusts the node count within a single cluster based on a CPU target (default 50%) and storage target. You can — and usually should — combine both: each cluster in a replicated instance gets its own autoscaler.

Q6: Can Bigtable run multi-row transactions like Firestore?

A6: No. Bigtable supports single-row atomic mutations (including check-and-mutate and read-modify-write counters) but has no multi-row, multi-table transactions. If you need cross-row ACID on GCP, choose Cloud Spanner for relational data or Firestore Native (up to 500 documents per transaction) for documents.

Q7: When should I use Bigtable instead of BigQuery for analytics?

A7: Use Bigtable when latency must stay under 10 ms per lookup, when writes arrive at hundreds of thousands per second, and when access is by row key or key range — typical of operational dashboards, real-time personalization, and time-series serving layers. Use BigQuery when you need SQL, aggregations across the entire dataset, or ad-hoc analytics; BigQuery scans are charged per byte and not optimized for point lookups. Many architectures pair them: Bigtable serves the hot real-time tier and Dataflow exports daily snapshots to BigQuery for long-horizon analytics.