Cloud Router and BGP

Introduction

Cloud Router is a fully managed Google Cloud service that exchanges routes between your VPC and external networks (on-premises, branch offices, or other clouds) using Border Gateway Protocol (BGP), defined in RFC 4271. Unlike static routes that you maintain by hand in the VPC route table, Cloud Router learns remote prefixes dynamically over a TCP/179 BGP session and re-advertises VPC subnets back to the peer. When a new subnet is added, when a region drops out of service, or when an Interconnect attachment fails over, Cloud Router updates the routing table without operator intervention.

Cloud Router lives in the control plane only. The actual packet forwarding happens on Google's Andromeda data plane, so a Cloud Router process failure does not by itself black-hole traffic — installed routes keep working until the BGP hold timer expires or until the peer withdraws them.

白話文解釋（Plain English Explanation）

Before unpacking ASNs, MED, and BFD, three analogies anchor what Cloud Router is actually doing on top of the underlying Google network.

Analogy 1 — A GPS Navigation System (Static vs Dynamic Routing)

Static routing is a printed paper map. You draw the freeway route from Taipei to Kaohsiung once and never touch it again. If the bridge over the Zhuoshui River collapses, the map still says "go straight" — your packets drive into the river. Cloud Router with BGP is Google Maps with live traffic. The map service talks to thousands of other cars (other BGP speakers) every second, learns that the bridge is out, and quietly reroutes you over Highway 3. When the bridge reopens, Maps notices the freeway is fast again and switches back. The gcloud compute routers update-bgp-peer command does not move packets; it tells the navigation service "please prefer this route" — the actual driving (forwarding) is done by Andromeda.

Analogy 2 — Two Customs Officers at a Land Border (BGP Peering Session)

A BGP session is two customs officers, one on each side of a land border, who shout paperwork at each other across the fence. One officer wears a Google Cloud uniform (Cloud Router on a cloud-router-ip like 169.254.0.1); the other wears your on-premises router's uniform (169.254.0.2). They first agree to talk only if they have different employee IDs (different ASNs) and the secret password matches (MD5 authentication). Every 20 seconds they pass a "still here" note (keepalive); if no note arrives for 60 seconds, they assume the other office has been bombed and stop accepting that customs paperwork (the hold timer expires, routes withdrawn). BFD is an extra walkie-talkie that beeps every 300 ms — much faster than the customs schedule — so if the line cuts, both officers know within a second instead of a minute.

Analogy 3 — A Bus Terminal With Multiple Loading Bays (Route Priority and MED)

Imagine a long-distance bus terminal where buses (packets) leave for Tainan from any of four bays (BGP sessions across two Cloud Routers and two on-prem peers). The terminal dispatcher has rules: lower BGP route priority (set with --advertised-route-priority=100) wins, so bay 100 is the primary, bay 200 is the backup. From the other direction — buses coming into the terminal from on-prem — the on-prem router uses MED (Multi-Exit Discriminator) to whisper "prefer the bay marked MED=100 over MED=200." If two bays have the same priority and MED, the dispatcher splits passengers between them (ECMP, up to 4 paths and 16 with custom learned route limits). AS Path prepending is a passive-aggressive way to make a bay look further away: the on-prem router writes its employee ID three times in a row on the paperwork ("65001 65001 65001") so Cloud Router thinks the route went through three offices and avoids it.

Cloud Router Architecture and the Control-Plane Boundary

Cloud Router runs as a regional managed resource. You create one with gcloud compute routers create my-router --region=asia-east1 --network=vpc-prod --asn=64512. Once it exists, you attach BGP peers — one peer per VPN tunnel or per VLAN attachment. The router itself is highly available across the zones of its region; Google does not expose the underlying replica count, but the service SLA assumes that no single zone failure takes a router offline.

Control Plane vs Data Plane

Dynamic Routing Mode (VPC-Level Setting)

Cloud Router behavior is governed by the VPC's dynamic routing mode, set with gcloud compute networks update vpc-prod --bgp-routing-mode=GLOBAL:

• REGIONAL (default): A Cloud Router only advertises subnets that live in the same region as the router, and only learned routes from this router are usable by VMs in that region.
• GLOBAL: A Cloud Router advertises subnets from every region in the VPC, and learned prefixes are installed in every region. This is the recommended setting for any multi-region production deployment using HA VPN or partner Interconnect.

Switching from regional to global is non-disruptive but causes a re-advertisement burst — schedule it in a maintenance window if you peer with sensitive carriers.

Regional Footprint and HA Pattern

Each region typically gets at least one Cloud Router for HA VPN (which itself uses two interfaces per gateway) and a second Cloud Router for any redundant Interconnect attachment. The standard HA pattern is: two HA VPN tunnels terminating on two interfaces of one Cloud Router, peering with two on-prem routers on different power and cooling domains. This yields the 99.99% SLA when both tunnels are up and both BGP sessions are established with non-overlapping prefixes.

ASN Selection: Private Ranges and 4-Byte ASNs

Every BGP session needs two distinct ASNs. Cloud Router supports both 2-byte ASNs (RFC 4893 private range 64512-65534) and 4-byte ASNs (4200000000-4294967294, RFC 6996). Public ASNs (anything not in those private ranges and not in the reserved blocks like 23456) are accepted by the API but should only be used if you actually own the ASN registered through a Regional Internet Registry such as ARIN or APNIC.

Choosing a 2-Byte vs 4-Byte ASN

• Pick a 2-byte private ASN when your on-prem network is small and you have no existing assignment. 64512 and 65000 are common starting points.
• Pick a 4-byte private ASN when you are running an enterprise WAN that already uses 64512-65534 for branches — you need a separate, non-conflicting range for cloud. 4200000000 is the typical anchor.
• Never reuse the same ASN on both sides. BGP requires distinct ASNs for an eBGP session, and Cloud Router rejects the peering attempt if peer-asn matches the router's own --asn.

Reserved ASN Numbers to Avoid

• 0 and 65535 are reserved by RFC.
• 23456 is the transition ASN for 4-byte advertisement to 2-byte speakers — never use it directly.
• 65534 was historically the default for some Google Cloud documentation examples; pick something else to avoid ambiguity in shared environments.

MD5 Authentication and BGP Security

Cloud Router supports RFC 2385 MD5 authentication on every BGP session. While MD5 is cryptographically weak by modern standards, it remains the BGP industry standard and meaningfully raises the bar against off-path TCP injection on Interconnect attachments where the link is shared with other tenants of the carrier.

Enabling MD5 on a Peer

gcloud compute routers add-bgp-peer my-router \
  --peer-name=peer-onprem-east-1 \
  --interface=interface-0 \
  --peer-ip-address=169.254.0.2 \
  --peer-asn=65001 \
  --md5-authentication-key="redacted-shared-secret-32-chars"

The key is stored in Google's encrypted metadata store and is never retrievable in cleartext via the API after creation. Rotation requires gcloud compute routers update-bgp-peer with a new key; the BGP session flaps briefly during the change.

Key Length and Rotation

Use a 20+ character random key — Google does not enforce a minimum but most security baselines require at least 16. Rotate keys whenever a network engineer with access leaves the team; treat the BGP MD5 key as a shared secret on par with a service account key.

Keepalive and Hold Timers

The two timers that govern BGP session liveness are negotiated when the session establishes — each side proposes a value, and the lower of the two is used.

Default Values and Tuning

• Keepalive: default 20 seconds on Cloud Router. Range accepted by the API is 20-60.
• Hold timer: default 60 seconds (3x the keepalive, per RFC 4271 recommendation). Range is 6-60 in 4-byte ASN sessions but Cloud Router enforces a minimum hold of 20.

Set with:

gcloud compute routers update-bgp-peer my-router \
  --peer-name=peer-onprem-east-1 \
  --advertised-route-priority=100 \
  --keepalive-interval=20

Note that Cloud Router does not expose a separate --hold-time flag; the hold time is derived as 3x keepalive, capped at 60. To get sub-second failure detection, use BFD, not aggressive BGP timers.

Custom Route Advertisement

By default, Cloud Router advertises all VPC subnet ranges in the relevant scope (regional or global, depending on dynamic routing mode). For most enterprise deployments this is too much — you want to advertise a summary range, hide internal-only subnets, or announce static external routes that are not part of any subnet (such as Google-managed services accessed via Private Service Connect).

Default vs Custom Mode

• Default mode: advertises all subnets visible to the router.
• Custom mode: advertises only what you list, optionally plus the default subnet set.

Switch with --advertisement-mode=CUSTOM:

gcloud compute routers update-bgp-peer my-router \
  --peer-name=peer-onprem-east-1 \
  --advertisement-mode=CUSTOM \
  --set-advertisement-ranges=10.100.0.0/16,10.101.0.0/16 \
  --set-advertisement-groups=ALL_SUBNETS

Advertising Non-VPC Prefixes

Custom advertisements can include prefixes that are not VPC subnets at all — for example, the 199.36.153.4/30 range for private.googleapis.com access from on-prem, or a 35.199.192.0/19 range for IAP TCP forwarding. This is how on-premises hosts learn to send Google API traffic through the HA VPN tunnel rather than the public internet.

Summarization and Filtering

Cloud Router does not perform automatic summarization. If you advertise 10.0.0.0/16 and 10.1.0.0/16 separately, the peer sees two routes. Summarize manually by advertising the supernet 10.0.0.0/15 and configuring your on-prem router to accept the supernet only.

Route Priority and MED

BGP route selection on Cloud Router follows the standard BGP best-path algorithm with two key Google-specific behaviors.

Inbound: Advertised Route Priority

The --advertised-route-priority flag on a Cloud Router peer is sent to the on-prem peer as the BGP MED attribute. Lower values win. Use this to make one tunnel primary and another backup:

# Primary tunnel
gcloud compute routers update-bgp-peer my-router \
  --peer-name=peer-onprem-tunnel-a --advertised-route-priority=100

# Backup tunnel
gcloud compute routers update-bgp-peer my-router \
  --peer-name=peer-onprem-tunnel-b --advertised-route-priority=200

The on-prem router now prefers tunnel A for return traffic. If A fails, BGP withdraws and B (priority 200) takes over.

Outbound: Inbound MED from the Peer

Symmetrically, when the on-prem router advertises its prefixes to Cloud Router, it sets MED on each route. Cloud Router honors MED and installs the lower-MED route as the active forwarding entry. If the on-prem side sends equal MED, Cloud Router falls back to AS Path length, then to router ID as the tiebreaker.

ECMP When Tied

When two paths have equal priority/MED, equal AS path length, and equal origin code, Cloud Router programs ECMP (Equal-Cost Multi-Path) with up to 4 paths by default. The hash is 5-tuple (src-ip, dst-ip, src-port, dst-port, protocol), so flows are sticky to a single tunnel but different flows spread across tunnels.

BFD (Bidirectional Forwarding Detection)

BGP timers detect failures in tens of seconds. For latency-sensitive applications — voice, real-time trading, synchronous database replication — that is unacceptable. BFD (RFC 5880) runs alongside BGP and detects path failure in sub-second time by sending lightweight UDP probes on port 3784.

Cloud Router BFD Parameters

• Mode: ACTIVE (Cloud Router initiates) or PASSIVE (waits for peer to initiate).
• min-transmit-interval: how often Cloud Router sends BFD packets. Range 1000-30000 microseconds (1ms-30s), default depends on attachment type.
• min-receive-interval: minimum interval Cloud Router will accept from the peer. Same range.
• multiplier: number of consecutive missed packets before declaring the session down. Range 5-10, default 5.

With transmit=300000us (300ms) and multiplier=5, failure detection is roughly 1.5 seconds — about 40x faster than the default BGP hold timer.

gcloud compute routers update-bgp-peer my-router \
  --peer-name=peer-onprem-east-1 \
  --bfd-session-initialization-mode=ACTIVE \
  --bfd-min-transmit-interval=300 \
  --bfd-min-receive-interval=300 \
  --bfd-multiplier=5

When BFD Is Worth the Effort

• Dedicated Interconnect: high-value, latency-sensitive — BFD is recommended.
• HA VPN over high-quality internet: BFD helps but is less critical because the underlying IPsec tunnel itself has heartbeats.
• Partner Interconnect: depends on the partner; some carriers do not support BFD end-to-end.

Cloud Router for HA VPN, Interconnect, and Inter-VPC

Cloud Router is the BGP termination point for every dynamic-routing attachment Google offers. The peer type changes, but the router itself is the same managed service.

HA VPN

HA VPN gateways have two interfaces (0 and 1), each terminating one tunnel. Both tunnels of a single gateway must peer with the same Cloud Router to get the 99.99% SLA. For 99.99%-plus geographic redundancy, deploy a second HA VPN gateway in a second region with its own Cloud Router and use BGP route priority to control which region is primary.

Dedicated and Partner Interconnect

Each VLAN attachment binds to exactly one Cloud Router. Cloud Router runs the BGP session over the layer-2 attachment using link-local addresses Google allocates from 169.254.0.0/16. You cannot specify the BGP IP yourself on an attachment — Google picks two adjacent /29 addresses.

Cross-Cloud Interconnect

Cloud Router peers with the remote cloud's BGP speaker (AWS Direct Connect VIF, Azure ExpressRoute Microsoft peering, OCI FastConnect). ASN selection becomes critical here because each cloud reserves certain private ASN ranges for its own use — pick a 4-byte ASN well outside any cloud-vendor reserved range.

Cloud Router and VPC Peering — They Do Not Mix

Cloud Router does not advertise routes over VPC Peering. VPC Peering exchanges subnet routes via the Google control plane directly; BGP plays no role. If you need transitive routing between three VPCs, use Network Connectivity Center, not chained peering.

Network Connectivity Center (NCC) Integration

Network Connectivity Center (NCC) introduces a hub-and-spoke abstraction for Google network connectivity. Cloud Router participates as a spoke when attached to HA VPN, Interconnect, or Router Appliance spokes.

Hub, Spokes, and Route Exchange

A hub is a regional or global container that re-advertises routes between its spokes. Each spoke is one of:

• HA VPN spoke: a pair of HA VPN tunnels.
• VLAN attachment spoke: an Interconnect VLAN.
• Router appliance spoke: a third-party NVA (e.g., a Palo Alto or Cisco VM) configured as a BGP peer.
• VPC spoke: a VPC network (currently limited to specific NCC modes).

Cloud Router is the BGP speaker on each non-VPC spoke. When you add a tunnel to a hub, the hub automatically learns the prefixes Cloud Router advertised and re-distributes them to every other spoke — turning Google's network into a transitive WAN backbone without you running iBGP yourself.

Use Cases

• Branch-to-branch over Google: each branch terminates HA VPN to Google; NCC stitches them together.
• Inserting a virtual firewall: a router-appliance spoke runs the NVA, NCC routes east-west traffic through it.
• Replacing MPLS: NCC plus Premium Tier delivers most MPLS-class behaviors over Google's backbone.

Route Propagation and BGP Dampening

How Routes Propagate Through a VPC

A learned BGP route on Cloud Router becomes a dynamic route in the VPC route table, visible with gcloud compute routes list --filter="nextHop:projects/.../routers/my-router". Dynamic routes have higher priority than the implicit default but lower than custom static routes (priority 1000 by default). They propagate to every VM in the relevant region (regional mode) or every region (global mode).

Route Propagation Latency

After a BGP UPDATE is received, Cloud Router typically programs the route into the data plane within seconds. There is no public SLA on propagation time, but field experience suggests 5-30 seconds end-to-end for a learned route to become forwarding-active across a global VPC.

BGP Dampening

If your on-prem peer implements dampening and is suppressing your VPC routes, the symptom is "all VMs lose connectivity to on-prem for 15 minutes after every BGP flap." Disable dampening on the on-prem side for cloud peers, or whitelist Cloud Router's advertised prefixes from dampening.

Monitoring, Logging, and Troubleshooting

Cloud Logging

Enable BGP session state logging via gcloud compute routers update my-router --set-advertisement-mode=... (logging is controlled per-router via the API field bgpPeers[].enable). BGP UPDATE messages flow into Cloud Logging under the cloud.googleapis.com/router_logs resource type.

Cloud Monitoring Metrics

Key metrics:

• router.googleapis.com/bgp/session_up: 1 if established, 0 if down.
• router.googleapis.com/bgp/sent_routes_count and received_routes_count: alert if these drop to zero unexpectedly.
• router.googleapis.com/bfd/session_up: separate from BGP — both should be 1.

Set an alert policy that fires when session_up is 0 for more than 60 seconds; ignore shorter blips that correspond to normal IPsec rekey events.

Connectivity Tests

Use Connectivity Tests (formerly Network Intelligence Center) to verify that a specific source/destination pair would actually reach via the expected Cloud Router. The test simulates forwarding without sending real packets, listing every routing hop including which Cloud Router programmed which route.

Exam Traps, Tips, and Best Practices

• Trap: Cloud Router is regional. If your dynamic routing mode is REGIONAL and you put the router in us-central1, VMs in europe-west1 will not see learned routes — switch to GLOBAL for any cross-region production.
• Trap: You cannot run BGP over VPC Peering. The exam will offer this as a distractor. Use NCC instead.
• Tip: For the 99.99% HA VPN SLA, both tunnels must terminate on the same Cloud Router with BGP established on both. Two routers gives 99.99% only if you also have two gateways.
• Tip: Use gcloud compute routers get-status my-router --region=... to see live BGP peer state and learned/advertised prefix counts — this is the fastest troubleshooting command.
• Best practice: Document every ASN in a shared registry. ASN collisions in M&A scenarios are the single most common cause of multi-day BGP outages.

FAQs

Q: What happens if I have two BGP paths with the same MED and AS-path length?

A: Cloud Router performs ECMP load balancing across up to 4 paths by default, using a 5-tuple hash so each flow is sticky to one path. You can request up to 16 ECMP paths via a quota increase.

Q: Can I run BGP over VPC Peering?

A: No. VPC Peering exchanges routes via the Google internal control plane, not BGP. For transitive routing between VPCs, use Network Connectivity Center, which lets Cloud Router participate as a spoke and stitch VPCs together.

Q: How do I change the Cloud Router ASN after it has been created?

A: You cannot. The ASN is set at router creation and is immutable. To change it, create a new Cloud Router with the desired ASN, migrate BGP peers one at a time (during a maintenance window because each migration flaps a session), then delete the old router.

Q: Does Cloud Router charge for itself?

A: The Cloud Router managed service is free. You pay for the underlying attachment — HA VPN tunnel-hours, Interconnect port and attachment fees, and egress data. There is no per-route, per-prefix, or per-BGP-session fee.

Q: Is there an SLA on Cloud Router itself?

A: Cloud Router availability is bundled into the SLA of the attachment it serves: 99.99% for HA VPN with both tunnels up, 99.9% or 99.99% for Dedicated Interconnect depending on topology, and 99.9% for Partner Interconnect. There is no separate Cloud Router SLA.

Q: Can Cloud Router peer with more than one on-prem router on the same tunnel?

A: No — one BGP session per tunnel or per VLAN attachment, one peer per session. For multiple on-prem peers, create multiple tunnels or multiple VLAN attachments, each with its own Cloud Router BGP peer entry.

Q: How fast is BFD failure detection in practice?

A: With min-transmit-interval=300ms and multiplier=5, failure detection is roughly 1.5 seconds. Aggressive settings (min-transmit-interval=100ms, multiplier=3) can push that to 300ms but risk false positives on noisy links.