Vertex AI Feature Store

Introduction to Vertex AI Feature Store

Vertex AI Feature Store is the managed feature platform on Google Cloud that turns raw tables into reusable model inputs. Teams use Vertex AI Feature Store to share features between training and inference, keep them fresh, and serve them with single-digit millisecond latency. PDE candidates need to know both the legacy product and the newer BigQuery-backed Vertex AI Feature Store, because exam questions still reference both designs.

白話文解釋（Plain English Explanation）

The library card catalog analogy

Picture a public library with two million books. Without a catalog, you cannot find anything. With a catalog, any librarian can locate a book in seconds, and every branch sees the same record. Vertex AI Feature Store is that catalog for machine learning data. Your raw transactions live in BigQuery the same way books live on shelves. The Feature Store records what each column means, who owns it, when it was last updated, and where to fetch the freshest copy. When a fraud model and a recommendation model both need customer_lifetime_value, neither team has to recompute it from scratch. They look it up in the catalog, and Vertex AI Feature Store hands them the same value with the same definition.

The catalog also stops chaos. Imagine if every patron rewrote the catalog card for every book they touched. Within a week the library would be unusable. Feature stores enforce a single source of truth so that the value cart_size_30d always means the same thing whether the data scientist trains on Friday or the inference server reads it on Sunday morning.

The restaurant prep station analogy

A busy kitchen never chops onions when an order arrives. Onions, garlic, and stock are prepped during slow hours and held in labeled containers near the line. When a ticket comes in, the cook grabs ingredients and plates the dish in two minutes. If the prep station were empty, every order would take twenty minutes and the restaurant would die during the dinner rush.

Online serving in Vertex AI Feature Store is the prep station. A nightly Dataflow job calculates user_30d_spend and writes it into the online store. When a checkout request arrives at 8pm, the recommendation model fetches that prepped value in under 25 milliseconds and returns a personalized list. Without the prep station, the model would scan a year of transactions for every page view, which is both slow and expensive. With the prep station, the kitchen serves a thousand covers a night without breaking a sweat.

The supermarket warehouse and storefront analogy

Supermarkets run on two locations. The warehouse holds pallets of every product the chain sells, organized by SKU, batch, and expiry date. The storefront holds a small selection of fast-moving items at eye level. Trucks restock the storefront from the warehouse on a schedule.

Vertex AI Feature Store has the same two-tier shape. BigQuery is the warehouse: cheap, vast, and fine for analytics queries that scan billions of rows. The online store (now Bigtable under the hood for the new product, Cloud Bigtable or optimized in-memory storage for legacy) is the storefront: small, expensive per gigabyte, and built for sub-25ms reads. A sync job moves features from BigQuery into the online layer on a schedule. Inference traffic only ever hits the storefront. Training jobs read straight from the warehouse with point-in-time joins. Once you internalize this two-tier model, the rest of the Vertex AI Feature Store API stops feeling abstract.

Core Concepts of Vertex AI Feature Store

Entities, features, and feature values

An entity is the thing you describe, such as a user, a product, or a credit card. A feature is a measurable attribute of that entity, such as total_purchases_7d. A feature value is the actual number stored at a specific timestamp. Vertex AI Feature Store keeps every value with its event time so that historical training queries can reconstruct exactly what the model would have seen.

Legacy Feature Store v1

The original Vertex AI Feature Store launched in 2021 as a fully managed service with its own storage layer. You created a Featurestore resource, then EntityType resources inside it, then Feature resources inside each entity type. Online and offline storage were both managed by Google, hidden behind the API. You ingested with ImportFeatureValues calls or streaming WriteFeatureValues requests. Most of the data on the public internet still describes this design, which is why exam writers occasionally ask about it.

The new BigQuery-backed Feature Store

In 2024 Google released a redesigned Vertex AI Feature Store. The new product treats BigQuery as the source of truth for offline data and uses Bigtable (or a managed online store) for low-latency reads. You define a FeatureGroup that points at a BigQuery table or view, then create Feature resources that reference columns. A FeatureOnlineStore plus FeatureView handles serving. The redesign cut storage costs and let teams use the same data in BigQuery ML, dashboards, and model training without copying it.

Feature groups and feature views

A feature group is a logical bundle of related features sourced from one BigQuery table. A feature view is the served projection: it joins feature groups, applies a schedule, and lands the result in the online store. Think of feature groups as cookbooks and feature views as the daily menu printed for the line cooks.

Point-in-time correctness

When you train a model, every label has a timestamp. The features used for training must reflect the world as it existed at that timestamp, not as it exists today. Vertex AI Feature Store enforces this with point-in-time joins that look up the most recent feature value strictly before the label timestamp. This single rule prevents the most expensive bug in machine learning: leaking future information into training data.

Online versus offline serving

Offline serving feeds training, batch scoring, and analytics. It runs against BigQuery and tolerates seconds or minutes of latency. Online serving feeds production inference. It runs against the online store and must respond in tens of milliseconds. The split exists because no single storage system is cheap enough for petabyte-scale history and fast enough for real-time lookups at the same time.

Architecture and Design Patterns

The hub-and-spoke pipeline

A common architecture places BigQuery in the center as the hub. Streaming sources land in Bigtable or Pub/Sub, then Dataflow aggregates them into BigQuery tables on a five-minute schedule. Batch sources land directly in BigQuery via Storage Transfer Service or scheduled queries. Vertex AI Feature Store registers feature groups against those tables. Feature views sync into the online store every fifteen minutes. Training jobs read from BigQuery with point-in-time joins. Inference services read from the online store via the Feature Store SDK.

Lambda-style fresh feature pattern

Some features must be near real-time, such as seconds_since_last_login. The lambda pattern keeps two copies. A streaming Dataflow job writes the freshest value directly to the online store every few seconds. A nightly batch job overwrites the same feature in BigQuery with the canonical aggregation. The streaming path keeps inference accurate; the batch path keeps training reproducible. Vertex AI Feature Store supports both writes through its ingestion APIs.

The shared-feature catalog pattern

Larger organizations pool features across teams. A central platform team owns the BigQuery dataset that backs the feature groups. Product teams contribute features by adding columns and submitting a small YAML config that registers them with Vertex AI Feature Store. IAM controls who can read which feature groups. This pattern eliminates the situation where five teams each compute customer_age_days slightly differently.

Embedding store pattern

The new Vertex AI Feature Store also serves embeddings. You can register a FLOAT[] column as an embedding feature and enable nearest-neighbor search on the feature view. This collapses what used to be three separate systems (BigQuery, Feature Store, Vector Search) into one query path for retrieval-augmented generation use cases.

GCP Service Deep Dive

How the legacy product stored data

Legacy Vertex AI Feature Store provisioned online_serving_nodes per Featurestore. Each node added throughput and online storage capacity at a fixed monthly cost. Offline storage was billed per GB-month. Ingestion ran via ImportFeatureValues for batch and WriteFeatureValues for streaming. The product is still supported but Google steers new workloads to the BigQuery-backed design.

How the new product stores data

The new Vertex AI Feature Store keeps offline data in your own BigQuery dataset. You pay BigQuery storage and query rates for everything offline. The online layer comes in two flavors. The optimized online store uses dedicated nodes for sub-25ms reads. The Bigtable online store uses an existing or autoprovisioned Bigtable instance for higher throughput. Sync from BigQuery to the online store is configured per feature view with a cron-style schedule.

Ingestion patterns

Batch ingestion is the simplest path. A scheduled query or a Dataflow job writes new rows to the BigQuery table that backs the feature group. The next sync picks them up automatically. Streaming ingestion uses Pub/Sub plus a Dataflow streaming job that writes either to BigQuery (with the streaming buffer or the Storage Write API) or directly to the online store via the Feature Store API. The Storage Write API gives you exactly-once semantics and cheap throughput, so it is the modern default for streaming into BigQuery.

ML.PREDICT and BigQuery ML integration

Because the new Vertex AI Feature Store sits on top of BigQuery, you can run ML.PREDICT directly against feature group tables. A SQL analyst can write SELECT * FROM ML.PREDICT(MODEL my_dataset.churn_model, TABLE my_dataset.feature_group_users) without exporting data. This integration matters for the PDE exam because it proves you understand that Vertex AI Feature Store is no longer a black box: it is a layer on top of standard BigQuery tables.

Feature monitoring

Vertex AI Feature Store ships built-in monitoring that tracks feature value distributions over time. You enable monitoring per entity type or feature view and set drift and skew thresholds. The service samples values, computes summary statistics, and writes them to a feature_monitoring table. Cloud Monitoring alerts fire when distributions shift beyond your threshold. This is how you catch a broken upstream pipeline before the model starts predicting garbage.

IAM and security

Three roles cover most use cases. roles/aiplatform.featurestoreUser lets a service account read feature values for inference. roles/aiplatform.featurestoreDataViewer grants read-only access for analysts. roles/aiplatform.featurestoreAdmin covers full management. Resource-level IAM lets you scope access down to a specific feature group, which matters when one team owns sensitive PII features and another team owns public catalog features inside the same project. CMEK is supported on both online and offline storage; configure it at the Featurestore or FeatureOnlineStore level when you need full control over encryption keys.

Common Pitfalls and Trade-offs

Treating the online store as a database

The online store is not a transactional database. It serves the latest value per entity per feature, full stop. You cannot run analytical queries against it, you cannot join it to other tables, and you cannot use it as a write-through cache for arbitrary workloads. Teams that try to repurpose the online store as a generic key-value store usually blow their budget within a month.

Forgetting about sync lag

A feature view with a fifteen-minute sync schedule means inference reads values that are at most fifteen minutes old. For most models that is fine. For real-time fraud or pricing it is a disaster. Decide upfront whether you need streaming writes to the online store or whether scheduled syncs are good enough, because retrofitting streaming after the fact requires rewriting the ingestion pipeline.

Underprovisioning online serving nodes

The legacy product used fixed nodes. Underprovisioning caused 503 errors during traffic spikes. The new product still has node-bound serving for the optimized online store, and the Bigtable variant inherits Bigtable's hot-key behavior. Load test before launch and set autoscaling alerts. The exam loves to ask which symptom (latency spikes versus error rate spikes) maps to which fix.

Mixing event time and processing time

Every feature value carries an event timestamp. The biggest source of training-serving skew is using CURRENT_TIMESTAMP() instead of the actual event time when ingesting. The bug is invisible until production, when the model performs worse than offline metrics suggested. Always carry the source event timestamp through the pipeline and write it as the feature timestamp.

Ignoring backfill semantics

When you add a new feature, the BigQuery table needs historical values for training to be useful. Backfilling is a manual SQL job, not something the Feature Store does for you. Plan the backfill query before you announce the feature to model teams.

Best Practices

• Use the new BigQuery-backed Vertex AI Feature Store for all greenfield work; only touch the legacy product if you are maintaining an existing pipeline.
• Source every feature group from a versioned BigQuery view, not a raw table, so that schema changes can be reviewed before they reach the online store.
• Carry the original event timestamp end to end and write it as the feature timestamp.
• Schedule feature view syncs based on actual freshness requirements, not a default cron.
• Enable feature monitoring on every production feature view; bake the alert threshold into the deployment review checklist.
• Use service accounts per environment and grant featurestoreUser at the feature group resource, not at the project.
• Document each feature with owner, refresh cadence, source query, and downstream consumers in the feature group description.
• Validate point-in-time joins against a known holdout before promoting a model to production.

Real-World Use Case

A mid-sized fintech in Singapore runs a credit underwriting platform that serves 800,000 applications a day across four product lines. Before adopting Vertex AI Feature Store, each product team maintained its own feature pipeline. The personal-loan team computed monthly_income_avg_6m one way; the credit-card team computed the same metric with a slightly different lookback. Audit found that 18 percent of features had silent definitional drift between teams.

The platform team rebuilt the stack on the new BigQuery-backed Vertex AI Feature Store. They created one BigQuery dataset called risk_features and registered seven feature groups inside it: applicant_profile, transaction_history, device_signals, bureau_data, behavioral, historical_decisions, and external_market. Each feature group sourced from a versioned dbt model that the data engineering team owned.

For online serving they provisioned an optimized online store with autoscaling enabled. Eight feature views sync from BigQuery every five minutes. The fraud feature view syncs every thirty seconds because real-time freshness matters there. The serving service account holds featurestoreUser on the four feature groups it needs and nothing else.

Training pipelines use the offline serving API to build point-in-time-correct training tables for every model. The team eliminated nine custom ETL jobs in the migration. Feature monitoring caught a broken upstream feed within ninety minutes when a partner API started returning nulls; before, the same incident would have polluted the model for days.

Total cost dropped 34 percent compared to the prior stack of legacy Vertex AI Feature Store plus a Redis cache plus custom Dataflow ingestion. Latency p99 for online reads sits at 18 milliseconds, well inside their 50ms budget.

Exam Tips

The PDE exam asks about Vertex AI Feature Store from three angles. First, design questions: which service combination meets a freshness requirement plus a cost constraint. Recognize that BigQuery handles offline at low cost and Bigtable or the optimized store handles online at low latency. Second, operational questions: how to fix latency or skew problems. Map symptoms to causes (drift means monitoring; skew means timestamp bugs; latency spikes mean undersized nodes). Third, integration questions: how Vertex AI Feature Store fits with BigQuery ML, Vertex AI Pipelines, and Dataflow.

Memorize that the new product uses BigQuery as the offline store. Memorize that point-in-time joins are the cure for training-serving skew. Memorize that streaming writes are needed when sync schedules cannot keep features fresh enough. Many wrong answers on the exam look right until you check whether the freshness requirement is met by the proposed sync schedule.

Frequently Asked Questions (FAQ)

What is the difference between Vertex AI Feature Store v1 and the new BigQuery-backed version?

Legacy Vertex AI Feature Store managed both online and offline storage internally and exposed proprietary ingestion APIs. The new product uses your own BigQuery dataset for offline storage, your own Bigtable instance or an optimized online store for serving, and standard BigQuery SQL for ingestion. The new design integrates with BigQuery ML, supports embeddings natively, and usually costs less. Google recommends the new product for all greenfield work; the legacy product remains supported but is not getting major new features.

How does Vertex AI Feature Store prevent training-serving skew?

Skew happens when training data and serving data differ in either definition or freshness. Vertex AI Feature Store solves the definition problem by storing one canonical value per feature that both training and serving read. It solves the freshness problem with point-in-time joins for training and consistent sync semantics for serving. The remaining sources of skew, such as application-level data drift, require feature monitoring rather than the store itself.

Can I use Vertex AI Feature Store without Vertex AI training?

Yes. The store does not care which training framework you use. You can read feature values into TensorFlow, PyTorch, scikit-learn, XGBoost, or BigQuery ML. The new product also lets you call ML.PREDICT directly against feature group tables, so a model trained in BigQuery ML can score against the same features used elsewhere without exporting data.

How fresh can online features be?

Three latency tiers exist. Scheduled feature view sync gets you minute-level freshness, configurable down to one minute as the minimum. Streaming writes via the Feature Store SDK get you sub-second freshness for the values you push. Lambda patterns combine both, using streaming writes for hot features and scheduled sync for the rest. Pick the tier that matches your business requirement; do not pay for streaming when daily sync is enough.

What IAM role does my inference service need?

The minimum role is roles/aiplatform.featurestoreUser scoped to the feature groups your service reads. Avoid granting it at the project level, because that exposes every feature in every group to the service. If your service also needs to write feature values back, add roles/aiplatform.featurestoreDataWriter. For administrative tasks like creating or deleting feature groups, use roles/aiplatform.featurestoreAdmin and grant it only to platform engineers.

How do I monitor feature quality in production?

Enable feature monitoring on the feature view or entity type. The service samples values on a schedule and computes summary statistics including mean, standard deviation, and value distribution. You set thresholds for drift (distribution changes over time) and skew (training versus serving differences). Cloud Monitoring fires alerts when thresholds are exceeded. Wire those alerts into PagerDuty or Slack and add a runbook step that compares the failing feature against its source BigQuery table to find the upstream break.

Does Vertex AI Feature Store support embeddings for RAG use cases?

Yes. Register a FLOAT[] column as an embedding feature, enable nearest-neighbor search on the feature view, and the online store will serve approximate nearest-neighbor queries alongside regular feature lookups. This collapses Vector Search and feature serving into one product, which simplifies architectures for retrieval-augmented generation, recommendations, and semantic search.

Related Topics

• BigQuery Data Modeling and Clustering covers how to design the BigQuery tables that back feature groups.
• ML Pipelines with Vertex AI explains how Vertex AI Pipelines orchestrate training jobs that consume Feature Store data.
• Streaming Pipelines with Dataflow covers the streaming ingestion patterns that feed real-time features.

Further Reading

• Vertex AI Feature Store overview (BigQuery-backed)
• Vertex AI Feature Store (Legacy) overview
• Serve feature values online
• BigQuery online feature serving with Bigtable
• Vertex AI pricing — Feature Store section