Model Evaluation and Selection

Why Model Evaluation and Selection Matters

For the Google Cloud Generative AI Leader exam, model evaluation and selection is the discipline of choosing the right model for a business problem and then proving that it actually works. As a Generative AI Leader, you are not the engineer who fine-tunes weights or writes evaluation code. Your job is to make a defensible business decision: which model, at what cost, with what quality guarantee, and how do we know it is good enough to put in front of customers.

This topic sits at the intersection of technology and economics. A model that produces beautiful output but costs ten times the budget is the wrong model. A model that is cheap and fast but hallucinates on regulated content is also the wrong model. Model evaluation and selection is the structured way to navigate that tension. The exam tests whether you can reason about trade-offs rather than reach for the most powerful option by reflex.

The single most important mindset shift this topic teaches is that model selection is a recurring trade-off, not a one-time pick. The model that wins today loses next quarter when a cheaper, faster variant ships, when your traffic grows, or when a new use case appears. Treating model evaluation and selection as a living process — with golden sets, repeatable evaluation, and periodic review — is what separates a mature GenAI program from a fragile one. Throughout this study note we ground every concept in Model Garden and Vertex AI evaluation tooling so you can connect the business framing to concrete Google Cloud products.

白話文解釋（Plain English Explanation）

Model evaluation and selection can sound abstract, but it maps cleanly onto decisions people make every day in business and in life. The following three analogies each illustrate a different facet of the discipline, and each is grounded in how Model Garden and Vertex AI evaluation actually work.

Analogy 1 — Hiring Against a Job Description, Not Hiring the Most Famous Candidate

Imagine a Taiwanese company filling a role. A hiring manager who simply picks the candidate with the most prestigious resume — a PhD from a top university — for a job that only needs someone to answer phones and route calls will overpay and probably keep a bored employee who soon leaves. A good hiring manager starts with the job description: what tasks, what volume, what response time, what budget. Only then do they shortlist candidates and interview against those criteria. Selecting a model from Model Garden works exactly the same way. Model Garden is the catalog — it lists Google's first-party models like Gemini, partner models, and open models, each with a description of strengths, context window, modality support, and pricing. The "most famous candidate" is the largest, most capable model, and reaching for it by default is the classic mistake. For a task like classifying support tickets into five categories, a small fast model such as Gemini Flash or Flash-Lite will answer in a fraction of the time and cost of a frontier model, with quality that is indistinguishable for that narrow job. The Generative AI Leader writes the "job description" first — define the task, the volume, the latency target, the budget, and the quality bar — and then evaluates candidates from Model Garden against it. Picking a model without that description is like hiring a star surgeon to staff a convenience-store checkout.

Analogy 2 — Tasting Before You Stock the Shelves

A night-market food stall owner deciding whether to add a new sauce supplier does not sign a year-long contract on the strength of a glossy brochure. They request samples, taste them, and ideally let regular customers taste them too, comparing the new sauce against the one they already use. Only after that side-by-side tasting do they commit. This is precisely what evaluation means for generative AI. You never select a model purely on its marketing benchmark scores. Instead you build a golden set — a fixed collection of representative inputs paired with the answers you would consider good — and you run each candidate model against that golden set. Vertex AI's Gen AI evaluation service lets you do this systematically: it can score model outputs for qualities like helpfulness, groundedness, safety, and instruction-following, and it supports pairwise comparison, where two models are judged side by side on the same prompts. The "regular customers tasting" step maps onto human review, where domain experts read sampled outputs and rate them. The "automated taste test" maps onto LLM-as-judge, where a capable model scores outputs at scale against a rubric. Tasting before stocking the shelves is the whole point: it converts a vague feeling that "this model seems good" into evidence you can show an executive sponsor.

Analogy 3 — Choosing Transport by the Length of the Trip

When you need to travel in Taiwan, you do not always book the High Speed Rail. For a trip to the corner store you walk. For crossing town you take the MRT or a scooter. For Taipei to Kaohsiung the HSR is worth its premium price. You instinctively match the mode of transport to the distance and urgency of the trip, balancing cost, speed, and comfort. Model evaluation and selection runs on the same logic, and this is the cost-latency-quality triangle at the heart of the exam. A frontier model like a top-tier Gemini Pro variant is the HSR: highest quality reasoning, highest cost per token, often higher latency. A small model like Gemini Flash-Lite is walking or the scooter: cheap, fast, perfect for short, simple, high-volume trips. You cannot maximize all three corners of the triangle at once — pushing quality up usually pushes cost and latency up too. The skill is matching the model to the "trip": a customer-facing chat assistant that must respond in under a second favors a fast model; a once-a-day legal-contract summarization job can tolerate a slower, more expensive, higher-quality model because latency does not matter and accuracy does. The Generative AI Leader, like a sensible traveler, picks the cheapest option that comfortably meets the requirement — and re-evaluates when the route changes.

The Cost, Latency, and Quality Triangle

Every model selection decision is a negotiation between three forces, and the exam expects you to reason about all three explicitly.

Cost

Generative AI on Vertex AI is generally billed per input token and output token — and prices differ sharply between model sizes. A frontier model can cost an order of magnitude more per token than a small, fast model. Cost is not just the sticker price: at scale, a use case handling millions of requests a day turns a small per-token difference into a six-figure annual line item. The Generative AI Leader thinks in total cost at expected volume, not cost per call.

Latency

Latency is how long the user waits for a response. Larger models and longer outputs take longer. For an interactive chatbot, latency is part of the product — slow responses feel broken. For a nightly batch job, latency is irrelevant. Latency requirements should be written down before model selection begins, because they often eliminate the largest models immediately.

Quality

Quality is whether the output is correct, relevant, safe, grounded, and well-formatted for the task. Unlike cost and latency, quality cannot be read off a price sheet — it must be measured against your own data. This is why evaluation, covered later in this note, is inseparable from selection.

Model Size Versus Task Fit

A foundational lesson for model evaluation and selection is that a bigger model is not automatically a better choice. Model capability scales with size, but so do cost and latency, and many real business tasks do not need frontier-level reasoning.

When a Small Model Wins

For tasks that are narrow and well-defined — classifying text into a fixed set of categories, extracting a few fields from a document, routing a query, generating short standardized replies, simple summarization — a small fast model such as Gemini Flash or Flash-Lite often delivers quality that is indistinguishable from a frontier model, at a fraction of the cost and latency. At high request volumes, the savings are decisive.

When a Large Model Earns Its Price

For tasks that require multi-step reasoning, nuanced judgment, long-context synthesis across many documents, complex code generation, or handling ambiguous instructions, a frontier Gemini Pro variant earns its higher price. Forcing a small model onto these tasks produces shallow or incorrect results, and the cheap model becomes expensive once you count the cost of bad outputs.

Routing and Tiered Strategies

Mature GenAI programs often do not pick a single model. They route: easy requests go to a small model, hard ones escalate to a larger one. This treats model selection as a portfolio rather than a single choice, and it directly serves the cost-latency-quality triangle. The capability differences between variants are covered in depth in Gemini models and capabilities.

Why Evaluating Generative AI Is Different from Traditional ML

Traditional machine learning has a comforting property: a single number tells you how good the model is. A fraud-detection classifier has an accuracy, a precision, and a recall, computed against a labeled test set. Generative AI breaks this comfort.

There Is No Single Correct Answer

When a GenAI model summarizes a report or drafts an email, there is no one "right" output. Many summaries are good; many are acceptable; some are wrong. You cannot compute accuracy because there is nothing to exactly match against. Quality becomes a matter of degree across several dimensions — helpfulness, groundedness, coherence, safety, tone, format — rather than a single pass/fail score.

Quality Is Multi-Dimensional and Often Subjective

A response can be factually correct but rude, or fluent but subtly hallucinated, or accurate but in the wrong format. Evaluating GenAI means scoring several qualities at once and accepting that some of them require human judgment. This is why GenAI evaluation borrows techniques like rubrics and side-by-side comparison from fields like editorial review rather than from classic statistics.

Outputs Are Non-Deterministic

The same prompt can produce different wording on different runs. Evaluation must therefore look at distributions of behavior across many inputs, not a single sample. The connection between non-determinism, factual error, and trust is explored in hallucinations and model limitations.

How to Evaluate Generative AI Output

Evaluation is the evidence layer of model selection. Four complementary methods appear on the exam, and a strong program uses several together.

Human Review

Domain experts read a sample of model outputs and rate them against criteria — is it accurate, is it safe, is it on-brand. Human review is the gold standard for judgment-heavy tasks and is essential for regulated or customer-facing content. Its weakness is that it is slow and expensive, so it cannot cover every output at scale.

Golden Sets

A golden set is a fixed, curated collection of representative inputs paired with reference answers or quality expectations. Every candidate model — and every later version of your chosen model — is run against the same golden set, so comparisons are fair and repeatable. The golden set is the backbone of model evaluation and selection because it turns "this feels better" into a measurable, re-runnable test.

Automated Evaluation

Automated metrics score outputs at scale without a human reading each one. Some measure overlap with reference text; more useful for GenAI are model-based metrics that score qualities like groundedness or instruction-following. Vertex AI's Gen AI evaluation service provides exactly these automated, repeatable scores.

LLM-as-Judge

LLM-as-judge uses a capable model to grade other models' outputs against a written rubric — for example, "rate this answer 1 to 5 for groundedness in the supplied document." Conceptually, it scales human-style judgment to thousands of examples cheaply. It is not perfect — the judge model has biases of its own — so it complements rather than replaces human review and golden sets.

Pairwise Comparison

Rather than scoring one model in isolation, pairwise comparison shows a judge (human or model) two candidates' answers to the same prompt and asks which is better. People and models are often more reliable at comparing than at assigning absolute scores, which makes pairwise comparison a practical way to choose between two finalists.

Benchmarks and Their Limits

Public benchmarks are standardized tests that score models on shared tasks, and they are useful for a quick first shortlist when comparing candidates in Model Garden.

What Benchmarks Are Good For

Benchmarks give a rough, comparable signal of general capability — reasoning, coding, multilingual ability — across many models. They help you eliminate clearly unsuitable candidates before you invest in a real evaluation.

Why Benchmarks Are Not Enough

Benchmarks are measured on generic public tasks, not your business problem. A model that tops a reasoning leaderboard may still perform poorly on your specific support tickets, your industry jargon, or your document formats. Benchmarks can also be gamed, and they say nothing about cost or latency at your volume. The exam wants you to treat benchmarks as a starting filter, never the final decision — the final decision comes from evaluating shortlisted models against your own golden set.

Benchmark Contamination and Staleness

Benchmark data sometimes leaks into model training data, inflating scores. Benchmarks also age as models improve. This is another reason a private golden set, refreshed over time, is the trustworthy measure for your specific use case.

Choosing a Model from Model Garden

Model Garden is the model catalog inside Vertex AI. It is where model selection physically happens on Google Cloud, and the exam expects you to know what it contains and how to navigate it.

What Model Garden Contains

Model Garden lists three broad families: Google first-party models such as the Gemini family, Imagen for images, and speech and embedding models; partner models from other providers offered through Vertex AI; and open models that can be deployed and managed within Vertex AI. Each entry carries a model card describing capabilities, supported modalities, context window, and pricing.

How to Shortlist in Model Garden

The selection workflow follows the job-description analogy. First, write the requirement: task type, modality, context-length needs, latency target, budget, and data-residency or compliance constraints. Then filter Model Garden to candidates that satisfy the hard constraints. Then run the shortlist through evaluation against your golden set. The model that clears the quality bar at the lowest cost and acceptable latency wins.

First-Party Versus Partner Versus Open Models

First-party Gemini models offer the tightest integration, multimodality, and managed scaling. Partner and open models can matter for specific licensing, transparency, customization, or portability needs. The Generative AI Leader should know all three exist in Model Garden and that the choice among them is itself part of model evaluation and selection.

Which Gemini Variant Should You Pick

A frequent, practical decision is choosing among Gemini variants rather than between vendors. The pattern mirrors the transport analogy.

Flash and Flash-Lite for Speed and Volume

Lightweight variants like Gemini Flash and Flash-Lite are optimized for low latency and low cost. They are the right call for high-volume, latency-sensitive, relatively simple workloads — chat front ends, classification, extraction, routing, short summaries.

Pro for Complex Reasoning

A frontier Gemini Pro variant is built for harder reasoning, long-context synthesis, complex code, and nuanced instruction-following. It costs more and may respond slower, but for complex or high-stakes tasks it earns the premium.

The Decision Is Per Use Case

There is rarely one Gemini variant for an entire organization. Each use case gets its own evaluation, and many programs use a small variant as the default with escalation to a larger one for hard cases. The capability matrix behind these variants is detailed in Gemini models and capabilities.

Build Versus Buy: Using a Foundation Model or Training Your Own

A recurring leadership decision inside model evaluation and selection is build versus buy.

The "Buy" Path: Use a Managed Foundation Model

For almost all business use cases, the right answer is to buy — consume a managed foundation model like Gemini through Vertex AI. You get state-of-the-art capability immediately, no infrastructure to run, no training data to assemble, and Google maintains and improves the model. This is the default the exam expects you to recommend.

The "Build" Path: Train or Heavily Customize a Model

Training a foundation model from scratch is extraordinarily expensive and is essentially never the right answer for a business outside of dedicated AI labs. Lighter customization — fine-tuning a foundation model on your data, or grounding it on your documents with retrieval — is far more common and far cheaper, and it lives between pure buy and pure build.

How to Frame the Decision

Frame build-versus-buy as a cost, speed, and differentiation question. Buy gets you to value fastest at the lowest risk. Customize when prompting alone cannot reach the quality bar. Build-from-scratch is almost never justified. Tying model choice to measurable outcomes is covered in measuring GenAI business value.

Model Selection as a Recurring Trade-Off

The exam wants you to internalize that model selection is never finished. New, cheaper, faster, more capable model variants ship regularly. Your traffic grows, changing the cost math. New use cases appear with different latency and quality needs. Regulations evolve. A model chosen confidently last quarter can be the wrong model today.

A mature program therefore treats model evaluation and selection as an ongoing operating process: a maintained golden set, a repeatable evaluation pipeline in Vertex AI, periodic re-benchmarking of new Model Garden entries, and a willingness to switch when evidence justifies it. The Generative AI Leader's deliverable is not a single model pick — it is a repeatable selection process the organization can run again and again.

Governance, Risk, and Compliance in Model Selection

Model selection is not purely about quality and cost. Leaders must also weigh governance and risk.

Safety and Responsible AI

Evaluation must include safety scoring — does the model produce harmful, biased, or unsafe content. Vertex AI provides safety filters and the Gen AI evaluation service can score safety-related dimensions. A model that scores well on helpfulness but poorly on safety fails the bar for customer-facing use.

Data Residency and Compliance

Some industries and regions require data to stay within specific boundaries. These hard constraints filter the Model Garden candidate list before any quality evaluation, and they can override a model that would otherwise win on cost and quality.

Explainability and Documentation

Model cards in Model Garden document each model's intended use and limitations. For audited or regulated deployments, the leader must ensure the chosen model's documentation, evaluation results, and golden-set performance are all recorded so the decision is defensible later.

A Practical Model Evaluation and Selection Workflow

Putting the pieces together, the workflow a Generative AI Leader should be able to describe is:

Step 1 — Define the Requirement

Write the job description: task type, modality, context-length needs, latency target, volume, budget, quality bar, and compliance constraints.

Step 2 — Shortlist in Model Garden

Filter Model Garden to candidates that satisfy the hard constraints, optionally using public benchmarks as a rough first cut.

Step 3 — Build a Golden Set

Curate representative inputs and reference expectations from real business data.

Step 4 — Evaluate the Shortlist

Run candidates through the Vertex AI Gen AI evaluation service, combining automated metrics, LLM-as-judge scoring, pairwise comparison, and targeted human review.

Step 5 — Decide on the Triangle

Choose the model that clears the quality bar at the lowest cost and acceptable latency — not the most powerful one.

Step 6 — Monitor and Re-Evaluate

Keep the golden set current, re-test when new models ship or traffic changes, and switch when evidence justifies it.

Common Exam Scenarios and the Right Reasoning

The Generative AI Leader exam loves scenario questions. A few patterns recur:

• A team wants to deploy "the most powerful model" for a simple classification task at high volume — the right answer is to evaluate a small fast variant first against a golden set, because cost and latency matter and the task is simple.
• An executive asks "how do we know the chatbot is good?" — the right answer references golden sets, human review, and automated or LLM-as-judge evaluation, not a single accuracy number.
• A vendor presents impressive benchmark scores — the right answer is that benchmarks are a shortlist signal only, and the decision needs evaluation on the company's own data.
• A team proposes training a foundation model from scratch — the right answer is to buy a managed foundation model and consider fine-tuning or grounding instead.

Frequently Asked Questions

Q: Why can't I just measure a generative model's accuracy like a traditional ML model?

A: Because generative tasks have no single correct answer. When a model writes a summary or drafts a reply, many outputs are acceptable and there is nothing exact to match against, so an accuracy percentage is meaningless. Generative quality is multi-dimensional — groundedness, helpfulness, safety, tone, format — and outputs are non-deterministic, varying run to run. Evaluation therefore relies on golden sets, human review, automated model-based metrics, and LLM-as-judge scoring rather than one number.

Q: Should I always pick the largest, most capable model?

A: No. The largest model is the most expensive and often the slowest. For narrow, high-volume tasks like classification, extraction, routing, or short summaries, a small fast variant such as Gemini Flash or Flash-Lite delivers equivalent quality at a fraction of the cost and latency. Reserve a frontier Gemini Pro variant for genuinely complex reasoning or long-context work. Match the model to the task using the cost-latency-quality triangle.

Q: What is a golden set and why does it matter?

A: A golden set is a fixed, curated collection of representative inputs paired with the answers or quality expectations you consider good, drawn from real business data. Every candidate model — and every future version of your chosen model — is evaluated against the same golden set, so comparisons are fair and repeatable. It turns a vague feeling that "this model seems better" into measurable, defensible evidence, and it is the backbone of sound model evaluation and selection.

Q: What is LLM-as-judge, and can it replace human review?

A: LLM-as-judge uses a capable model to grade other models' outputs against a written rubric — for example, scoring groundedness from 1 to 5. It scales judgment cheaply to thousands of examples, which human review cannot. But it has biases of its own and is not a perfect substitute, so it complements human review and golden sets rather than replacing them. For regulated or customer-facing content, keep humans in the loop for sampled review.

Q: How reliable are public benchmarks for choosing a model?

A: Benchmarks are useful only as a first filter to build a shortlist. They measure generic public tasks, not your business data, can be gamed or contaminated by training-data leakage, age as models improve, and say nothing about cost or latency at your volume. A model that tops a leaderboard may still underperform on your support tickets or document formats. The final decision must come from evaluating shortlisted models against your own golden set in the Vertex AI Gen AI evaluation service.

Q: Where do I actually choose a model on Google Cloud, and is model selection a one-time decision?

A: You browse and select models in Model Garden inside Vertex AI, which catalogs Google first-party models like Gemini, partner models, and open models, each with a model card and pricing. Model selection is not one-time. New, cheaper, faster, more capable variants ship regularly, traffic grows, and new use cases appear, so a model chosen last quarter can be wrong today. Treat model evaluation and selection as a recurring process with a maintained golden set and periodic re-evaluation.

Summary: Model Evaluation and Selection for the Generative AI Leader

For the Generative AI Leader exam, model evaluation and selection is the disciplined practice of choosing the right model and proving it works. Master the cost-latency-quality triangle, understand that a bigger model is not automatically better and small fast variants often win simple high-volume tasks, and know why generative evaluation differs from traditional ML — no single correct answer, multi-dimensional quality, non-deterministic output. Build evaluation on golden sets, human review, automated metrics, LLM-as-judge, and pairwise comparison, treat benchmarks only as a shortlist signal, navigate Model Garden with a clear job description, default to buying a managed foundation model over building one, and remember that model selection is a recurring trade-off, not a one-time pick. With that framework, you can recommend GenAI model strategy with confidence — and answer any model evaluation and selection question on the exam.