Hallucinations and Model Limitations

What Hallucinations and Model Limitations Mean for a GenAI Leader

For the Google Cloud Generative AI Leader exam, hallucinations and model limitations are not obscure technical footnotes — they are central to whether an organization can adopt generative AI safely and profitably. A hallucination is when a large language model (LLM) produces an answer that is fluent, confident, and completely wrong. The model is not lying and it is not broken; it is doing exactly what it was designed to do, which is to predict plausible-sounding text. As a Generative AI Leader, your job is not to debug a neural network. Your job is to know why hallucinations happen, where they create business risk, and which Google Cloud capabilities turn that risk into something manageable.

Why the Exam Weights This Topic Heavily

This is one of the most heavily weighted concepts on the exam because it sits at the intersection of technology, governance, and business value. An executive who does not understand hallucinations will either reject generative AI entirely out of fear, or deploy it recklessly into a high-stakes workflow and damage the company. The Generative AI Leader is expected to chart the middle path: treat hallucination as a managed, mitigated risk, the same way a business treats credit risk, fraud risk, or supply-chain risk. You do not avoid lending money because some loans default; you build controls. Generative AI is the same. The exam wants to confirm you can talk about hallucinations and model limitations with that calm, business-leader framing rather than either hype or panic.

What This Study Note Covers

Across this study note we will explain what hallucinations are, why LLMs hallucinate, and the related limitations every leader must know: knowledge cutoff, training-data bias, non-determinism, prompt sensitivity, and context-window limits. We will then cover mitigations at a conceptual level — grounding, retrieval-augmented generation (RAG), citations, and human review — and point you to deeper topics for the technical mechanics.

白話文解釋（Plain English Explanation）

Hallucinations sound mysterious, but the underlying behaviour is something you have already experienced from people in everyday life. The model limitations a Generative AI Leader must understand map cleanly onto familiar situations. The following analogies make hallucinations concrete and tie them back to Google Cloud generative AI products like Gemini, Vertex AI, and grounding.

Analogy 1 — The Smooth-Talking Salesperson Who Bluffs

Imagine a sales representative who is brilliant at conversation. They never pause, never look unsure, and always have an immediate, polished answer to any question a customer throws at them. The problem is that when they do not actually know something — a delivery date, a technical spec, a warranty clause — they do not say "let me check". They simply produce a confident, plausible-sounding answer on the spot, because sounding helpful and fluent is what they are rewarded for. Most of the time the answer is right. Occasionally it is completely invented, and the customer has no way to tell the difference because the bluff is delivered with the exact same confidence as the truth.

A large language model behaves like this salesperson. Gemini on Vertex AI is trained to produce the most plausible continuation of your prompt, not to look up a verified fact. When the model has seen enough relevant data, the plausible answer and the correct answer are the same thing. When it has not, it still produces something fluent — that is the hallucination. The critical lesson for the exam is that a confident-sounding answer and a correct answer are not the same thing. The model's tone of certainty carries zero information about accuracy. This is why a Generative AI Leader never lets the salesperson close the deal alone: you ground the model in real catalogue data and you keep a human in the loop for anything high-stakes.

Analogy 2 — The Exam Candidate Answering From Memory

Now picture a student sitting a closed-book exam. They studied months ago, they cannot look anything up, and the exam asks about a detail they half-remember. A blank answer scores zero, so the rational move is to write the most likely-sounding answer based on a fuzzy impression of what they once read. Sometimes that impression is accurate; sometimes they confidently write down a date or a name that is simply wrong. They are not cheating — they are reconstructing from compressed memory under pressure.

This is exactly how an LLM works at inference time. The model's "studying" was its training run, which finished at a fixed point — its knowledge cutoff. After that, it sits the exam closed-book: it cannot see today's news, your company's internal documents, or a price that changed yesterday. It answers from a compressed, statistical impression of its training data. This explains both the knowledge cutoff limitation and a major source of hallucination. The Generative AI Leader's fix is to turn the closed-book exam into an open-book exam. That is precisely what grounding and retrieval-augmented generation do on Vertex AI: before the model answers, the system retrieves the relevant authoritative passages — from Google Search or from your private data in Vertex AI Search — and places them in front of the model. The candidate is no longer guessing from memory; they are reading from the textbook.

Analogy 3 — The Out-of-Date Travel Guidebook

Think of a printed travel guidebook bought three years ago. It is beautifully written, well organised, and was accurate the day it was printed. But restaurants have closed, museum prices have risen, a train line has been rerouted, and a hotel has changed its name. The book itself has not changed a word — it is simply frozen at its publication date. If you follow it blindly you will arrive at a restaurant that no longer exists, and the book will give you no warning, because it has no idea time has passed.

A pretrained foundation model is a guidebook frozen at its training cutoff. Gemini does not "know" that today is later than its training data; it confidently presents stale information as current. This is why knowledge cutoff and stale facts are core model limitations, distinct from pure hallucination but often confused with it. For a Generative AI Leader, the takeaway is twofold. First, never assume a base model knows recent events, current prices, or this quarter's policy. Second, the mitigation is to attach a live information source: grounding with Google Search gives the model a fresh "internet edition" of the guidebook, while grounding on your own data in Vertex AI Search gives it your company's continuously updated internal edition. The model stays frozen, but the facts it reads are current.

Why LLMs Hallucinate: Prediction, Not Retrieval

The single most important sentence in this entire topic is: an LLM predicts plausible text, it does not retrieve verified facts. Everything about hallucination follows from this.

When you send a prompt to Gemini, the model is not searching a database for the right answer. It is calculating, token by token, which word is the statistically most likely continuation given everything it learned during training. Language is the medium, plausibility is the objective. Because human language is full of true statements, the most plausible continuation is usually also true — which is why these models are useful at all. But "usually true" is not "always true". When the model lacks solid training signal for a specific fact, it still produces a fluent, grammatical, confident answer, because producing fluent text is the only thing it does. That fluent-but-wrong output is the hallucination.

Prediction Versus a Database Lookup

This is a profound mental shift for business leaders. A database, when it does not have a record, returns "not found". An LLM, by its nature, almost never returns "not found" — it returns a guess dressed as an answer. Recognising that hallucination is a structural property of how generative models work, not a bug to be patched out, is what separates a credible Generative AI Leader from someone repeating headlines.

Knowledge Cutoff: The Model Is Frozen in Time

Every foundation model is trained on a snapshot of data that ends at a specific date, known as the knowledge cutoff. After that date the model knows nothing — not yesterday's news, not last week's regulation, not the price your company changed this morning. The model also has no built-in awareness that time has passed, so it will confidently answer questions about "current" events using information that may be months or years old.

For a Generative AI Leader, knowledge cutoff matters because many of the most valuable enterprise use cases involve recent or live information: customer-support agents quoting current policies, sales assistants citing current pricing, analysts summarising this quarter's results. A base model cannot serve these reliably on its own. The Google Cloud answer is grounding: connecting the model to a live, authoritative source so that the answer reflects the present, not the training snapshot. We cover the mechanics in the grounding and RAG topic, and the choice of which model fits which freshness requirement in the model evaluation and selection topic.

Training-Data Bias: The Model Reflects Its Source

An LLM learns from enormous volumes of human-created text and images. Whatever patterns, assumptions, stereotypes, and gaps exist in that source data are absorbed into the model. This is training-data bias, and it is a model limitation distinct from hallucination, though both undermine trust.

Bias shows up in business in concrete, costly ways: a hiring-assistant summary that subtly favours certain demographics, a marketing generator that defaults to stereotypes, or a model that performs noticeably worse for under-represented languages or regions because there was less training data for them. For a Generative AI Leader, the key point is that bias is not malicious intent by the model — it is an inherited reflection of the data and the world that produced it. Mitigation is therefore a governance discipline: representative evaluation across demographic slices, human review of sensitive outputs, and the practices covered in the responsible AI and SAIF topic. Google Cloud's Responsible AI program and Vertex AI safety filters provide tooling, but leadership ownership is what makes bias mitigation real.

Confident Tone Versus Correct Answer

One of the most dangerous traps in adopting generative AI is mistaking fluency for accuracy. Because models like Gemini produce grammatically perfect, well-structured, authoritative-sounding prose, both employees and customers instinctively trust the output. But the model's confident tone is generated by the same plausibility engine that produced the words — it is a stylistic feature, not a measure of truth.

Why This Matters for High-Stakes Outputs

This matters enormously for high-stakes outputs. A hallucinated answer about a refund policy, a medical dosage, a legal clause, or a financial figure does not arrive flagged in red. It arrives in the same calm, helpful voice as a correct answer. A Generative AI Leader must educate the organisation that the model's confidence is not evidence. The practical defences are citations — making the model show its source so a human can verify — and human review for any output where being wrong has real consequences.

Non-Determinism: The Same Prompt, Different Answers

Traditional software is deterministic: the same input produces the same output every time. Generative AI is different. By design, LLMs sample from a probability distribution when choosing each token, so the same prompt can produce different answers on different runs. Parameters such as temperature control how much randomness is introduced — higher temperature means more varied, creative output; lower temperature means more focused, repeatable output.

For a Generative AI Leader, non-determinism has real operational consequences. It complicates testing, because you cannot simply check for one fixed correct string. It complicates compliance, because two customers asking the same question may get differently worded answers. And it interacts with hallucination, because variation means an answer that was fine in a demo may differ in production. The mitigation is not to be alarmed but to design for it: use low temperature for factual, consistent use cases; use systematic evaluation across many runs rather than a single spot-check; and ground the model so that, however the wording varies, the underlying facts come from a fixed authoritative source.

Temperature and Output Settings

Temperature is the most commonly discussed control, but related settings include top-k and top-p sampling and an output-token limit. A Generative AI Leader does not need to tune these personally, but should understand the trade-off: more randomness gives more creative range and a higher hallucination surface; less randomness gives more reliability and less originality. The right setting depends entirely on the use case — a brainstorming tool and a policy-answering agent sit at opposite ends.

Why Non-Determinism Is Not a Defect

It is worth stressing to stakeholders that non-determinism is intentional. The same property that lets Gemini write three different headline options, or rephrase an explanation for a confused customer, is the property that makes outputs vary. You manage it; you do not eliminate it.

Prompt Sensitivity: Small Wording Changes, Large Output Changes

LLMs are highly prompt-sensitive. A small change in how a question is phrased — a different word order, an added instruction, a missing piece of context — can produce a meaningfully different answer, and sometimes the difference between a grounded answer and a hallucinated one. This is why prompt engineering has become a discipline: clear, specific, well-structured prompts measurably reduce hallucination and improve relevance.

For a Generative AI Leader, prompt sensitivity has two implications. First, it is a limitation: you cannot assume every employee will phrase prompts well, so user-facing applications should wrap the model in carefully designed system prompts and templates rather than exposing a raw text box. Second, it is a lever: investing in prompt design, examples, and guardrails is one of the cheapest, fastest ways to improve output quality before reaching for more expensive options like fine-tuning. Vertex AI Studio on Google Cloud lets teams iterate on and save effective prompts so good patterns are reused rather than rediscovered.

Context-Window Limits: The Model Can Only Hold So Much

Every LLM has a context window — the maximum amount of text (measured in tokens) it can consider at once, covering both your input and its generated output. Modern Gemini models on Vertex AI offer very large context windows, but the limit is never infinite, and it shapes what is realistic.

Context-window limits create several practical effects a Generative AI Leader should recognise. If you paste a document larger than the window, the model simply cannot see all of it, and may hallucinate to fill the gap. In a long conversation, earlier turns can fall out of the window, so the model "forgets" what was said earlier and may contradict itself. And cramming a window full of marginally relevant text can dilute the signal so the model focuses on the wrong parts. The mitigation is, again, grounding and retrieval: instead of stuffing everything into the prompt, a RAG system retrieves only the most relevant passages and supplies those, keeping the context focused and within budget. Larger context windows reduce the pain but do not remove the need for disciplined retrieval, which is covered in the grounding and RAG topic.

Tokens, Cost, and the Context Budget

Context also has a cost dimension. Generative AI on Vertex AI is billed largely by input and output tokens, so a bigger context window used carelessly increases both latency and spend. A Generative AI Leader frames the context window as a budget to be spent wisely on the most relevant information, not a bucket to overfill.

Why GenAI Must Not Be the Sole Decision-Maker for High-Stakes Outputs

Pulling the limitations together leads to a governance principle the exam expects you to state clearly: generative AI must not be the sole, unchecked decision-maker for high-stakes outputs. Because hallucination is structural, because the model is frozen at a cutoff, because it can be biased, non-deterministic, and prompt-sensitive, any workflow where being wrong causes real harm needs a human or a deterministic system in the loop.

Defining High-Stakes Versus Low-Stakes Use Cases

"High-stakes" means decisions affecting health, safety, legal rights, finances, employment, eligibility, or anything regulated. In those domains generative AI is best used as an assistant that drafts, summarises, and proposes, while a qualified human reviews, verifies, and owns the final decision. This is the "human-in-the-loop" model and it is a recurring theme across Google Cloud's Responsible AI guidance and the Secure AI Framework discussed in the responsible AI and SAIF topic. Low-stakes uses — brainstorming marketing copy, drafting an internal summary, suggesting code — can run with lighter oversight. Matching the level of human review to the level of risk is the leader's core judgement call.

Mitigations: Turning Hallucination Into a Managed Risk

A Generative AI Leader is not expected to implement the mitigations, but must know what they are and when each applies. There are four conceptual levers, and they stack.

Grounding

Grounding connects a model to an authoritative source of truth so its answers are anchored in real, retrievable information rather than memory. On Google Cloud you can ground Gemini with Google Search for public, current information, or ground it on your private corporate data using Vertex AI Search. Grounding directly attacks both the knowledge-cutoff limitation and the main source of hallucination, and is usually the highest-impact mitigation.

Retrieval-Augmented Generation (RAG)

RAG is the pattern that operationalises grounding for enterprise data: when a question arrives, the system searches a knowledge base, retrieves the most relevant passages, and supplies them to the model along with the prompt. The model then answers from the retrieved text instead of guessing. RAG turns the closed-book exam into an open-book one. The mechanics are covered in depth in the grounding and RAG topic.

Citations and Source Attribution

When a model is grounded, it can return citations — links back to the source passages it used. Citations do not make the model more accurate by themselves, but they make verification fast and cheap: a reviewer can click through and confirm, and an end user can judge trustworthiness. Citations are a key part of building user trust and are surfaced by Google Cloud grounding features.

Human Review and Evaluation

Finally, human review remains the backstop for high-stakes outputs, and systematic evaluation — testing the model against representative cases and metrics before and during deployment — catches problems at scale rather than one demo at a time. Choosing the right model and proving it is good enough is the subject of the model evaluation and selection topic, and the broader governance wrapper is in the responsible AI and SAIF topic.

Safety Filters and Guardrails

Beyond accuracy, Vertex AI provides configurable safety filters that screen for harmful, unsafe, or policy-violating content. These address a different risk than hallucination — harmful output rather than incorrect output — but a leader should know both controls exist and complement each other.

The Business Framing: Hallucination Is Risk, Not a Dealbreaker

The conclusion the exam wants from a Generative AI Leader is balanced and pragmatic. Hallucination and model limitations are real, but they are not a reason to avoid generative AI. Every transformative business technology carries managed risk — payment systems carry fraud risk, lending carries default risk, manufacturing carries defect risk — and organisations succeed by building controls, not by abstaining.

Three Practical Steps to Manage the Risk

Treating hallucination as a managed risk means three things in practice. First, match the use case to the risk tolerance: deploy generative AI freely for low-stakes drafting and ideation, and add grounding, citations, and human review as the stakes rise. Second, invest in the mitigation stack as a normal cost of doing business, the same way you invest in security or quality assurance. Third, set expectations honestly with employees, customers, and regulators — communicate that the system assists rather than decides, and that outputs are verified where it matters. A Generative AI Leader who can hold this balanced position, neither hyped nor fearful, is exactly what the certification is designed to validate.

How This Connects to the Rest of the Exam

Hallucinations and model limitations are a connective topic. They explain why grounding and RAG exist, so study the grounding and RAG topic next for the technical depth. They are the practical reason responsible AI governance matters, covered in the responsible AI and SAIF topic. And the limitations described here — knowledge cutoff, context window, cost per token — directly drive which model an organisation should pick, which is the subject of the model evaluation and selection topic. Expect the exam to test hallucinations not in isolation but woven into scenario questions across all three of those areas.

Frequently Asked Questions

Q: What exactly is an AI hallucination, in business terms?

A: An AI hallucination is when a generative model such as Gemini produces an answer that sounds fluent and confident but is factually wrong or fabricated. It happens because large language models predict the most plausible next words rather than retrieve verified facts. In business terms, treat hallucination the way you treat fraud or default risk — a known, structural risk you manage with controls (grounding, citations, human review), not a defect that can be fully eliminated.

Q: Why do large language models hallucinate at all?

A: Because of how they work. An LLM is trained to generate the most statistically plausible continuation of your prompt, not to look up an answer in a database. When the model has strong training signal for a topic, plausible and correct usually coincide. When it does not, it still produces fluent, grammatical text — and that fluent-but-wrong output is the hallucination. It is a structural property of generative models, which is why mitigation focuses on grounding and oversight.

Q: What is a knowledge cutoff, and why does it matter?

A: A knowledge cutoff is the date when a model's training data ends. After that date the model knows nothing — not recent news, not new regulations, not prices that changed yesterday — and it has no awareness that time has passed, so it presents stale information confidently. It matters because many valuable use cases need current information. The Google Cloud fix is grounding the model on a live source, such as Google Search or your own data in Vertex AI Search.

Q: If a Gemini answer sounds confident and detailed, can I trust it?

A: No — confidence is not evidence. An LLM's authoritative tone is produced by the same next-token prediction process that produces hallucinations, so it carries no information about factual accuracy. A confident-sounding answer and a correct answer are different things. For any high-stakes output, require citations so a human can verify the source, and apply proportional human review before the output is published or actioned.

Q: Can hallucinations be completely eliminated?

A: Not entirely. Because hallucination is a structural characteristic of how generative models predict text, it cannot be fully removed, but it can be reduced dramatically. Grounding and RAG anchor answers to real sources, citations make verification fast, careful prompt design reduces errors, and human review catches what remains. The Generative AI Leader goal is to drive residual risk low enough for the specific use case, not to chase an unachievable zero.

Q: When should generative AI not be the decision-maker?

A: Generative AI should never be the sole, unchecked decision-maker for high-stakes outputs — anything affecting health, safety, legal rights, finances, employment, or regulated eligibility. In those domains the model should draft, summarise, and propose while a qualified human reviews, verifies, and owns the final decision. Low-stakes uses such as brainstorming or internal drafting can run with lighter oversight. Matching the level of human review to the level of risk is the leader's key judgement.

Q: What is the difference between hallucination, bias, and non-determinism?

A: They are three distinct model limitations. A hallucination is a fabricated or incorrect fact stated confidently. Training-data bias is the model reflecting unfair patterns or gaps inherited from its source data. Non-determinism is the model producing different answers to the same prompt because it samples randomly when choosing words. All three reduce trust, and a Generative AI Leader should be able to name each one and its mitigation rather than lumping every problem under "hallucination".

Summary: Hallucinations and Model Limitations for the Generative AI Leader

A Generative AI Leader does not eliminate hallucinations — they manage them. Understand that LLMs predict plausible text rather than retrieve facts, which makes hallucination structural. Know the related limitations: knowledge cutoff freezes the model in time, training-data bias is inherited, outputs are non-deterministic and prompt-sensitive, and the context window is finite. Internalise that a confident tone is not proof of accuracy, and that generative AI must never be the sole decision-maker for high-stakes outputs. Then apply the four-lever mitigation stack — grounding, RAG, citations, and human review and evaluation — delivered on Google Cloud through Gemini grounding, Vertex AI Search, and Vertex AI evaluation tooling. With that balanced, business-leader framing, hallucination becomes a managed risk rather than a dealbreaker, and you can confidently answer any model-limitations question on the Generative AI Leader exam.