Hallucinations in Generated Code: Five Signs to Catch Before You Lose an Afternoon Debugging

An AI assistant writing code cannot tell the difference between "this exists" and "this sounds like it should exist." When you ask it to use a library it only half knows, it produces the most plausible call according to its training, not the real one. The result compiles in the model's own reasoning — and sometimes literally, if the language is dynamic — but it does not exist in any installed package.

We call this a code hallucination: the model invents an API, a method, a parameter, or an import with the same confidence it uses to write a real one. This isn't an isolated failure of one particular provider, and it doesn't depend on model size; it's a consequence of how probability-based text generation works. What does depend on the team is how long it takes to catch it. These are the five signs that have saved us the most debugging hours, in the order we check them before accepting any generated function as done.

1. Plausible but fake method names

The most common sign isn't an obvious error — it's a name that "fits" the rest of the library. If an HTTP client exposes client.get(), client.post(), and client.delete(), a model might generate client.patchAsync() even though that specific library doesn't follow that suffix convention. The name isn't absurd; it's consistent with the pattern the model learned from similar libraries, which is exactly what makes it dangerous — it survives a quick read.

The way to catch it isn't reading more carefully, it's letting the right tool catch it for you. An editor with IntelliSense or autocomplete based on the real types of the installed package flags any nonexistent method in red before you run a single line. If the editor doesn't complain but the method also doesn't show up when you type client. and trigger autocomplete, that's the first alarm.

// The editor doesn't offer 'patchAsync' in the autocomplete for 'client.'
// That alone is the signal, before a single line runs.
const response = await client.patchAsync('/orders/123', payload);

2. Imports that don't resolve

When the model isn't sure of a module's exact path, it tends to construct one that "sounds" right based on conventions it has seen in similar projects. This is common in ecosystems with many subpackages, like Java/Spring or Python's utility libraries, where the real path can differ from the intuitive one by a single folder or package name.

# Plausible, but in many versions the real module lives elsewhere
from utils.validators.email import validate_email_format

Here the check is mechanical and doesn't require careful reading: an incremental build or a live import fails immediately if the path doesn't exist. The key is not letting a failed import pile up alongside other unreviewed changes; if you compile or run the module right after the assistant proposes the import, the error shows up isolated and is trivial to diagnose. If you wait until twenty more changes have stacked on top, the same error takes ten times longer to locate.

3. Signatures that mix versions of the same library

This is the hardest sign to catch with a surface-level read, because each piece, on its own, is real. The model can generate a call that matches an older version's API mixed with a parameter that only exists in a later version, because both showed up in its training and it has no way of knowing which one matches the version actually installed in your project.

// 'timeout' as the constructor's second argument: an older version's pattern.
// 'retryPolicy' as a property: a later version's pattern.
// No real version accepts both at once.
const client = new ApiClient(baseUrl, { timeout: 5000 });
client.retryPolicy = { attempts: 3 };

Static type checking (TypeScript, mypy, or the compiler itself in typed languages) usually catches half of this problem — the half that produces a direct type error. The other half — a signature that's syntactically valid but semantically from a different version — is only caught by checking against the documentation for the exact installed version, not against "that library's documentation" in general. Pinning the version in package.json or requirements.txt before asking for code doesn't stop the model from mixing versions, but it does give you a single reference to verify against.

4. Behavior that passes the smoke test but fails the real case

This sign shows up one step later, when the code does compile and does run, but does something different from what the function's name promises. It's more dangerous than the previous three because neither the linter nor the compiler catches it: the code is valid, it's just wrong.

The pattern we keep seeing is a function with a correct name and a correct happy path that ignores an edge case that was never explicit in the prompt. Before accepting a generated function as finished, we run the tests that already exist for that module first — not new tests written by the same assistant that wrote the function. A test written by the model that generated the code tends to validate exactly what the code does, not what it should do, so it confirms the problem instead of exposing it.

5. Narrative confidence without verification

The last sign isn't in the code, it's in how the model presents it. When an assistant describes its own solution with phrases like "this correctly handles all error cases" or "this is the standard way to do it in this library," that claim isn't evidence of anything: it's the same generative process that produced the code, now applied to describing the code. A model hasn't run the function against the real case or checked the official documentation before writing that sentence; the sentence sounds confident because that's the register training associates with technical explanations, not because there's verification behind it.

We treat any claim of that kind as a hypothesis to check, not a reported fact. If the assistant says a library exposes some behavior, the check is reading the documentation for the installed version or the package's own source code, not trusting the confidence with which it was explained.

The order that actually saves time

None of these signs require expensive tooling or a new process. What changes the detection time is the order in which you apply the checks already available in any project: first the linter and type checker, because they're free and catch signs one and two almost instantly; then an incremental build, because it exposes broken imports before you write another line on top; then the existing tests for the affected module, run before any new tests, because they reveal whether the actual behavior matches what's expected; and only at the end, if everything above passed, a human read focused specifically on the claims the assistant made about its own code, not on the code itself.

That order matters because each step is cheaper than the next. Running a linter costs seconds. Debugging a method that never existed in production costs an afternoon. The difference between those two outcomes is almost never the quality of the model that wrote the code — it's the point in the process where someone, or something, checked that what sounded right actually was.