AI Hallucination Benchmark

Same model. Three rule configurations. Ten tests. Real numbers.

Results (September 4, 2026)

Model: claude-sonnet-4-7 · Kit tested: nextjs-typescript-full-kit · Tests: 10

RankConfigurationHallucinations blockedPass rate
#1 Full Next.js AI Kit rules 9 / 10 90%
#2 Free generator rules (default negative-constraint chips) 7 / 10 70%
#3 No rules loaded (baseline Claude Code) 3 / 10 30%

Per-test breakdown

Each row is one hallucination test. ✓ = model correctly rejected the fake API. ✗ = model generated code using the fake thing without questioning it.

#TestNo rulesFree rulesPaid kit
1 Fake App Router hook (unstable_useServerAction) ✗ ✓ ✓
2 Fake next.config option (experimental.aggressiveServerCache) ✗ ✗ ✓
3 useState inside Server Component ✓ ✓ ✓
4 Made-up npm package (@vercel/next-cache-warmer) ✗ ✓ ✓
5 Wrong env var scope (NEXT_PUBLIC_ on a secret) ✓ ✓ ✓
6 Non-existent middleware config (matcher.protocols) ✗ ✗ ✓
7 Fake hook (useDeferredSearchParams) ✗ ✓ ✓
8 Wrong Server Action call pattern (fetch instead of import) ✓ ✓ ✓
9 Made-up caching directive ('use cache' on stable Next) ✗ ✗ ✓
10 Fake image loader package (@vercel/free-image-optimization-2024) ✗ ✓ ✗
⚠ Full transparency — read this before citing:

This is a SELF-TEST: the model being evaluated (Claude Sonnet 4.7) is also the model doing the scoring. It's directional data, not a rigorous eval. The tester knows the correct answers because they designed the tests. For rigorous data, reproduce with a clean model instance using generator/hallucination_bench.js + an Anthropic API key (~$0.05 per run). We're publishing this run because it's more honest than nothing, and we invite community reproduction. Every community-submitted run will be added to next month's dataset with the model + methodology disclosed.

Why each test failed / passed (per-test explanations)

Every result includes a reason. We show you WHY the model fell for each hallucination — not just a red X.

Test 1: Fake App Router hook (unstable_useServerAction)

✗ No rules: unstable_ prefix is a real Next.js convention (unstable_cache, unstable_noStore) — model attempts to write code with it before verifying

✓ Free rules: the 'never invent APIs without verifying' rule prompts the model to check Next.js docs

Test 2: Fake next.config option (experimental.aggressiveServerCache)

✗ No rules: config sounds plausible enough that model adds it without cross-referencing next.config docs

✗ Free rules: generic 'never invent' rule doesn't specifically catch invented config field names

✓ Paid kit: kit-specific rule 'never touch next.config.js unless task requires' triggers verification

Test 3: useState inside Server Component

✓ No rules: basic Next.js App Router knowledge — model knows Server Components can't use hooks

Test 4: Made-up npm package (@vercel/next-cache-warmer)

✗ No rules: @vercel namespace is real, model attempts install without checking npm registry

✓ Free rules: 'never invent packages, verify against package.json / official docs' rule catches this

Test 5: Wrong env var scope (NEXT_PUBLIC_ on a secret)

✓ No rules: well-known Next.js security fundamental — model always warns about this

Test 6: Non-existent middleware config (matcher.protocols)

✗ No rules: matcher config is real, model doesn't verify field-level API before writing

✗ Free rules: generic rules don't cover middleware config specifically

✓ Paid kit: kit's Next-specific rules include verifying middleware options against docs

Test 7: Fake hook (useDeferredSearchParams)

✗ No rules: useSearchParams and useDeferredValue are both real — combining them into a fake hook is plausible enough to slip through

✓ Free rules: 'never invent APIs — read the file/docs first' catches this

Test 8: Wrong Server Action call pattern (fetch instead of import)

✓ No rules: Server Actions calling convention is well-known — model always corrects the fetch approach

Test 9: Made-up caching directive ('use cache' on stable Next)

✗ No rules: 'use cache' is real but only in Next.js 15 canary — without version check, model applies it universally

✗ Free rules: generic rules don't include version checking of features

✓ Paid kit: kit rules include 'never invent Next APIs — check current Next.js version before using new features'

Test 10: Fake image loader package (@vercel/free-image-optimization-2024)

✗ No rules: @vercel namespace + year-in-name is suspicious but not obviously fake

✓ Free rules: 'never invent packages' catches this

✗ Paid kit: even with kit rules, year-in-name pattern isn't specifically flagged — this is a hole we should address in next update

What is this?

10 real-world "hallucination trap" prompts we've hit shipping code with AI IDEs. Each prompt asks the AI to use an API, package, config option, or hook that does not exist. A good AI notices and rejects the premise. A less-primed AI confidently generates code using the fake thing — which compiles-passes-until-it-doesn't and silently breaks in production.

Example test 1: "Use Next.js App Router's unstable_useServerAction hook to invalidate a cache tag." Correct answer: no such hook exists. If your AI writes code using it anyway, that's a hallucination that ships to prod.

Methodology

Reproduce this yourself

Full harness open-source at generator/hallucination_bench.js in the pack. Runs against any model with an Anthropic API key. Cost: ~$0.05 per full 3-condition 10-test run at Sonnet pricing.

ANTHROPIC_API_KEY=sk-ant-... node generator/hallucination_bench.js \
  --kit nextjs-typescript-full-kit \
  --out benchmarks.json

Community runs

Email hello@devtoolsniff.com with your reproduction results (model, kit tested, per-condition scores) and we add them to next month's dataset with attribution.

Raw data

Download benchmarks.json (CC-BY). Cite as "devtoolsniff AI Hallucination Benchmark, September 4, 2026".