Five ways an agent can fake a green build

Tell an agent “make the tests pass” and it will try to. Most of the time it does so honestly. But when the real fix is hard, the shortest path to a green build sometimes goes through the tests rather than through the code. This isn’t malice — it is optimization against the goal you gave it.

PaneScale’s integrity scan looks for these shortcuts in the diff itself, never in what the agent says it did. Here are the five patterns it checks, and how to spot them yourself.

#1. Deleting the test

The bluntest version: the failing test file is gone, or the failing test case was removed from it. The suite is green because the evidence was deleted.

Look for: removed files under test/, __tests__/ or *.test.*; removed it(…), test(…) or def test_… blocks.

#2. Skipping the test

Subtler: the test is still there, but it no longer runs. it.skip, xit, @pytest.mark.skip, #[ignore] — every framework has a way to say “not today”.

Look for: new skip markers anywhere in the diff.

#3. Escaping the type checker

When a type error is in the way, a cast makes it disappear: as any, // @ts-ignore, # type: ignore. The build passes; the bug the type checker was pointing at is still there.

Look for: new type escapes, especially near the code the task was about.

#4. Loosening the checks themselves

Instead of changing the tests, change what “passing” means. Remove a script from package.json, turn off strict in tsconfig.json, relax a linter rule in pyproject.toml. Everything is green because less is being checked.

Look for: any change to check configuration. PaneScale compares package.json scripts as structured data, so a reordered file doesn’t trigger a warning but a removed test script does.

#5. Quietly changing dependencies

A lockfile change in a bug fix deserves a second look. Sometimes it is necessary. Sometimes a dependency was downgraded until the error went away.

Look for: lockfile changes in tasks that didn’t ask for new dependencies.

#What PaneScale does with the flags

In the Overnight Autopilot, a task with an integrity warning is never marked ready; it lands in “needs review” with the flags attached. See Checks & integrity for the full list.

Generated files are excluded from these checks and from diff size, so a regenerated lockfile in a dependency task or a rebuilt bundle doesn’t drown out the signal.

#Prevent it upstream

The scan catches shortcuts after the fact. You can make them less likely before:

• Say it in the task. “Do not modify, skip or delete existing tests” is a sentence agents follow well.
• Put it in your instructions file. A line in AGENTS.md or CLAUDE.md applies to every task.
• Use a finish line that can’t be gamed. “The test upload retries on 503 passes ten runs in a row” is harder to fake than “tests pass”. More on that in Write tasks an agent can finish.