Five ways an agent can fake a green build
Agents rarely cheat on purpose, but under pressure to make tests pass they find shortcuts. Here is what to look for — and what PaneScale flags automatically.
Write tasks an agent can finish
The difference between a task that runs overnight and one that wanders is usually a single sentence — the finish line. A short template and examples.
“Improve the upload flow” is a fine thing to say to a colleague over coffee. Given to an agent at 11 pm with nobody watching, it is an invitation to wander: it might refactor the whole module, rename half the functions and stop when it runs out of ideas — or tokens.
Agents are good at reaching a goal. They are much worse at deciding what the goal is. So give them one they can check.
The finish line
A finish line is a condition the agent can verify itself, with a command, before it stops. Compare:
| Wanders | Finishes |
|---|---|
| Improve upload error handling | npm test -- upload passes, including a new test for a 503 response that retries twice |
| Make the settings page faster | The settings page renders in under 100 ms in npm run bench:settings |
| Clean up the date utils | src/utils/date.ts has no any, npm run typecheck passes, behavior unchanged — all existing tests green |
| Fix the flaky test | upload.retry.test.ts passes ten runs in a row with npm test -- --repeat 10 |
The right column is longer. That’s the point: the extra words are the ones the agent would otherwise have to guess.
A template
Task template
Goal: <one sentence, the outcome, not the activity>
Done when:
- <a command that must pass>
- <a second check, if any>
Scope:
- Only change files in <folder or module>.
- Do not modify, skip or delete existing tests.
Context:
- <the one thing a newcomer wouldn't know>Four parts, each with a job:
- Goal keeps the agent pointed in one direction.
- Done when gives it a way to stop. Without it, agents either stop too early (“I have made the changes”) or too late (“I also refactored…”).
- Scope keeps the diff reviewable and makes integrity problems less likely.
- Context saves the agent from rediscovering what you already know — and saves you the tokens.
Make the checks real
A finish line only works if the command exists and works on the starting commit. Before you queue a task overnight, run the command yourself once. If it fails for unrelated reasons, the agent will spend its attempts on the wrong problem.
PaneScale helps here: the autopilot runs your checks on the starting commit first — the baseline — so failures that were already there don’t count. The autopilot preflight refuses to start a night without at least one required check, because “ready” has to be proven.
Split what can’t be finished
Some tasks don’t have a finish line yet — “design a better onboarding” is a conversation, not a task. Have that conversation in Chat or with a teammate in Plan mode, write down the result, and turn each piece into a task with its own finish line. Then queue those.
A quick test
Before handing a task to an agent, ask: if I came back in four hours, could I tell in one minute whether it’s done? If the answer is yes, it’s ready to go.