Checks & integrity
How PaneScale proves that agent work is done — your own commands as checks, a baseline on the starting commit, a second run for flaky tests and an integrity scan that catches faked green builds.
An agent’s summary is a claim. A check is evidence. Wherever PaneScale decides whether agent work is good enough — the Overnight Autopilot marking a task ready, Point & Fix running its quick check — it runs your commands in the agent’s worktree and records the result. No language model is involved in that decision.
Checks
A check is any shell command that succeeds or fails: a build, a test suite, a type checker, a linter, a script of your own. PaneScale proposes checks from the scripts in your project’s package.json:
| Script | Proposed check | Required |
|---|---|---|
build | npm run build | yes |
test | npm test | yes |
typecheck | npm run typecheck | yes |
lint | npm run lint | optional |
Edit, add or remove checks freely in the autopilot setup — cargo test, pytest, go test ./... or make check work just as well. Reset to detected brings back the proposal.
- Required checks must pass for a task to count as ready.
- Optional checks are run and reported, but don’t block.
How checks run
- One after another, in the agent’s worktree, with a time limit — 15 minutes per check in the autopilot, 10 minutes in Point & Fix.
- In a clean environment:
CI=1andNO_COLOR=1are set, so test runners behave like on a build server and their output is readable. - The log is kept at up to 2 MB — the beginning and the end — with secrets removed before it is stored.
- A command that isn’t found counts as an error, not as a failure of the agent’s work.
Each check ends as passes, failed, timeout, error or skipped. PaneScale reads the summaries of node:test, Jest, Vitest, pytest, cargo and Go, so a report can say “41/41” instead of just “passed”.
Baseline
Before the first task of a night, the autopilot runs every check once on the unchanged starting point, in a worktree of its own. This is the baseline.
- A check that passed at the baseline and fails after the agent’s change is a regression — the task is not ready, whatever else passed.
- A check that already failed at the baseline is reported in the night’s alerts. If it is a required check, a task can only become ready if the agent fixes that failure too — so either fix it first, or make it optional for the night.
Flaky tests
When a required check fails, the autopilot runs it a second time. If the second run passes, the result counts — and the report marks it as passed on the second run, so flaky tests don’t go unnoticed.
Integrity scan
An agent can make checks pass by weakening them. The integrity scan looks at the diff between the starting commit and the agent’s result — never at what the agent says — and flags:
| Flag | Examples |
|---|---|
| Deleted tests | Test files deleted or renamed away, lines deleted from existing test files (*.test.*, *.spec.*, tests/, __tests__/, *_test.go, *_test.py …) |
| New skip markers | it.skip, xit(, describe.skip, test.skip, pytest.mark.skip, @unittest.skip, skipif, t.Skip, #[ignore] in added lines |
| Type and lint escapes | as any, @ts-ignore, @ts-expect-error, eslint-disable, # noqa, # type: ignore, allow(dead_code) in added lines |
| Check configuration | Changed scripts in package.json, loosened strictness in tsconfig.json, changed tool settings in pyproject.toml, and any change to pytest.ini, Jest, Vite or Vitest configs or GitHub workflows |
| Lockfiles | Changes to package-lock.json, yarn.lock, pnpm-lock.yaml, Cargo.lock, poetry.lock or uv.lock |
Markers only count as whole words — process.exit( is not mistaken for xit(. Generated files such as *.min.js, source maps, snapshots and generated/ folders are not scanned for markers.
In the autopilot, any integrity warning keeps a task from being ready, even if every check passed: it lands in needs review with the flags attached, and you decide. See When a task counts as ready.
Point & Fix quick check
Point & Fix uses a lighter version: after the agent’s change it runs your type check and lint — plus tests when you asked for a bug fix — or, if the project has none of these, the build. If the quick check fails, Apply is blocked until you either refine the change or choose Apply anyway. See Point & Fix.