Verify a diff

verify-diff takes a change (your AI agent’s, a codemod’s, or your own) and answers one question: did it preserve behavior, and can you prove it? Refactron applies the diff in an isolated shadow tree, runs the syntax, imports, and test gates against it, checks whether your tests actually exercise the changed statements, and returns one of three verdicts. Your working tree is never touched.

A code diff flows into Refactron, through the syntax, imports, and test gates in an isolated shadow tree, and out to a SAFE, UNSAFE, or UNPROVEN verdict — A diff enters, the gates run in an isolated shadow tree, and one of three verdicts comes out.

The verdict, in one line

Verdict	Meaning	Exit
`SAFE`	Every gate passed and your tests exercise the changed code (at least one changed statement per file).	`0`
`UNSAFE`	A gate failed: the change broke something.	`1`
`UNPROVEN`	Tests pass, but the changed code isn’t exercised (or coverage couldn’t be assessed).	`0`

UNPROVEN is the honest verdict. “Tests pass” is not the same as “this change is proven safe”: if nothing runs the lines you changed, a green suite proves nothing about them. Refactron says so out loud and tells you which test to add. See Verdicts for the full model.

Coverage is Python-only (via coverage.py). A TypeScript or mixed-language diff can never earn SAFE today; it returns UNPROVEN with the reason “coverage of the changed code could not be determined.” The gates still run; only the coverage half is Python-only. Coverage also cannot see code that runs only in a subprocess, so a change exercised solely through subprocess.run or a multiprocessing worker reads as UNPROVEN until in-process subprocess coverage lands.

If your project is installed into the environment (an editable pip install -e . counts), your tests may import that copy instead of the one being verified, in which case the run proves nothing about your change. Put the verified tree first on sys.path by prefixing the test command: PYTHONPATH=. python3 -m pytest -q, or PYTHONPATH=src for a src layout. Use module form rather than a bare pytest, which cannot always be measured. See Make sure the tests run the code being verified.

Install

Requires Node.js ≥ 18, plus Python 3.8+ with coverage.py for the coverage half.

npm install -g refactron

That puts refactron (and refactron-mcp) on your PATH. To skip the install, run npx refactron verify-diff ... instead.

pip install refactron==0.3.0 installs a thin refactron shim that shells out to the npm CLI. It is not a Node-free path: you still need Node.js ≥ 18 and npm install -g refactron. When the npm CLI is missing, the shim prints the exact matching install command and exits non-zero.

Build from source (contributors)

git clone https://github.com/Refactron-ai/Refactron_Lib_TS
cd Refactron_Lib_TS
npm install
npm run build

The CLI is then node dist/cli/index.js <command>. Substitute that for refactron in every example below.

Authenticate

verify-diff is auth-gated. Log in once on this machine:

refactron login

In CI, skip the browser and export a token instead:

export REFACTRON_TOKEN=refactron_live_...

An unauthenticated run exits 7.

Run it

refactron verify-diff [repoRoot] --diff <file> [flags]

repoRoot defaults to .. The --diff file is a standard unified (git) diff.

# A change your agent produced, saved to change.diff:
refactron verify-diff . --diff change.diff

Output is the verdict, its reason, and (for UNPROVEN by lack of coverage) the uncovered lines:

[UNPROVEN] Tests pass, but the changed code is not exercised by any test.
  uncovered: calc.py:14

Flags

Flag	Type	Default	Description
`--diff <file>`	path	none	Required. The unified diff to verify.
`--test-cmd <cmd>`	string	auto-detect	Override the test command (e.g. `"python3 -m pytest -q"`). Drives both the test gate and the coverage run.
`--json`	bool	false	Emit the full report as JSON instead of the one-line verdict.

Exit codes

Code	Meaning
`0`	`SAFE` or `UNPROVEN`.
`1`	`UNSAFE`: a gate failed. A tests-gate failure prints the failing-test tail; a syntax or imports failure prints that gate’s reason (the offending file or the unresolved import).
`2`	Bad input: unknown flag, missing `--diff`, the diff didn’t apply (e.g. `diff did not apply to calc.py (stale base?)`), or the diff uses an operation verify-diff does not support yet (see Unsupported operations).
`7`	Not authenticated.

Unsupported operations

v1 verifies content edits. A diff that deletes, renames, or copies a file, or changes a binary file, is refused with exit 2 rather than partially verified. This is deliberate: a diff that deleted a module while also making one innocuous edit once verified SAFE, yet applying it broke every import in the package. A partial verdict on the verifiable half must never read as a verdict on the whole diff.

Operation	Message
File deletion	`diff deletes <path>; file deletions are not supported yet, verify that change manually`
File rename	`diff renames <old> to <new>; renames are not supported yet`
File copy	`diff copies <old> to <new>; copies are not supported yet`
Binary only	`diff contains only binary changes; nothing verifiable`
Binary plus text	`diff contains binary changes alongside text edits; binary changes cannot be verified`

Detection is belt and braces: both the parsed diff and a raw scan of the diff text are checked, so a pure rename or deletion that the diff parser drops entirely is still caught. Full deletion and rename support is planned; until then, verify those changes manually.

Test files touched

When a diff changes files that look like tests (a tests/ or test/ path segment, test_*.py, *_test.py, conftest.py, *.test.ts, or *.spec.ts), the human output prints one advisory line and the JSON report carries the list under testFilesChanged:

[SAFE] Tests pass and the changed code is covered.
note: this diff modifies test files (1): tests/test_calc.py

This is a note, not a verdict change. A green verdict on a diff that also weakens its own tests is still green; the note is there so you look before you trust it.

The JSON report

--json emits a reproducible record of the verdict: the same structure the MCP tool returns:

{
  "reportVersion": 1,
  "verdict": "UNPROVEN",
  "gates": {
    "syntax": { "passed": true, "durationMs": 12 },
    "imports": { "passed": true, "durationMs": 9 },
    "tests": { "passed": true, "durationMs": 3140 }
  },
  "changedFiles": ["calc.py"],
  "testFilesChanged": [],
  "coverage": {
    "tool": "coverage.py",
    "changedLinesCovered": false,
    "changedStatements": { "total": 3, "covered": 2 },
    "uncovered": [{ "file": "calc.py", "line": 14 }],
    "filesWithUncovered": 1,
    "inertOnlyFiles": []
  },
  "reason": "Tests pass, but the changed code is not exercised by any test.",
  "missingTests": [{ "file": "calc.py", "hint": "add a test exercising calc.py:14" }]
}

reportVersion: schema version of this report. The shape below is a public contract; the version tells a consumer that stores reports which shape it is holding.
gates: each gate’s passed flag and wall-clock. A failed gate also carries a blockingReason.
testFilesChanged: changed files matching test conventions. A note that the diff touched tests, never a verdict input.
coverage.tool: coverage.py when coverage was assessed, none when it couldn’t be (non-Python diff, or no coverage.py installed).
coverage.uncovered: one entry per unexercised statement, at the statement’s first line, mapped from the changed line by AST containment. A multi-line statement reports once rather than once per wrapped line. Present on SAFE too: the per-file rule clears SAFE on one exercised statement per file, so a SAFE change can still hold statements no test ran, and hiding them would make the verdict read stronger than it is. An entry marked "excluded": true sits in a # pragma: no cover or if TYPE_CHECKING: block that no test can reach.
coverage.changedStatements: { total, covered } across the whole diff, so you can read the ratio (“12 of 40 changed statements exercised”) rather than only the boolean.
coverage.filesWithUncovered: distinct files with at least one uncovered statement, counted before the cap.
coverage.inertOnlyFiles / coverage.removalOnlyFiles: changed files with nothing for coverage to attest, because their added lines are all blank/comment lines, or because the file only had lines removed.
missingTests: present on UNPROVEN-by-coverage, a concrete hint per uncovered statement, capped at 50. coverage.uncoveredTruncated and missingTestsTruncated ({ shown, total }) appear whenever a list was capped, so a truncated report never reads as a complete one.

Verify in CI

Run verify-diff on a pull request’s diff and let the exit code gate the merge. UNSAFE returns 1 and fails the job; SAFE and UNPROVEN return 0.

A pull request diff running through Refactron as a CI gate, blocking the merge on an UNSAFE verdict — verify-diff as a merge gate: UNSAFE fails the job, SAFE and UNPROVEN pass.

git diff origin/main...HEAD > pr.diff
refactron verify-diff . --diff pr.diff --test-cmd "python3 -m pytest -q"

verify-diff has no --fail-on-unproven flag. To fail CI on UNPROVEN, read the verdict field from --json output and set the exit code yourself; the process exits 0 for UNPROVEN by design, so a green suite on untested lines never silently blocks a merge unless you choose to.

Verdicts

The three verdicts, coverage fusion, and the Python-only limitation in depth.

MCP server

Give your AI agent the same gate as a verify_change tool call.

Safety model

How the shadow tree and the three gates work.

Getting Started

Verification

Concepts

Reference

The verdict, in one line

Install

Build from source (contributors)

Authenticate

Run it

Flags

Exit codes

Unsupported operations

Test files touched

The JSON report

Verify in CI

Next

Verdicts

MCP server

Safety model

​The verdict, in one line

​Install

​Build from source (contributors)

​Authenticate

​Run it

​Flags

​Exit codes

​Unsupported operations

​Test files touched

​The JSON report

​Verify in CI

​Next

Verdicts

MCP server

Safety model

The verdict, in one line

Install

Build from source (contributors)

Authenticate

Run it

Flags

Exit codes

Unsupported operations

Test files touched

The JSON report

Verify in CI

Next