Verdicts

Every verification returns exactly one of three verdicts. Two of them exist in most tools. The third, UNPROVEN, is the one that makes Refactron trustworthy.

SAFE

Gates pass and your tests exercise the changed statements.

UNSAFE

A gate failed. The change broke something.

UNPROVEN

Tests pass, but the changed code isn’t proven safe.

How a verdict is decided

Refactron runs two independent checks and fuses them:

The gates: syntax, then imports, then your test suite, all against the change applied in an isolated shadow tree. A gate either passes or fails.
Changed-statement coverage: did your tests actually execute the statements the change touched? Each changed line is mapped to the statement containing it first. Assessed only when the gates pass.

The fusion rule:

Gates	Changed statements covered?	Verdict	Reason
A gate failed	n/a	`UNSAFE`	The failing gate’s blocking reason.
All passed	yes	`SAFE`	”Tests pass and the changed code is covered.”
All passed	no	`UNPROVEN`	”Tests pass, but the changed code is not exercised by any test.”
All passed	couldn’t tell	`UNPROVEN`	”Tests pass, but coverage of the changed code could not be determined.”

SAFE

Every gate passed, and every changed statement your suite could reach was executed by it. The change is on a tested path, and that path is green. SAFE is a per-statement check. Every changed statement your suite could have executed did execute. A file with some exercised and some unexercised changed statements reads as UNPROVEN, and the reason names the ratio:

[UNPROVEN] Tests pass, but only 3 of 14 changed statements were exercised.

Statements coverage.py excluded are subtracted from the count rather than held against you. # pragma: no cover bodies and if TYPE_CHECKING: blocks can never be executed by any test, so requiring them would put SAFE out of reach for any change that adds a typing-only import. They are still listed in coverage.uncovered, tagged excluded, with a hint telling you to review them by hand instead of asking for an impossible test. A change consisting entirely of excluded statements does not reach SAFE: there is nothing a test could have proven about it. SAFE also requires that changed conditionals were fully branched. A changed if/elif whose header ran but one of whose branches no test entered floors at UNPROVEN, even when statement coverage is complete: the behaviour that changed sits in a branch nobody tested. The reason names the line so you know which test to add:

[UNPROVEN] Tests pass, but a changed conditional has a branch no test took
(calc.py:3). Add a test that enters the other branch.

A conditional whose branches are all exercised still reaches SAFE, so this never blocks a fully-tested change. Branch coverage is Python-only, like the rest of the coverage check. This covers statement-level branches — if, elif, while, for/else, match. It does not cover an untested path inside a comprehension filter, a ternary, or a boolean short-circuit (a and b): coverage.py does not report those as branches, so a changed expression of that kind can still read SAFE with a sub-path untested.

`--mutate`: was the change actually asserted?

Coverage proves a changed statement ran. It does not prove any test would fail if that statement’s behaviour changed. A line a test executes but never asserts on counts as covered and earns SAFE. refactron verify-diff --mutate closes that gap. It perturbs the operators and constants in your changed statements — a boundary (<= → <), an arithmetic sign, a boolean and/or, or a constant (a number, a string, True/False/ None) — reruns your suite against each, and if any mutant survives (the suite still passes), the verdict floors at UNPROVEN, naming the survivor:

[UNPROVEN] Tests pass, but a mutant of a changed statement survived
(calc.py:2, + to -): no test failed when its behaviour changed. Add a test
that asserts on it.

It is opt-in and slower — it reruns your suite once per mutant — so it is a deep check, not the default. It is also downgrade-only: a surviving mutant can move SAFE to UNPROVEN, but a clean mutation run never lifts a verdict. An inconclusive mutant (one that times out) is skipped, not held against you. Python-only, and bounded to the statements the diff changed.

This is stricter than it was in 0.4.0. The old rule cleared a whole file as soon as one changed statement in it ran, so a diff changing 40 statements with 1 exercised returned SAFE. If you are comparing against stored reports, a SAFE whose coverage.changedStatements shows covered < total was earned under the old rule.

`--flaky-check`: was the green stable, or lucky?

A default SAFE assumes your tests are deterministic. The suite runs once; a test that passes because of randomness, ordering, timing, or hash-seed dependence — not because behaviour is preserved — still counts toward SAFE. The single-run fast path cannot tell a stable green from a lucky one. refactron verify-diff --flaky-check closes that gap. After a would-be-SAFE verdict, it reruns your suite several times on fresh trees, each under a different PYTHONHASHSEED, and if any test’s outcome varies across the reruns, the green was never stable, so the verdict floors at UNPROVEN, naming the flaky test:

[UNPROVEN] Tests pass, but a test outcome varied across reruns
(tests/test_x.py::test_scale); the green is flaky, not stable. Fix the
flakiness or the verdict cannot be SAFE.

Like --mutate it is opt-in and slower — it reruns your whole suite K times (default 3) — so it is a deep check, not the default. It is also downgrade-only: a varied test can move SAFE to UNPROVEN, but a run where every rerun agrees never lifts a verdict. A rerun that times out is inconclusive (skipped), not held against you. Varying PYTHONHASHSEED catches dict/set order-dependence deterministically; timing, network, and random-based flakes are probabilistic, so K reruns give K chances rather than a guarantee.

Because the default gate runs the suite once, a default SAFE cannot detect a lucky-pass flake. It is an honest verdict on the run it observed. If your suite has any non-determinism, use --flaky-check before trusting a SAFE.

SAFE also requires that the suite was not narrowed. If your testCmd names a subset of the suite, the verdict floors at UNPROVEN however well the changed code is covered, because a green run of the tests you selected says nothing about the tests you did not. See A narrowed test command cannot be SAFE below. SAFE reports still list what they did not prove. coverage.uncovered is always populated, and coverage.changedStatements gives the ratio outright:

"coverage": {
  "changedLinesCovered": true,
  "changedStatements": { "total": 40, "covered": 39 },
  "uncovered": [{ "file": "calc.py", "line": 31, "excluded": true }]
}

Thirty-nine of forty changed statements ran. The fortieth sits in a # pragma: no cover or if TYPE_CHECKING: block that no test can reach, so it is subtracted from what SAFE requires and still listed rather than hidden. That is the only shape of gap a SAFE can now carry: a verdict that concealed it would be easier to read and worth less. A report showing changedLinesCovered: true with a genuinely unexercised statement ("total": 40, "covered": 12 and no excluded flag) was produced by Refactron 0.4.0 or earlier, under the older per-file rule noted above. SAFE requires coverage, and coverage is Python-only. A change that isn’t entirely Python can’t reach SAFE today.

UNSAFE

A gate rejected the change:

Syntax: the changed file no longer parses.
Imports: an import in the changed content doesn’t resolve, or a previously-resolving import now fails.
Tests: your suite went red on the change.

UNSAFE exits 1, and a failing test gate prints the test output so you can see what broke. This is a real signal that the change is wrong, not merely UNPROVEN.

One case looks like a test failure but isn’t a real one: if your suite is already red before the change, or no test runner is detected, Refactron can’t blame the change for anything, so it returns UNPROVEN, not UNSAFE. A broken baseline is a “can’t prove it” situation, not evidence the diff broke something.

UNPROVEN

The gates passed, but Refactron won’t claim the change is safe, because the evidence isn’t there. This is the verdict no other gate gives you honestly. There are two ways to land here:

The changed code isn’t exercised. Your tests pass, but none of them run the code you changed. A green suite tells you nothing about untested lines. Refactron lists each uncovered statement and, in the JSON report, a missingTests hint:
"missingTests": [ { "file": "calc.py", "hint": "add a test exercising calc.py:14" } ]
Coverage couldn’t be assessed. The change isn’t entirely Python, or coverage.py isn’t installed. Refactron can’t measure whether the changed statements ran, so it declines to certify SAFE. The reason reads “coverage of the changed code could not be determined.”

UNPROVEN exits 0. It is a warning, not a rejection: nothing is known to be broken, but nothing is proven either. The right response is to add the missing test, then re-run and earn SAFE.

UNPROVEN turns your test suite’s blind spots into a to-do list. Every uncovered statement is a test you could write to move the verdict to SAFE.

Coverage is judged per statement, not per line

coverage.py records execution against the first line of a statement. A statement wrapped across several lines, which is what any formatter produces, has continuation lines, closing brackets, and trailing commas that coverage.py never marks at all. Refactron therefore maps each changed line to the statement that contains it, using the Python AST, before judging it. A changed continuation line counts as exercised when the statement it belongs to ran, and one unexercised multi-line statement produces one entry, at the line you would actually write a test against, not one entry per physical line. Without this, a reformat that only rewraps code reports every wrapped line as uncovered: a black run over 28 files once produced 3666 such entries, almost all of them for code that provably executed. Containment is the load-bearing word. The cheaper version of this idea, “walk back to the nearest statement start at or above the changed line,” is wrong in a way that manufactures false SAFE verdicts: it cannot tell a continuation line of that statement from a blank line, a comment, or a dead-branch line that merely follows it and belongs to somewhere else entirely. Under that rule an executed def vouches for a body that never ran. Real extents from the AST answer the question exactly.

Blank lines and comments prove nothing, and are asked to prove nothing

A changed line carrying no code at all, a blank line or a comment-only line, is inert. It cannot change behavior, so Refactron never reports it as uncovered; and it cannot be exercised by a test, so it never counts toward a file’s coverage either. Formatters move blank lines constantly, and a mechanism that let them vouch for their neighbours would turn every reformat into a free SAFE. A file whose changed lines are all inert has nothing to attest, and gets its own reason rather than a pass:

[UNPROVEN] Tests pass. The change only touches comments and blank lines; there are no added
statements for coverage to attest.

That is deliberately conservative. A diff exposes only the lines it adds, so “every added line is inert” is not the same claim as “this file is unchanged”: a deleted statement next to a moved blank line looks identical from here. Removal-only files are treated the same way and for the same reason. A docstring is a real statement in Python, so a docstring-only edit lands on its own statement and is judged like any other. coverage.py does not track function docstrings, so such a change typically reads as unexercised, which is the honest answer rather than a convenient one.

Code your suite is not allowed to reach

Some statements can never be exercised, by design. A # pragma: no cover block, or an import under if TYPE_CHECKING:, is excluded from coverage.py’s judgement and never executes under test. A diff that touches only such code therefore cannot reach SAFE, no matter how good your suite is: there is no execution to observe, and Refactron will not certify what it did not see. These entries are marked, and their hints say what is actually true instead of asking for a test that cannot exist:

"coverage": {
  "uncovered": [{ "file": "gated.py", "line": 6, "excluded": true }]
},
"missingTests": [
  {
    "file": "gated.py",
    "hint": "gated.py:6 is excluded from coverage (e.g. `# pragma: no cover`, `if TYPE_CHECKING:`), so no test can exercise it; review this change by hand"
  }
]

The right response is a human review of that hunk, not a new test. Large diffs are capped so the report stays readable. When Refactron truncates, it says so rather than shipping a short list that looks complete:

"coverage": {
  "uncoveredTruncated": { "shown": 200, "total": 412 },
  "filesWithUncovered": 37
},
"missingTestsTruncated": { "shown": 50, "total": 412 }

Each file is guaranteed a share of the list before any file takes a second helping, so one pathological file cannot consume every slot and push later files out of the report entirely. filesWithUncovered counts distinct files before the cap, so you can always tell whether the list you are reading spans the whole diff.

The Python-only limitation

Coverage fusion depends on coverage.py, so it is Python-only today:

A diff where every changed file is .py, with coverage.py available, can be assessed, and can reach SAFE.
A diff touching any TypeScript (or any non-Python) file, or run without coverage.py, returns UNPROVEN with coverage.tool: "none". The gates still run; only the coverage half is unavailable.

Refactron never guesses here. Reporting a non-Python change as “covered” would let an unverified change through as SAFE: a false SAFE, which the engine forbids. When it can’t measure, it says UNPROVEN.

Coverage cannot see subprocesses

Coverage fusion measures the lines your test process executes. Code that runs only in a child process, launched through subprocess.run, a multiprocessing worker, or a spawned server, is invisible to coverage.py unless you wire up subprocess coverage yourself: a COVERAGE_PROCESS_START environment variable plus a coverage.process_startup() call in sitecustomize. Without that wiring, a change whose only exercise happens inside a subprocess reads as UNPROVEN. The gates still pass, but the changed statements never register as covered, so Refactron declines to certify SAFE. This is the honest result, not a defect. First-class subprocess coverage is a planned fast-follow; until it lands, exercise the changed code in-process in at least one test so the verdict can reach SAFE.

SAFE means suite-approved, not proven correct

SAFE says your suite ran the changed code and stayed green. It does not claim the change is correct in some absolute sense: it inherits exactly what your suite checks. That includes side effects your suite never observes, an extra or missing database write, an email that stopped sending, a log a downstream system parses: Refactron sees a change only through your tests, so behavior no test watches is outside the verdict. Audits of AI agent patches (for example on SWE-bench) find that a substantial share of patches which pass the project’s tests are still wrong, because the suite was too weak to catch the defect. Refactron cannot turn a weak suite into a strong one, but it refuses to overstate what a green run proves. That is why UNPROVEN exists, and why every uncovered statement ships with a missingTests hint: the path to a SAFE you can trust is a suite that actually exercises the behavior you care about. A worked example from our own hardening runs, on a real library. Jinja2’s truncate filter guards its early return with if len(s) <= length + leeway. Change that <= to < and the behavior genuinely changes: at the exact boundary the string is now truncated instead of returned whole. All 911 tests in Jinja2’s suite still pass, because none of them lands on that precise boundary, and the changed line is covered, so the verdict is SAFE. That verdict is correct about what it claims (the suite ran this line and stayed green) and it is still not a proof of correctness. Boundary conditions are exactly where suites tend to be thin. Read SAFE as “your tests approve this change”, then decide separately whether your tests are strong where this change lives.

A narrowed test command cannot be SAFE

If you pass a testCmd that names a subset of the suite, the verdict floors at UNPROVEN. Coverage can report the changed code as fully exercised while the one test that would have caught the change was never selected.

What this check can and cannot see. Refactron reads the command string, the environment, and your pytest configuration. It recognises pytest, unittest, vitest and jest and their common flags — but not every flag of every plugin. A command carrying an option it does not recognise is reported unknown rather than full, and unknown does not floor the verdict. Treat this as a strong check on the shapes it knows, not a guarantee that no narrowing can ever reach SAFE. If you need certainty, run the bare command.

A worked example. One file, one change (return x * 2 to return x * 3), and two tests: test_scale executes the changed line without pinning its value, test_report pins it and fails. Only the command differs:

# runs both tests; test_report catches the change
python3 -m pytest -q                       # UNSAFE

# runs only the weak test; the change sails through
python3 -m pytest -q tests/test_scale.py   # UNPROVEN (previously SAFE)

Every changed statement executed in the second run, so coverage alone cannot tell the two apart. The scope of the run is what distinguishes them. Refactron classifies your command into three buckets, and reports the result on testScope:

Scope	Meaning	Can be SAFE
`full`	Parsed, and names no filter	yes
`narrowed`	Names a path, node id, `-k`, `-m`, `-t`, `--onlyChanged`, …	no
`unknown`	Not recognised (`make test`, `npm test`, a flag on a wrapper script)	yes

Naming a directory counts as narrowing, including pytest tests/. Tests can live outside any single directory, so Refactron cannot treat that as the whole suite without risking a false SAFE. Run the bare command (pytest -q) to get a SAFE-eligible verdict. pytest . is fine: . collects from the root, so it is never narrower than the default. For unittest the canonical whole-suite form is different, and Refactron treats it accordingly: python3 -m unittest discover -s tests is full. Bare discover starts from the current directory, so pointing it at the test directory is how a complete unittest run is normally written, not a narrowing of one. Naming a module, class or method (python3 -m unittest tests.test_scale) is narrowed. A run that executes no tests is narrowing too. --collect-only and --help exit successfully while selecting zero tests, and they still import your test modules, so coverage can mark module-level changed lines as executed. Those are classified narrowed. unknown does not floor the verdict. Most commands that land there are a full suite carrying a flag we do not recognise (pytest --doctest-modules), and refusing SAFE for every unrecognised plugin flag would be worse than the gap it closes. The CLI says so explicitly rather than staying quiet:

  note: could not determine whether the test command runs the whole suite
        (make is not a recognised test runner, so the scope is unknown);
        the verdict assumes it does.

Refactron recognises pytest, unittest, vitest and jest by name, so python -m unittest tests.test_foo and python3 tests/runtests.py auth are both narrowed. A flag on an unrecognised script stays unknown and is not floored: python3 tests/runtests.py --parallel 4 could be a whole suite, and we do not know that flag well enough to guess. An exported PYTEST_ADDOPTS (or VITEST_ADDOPTS / JEST_ADDOPTS) is seen: its value is scanned with the same rules as the command line, and it is checked against the runner that variable actually feeds, so a PYTEST_ADDOPTS left in your shell will not floor a vitest project. Narrowing configured in your pytest configuration is also seen. addopts and testpaths are read from pytest.ini, tox.ini, setup.cfg and pyproject.toml, and addopts is scanned with the same rules as the command line — so addopts = -q --strict-markers stays full, while addopts = -k foo does not. testpaths always counts as narrowing: it restricts collection, and whether the restriction excludes anything cannot be known without collecting. Only the repository root is read. pytest walks up to find its rootdir, but your suite runs inside an isolated copy whose parent is a temporary directory, so a config file above your repository does not reach the run being judged. A vitest include or a jest testMatch is still not seen. Those are JavaScript and would have to be executed rather than parsed. In practice they cannot reach a false SAFE today, because coverage is Python-only and a JavaScript or TypeScript change already caps at UNPROVEN. full therefore means “no filter was found in the command, the environment, or your pytest config”. It is still not a proof that the whole suite ran: a runner plugin or a conftest.py can deselect tests in ways nothing here inspects. The PYTHONPATH= prefix recommended below is not narrowing. PYTHONPATH=. python3 -m pytest -q classifies as full.

Make sure the tests run the code being verified

Refactron verifies a change in an isolated copy of your project. If your tests import the package from somewhere else, they will exercise the original code and the run proves nothing about your change. The common cause is a project installed into the environment, including an editable install (pip install -e .), because import yourpackage then resolves to the installed location rather than the copy under verification. We hit this on Django: the same diff read UNPROVEN when the tests loaded the installed copy, and UNSAFE (correctly, since the auth suite catches the change) once the verified copy came first on sys.path. Refactron detects the case rather than guessing: if a changed file is never even measured by coverage, the verdict reports that coverage could not be determined instead of claiming the code is untested, and the report carries the reason and this remedy in coverage.unknownReason. To get a real verdict, make the verified tree win on sys.path. Either export PYTHONPATH=. before you run Refactron, or prefix the test command itself:

PYTHONPATH=. python3 -m pytest -q      # or PYTHONPATH=src for a src layout

Both forms work from the CLI. Over MCP only the prefix form is available, because the client spawns refactron-mcp itself and there is no shell of yours to export into. Relative paths are resolved from the copy being verified, which is what you want here. Write the command in module form (python3 -m pytest) rather than as a bare console script (pytest). Coverage has to run the same program the test gate ran, and a console script is only runnable under coverage when it resolves to a Python file. It does not on Windows, where console scripts are native .exe launchers, nor under pyenv, asdf or nix, which install shell shims. Refactron declines to measure rather than measure a different program, so a bare console script reports that coverage could not be determined on those setups. Module form is measurable everywhere.

Exit codes

The verdict maps to a process exit code so it can gate CI directly:

Verdict	Exit code
`SAFE`	`0`
`UNPROVEN`	`0`
`UNSAFE`	`1`

Both SAFE and UNPROVEN pass, so a green suite on untested lines never silently blocks a merge. To fail CI on UNPROVEN too, read the verdict field from the JSON report and decide for yourself.

Bad input (a diff that doesn’t apply, a missing flag) exits 2; an unauthenticated CLI run exits 7. Those are operational errors, not verdicts.

Getting Started

Verification

Concepts

Reference

SAFE

UNSAFE

UNPROVEN

How a verdict is decided

SAFE

`--mutate`: was the change actually asserted?

`--flaky-check`: was the green stable, or lucky?

UNSAFE

UNPROVEN

Coverage is judged per statement, not per line

Blank lines and comments prove nothing, and are asked to prove nothing

Code your suite is not allowed to reach

The Python-only limitation

Coverage cannot see subprocesses

SAFE means suite-approved, not proven correct

A narrowed test command cannot be SAFE

Make sure the tests run the code being verified

Exit codes

SAFE

UNSAFE

UNPROVEN

​How a verdict is decided

​SAFE

​--mutate: was the change actually asserted?

​--flaky-check: was the green stable, or lucky?

​UNSAFE

​UNPROVEN

​Coverage is judged per statement, not per line

​Blank lines and comments prove nothing, and are asked to prove nothing

​Code your suite is not allowed to reach

​The Python-only limitation

​Coverage cannot see subprocesses

​SAFE means suite-approved, not proven correct

​A narrowed test command cannot be SAFE

​Make sure the tests run the code being verified

​Exit codes

How a verdict is decided

SAFE

`--mutate`: was the change actually asserted?

`--flaky-check`: was the green stable, or lucky?

UNSAFE

UNPROVEN

Coverage is judged per statement, not per line

Blank lines and comments prove nothing, and are asked to prove nothing

Code your suite is not allowed to reach

The Python-only limitation

Coverage cannot see subprocesses

SAFE means suite-approved, not proven correct

A narrowed test command cannot be SAFE

Make sure the tests run the code being verified

Exit codes