Skip to main content
Every verification returns exactly one of three verdicts. Two of them exist in most tools. The third, UNPROVEN, is the one that makes Refactron trustworthy.

SAFE

The SAFE verdict markGates pass and your tests exercise the changed statements.

UNSAFE

The UNSAFE verdict markA gate failed. The change broke something.

UNPROVEN

The UNPROVEN verdict markTests pass, but the changed code isn’t proven safe.

​
How a verdict is decided

Refactron runs two independent checks and fuses them:
  1. The gates: syntax, then imports, then your test suite, all against the change applied in an isolated shadow tree. A gate either passes or fails.
  2. Changed-statement coverage: did your tests actually execute the statements the change touched? Each changed line is mapped to the statement containing it first. Assessed only when the gates pass.
The fusion rule:

​
SAFE

Every gate passed, and at least one changed statement in every changed file was executed by your test suite. The change is on a tested path, and that path is green. SAFE is a per-file check, not a per-statement one. It means every changed file had at least one changed statement run by your suite; it does not mean every changed statement ran. A file with some exercised and some unexercised changed statements still reads as covered, so SAFE can leave individual statements unexercised. Requiring all changed statements, not one per file, is the roadmap item here. Until then, read SAFE as “the change is on a tested path,” not “every changed statement is tested.” SAFE reports still list what they did not prove. coverage.uncovered is always populated, and coverage.changedStatements gives the ratio outright:
Twelve of forty changed statements ran. That is a real SAFE under the per-file rule and also a clear picture of its limits. A verdict that hid the other twenty-eight would be easier to read and worth less. SAFE requires coverage, and coverage is Python-only. A change that isn’t entirely Python can’t reach SAFE today.

​
UNSAFE

A gate rejected the change:
  • Syntax: the changed file no longer parses.
  • Imports: an import in the changed content doesn’t resolve, or a previously-resolving import now fails.
  • Tests: your suite went red on the change.
UNSAFE exits 1, and a failing test gate prints the test output so you can see what broke. This is a real signal that the change is wrong, not merely UNPROVEN.
One case looks like a test failure but isn’t a real one: if your suite is already red before the change, or no test runner is detected, Refactron can’t blame the change for anything, so it returns UNPROVEN, not UNSAFE. A broken baseline is a “can’t prove it” situation, not evidence the diff broke something.

​
UNPROVEN

The gates passed, but Refactron won’t claim the change is safe, because the evidence isn’t there. This is the verdict no other gate gives you honestly. There are two ways to land here:
  • The changed code isn’t exercised. Your tests pass, but none of them run the code you changed. A green suite tells you nothing about untested lines. Refactron lists each uncovered statement and, in the JSON report, a missingTests hint:
  • Coverage couldn’t be assessed. The change isn’t entirely Python, or coverage.py isn’t installed. Refactron can’t measure whether the changed statements ran, so it declines to certify SAFE. The reason reads “coverage of the changed code could not be determined.”
UNPROVEN exits 0. It is a warning, not a rejection: nothing is known to be broken, but nothing is proven either. The right response is to add the missing test, then re-run and earn SAFE.
UNPROVEN turns your test suite’s blind spots into a to-do list. Every uncovered statement is a test you could write to move the verdict to SAFE.

​
Coverage is judged per statement, not per line

coverage.py records execution against the first line of a statement. A statement wrapped across several lines, which is what any formatter produces, has continuation lines, closing brackets, and trailing commas that coverage.py never marks at all. Refactron therefore maps each changed line to the statement that contains it, using the Python AST, before judging it. A changed continuation line counts as exercised when the statement it belongs to ran, and one unexercised multi-line statement produces one entry, at the line you would actually write a test against, not one entry per physical line. Without this, a reformat that only rewraps code reports every wrapped line as uncovered: a black run over 28 files once produced 3666 such entries, almost all of them for code that provably executed. Containment is the load-bearing word. The cheaper version of this idea, “walk back to the nearest statement start at or above the changed line,” is wrong in a way that manufactures false SAFE verdicts: it cannot tell a continuation line of that statement from a blank line, a comment, or a dead-branch line that merely follows it and belongs to somewhere else entirely. Under that rule an executed def vouches for a body that never ran. Real extents from the AST answer the question exactly.

​
Blank lines and comments prove nothing, and are asked to prove nothing

A changed line carrying no code at all, a blank line or a comment-only line, is inert. It cannot change behavior, so Refactron never reports it as uncovered; and it cannot be exercised by a test, so it never counts toward a file’s coverage either. Formatters move blank lines constantly, and a mechanism that let them vouch for their neighbours would turn every reformat into a free SAFE. A file whose changed lines are all inert has nothing to attest, and gets its own reason rather than a pass:
That is deliberately conservative. A diff exposes only the lines it adds, so “every added line is inert” is not the same claim as “this file is unchanged”: a deleted statement next to a moved blank line looks identical from here. Removal-only files are treated the same way and for the same reason. A docstring is a real statement in Python, so a docstring-only edit lands on its own statement and is judged like any other. coverage.py does not track function docstrings, so such a change typically reads as unexercised, which is the honest answer rather than a convenient one.

​
Code your suite is not allowed to reach

Some statements can never be exercised, by design. A # pragma: no cover block, or an import under if TYPE_CHECKING:, is excluded from coverage.py’s judgement and never executes under test. A diff that touches only such code therefore cannot reach SAFE, no matter how good your suite is: there is no execution to observe, and Refactron will not certify what it did not see. These entries are marked, and their hints say what is actually true instead of asking for a test that cannot exist:
The right response is a human review of that hunk, not a new test. Large diffs are capped so the report stays readable. When Refactron truncates, it says so rather than shipping a short list that looks complete:
Each file is guaranteed a share of the list before any file takes a second helping, so one pathological file cannot consume every slot and push later files out of the report entirely. filesWithUncovered counts distinct files before the cap, so you can always tell whether the list you are reading spans the whole diff.

​
The Python-only limitation

Coverage fusion depends on coverage.py, so it is Python-only today:
  • A diff where every changed file is .py, with coverage.py available, can be assessed, and can reach SAFE.
  • A diff touching any TypeScript (or any non-Python) file, or run without coverage.py, returns UNPROVEN with coverage.tool: "none". The gates still run; only the coverage half is unavailable.
Refactron never guesses here. Reporting a non-Python change as “covered” would let an unverified change through as SAFE: a false SAFE, which the engine forbids. When it can’t measure, it says UNPROVEN.

​
Coverage cannot see subprocesses

Coverage fusion measures the lines your test process executes. Code that runs only in a child process, launched through subprocess.run, a multiprocessing worker, or a spawned server, is invisible to coverage.py unless you wire up subprocess coverage yourself: a COVERAGE_PROCESS_START environment variable plus a coverage.process_startup() call in sitecustomize. Without that wiring, a change whose only exercise happens inside a subprocess reads as UNPROVEN. The gates still pass, but the changed statements never register as covered, so Refactron declines to certify SAFE. This is the honest result, not a defect. First-class subprocess coverage is a planned fast-follow; until it lands, exercise the changed code in-process in at least one test so the verdict can reach SAFE.

​
SAFE means suite-approved, not proven correct

SAFE says your suite ran the changed code and stayed green. It does not claim the change is correct in some absolute sense: it inherits exactly what your suite checks. Audits of AI agent patches (for example on SWE-bench) find that a substantial share of patches which pass the project’s tests are still wrong, because the suite was too weak to catch the defect. Refactron cannot turn a weak suite into a strong one, but it refuses to overstate what a green run proves. That is why UNPROVEN exists, and why every uncovered statement ships with a missingTests hint: the path to a SAFE you can trust is a suite that actually exercises the behavior you care about. A worked example from our own hardening runs, on a real library. Jinja2’s truncate filter guards its early return with if len(s) <= length + leeway. Change that <= to < and the behavior genuinely changes: at the exact boundary the string is now truncated instead of returned whole. All 911 tests in Jinja2’s suite still pass, because none of them lands on that precise boundary, and the changed line is covered, so the verdict is SAFE. That verdict is correct about what it claims (the suite ran this line and stayed green) and it is still not a proof of correctness. Boundary conditions are exactly where suites tend to be thin. Read SAFE as “your tests approve this change”, then decide separately whether your tests are strong where this change lives.

​
Make sure the tests run the code being verified

Refactron verifies a change in an isolated copy of your project. If your tests import the package from somewhere else, they will exercise the original code and the run proves nothing about your change. The common cause is a project installed into the environment, including an editable install (pip install -e .), because import yourpackage then resolves to the installed location rather than the copy under verification. We hit this on Django: the same diff read UNPROVEN when the tests loaded the installed copy, and UNSAFE (correctly, since the auth suite catches the change) once the verified copy came first on sys.path. Refactron detects the case rather than guessing: if a changed file is never even measured by coverage, the verdict reports that coverage could not be determined instead of claiming the code is untested, and the report carries the reason and this remedy in coverage.unknownReason. To get a real verdict, make the verified tree win on sys.path. Either export PYTHONPATH=. before you run Refactron, or prefix the test command itself:
Both forms work from the CLI. Over MCP only the prefix form is available, because the client spawns refactron-mcp itself and there is no shell of yours to export into. Relative paths are resolved from the copy being verified, which is what you want here. Write the command in module form (python3 -m pytest) rather than as a bare console script (pytest). Coverage has to run the same program the test gate ran, and a console script is only runnable under coverage when it resolves to a Python file. It does not on Windows, where console scripts are native .exe launchers, nor under pyenv, asdf or nix, which install shell shims. Refactron declines to measure rather than measure a different program, so a bare console script reports that coverage could not be determined on those setups. Module form is measurable everywhere.

​
Exit codes

The verdict maps to a process exit code so it can gate CI directly: Both SAFE and UNPROVEN pass, so a green suite on untested lines never silently blocks a merge. To fail CI on UNPROVEN too, read the verdict field from the JSON report and decide for yourself.
Bad input (a diff that doesn’t apply, a missing flag) exits 2; an unauthenticated CLI run exits 7. Those are operational errors, not verdicts.