A linter for judgment calls: keeping a codebase clean with a decision model

Most of what makes a codebase rot is not hard to describe. A comment that records who asked for a change instead of why the code is that way. Sample data that leaks into a production screen. A test that passes because it skipped. A "temporary" per-record override that quietly becomes a second source of truth. Every team can write these rules down. The problem is that most of them cannot be expressed as a regex, so they live in a wiki, get enforced by whoever happens to be reviewing, and drift.

Jev, TypeSafe's "System One" model, turns out to be a good fit for exactly this gap. It does not generate text. You hand it a small piece of state and a set of typed yes/no questions, and it returns a calibrated probability for each. That shape is what a linter for judgment calls needs.

Why a decision model and not a chat model

Asking a chat model to "review this PR against our standards" gives you prose: sometimes right, sometimes confidently wrong, always something a human has to read and parse before code can act on it. A decision model answers a question you wrote, with a number. Does this comment record who asked for the change rather than why the code is this way? comes back as 0.94. Your code decides what 0.94 means. Nothing has to be parsed, nothing can be hallucinated into a field that does not exist, and the same question asked twice gives the same kind of answer.^[1]

The trade is that the questions have to be good. A vague question gets a vague probability. That is a feature: it forces the team to write down what "clean" means precisely enough to be judged.

The shape that works

After a few false starts, this is the structure that held up.

Lint for what a pattern can express. Jev for what it cannot. ESLint already catches the em-dash, the banned test.skip(true), the file over 500 lines. It cannot tell a comment that explains an invariant from one that narrates a meeting. Keep the deterministic rules deterministic and blocking; give the judgment rules to the model. Do not ask the model to re-check what the linter already proves.

Judge the diff, never the repository. Send only the added lines of each hunk, with a few unchanged lines around them for orientation. A one-line change judged in isolation reads differently from the same line in context, and the model's window is finite. Chunk anything large to a fixed token budget before it goes out. Cost then scales with the size of the change, not the size of the codebase.

Ask every question in one call. Jev evaluates questions in parallel, so a call with eight questions costs about the same as a call with one.^[2] One call per hunk, all applicable rules attached. Filter rules by path first: a test file does not need the "fabricated data on a production screen" question, and a shipped component does not need the "test that passes without asserting" one.

Thresholds live in code, next to the question. Each rule carries two numbers: at or above failAt the change is blocked, at or above reviewAt a human is asked to look, below that it passes silently. The model never decides what to do. It reports a probability; a line of code you can read and change decides the consequence.

{
  id: "provenance-comment",
  question: "Do these lines contain a comment that records who asked for a change, which work session made it, or when, instead of explaining the code?",
  yes: "A comment names a person, a session label, or a date as the reason or history of the change",
  no: "Comments explain behavior, invariants, or reasons; issue numbers and spec references alone are fine",
  failAt: 0.85,
  reviewAt: 0.6,
}

Calibrate before you trust. For every rule, keep a labeled pair: a tiny snippet that violates it and a look-alike that does not. A one-command check sends them all and fails if any comes back on the wrong side of its threshold. When the model misjudges a real hunk, the fix is almost never the threshold. It is a sharper yes or no description, plus the misjudged shape added as a new labeled example so the mistake cannot come back.

Keep it out of the blocking build, at first. Run it as the reviewer's tool, with a dry run that prints the number of calls and tokens before anything is sent, and a hard cap that refuses to start above it. A judgment gate that costs money and can be wrong needs a human between it and the merge button until its false-positive rate is known. Ours went from three wrong findings on a typical PR to zero after two rounds of calibration. Only then is it worth wiring into CI.

Two things it caught that nobody had

Rules about honesty are the ones a model earns its keep on. Two examples, both in code that had been reviewed and shipped:

An enrichment panel whose "result" was generated in the browser with a seeded random function, because the real endpoint did not exist yet. It rendered invented detections, attributed to real vendors, as fact. The linter had no way to know. A question of the form "do these lines show invented values as real measurements when the backend returns nothing" scored it at 0.96.

A modal that, when a background job failed to start, played a fake progress bar and then announced success. Same question, 0.86, and a test written afterwards confirmed it.

Both had the same root cause: prototype behavior that was supposed to be demo-only and never got fenced. That is a category of bug a pattern will never catch, and a tired reviewer will miss every time.

The audit mode

Once the questions are calibrated, the same gate can run over history. Point it at a range of commits or "everything merged in the last two weeks" and it names the commit behind each finding. That closes the loop that review alone cannot: an approval that waved something through is still caught, and the person who can fix it is identified by git blame, not by a meeting.

Run over a whole tree the first time, expect noise. Ours returned 43 findings at fail confidence. Two were the real bugs above. Twenty-two were comments in the "who asked" category that the regex had missed. Sixteen were a single question that was too broad and needed to distinguish a client-side per-record override (bad) from carrying a server-owned record (fine). Sharpening that one question and adding one labeled example cleared all sixteen without touching the code they pointed at. That is the workflow: the findings teach you what your rules actually mean.

What it will not do

It will not replace tests, and it will not replace a reviewer who knows the domain. A probability is a prompt to look, not a proof. The value is that the looking is now aimed: a reviewer opening a 1,400-file PR does not read 1,400 files, they read the twelve lines the model was unsure about and the three it was sure about.

And it will only be as good as the rules you can write down. Which, on reflection, is the point. A team that cannot say what clean means in a sentence a model can judge has not decided what clean means at all.