The harness is the product

Every firm asking “which model should we use?” is asking the wrong question. In legal work, accuracy is not a property of a model — it is a property of the scaffolding you build around one. This is a long read on what an evaluation and verification harness actually is, the eight mechanisms by which it makes legal output more accurate, what it cannot fix, and how to build one inside ninety days without buying anything you do not already have.

Start with the observation that makes the rest of this article necessary. Two firms deploy the same model, on the same hardware, for the same task — extracting termination rights from a set of commercial leases. One firm's output is materially more accurate than the other's, consistently, over hundreds of documents. Nothing about the weights explains the difference.

What explains it is everything else: how the task was specified, what was retrieved before the model was asked anything, what the output was required to contain, what happened when the model could not answer, what was checked automatically before a human saw it, what was logged, and whether anyone re-ran the whole thing after the model was updated. That scaffolding is a harness. In practice it is the difference between a demonstration and a capability, and it is where almost all of the engineering — and almost all of the legal-quality gain — actually lives.

§ 01What a harness actually is

A harness is the layer between a fee-earner's question and a model's output. It has seven parts, and the omission of any one of them is a predictable failure. Firms that think they have “an AI tool” usually have parts one and two, and none of the rest.

Two clarifications that prevent a common misunderstanding. First, a harness is task-specific: a harness for lease abstraction does not transfer to disclosure triage, because the output schema, the gold answers and the checks are different. Second, a harness is not the same as a policy. A policy says what staff may do. A harness makes the compliant path the easy one and produces the record that the policy was followed.

§ 02The eight mechanisms that improve legal accuracy

“Improves accuracy” is not a slogan here; each mechanism prevents a specific, nameable failure. Eight of them, with the failure each one removes:

1 · Abstention enforcement removes invented answers

The single largest source of legal error in generative systems is not bad reasoning — it is a fluent answer to a question the documents do not answer. A harness addresses this structurally: the prompt requires refusal when the retrieved passages are silent, the verification layer rejects any output containing a legal proposition with no citation in the retrieved set, and the guardrail converts the refusal into a task. The model is no longer asked to be honest about its limits; the system is built so that dishonesty fails a check.

2 · Citation binding removes uncited assertion

If every proposition must carry a reference to a passage that was actually retrieved, then verifying the output becomes mechanical rather than a re-performance of the work. This is the mechanism that changes supervision economics: a supervisor can check forty citations in the time it takes to read one opinion from scratch. It also catches the most dangerous failure, the plausible-looking citation to an authority that does not exist or says the opposite.

3 · Output schemas remove silent omission

Free prose hides what was missed. A schema — parties, dates, obligations, termination rights, governing law, each with a source reference and a confidence — makes omission visible: an empty field is a question, not a paragraph you did not notice. For volume work this is the difference between “read everything, hope you spot the gap” and “validate the empty cells”.

4 · Scoped retrieval removes wrong-law answers

A model answering from its own memory of contract law is guessing in a suit. Retrieval scoping means the model reasons from the documents and the authorities you selected, with the firm's permissions applied before ranking. It also makes the answer defensible: the file shows what the answer was based on.

5 · Pinned versions remove unrepeatable results

Record the model release, the sampling settings, the prompt version and the ruleset version, and an output becomes reproducible — which means it can be re-run, challenged and defended six months later. An evaluation that does not name a version cannot be reproduced, and an unreproducible evaluation is a memory, not evidence.

6 · Golden sets remove the illusion of quality

The only meaningful measure of accuracy is performance against answers you already know are right. A gold set — fifty to 150 tasks from your own matters with the correct output recorded by an experienced fee-earner before the model sees anything — converts opinion into a number that moves when something changes, up or down.

7 · Two-pass verification removes single-point failure

One model proposing and a second, cheaper check catching is a pattern borrowed from engineering and it works on text: the first pass produces the answer, the second pass answers a narrower question — “does each citation exist and say what is claimed?” Narrow verification is more reliable than broad generation, and it costs a fraction of the first pass.

8 · Regression suites remove silent decay

Models change. Retrieval changes when a firm reorganises its document store. A prompt edit made for one matter quietly breaks another. A harness re-runs the gold set on every change and records the score. This is the mechanism that turns a pilot into a managed capability, and its absence is why so many AI rollouts look excellent in month one and unexplainedly worse by month nine.

Model capability sets the ceiling. The harness sets the floor — and in legal work the floor is what gets reviewed, billed and defended. — the position we take in every deployment review

§ 03One task, through the whole harness

Concrete example. A firm wants every termination right across a set of twenty commercial leases, with the clause text and the page reference, ready for a schedule. Here is what each layer contributes, and what it catches.

Notice what has changed about the human's role. They are no longer reading model output hoping to spot a problem; they are adjudicating flagged items and spot-checking a measured system. That is a different job, and a much smaller one — which is how a twenty-file review becomes days rather than weeks without the evidence standard dropping.

§ 04What a harness cannot fix

The honest section, and the one that matters when a vendor claims their harness solves everything.

• Model capability. A harness can prevent a weak model from asserting, but it cannot make it capable of hard legal reasoning. Below a certain capability level, the honest output is “I cannot do this”, and a harness will faithfully deliver that message — which is a good outcome, not a fix.
• Bad source documents. If the file lacks the verification record, no scaffolding invents it. A harness makes the absence visible; it cannot make it compliant.
• Ambiguous law. Where two reasonable readings exist, the harness can require both to be surfaced, but choosing between them is judgement — and in some cases legal advice, which we do not provide and a model must not appear to.
• Retrieval you got wrong. A harness cannot find the document that was never in scope. It can, and should, report how many passages the answer relied on, which is how you notice that a comprehensive answer rests on four paragraphs.
• Supervision culture. A harness generates the record of a human decision. If nobody makes one, the record documents the absence — which is at least informative when the regulator asks.
• The judgement of the fee-earner. Which is the point: the harness exists to protect the parts of the work that require a person by removing the parts that do not.

§ 05Building one in ninety days

Sequence matters more than tooling. This is the order we use, and the reason for each step.

§ 06What it costs, in the units that matter

The infrastructure costs are the easy part. Running 150 evaluation tasks against a mid-range open model on rented GPU time is a matter of a few GPU hours — the tables in our open-model reviews give per-hour and per-month figures for each class of hardware, including the storage line that billing departments forget. What costs real money is the gold set, and it always will be.

Set against that: the cost of a single inaccurate schedule sent to a client, one mis-cited authority in a skeleton argument, or one file in which nobody can show what was checked. The harness is not a technology purchase. It is the cheapest insurance a firm can write on its own supervision.

§ 07Why this is a governance story, not an engineering one

The regulator's themes and the harness's outputs are the same list. The SRA's August 2026 warning notice on AI misuse centres on supervision and record-keeping rather than technology; the ICO expects assessment, transparency and human oversight where personal data is processed; ISO/IEC 42001 asks for documented controls, evaluation and a review cycle.

A harness is how a firm answers all three with artefacts rather than assurances:

• the gold set and scores are the evidence of monitoring and measurement;
• the verification checks are documented controls, with pass and fail rates;
• the decision log is the supervision record — who confirmed, rejected or escalated what, and when;
• the regression suite is the review cycle, and it runs on evidence of change rather than on a calendar;
• the pinned versions are reproducibility, which is what makes any of the above stand up to a challenge six months later.

This is also the honest answer to the question we are asked most often by COLPs: how do we evidence oversight of a model we did not build and cannot inspect? You do not inspect the model. You inspect the harness, and you check that it is running. A firm with a harness, a score and a decision log can answer that question in one document. A firm with a subscription and good intentions cannot answer it at all.

The short version, for the partner who will read only this paragraph. A model is a component. Accuracy in legal work comes from what you put around it: a task stated so failure is recognisable, retrieval scoped so the model reasons from your documents, checks that reject uncited assertions before a human sees them, a route for “the documents do not answer this”, answers you already know so accuracy is a number, versions pinned so results are reproducible, and a suite that re-runs when anything changes. Build it for one task, in ninety days, with twenty tasks if that is all you can spare. Then do the next task.