Instrument Review

People accept what an AI judge, scanner or code reviewer says without seeing inside it. Instrument Review measures that instrument independently, for one named decision, on cases where the right answer is already known, so the people who rely on it know how often it is wrong and where it stops being dependable.

Who it is for

• Teams that build an AI judge, scanner or reviewer that customers rely on, and want it measured by someone else.
• Firms whose reports or assessments rest on their own automated tools.
• Platform and security teams choosing between instruments before they commit to one.

What we measure

• How often the instrument passes something that is wrong, and fails something that is right.
• Where it answers when it should have said it could not tell.
• How its verdict changes with the evidence and input it is given, including inputs longer than it can read.
• Where you hold reference judgments you trust, how closely it agrees with them.
• The limits of the result: which cases, settings and versions it covers, and which it does not.

How it runs

1. We agree the decision the instrument informs and what counts as a wrong answer for it.
2. We build cases where the right answer is known, and freeze the cases, our written predictions and the scoring before anything runs.
3. We run the instrument at pinned versions and settings, and produce a record a third party can rerun.
4. You see the results first and choose whether they are published, not what they say.

The result is evidence about one instrument, for one decision, on the cases tested. It is not a certification of the instrument and not a general ranking.

Published examples

Our public studies use the same method, with every number traceable to its record:

• A commercial AI judge (Jev, TypeSafe AI): passed 9 in 10 false reports across 430 test cases; showing it the evidence helped with only one kind of error. Three of our five written predictions were wrong, and we published anyway.
• An open model (Laya, Convai Innovations): cleared 84 to 88 in 100 false reports even with the evidence attached; most inputs exceeded its context window, so it judged only the start of each report.
• Reading is not checking: a model judge and a grammar with no model (built by Barg Labs) both missed false claims that were not found in the prose; once a claim was isolated, both mostly checked it.

Price

From C$12,500, plus applicable taxes. The fee depends on the amount of work, mainly the cases to build and the instruments and settings to test, and is fixed in writing before work starts. Canadian dollars. Price list version 2026-10-27.1.

Start

Email team@barglabs.ai with the instrument, the decision it informs, and what you would count as a wrong answer. If you are accepting code rather than a tool, see the Repository Evidence Review.