The test rig that automatically checks an AI system’s output against a fixed set of examples every time you change something. For agencies, it is what turns a clever prompt into a workflow you can trust to run without someone reading every result.
Also known as eval harness, test harness, evaluation scaffold
An evaluation harness is the code and scoring rules that sit around an AI system and grade it. You give it a set of inputs with known good answers, it runs the model or agent against all of them, and it reports a score. Change the prompt, swap the model, add a new step, and you re-run the harness to find out whether you made things better or quietly broke something that used to work.
The distinction worth holding onto is that an eval is a single test and the harness is the machinery that runs hundreds of them on demand. Term coinage around loop engineering in mid-2026 pushed harness work from a research nicety into the thing practitioners point at when they explain why one team’s agent is reliable and another team’s is not.
Most agency AI work never gets tested, which is why so much of it stalls at the pilot stage.
It replaces vibes with a number. Right now the standard method for deciding whether an AI workflow is good enough is that someone senior skims twenty outputs and says it feels fine. A harness gives you a score you can defend in a room, and a score you can watch move when you change something.
It makes model swaps survivable. Model providers ship new versions constantly, and behavior shifts with them. Without a harness, an upgrade is a coin flip. With one, you re-run the suite in ten minutes and know exactly which parts of your workflow degraded.
It protects the client relationship. The risk in automated creative and copy work is not that it fails loudly, it is that it drifts off-brand slowly. A harness that includes brand and tone checks catches drift before a client does.
A studio has an agent that drafts paid social variants from a brief. Before it goes near a live account, the team assembles fifty past briefs paired with the copy that actually shipped and approved, then writes scoring rules: does the output hit the character limits for each placement, does it avoid the client’s banned claims list, does it include the mandated legal line, and does a cheap model rate it as on-brand against the tone guide. That set becomes the harness. When someone rewrites the system prompt to fix a recurring problem with headlines, they re-run all fifty and see that headlines improved but the legal line started dropping out on two placements. They fix it before anyone outside the studio ever sees a draft.
The automations and agents module of the workshop teaches you how to build AI workflows that compress the busywork without taking the craft out of the studio.