AI Harnesses & Evaluation
A prompt or an agent that "seems better" after a change is an opinion. A harness turns it into a measurement: the same inputs, the same scoring, run again after every change, so an improvement is something you can demonstrate rather than something you believe.
How I use it
Testing prompt variants against each other
Built a harness for running variants over a fixed input set and comparing results, rather than editing a prompt and assuming it improved. A meaningful share of confident prompt changes turn out to be neutral or worse once they are actually measured.
Held-out inputs
Evaluating on the same examples used to tune something measures memorisation, not capability. Keeping inputs back is the only way to know whether a change generalises or was fitted to the cases in front of you.
Gates in the pipeline
Output that reaches a customer clears a check first. Without one you are not running an AI feature, you are publishing whatever the model produced that day and finding out from the customer.
Using an existing harness versus building one
Off-the-shelf evaluation covers the general cases well. It stops being enough when the thing you need to score is specific to your domain — at that point the scoring function is the hard part, and it has to be built.
What I use
- Claude API
- TypeScript
- GitHub Actions
Related skills
Want the detail behind any of this?
I can walk through any of it in as much depth as you want.