Evaluation Harness

Introducing Iris, an AI evaluator that tests your agents in minutes and fixes what fails by updating their prompts.

Request AccessWatch How It Works →

100K+
LLM-judge evaluations run
50K+
Test cases generated for evaluation datasets
94%
LLM Judge agreement with your domain experts
Demo

A Full Evaluation, Start to Finish

From a request in plain language to a calibrated judge, in eight steps.

Custom Rubrics

Define the criteria that matter for your domain, so a score means what your reviewers mean by it.

Regression Runs

Re-run the same graded cases after a change and see exactly what moved.

Continuous Evaluation

Evaluators score new production traces as they arrive, between test runs.

Judge Calibration

Score a sample blind, chosen at random or by the agent, until the judge agrees with your experts.

Iris

What Iris Does

It reads every run your agents produce and names exactly where they broke: a failed tool call, a knowledge source that came back empty, a sub-agent that errored or a context window that dropped the evidence. What matters becomes a graded set the next version has to pass.

  • Shows Its WorkEvery tool call and reasoning step is one click away.
  • Asks Before It Changes AnythingNothing is written until you say yes.
  • Scales Your Reviewers' JudgmentExpert-level scoring on every run, at a fraction of the cost.
Three specialized sub-agents, one conversation
IrisPlans the work and owns every write

Dataset Writer

Writes the test set, every category at once.

Score Analyst

Groups the failures, names the pattern, and drafts the fix.

Trace Analyst

Pulls the trace behind any case you question.

Run phases
The seven phases of an evaluation runA rail of seven stops — plan, dataset, template, running, analysis, iteration, calibrating. Stops up to analysis are complete, analysis is the current phase, later stops are hollow. Confirmation gates sit between the stops that write anything.PlanDatasetTemplateRunningAnalysisIterationCalibratingConfirmConfirmConfirmConfirm
Try askingProvide an analysis of my traces from the past weekRun an evaluation for my agentAnalyze my recent evaluation resultsCompare two prompt versionsGenerate a test dataset for evaluating a specific agent
Calibration

Score It Blind, Then Train the Judge

Score a sample yourself, measure the gap, and improve the judge until it agrees with your experts.

An evaluator template: the judge prompt with its input, output and retrieved_context variables highlighted, plus score-range and reasoning prompts
The default evaluation templates, and one trace graded on them: Conciseness 0.90, Contextcorrectness 0.99, Hallucination 0.05
The Scores page in blind review: agent scores hidden, manual scores entered per case
Judge vs. human, 10 casesScore 0 → 1
Judge calibration — the gap between human and model scores closingEight cases, each with a hollow dot for the human score and a filled green dot for the judge score. On load the judge dots move from their first-version positions to their calibrated positions and the disagreement lines collapse. Case 6 is an exact match, marked with a ring.0.000.501.00ExactCase 1Case 2Case 3Case 4Case 5Case 6Case 7Case 8Case 9Case 10
HumanJudge
Alignment
Mean gap0.053
✓ Threshold < 0.3
Agreement94.7%
✓ Threshold > 80%
Agreement beyond chance (κ)0.71
✓ Threshold > 0.6
Sample10 cases
Scored blind by you
Iteration

From Failed Cases to a New Prompt Version

1

See Why Cases Failed

Iris groups the low scores and names the pattern behind them.

2

Approve the Proposed Change

It drafts the prompt edit and waits for your yes.

3

Re-run the Same Cases

The same dataset runs against the new version, so the scores compare directly.

IrisProposal

Pattern: three cases scored low, all on well-established topics the agent should know cold.
Hypothesis: the prompt lets it hedge on obscure topics, and it is hedging on familiar ones too.
Change: answer confidently on established policy; keep the hedge for the genuinely obscure. Want me to stage this?

Version Comparison on the Prompts page: the compliance assistant prompt, v2 compared with v3Iris comparing v2 and v3 on the same 25 cases: average 0.894 to 0.910, three target cases improved, one regression caughtPrompt v2 compared with v3
Re-ran the same 25 casesAvg 0.894 → 0.910 · 1 regression caught
How it works

The Evaluation Loop

1

Create Custom Evaluations

Create a custom evaluator and choose what evidence the judge can read.

2

Evaluate and Analyze

Run an LLM evaluation and analyze the results with Iris.

3

Score Live Traffic

Evaluators score new production traces as they arrive.

4

Calibrate the Judge

Score a sample yourself and improve the evaluator until it agrees with you.

BEFORE RELEASEIN PRODUCTIONReleaseRepeatEvaluationHarnessTHINKSTACK1234
Per-case progress saved, so a crashed run shows how far it got
Prompt version recorded on every run
Alignment history kept for every evaluator version

Related Modules

Governance
Policy & Guardrails
Policy, PII, and safety checks that execute inside the runtime, in both directions.
Run & Deploy
Agent Runtime Environments
Promote the same agent spec from sandbox to production without a rewrite.
Build
Agent Harness Builder
Compose model, prompt, tools, and guardrails into one governed, versioned spec.

Frequently Asked Questions

Common questions about Evaluation Harness.

What does an evaluation harness actually do?+
ThinkStack's evaluation harness scores agent responses against the evaluation templates your team defines, and re-runs the same graded set after a prompt, model, or tool change. It also reads production runs to find where they broke, such as a failed tool call, a knowledge source that returned nothing, or a sub-agent that errored, and turns those failures into test cases the next version has to pass.
How do we know the model grading our agents is any good?+
You check it, and the harness measures the gap. Score a sample of cases yourself with the agent's scores hidden so your judgment isn't anchored to the model's. ThinkStack then reports the gap between your scoring and the judge's, including which direction the judge leans, proposes an edit to the evaluator, and re-runs the sample so you can see whether the gap actually closed. Evaluators are versioned.
Can the judge see the evidence, or only the final answer?+
Both. Evaluator templates can read the evidence behind a response: the retrieved context, the knowledge lookups and what they returned, the tool calls, including ones that errored, and results from other agents. A judge that sees only the final answer can't tell a lucky answer from a grounded one, so grounding is scored against what the agent actually retrieved.
When cases fail, do we get a report or a fix?+
A fix. The Score Analyst groups the failing cases, names the pattern behind them, and drafts a prompt change. Once you approve it, Iris updates the agent's prompt and re-runs the same graded set, so the score change is measured on identical cases.
What does it take to set up, and who has to do it?+
Three steps. Create a custom evaluator and choose what evidence the judge can read. Run the evaluation and review the failures Iris groups for you, each with a proposed prompt change. Then score a sample yourself to calibrate the judge, and version the evaluator until it agrees with your reviewers. Iris can generate the test set for you, by category.