Evals
Evaluations built for your workflows, scored over the whole agent.
We don't republish industry benchmarks. We build evaluation suites specific to your organization: your cases, your rubric, your data, in your environment. Every result is scored over the full agent, the model, the harness around it, and the environment it runs in, so a change to any one of the three shows up as a number. Access to the evaluation database is by engagement. Ask us.
- Healthcare dataThe first domain. Clinical and provider-side workflows, in your governed environment
- Agentic systemsMulti-agent workflows and retrieval pipelines, scored end to end
- How an engagement runsName the problem, meet the person, build in your stack, hand it over
We hold no HIPAA attestation of our own — we work inside your governed environment, under your BAA.
What we build
Evaluations that belong to you, not a leaderboard
Public leaderboards score models on public tasks. Your deployment is an agent on your tasks. The numbers that decide whether it ships are yours, and nobody publishes them for you.
The suite
Cases and a rubric from your own work
Cases drawn from the workflows you actually run, scored against rubric points your own experts write. Use cases, case types, and difficulty bands you name, so the result reads in your terms.
- Your cases, de-identified where they must be
- Your rubric, written with your reviewers
- Versioned, so a re-run is comparable
The unit
The whole agent, not the model
A model alone scores nothing in production. We score the model, the harness around it (retrieval, tools, planner and verifier, a human gate), and the environment it runs in, as one agent, and re-run when any of the three changes.
- Swap the model, keep the harness, read the difference
- Hosted endpoint against on-prem, on the same cases
- Cost, tokens, and time per task beside the score
The practice
Consulting around the evaluations
Choosing what to measure, building the harness, running the suite in your environment, and turning the thresholds into monitoring. Handover is in scope. You own the deliverables once they're paid for. We keep the tools and frameworks we brought with us.
- Hourly or under a scoped statement of work
- Run on every change, not once
- Thresholds handed to your monitoring
Access to the evaluation database is by engagement. Get in touch and name the workflow.
A demo suite
What a suite looks like, for a fictional health system
Twelve agents on one proprietary-shaped suite for Harborlight Health, a fictional specimen. Every mark is a full agent - model, harness, and environment - with score, cost, tokens, and speed beside each other. Every number is a demo value, never a measured client result.
That specimen name also appears in Publications method drafts (hardware seating and population anatomy). Those write-ups use public-eval stand-ins and nested tiers. This page's figures are a different pack: a proprietary-shaped suite of twelve agents. Do not read one as the measured output of the other.
The interactive figures (score bars, score vs cost, heatmap by rubric slice, token and cost stacks, and more) have their own page, outside this layout:
We hold no HIPAA attestation of our own — we work inside your governed environment, under your BAA.
How an evaluation engagement runs
Name the workflow. Write the rubric. Score the agents. Hand it over.
Name the workflow
A 30-minute call with Shaun on one workflow you want to trust. We come back with the consultants who have built evaluation frameworks before.
Write the cases and the rubric
With your reviewers, from your own work. Use cases, case types, and difficulty bands in your terms. If the data will not carry it, we say so early and in writing.
Score the agents in your environment
Model, harness, and environment as one unit, on your infrastructure, under your access. Score, cost, tokens, and time per task, and the slices behind the overall number.
Hand it over
The evaluation set, the harness, the thresholds in your monitoring, and the documentation. You own the deliverables once they're paid for. We keep the tools and frameworks we brought with us.
Ask about access to the evaluation database.
Shaun takes the call. Name the workflow you want to trust, and you leave knowing which shape fits, hourly or SOW, and who in the practice would build the suite.