AI Evaluation Design
Making a cheaper AI model reliable, and choosing tools by measuring them.
Making a cheaper AI model reliable enough for production, putting every prompt under version control, and building the evaluations that picked Exa over Tavily, for Motley, an NYC school and neighborhood guide for families.
Evaluation Design · AI Engineering · Tool Selection

Motley runs on a cheap AI model that makes mistakes. Its prompts were written into the code with no version numbers. And there were two web search APIs to choose from, but nobody had compared them.
Keep the cheap model, and back up every choice with data, so anyone who asks why gets a real answer. Usually the person asking is me.
What Motley Is
Motley helps families compare neighborhoods and schools. An AI model turns city data into neighborhood descriptions, matches schools to what a family cares about, and searches the web to fill in what the school database is missing.

Districts and the schools in them.

A neighborhood description written by the model.
Starting point
The model was already chosen. MiniMax 2.7 had won an earlier evaluation on Agent Church, where it beat Groq Fast and Claude Haiku at a five-step job that used tools, at a price that fit the budget.
That left three questions, and they became this study. Could a cheap model work reliably with nobody watching it? Could anyone tell which prompt produced which answer? And which search API should Motley use to find the school information the city doesn't publish?

Onboarding: fourteen quick steps. The model only summarizes, so the calls stay cheap.

School-match: the top schools for a family, each with a reason.

Enrichment: overnight, the model searches the web to fill in details about each school.
1 · Durability
MiniMax 2.7 costs much less than a top-tier model and handles multi-step tool calls well. But it makes mistakes: it puts a field under the wrong name, writes past the length limit, or describes the wrong neighborhood.
Instead of paying for a more expensive model, I built a layer of checks around this one.
01
Inngest saves each step of a job. If step four of five fails, the retry starts at step four, so the first three steps don't run, or cost money, a second time.
02
The answer format is defined once as a schema and sent to the model with each request. Every answer is checked against it. If the model gets a field name almost right, the code fixes the name before checking.
03
If the model is asked about one neighborhood and describes another, the answer is thrown out and the request runs again with a note explaining the mistake.
04
Each failure is sorted into one of seven types, so when things go wrong I can see what happened instead of guessing.

One school's enrichment job, with every step saved and every retry recorded.
With these checks, the cheap model runs overnight with nobody watching. Without them, the money saved would go to time spent fixing its mistakes.
The model was now reliable. The next problem was the prompts: when an answer went wrong, nobody could tell which version of the prompt had produced it.
2 · Prompts
Prompts used to be written inside the code. Changing one word meant redeploying the app, and nobody could tell which wording produced which answer. Now every prompt is a plain file in git with a version number, and that version is logged with every call.



The prompt folder, one system prompt, and its examples.
System prompts are plain text files, user prompts are templates with {{ placeholders }}, and examples are JSON. All of it is tracked in git and editable without touching the code.
The prompt's name and version are logged with every model call, so any answer can be traced back to the exact wording that produced it, and two versions can be compared side by side on speed, tokens and errors.
The tests make sure every prompt has a version, every placeholder gets filled in, and no unfilled {{ placeholder }} is ever sent to the model. They only read the files and never call the model.
| Change | Version bump | Example |
|---|---|---|
| A wording fix | Patch | 2.2.0 → 2.2.1 |
| A new rule or a change in tone | Minor | 2.2.0 → 2.3.0 |
| A change in what the prompt is for | Major, and a new file | _v2.txt → _v3.txt |
When a new major version is added, the old file stays, so answers from before the change can still be compared with answers after it.
With prompts in versioned files, every change can be measured in the next evaluation, and none of them needs a redeploy.
Now that every call logged its prompt, cost and failures, I could use the log to make a real decision: which search API to use.
3 · The evaluation system
Every model call and search call now writes one row to a log: what it cost, how long it took, which prompt it used, and what went wrong if it failed. That log is the evaluation system.
Motley has four AI jobs. Before picking a tool, I listed what each one needs. That answered the first question right away: only one job searches the web, so only that job needs a search API. It's enrichment, the job that fills in details for every private school in the city.
| What each job needs | Onboarding | Snapshot | School-Match | Enrichment |
|---|---|---|---|---|
| Median response time | 7.3s | 33.7s | 21.2s | 14.7s |
| Calls tools | — | — | ✓ | ✓ |
| Answers in a fixed format | — | ✓ | ✓ | ✓ |
| Searches the web | — | — | — | ✓ |
| Maximum cost per call | $0.01 | $0.02 | $0.03 | $0.02 |
| Same job, real production calls | Exa | Tavily |
|---|---|---|
| Cost per call | $0.007 | $0.100 |
| Median response | 1.2s | 0.6s |
| Error rate | 0.24% | 0.20% |
| Finds pages similar to a given page | ✓ | — |
| Writes an answer for you | — | ✓ |

Exa
5,043 calls · $29.28

Tavily
497 calls · $49.60
Why I chose Exa.
Tavily is faster and just as reliable. But Exa is about 14× cheaper per call, and the enrichment job makes thousands of calls. Exa can also find more schools like a given one, which is what the job needs most. Tavily's written answer doesn't help here, because the model rewrites everything into structured fields anyway.

What the enrichment job filled in, across 906 private schools.
Durability
Before this work, every choice of model or search API came down to opinion, and each new option started the debate over.
New models and search APIs come out all the time. Now, when a new one appears, I can test it on the same jobs, compare it against the same log, and let the numbers decide. MiniMax and Exa are the best choices today. The system that chose them will keep working when better options arrive.
Outcomes
A cheap model can work if you add checks
The savings only count if nobody has to watch every call. Retries, format checks and labeled failures are what made it possible to keep the cheaper model.
Search APIs aren't interchangeable
Exa and Tavily look the same until you have a real job with real volume and a budget. Then it's clear they're built for different things.
Measure before you choose
Without the log, every choice is just an opinion. With it, the data for the next evaluation is already collected.
The model didn't get smarter. What changed is that every decision about it is now backed by data anyone can check.