AI Evaluation Design

Motley

Making a cheaper AI model reliable, and choosing tools by measuring them.

Making a cheaper AI model reliable enough for production, putting every prompt under version control, and building the evaluations that picked Exa over Tavily, for Motley, an NYC school and neighborhood guide for families.

Evaluation Design · AI Engineering · Tool Selection

Motley landing page — Family Intelligence for NYC
What's The Problem?

Motley runs on a cheap AI model that makes mistakes. Its prompts were written into the code with no version numbers. And there were two web search APIs to choose from, but nobody had compared them.

What's The Scope?
  • 1.Durability: make a cheaper model reliable enough to run with nobody watching
  • 2.Prompts under version control: save every prompt as a file with a version number, and log that version with every call
  • 3.The evaluation system: log every call, then use the numbers to pick a search API
What's The Challenge?

Keep the cheap model, and back up every choice with data, so anyone who asks why gets a real answer. Usually the person asking is me.

What Motley Is

A guide to NYC for families.

Motley helps families compare neighborhoods and schools. An AI model turns city data into neighborhood descriptions, matches schools to what a family cares about, and searches the web to fill in what the school database is missing.

Motley District 15 page with public, charter, and private school map

Districts and the schools in them.

Motley Lower East Side neighborhood overview

A neighborhood description written by the model.

Starting point

What we were working with.

The model was already chosen. MiniMax 2.7 had won an earlier evaluation on Agent Church, where it beat Groq Fast and Claude Haiku at a five-step job that used tools, at a price that fit the budget.

That left three questions, and they became this study. Could a cheap model work reliably with nobody watching it? Could anyone tell which prompt produced which answer? And which search API should Motley use to find the school information the city doesn't publish?

Motley onboarding step 2 of 14 — Your Situation: Years in NYC, what brings you here, where do you live now

Onboarding: fourteen quick steps. The model only summarizes, so the calls stay cheap.

Motley School Smart Match card — grade-aware candidates, family-priority weights, top 3 with why-it-fits narratives

School-match: the top schools for a family, each with a reason.

Motley School Details card — Brooklyn Prospect Charter School, district 13, K-5 elementary, 6-8 middle school, 657 students

Enrichment: overnight, the model searches the web to fill in details about each school.

1 · Durability

Making a cheaper model reliable.

MiniMax 2.7 costs much less than a top-tier model and handles multi-step tool calls well. But it makes mistakes: it puts a field under the wrong name, writes past the length limit, or describes the wrong neighborhood.

Instead of paying for a more expensive model, I built a layer of checks around this one.

01

Retries that pick up where they left off

Inngest saves each step of a job. If step four of five fails, the retry starts at step four, so the first three steps don't run, or cost money, a second time.

02

Every answer is checked against a format

The answer format is defined once as a schema and sent to the model with each request. Every answer is checked against it. If the model gets a field name almost right, the code fixes the name before checking.

03

Wrong answers are retried too, not just errors

If the model is asked about one neighborhood and describes another, the answer is thrown out and the request runs again with a note explaining the mistake.

04

Every failure is labeled

Each failure is sorted into one of seven types, so when things go wrong I can see what happened instead of guessing.

Malformed JSONWrong shapeHallucinationTimeoutRate limitProvider errorEmpty answer
Inngest run trace for enrich-private-school job, 24.104 seconds

One school's enrichment job, with every step saved and every retry recorded.

With these checks, the cheap model runs overnight with nobody watching. Without them, the money saved would go to time spent fixing its mistakes.

The model was now reliable. The next problem was the prompts: when an answer went wrong, nobody could tell which version of the prompt had produced it.

2 · Prompts

Prompts under version control.

Prompts used to be written inside the code. Changing one word meant redeploying the app, and nobody could tell which wording produced which answer. Now every prompt is a plain file in git with a version number, and that version is logged with every call.

Editor file tree — prompts/system_prompts/ and prompts/few_shot_examples/
private_school_enrichment_v1.txt open in the editor showing extraction rules
private_school_enrichment_examples.json showing input/output few-shot pairs

The prompt folder, one system prompt, and its examples.

A prompt is a file, not code

System prompts are plain text files, user prompts are templates with {{ placeholders }}, and examples are JSON. All of it is tracked in git and editable without touching the code.

Every call records its version

The prompt's name and version are logged with every model call, so any answer can be traced back to the exact wording that produced it, and two versions can be compared side by side on speed, tokens and errors.

Tests check the prompt files

The tests make sure every prompt has a version, every placeholder gets filled in, and no unfilled {{ placeholder }} is ever sent to the model. They only read the files and never call the model.

ChangeVersion bumpExample
A wording fixPatch2.2.0 → 2.2.1
A new rule or a change in toneMinor2.2.0 → 2.3.0
A change in what the prompt is forMajor, and a new file_v2.txt → _v3.txt

When a new major version is added, the old file stays, so answers from before the change can still be compared with answers after it.

With prompts in versioned files, every change can be measured in the next evaluation, and none of them needs a redeploy.

Now that every call logged its prompt, cost and failures, I could use the log to make a real decision: which search API to use.

3 · The evaluation system

Choosing between Exa and Tavily with real numbers.

Every model call and search call now writes one row to a log: what it cost, how long it took, which prompt it used, and what went wrong if it failed. That log is the evaluation system.

Motley has four AI jobs. Before picking a tool, I listed what each one needs. That answered the first question right away: only one job searches the web, so only that job needs a search API. It's enrichment, the job that fills in details for every private school in the city.

What each job needsOnboardingSnapshotSchool-MatchEnrichment
Median response time7.3s33.7s21.2s14.7s
Calls tools——✓✓
Answers in a fixed format—✓✓✓
Searches the web———✓
Maximum cost per call$0.01$0.02$0.03$0.02

Exa searches by meaning, and it can find pages similar to one you already have. Tavily searches the live web and can write a short answer for you. On paper, they look like they do the same job.

Same job, real production callsExaTavily
Cost per call$0.007$0.100
Median response1.2s0.6s
Error rate0.24%0.20%
Finds pages similar to a given page✓—
Writes an answer for you—✓
Exa usage dashboard — 5,043 total calls, 4,053 searches, 990 contents, $29.28, 0.2% error rate; private school enrichment 3,710 calls

Exa

5,043 calls · $29.28

Tavily usage dashboard — 497 total calls, 496 credits used, $49.60, 0.2% error rate; private-school-enrichment 444 calls, 925ms avg latency

Tavily

497 calls · $49.60

Why I chose Exa.

Tavily is faster and just as reliable. But Exa is about 14× cheaper per call, and the enrichment job makes thousands of calls. Exa can also find more schools like a given one, which is what the job needs most. Tavily's written answer doesn't help here, because the model rewrites everything into structured fields anyway.

Private school enrichment coverage by field — enrichment score 87.4%, philosophy 84.7%

What the enrichment job filled in, across 906 private schools.

Durability

Future choices can use the same data.

Before this work, every choice of model or search API came down to opinion, and each new option started the debate over.

New models and search APIs come out all the time. Now, when a new one appears, I can test it on the same jobs, compare it against the same log, and let the numbers decide. MiniMax and Exa are the best choices today. The system that chose them will keep working when better options arrive.

Outcomes

What it changed.

A cheap model can work if you add checks

The savings only count if nobody has to watch every call. Retries, format checks and labeled failures are what made it possible to keep the cheaper model.

Search APIs aren't interchangeable

Exa and Tavily look the same until you have a real job with real volume and a budget. Then it's clear they're built for different things.

Measure before you choose

Without the log, every choice is just an opinion. With it, the data for the next evaluation is already collected.

Choose tools by measuring them.

The model didn't get smarter. What changed is that every decision about it is now backed by data anyone can check.