Motley

Before any tool gets evaluated, the loops need a shared list of abilities. Five loops, eight criteria, one matrix. The matrix decides which loops need a search API at all (only the two that touch the open web) and which abilities a model has to satisfy per loop.

The throughput cap row isn't aspirational — it's enforced in production by a sliding-window limiter on the matching API routes (/api/agent/snapshot 10/hr, /api/comparable 20/hr, /api/schools/summaries/generate 3/hr), shaped to mirror @upstash/ratelimit for a one-line Redis swap later.

The model isn't a variable in this evaluation. It was decided before this case study began. MiniMax 2.7 is the production model for every loop, inherited from an earlier evaluation where it did better than Groq Fast and Claude Haiku at running a five-step job that uses tools.

MiniMax 2.7 boils down to inexpensive agent ability that works for what Motley needs from it... for now.

The lever for the next eval.

Every task currently runs on MiniMax 2.7 through the global default. If a future eval picks differently for one loop, an override file maps the loop ID to the model it should use. Editing the file and restarting the service is enough — no code change, no redeploy.

# .env (production)
# Default: MiniMax 2.7 for every loop currently.
DEFAULT_PROVIDER=minimax
DEFAULT_MODEL=minimax-2.7

# Per-loop overrides. The day an eval picks differently for one
# loop, add a line here. One env var, no redeploy.
MODEL_OVERRIDES='{
  "snapshot-generate": "claude-haiku-4-5",
  "school-match":      "groq-llama-3.3-70b"
}'

Prompts moved out of TypeScript into a flat directory — visible in the editor below. System prompts as .txt files, user templates with {{ placeholders }}.

Tree, system prompt, and few-shot pairs — all editable without a deploy.

Every prompt has a version number. The version is logged with every model call, so a future evaluation can tie the quality of an answer back to the exact prompt that produced it.

Automated tests check that every prompt has a version, that placeholders get filled in correctly, and that no {{ placeholder }} ever leaks through to the final text. The tests don't call the model — they only check the prompt files.

Inngest smooths MiniMax.

MiniMax is the cheap model. Inngest is what makes it usable — retries, rate limits, and step-level failures absorbed across long-running jobs. Without it, the price-per-token advantage gets eaten babysitting calls; with it, the rough-edged model runs unattended overnight.

MiniMax 2.5 → 2.7: shape in code, not in the prompt.

The old loop ran on MiniMax 2.5 — the model returned a paragraph, and the runtime fished the JSON out of it. The current loop runs on MiniMax 2.7 with the Vercel AI SDK and Zod 4. The shape we want is written once as a Zod schema, the SDK hands it to the model along with the request, and the response comes back already in that shape. No text-to-JSON cleanup at the door.

That's the upgrade. The catch: the right shape isn't the same as the right answer. The model still slips — a key with the wrong capitalization, a field that runs past its length cap, content written for the wrong neighborhood. Three guardrails close the gap.

1. Match the keys before failing.

If the model returns the right data under slightly wrong names — neighborhoodName instead of neighborhood_name, that kind of thing — coerceFields renames the keys back to what the schema expects, then validates.

// generate-object-robust.ts (excerpt)
const keyMap = new Map<string, string>();
for (const key of Object.keys(shape)) {
  keyMap.set(key, key);
  keyMap.set(key.replace(/_/g, ""), key);
  keyMap.set(
    key.replace(/_([a-z])/g, (_, c) => c.toUpperCase()),
    key,
  );
}
// remap incoming keys, then schema.safeParse(remapped)

When something still won't fit, the runtime throws a typed error instead of crashing. Every failure ends up in one of the seven buckets below.

2. Retry on hallucination, not just on errors.

The neighborhood snapshot loop has a specific failure mode: a neighborhood goes in, the model writes about a different one. The validator compares the name in the response to the name in the input. On a mismatch, the runtime logs a HALLUCINATION retry row, appends an ## ISSUES WITH PREVIOUS ATTEMPT block telling the model exactly what it got wrong, and runs it again. The successful retry writes its own success row, so ai_llm_calls shows the full pair instead of pretending nothing went wrong.

// snapshot-generate.ts (excerpt)
const nameMatch = snapshot.neighborhood_name
  .toLowerCase()
  .includes(ctx.ntaData.ntaName.toLowerCase().split("-")[0]);

if (!validation.valid || !nameMatch) {
  issues.push(`CRITICAL: You generated content for
    "${snapshot.neighborhood_name}" but the neighborhood is
    "${ctx.ntaData.ntaName}". Generate content ONLY for
    ${ctx.ntaData.ntaName}.`);
  logLlmCall({ status: "retry", errorType: "HALLUCINATION" });
  // re-run the generator with the augmented prompt
}

3. Sort failures into seven buckets.

Inngest retries everything retryable; the metrics page needs to know why. Every failure path tags itself before bubbling up.

enum LLMErrorType {
  JSON_PARSE,          // model returned malformed JSON
  SCHEMA_VALIDATION,   // shape was wrong
  HALLUCINATION,       // content failed a domain check
  TIMEOUT,             // provider didn't respond in time
  RATE_LIMIT,          // upstream throttled us
  PROVIDER_ERROR,      // 5xx from the model API
  EMPTY_RESPONSE,      // string was blank
}

The buckets feed Pillar 6's error_type column directly.

The two APIs are for opposite jobs. Exa is for known questions with indexed answers. Tavily is for long-tail questions, freshly written pages, anything outside any index. The table below resolves the tradeoffs against the enrichment loop's needs.

What the dashboards show.

Same enrichment loop, near-identical error rate (0.24% Exa vs 0.20% Tavily). Exa ran 5,043 calls at $29.28; Tavily ran 497 at $49.60. Per call, Exa is roughly 14× cheaper. Tavily is actually the faster API — p50 647ms vs Exa's 1,215ms — but the cost gap dwarfs the latency advantage at fan-out volume. The picks fall out of the rows.

Exa

5,043 calls · $29.28 · 0.2% errors

Tavily

497 calls · $49.60 · 0.2% errors

Why Exa wins.

Two answers. First: cost. Exa charges $0.007 per call, Tavily charges $0.100 — ~14× difference. At enrichment volume that's the deciding row. Second: semantic search reads meaning, not keywords. Motley isn't looking for “the page that contains the words progressive Brooklyn elementary” — it's looking for “pages about schools like this one.” For a loop that asks “find more like this one” thousands of times, the engine that understands “like this one” wins by default.

Technically.

Exa exposes three endpoints, each doing one job. /search takes a natural-language query and returns ranked URLs with cleaned text. /findSimilar takes a known-good URL and returns semantically similar pages — Tavily has no equivalent. /contents takes a list of URLs and returns the readable main text from each. The three compose cleanly because they don't assume you want a written answer. Tavily bundles search and synthesis into a single call — convenient when you want a paragraph, wasted work when the LLM is about to overwrite that paragraph into typed fields. Reliability is a wash: 0.24% error rate over 5,043 Exa calls, 0.20% over 497 Tavily calls.

Private-school enrichment — 906 schools, avg 55.1.

Picks age. The next evaluation needs real numbers — how long each call took, how often the model returned the wrong shape, how often it made up an answer — pulled from production calls instead of a fresh probe. The ai_llm_calls table is the place those numbers live: fifteen columns, one row per model call. The row is written after the response goes back to the user, so the logging can never slow a request down.

CREATE TABLE ai_llm_calls (
  id              uuid PRIMARY KEY,
  prompt_id       text NOT NULL,
  prompt_version  text NOT NULL,
  feature_id      text,
  user_id         uuid,
  model           text NOT NULL,
  input_tokens    integer,
  output_tokens   integer,
  latency_ms      integer,
  cached          boolean NOT NULL DEFAULT false,
  status          text NOT NULL,         -- ok | retry | error
  error_type      text,                  -- enum below
  created_at      timestamptz NOT NULL DEFAULT now()
);

Every model call in the codebase passes a prompt ID and a feature ID with it, so every row in the log knows which prompt produced it and which loop it served.

A second table tracks the parent job: how many steps finished, how long the whole job took, how many tokens it used, and the Inngest job ID. Every model-call row points at its parent job, so any single ID pulls up the full tree of calls underneath it.

ai_agent_runs ─┬─ run_id: 9f3a…
               ├─ status: completed
               ├─ steps: 5/5
               └─ children:
                   ├─ ai_llm_calls (generate)        ok    1842 ms
                   ├─ ai_llm_calls (validate)        ok      94 ms
                   ├─ ai_llm_calls (retry on hallu.) retry 1610 ms
                   ├─ ai_llm_calls (summarize)       ok    2103 ms
                   └─ ai_llm_calls (smooth)          ok     487 ms

Coverage, by field, by source.

Charter URL & data — 289 schools, 34 networks.