In September, TypeSafe released Jev, an AI model that doesn’t write. You hand it some text and a few typed questions, and it returns an answer to each with a probability, usually in under a third of a second. We put it on one of the most repetitive jobs in our newsroom, Radar’s link filter, and graded it against the model we use today and against Claude Opus 5.5.
Jev beat our current filter by 5 points at the same price and twice the speed. It came within 3 points of Opus at a seventy-fifth of the cost. The bigger surprise was what the test exposed about our own definition of a good link.
If you’re new here, we’re ZURB. We’ve spent 25 years asking how teams make better product decisions faster, and lately we’ve been asking it of our own machines: which decisions in an AI system deserve the expensive model, and which ones don’t? We’ve written before about Radar, our design and product newsroom that runs about 95% on its own, and about how the filter at its front is just a question we ask of every link. Jev gave us a reason to test that claim. It held up, though not in the way we expected.
The job we gave Jev
Radar's watch feed pulls a few hundred links a day from about 160 design, product and tech news sources. Before anything gets grouped into topics or written up, a small, cheap model reads each headline and summary and answers one question: would a busy product or design leader care, and could they act on it or form a point of view?
It's what we call a system closed loop. We wrote the instructions, stepped out, and only go back in when it drifts. The problem is that it makes it easy to forget. Hundreds of times a day it decides what our editors and our topic grouper ever get to see, and nobody checks its work. If it's too strict, good material disappears before a human could notice it was there.
Today that model is gpt-4o-mini (we use over 6 different models depending on the stage in Radar’s workflow). It writes a score from 0 to 100 for each link determining its relevancy to the question would a design or product leader care, and anything under the line gets hidden. We'd never really asked whether it was good at the job. A yes-or-no call on a headline, made hundreds of times a day, is exactly what Jev was built for.
A model built to answer one kind of question
The question was simple. Can a model built for a single job beat the general model we already pay for, on the one call it’s built for?
We went in skeptical. Price per call is the easy part of the cost. A new model has to be wired in, watched, and kept honest inside the system, so it has to earn its place on accuracy, not just on the bill. Before we ran anything, we wrote down twelve predictions about how it would do.
Several of them assumed it would stumble.
The answer key is the real work
To grade a filter, you need to know the right answer for each link. We pulled 100 real links from Radar's feed, on purpose heavy on the hard ones: links our current filter scored right near its line, and links from design-named outlets that publish a lot of gadget posts.
The first attempt at an answer key is worth telling, because it failed in a useful way. We had Claude Opus draft a label for each link, built a review page, and Ben clicked through it. He agreed with 99 of 100 drafts in under five minutes to save time (bad). We saw the first iteration of the results, took a step back because he knew that not having real human input can actually be a pretty big problem when building out a system.
So we rebuilt the page and did it again. This time Ben made each call before he could see any model's answer, and only then saw what Opus had said. It took real time, and he disagreed with Opus on 15 links. Those 100 blind calls became the answer key.
The lesson carries well past this test. If a person is going to be the standard a machine is measured against, the review tool has to make agreeing harder than thinking. Ours made it easier, and the first version of the data was worthless because of it.
The one-job model held its own
Four contenders took the same test, with the same definition of the reader and what counts. Jev answered one yes-or-no question. gpt-4o-mini ran the exact prompt Radar uses today. Claude Opus 5.5 stood in as the bar for how good this call can get; it writes our articles, and we'd never run it on every incoming link. And as a floor, a filter that calls everything noise, which already scores 59% because most of these links weren't relevant.
Jev beat the filter we run today by 5 points, at the same price and about twice the speed. It came within 3 points of Opus at a seventy-fifth of the cost and fourteen times the speed. It was also th`e best of the four at ranking links from most to least relevant, which matters more than it sounds, because a filter is a ranking with a line drawn through it.
The detail we keep coming back to: Jev and gpt-4o-mini gave the same answer on 78 of the 99 links. The whole difference came from the 21 where they split, and on those Ben sided with Jev 13 times and with today's filter 8.
Opus didn't buy much here. Given time to think and the full article, it matched Ben 85% of the time. Asked the quick way, with only the headline and summary, also 85%. For a yes or no on a headline, a model built for the job got most of the way there for pennies, and that frees the expensive model for the work only it can do.
We also went looking for the weaknesses other people have reported. Putting the outlet name in front of the headline, to see if "Yanko Design" would trick it into calling a gadget post a design story, changed nothing. Shuffling the order of its answer options barely moved it. Sending the identical request five times gave nearly the same number every time. One idea backfired: splitting the question into three narrower ones made it worse. The single question carried the full definition, and the narrow ones lost it.
The question mattered more than the model
The most useful finding wasn't about Jev at all.
Our current filter caught only 54% of the links Ben actually wanted. Opus caught 66%. Every model we tried was stricter than our own editor. And all 15 of Ben's disagreements with Opus went the same direction. He kept stories it called consumer noise: a $40 bundle of AI chat tools, a dating app getting into audiobooks, a roundup of cheap running watches. Each one shows a product, pricing or experience decision a product leader can learn from.
Our definition says gadgets and deals are noise, full stop. Ben's calls say it depends on whether there's a product decision inside the story. That's the drift we said we'd go back in for, and we'd never have seen it without building an answer key. In July we said the question is what makes the filter useful. It turns out the question was also where most of the error was hiding.
What we're doing with it
We're not swapping models tomorrow. 99 links is enough to take Jev seriously and not enough to bet the feed on it. The next step is running it next to the current filter on live traffic for a few weeks, same links, both answers, and comparing them against fresh editor calls.
Before that, we're fixing the question. The definition gets a rewrite to make room for consumer stories with a product decision inside them, and then both models get retested against the same 99 links. We kept every one of Ben's calls and notes for exactly this.
And we'll draw the line from labels, not from the model. None of these models' raw scores are probabilities you can take at face value. The cutoff belongs wherever labeled links say it does.
What we'd tell a fellow builder
Look for the small yes-or-no calls inside your loops. Tagging, routing, triage, deciding which of a thousand ideas deserves a closer look. They run constantly, they're cheap enough per call that nobody checks them, and they shape everything downstream. Those are the calls where a model built for one job can earn its keep, and where the expensive general model is mostly overhead.
But the model isn't the first thing to test. Write down what a right answer looks like, have a person make those calls honestly, and measure your current setup against them before you invest in anything new. We went in to evaluate a model and came out with a better question, which was the more valuable of the two.
Radar is live at radar.zurb.com, built by the team behind Glare. If you're testing small models inside your own loops, or you want to argue with any of this, the forum is open.
