Arthur, my personal assistant, routes with a small model and converses with a large one — a split never measured. Three concrete failures, and a question that changed shape: not whether the model is right, but whether I can prove it.
The model split wasn't a measurement
Arthur is a personal multi-agent assistant I use daily. Two models run inside it. mistral-small-latest classifies the request: is it for the Intendant, which files and retrieves, or for the Mentor, which tracks projects? mistral-large-latest then carries the whole conversation.
This split comes from reasoning, not measurement. The large model seemed necessary to me because of the number of tools, the preferences to keep in mind, and the decisions to chain together. The small one seemed sufficient to decide between two destinations. None of it was ever compared.
What "doing the job" means here
Arthur serves as my second organizational memory. It notes reminders, dates, times. It reports them back on request, in writing or out loud.
So the number-one design criterion isn't performance: it's accuracy. Noting exactly what I said, without rephrasing, without shifting anything.
Three measured failures
A substituted word, on August 20th. I dictate: "remind me to resend the file." What gets written to the database: "send the file." One word, and a second time becomes a first time. The write succeeded, the deadline was correct, the record looked perfect. That same morning, the identical sentence had produced the correct wording.
A wrong day, same day. I ask for "Friday," the context shows Thursday the 20th. The reminder gets created for Saturday the 22nd.
An unstable routing. The same sentence went to the Intendant, then to the Mentor twenty minutes later. The router also produced a truncated word that brought down the entire response.
The first and second come from the large model. The third, from the small one.
Reliability didn't come from a bigger model
For dates, I took the calculation away from the model. A code function resolves relative days, with its own tests. It's not that a bigger model would get it wrong more often: it's that I can't prove it either way. A model is stochastic, it can give two different answers to the same question. A deterministic function always returns the same result for the same input, and that can be tested.
The question changed shape. Not "can the model be right?", but "can I prove it's right?"
The same reflex produced the rest: when a tool refuses, the code writes the response, not the model. It can no longer comment on what it just failed to do.
The substituted word, though, remains an open question. None of these levers catch it: nothing failed, so nothing triggers. Comparing the dictated text against the written text in code would wrongly flag legitimate rewordings, like a shift to the infinitive. I want to first measure how often it happens.
What hasn't been measured
Is the large model necessary? That hasn't been measured. I can describe what it does well — chaining several tools across several turns, spotting the hidden action in a sentence — but I've never verified that a smaller one would fail at it.
Yet the test is cheap: swap the conversation model, replay the same scenarios in a fresh conversation, compare. Only one variable changes.
Ten exchanges traced on September 8th and 9th come out to $0.003 to $0.008 each, under a €10 monthly cap. The stakes aren't the amount. It's that an unmeasured conviction remains a conviction.