We tested two AI models on the same ten builds. The numbers picked one. We picked the other.
Kreova's builder runs on a large AI model. Until this week it ran on exactly one, from one provider. That is a risk for a platform: if that provider has an outage, changes its prices or falls behind, every build on Kreova goes with it. So we did the work to make the model a choice rather than a fixture, and wired up a second provider's model as a drop-in. Every tool the builder has, its web search, its progress reporting, all of it works the same on both.
Then we had to answer the obvious question: which one is better?
The test
We wrote ten prompts, the kind of requests real people send, and froze them. A booking site for a barbershop from a one-line brief. A follow-up edit on that site. A question about it that should change nothing. A quote form that has to email a confirmation. An expense dashboard that has to use real database totals rather than adding numbers up in the browser. A site built from an attached logo whose colours it has to match. A request for photos it cannot possibly have, where the right answer is to ask for them. A brief big enough to run the builder out of turns. Two requests that need services the project has not connected yet, where honesty is the test.
Each prompt runs as a real build on a throwaway project, from cold, exactly the way a customer's would. A machine then checks what a machine can check: did it commit, did it call the tools the job needed, did it invent a payment link or a spreadsheet address, did it hotlink an image from somewhere else, did it use the colours it was shown. We ran the same ten on both models, minutes apart, on the same code.
What the numbers said
The second model won on paper. It passed all ten mechanical checks against nine for the incumbent. It was about forty percent faster and about forty-five percent cheaper per build. On the one prompt the incumbent failed, it had spent all of its turns building a login system and never got to the dashboard the brief was actually about. The challenger built the dashboard in six turns.
We moved the platform to the challenger that afternoon.
What a real job said
That evening we gave both models the same real task: take an existing dealership website and rebuild it on Kreova, with its actual photos and brochures. Same prompt, same tools, fresh project each.
The incumbent read the home page, found the brochures page, found the site's configuration file, and downloaded all eight photos and all seven brochures in one build. It needed one fix afterwards, a popup that would not close.
The challenger read the home page, downloaded one file, and stopped. It needed to be told where the photos were. It never found the brochures. It then needed three rounds of fixes for things that were visibly wrong on the page.
The challenger's attempt cost about twenty cents less. It also cost most of an hour and did not finish.
We moved the platform back the same evening.
What we learned
The mechanical checks were not wrong. Every one of them measured something real, and the challenger really did pass them. What they could not see was thoroughness: whether the model went looking for what the job needed or stopped at the first thing it found. On a well-specified prompt that difference barely shows. On an open-ended job like "move my site here", it is the whole job.
That is also the honest reading of the incumbent's one failure in the suite. It was thorough about the wrong thing, building a complete login system before the dashboard. Thoroughness is a strength that needs steering, not a weakness to trade away for speed.
So the test stays, and it now has a rule attached: the numbers tell you a build did not break. A person tells you it is good. Both models remain wired in, and switching between them is one line and one deploy, so the day the answer changes, we can act on it in an afternoon. We will run the ten prompts again whenever either provider ships something new, and we will publish what we find.
What this means for you
Nothing changes in how you use Kreova. Your builds run on the model that finished the job. If that ever changes, it will be because a real build, checked by a real person, showed us something better, not because a spreadsheet did.

Comments
Loading comments…