measuredlocally.comLocal LLM leaderboard round two: why AI is not a reliable judge of AI
I rebuilt my leaderboard of the AI models you can run at home after round one's tests turned out too easy. When I judged the builds myself, the AI judges had been kinder than me in nearly every case, so my own marks now make up 80% of each score.
Plays here, from YouTube
Last week I started ranking the AI models you can run at home, on my own machines, here on measuredlocally.com. Round one had eight tests, from a Breakout game and a 3D lighthouse to a bug fix and six tough questions.
Why round one was too easy
Within a few days, four models were within about two points of each other at the top. Those tests only asked whether each build worked, and a build can work and still be poor. GLM's spinning heptagon scored full marks with its balls passing straight through the walls, when the prompt said that should not be allowed.
Round two
So I expanded the tests. Round two has ten tasks, and every build is now checked feature by feature, so the instructions in the prompts have to be followed to the letter: a pricing option asked for on a website has to work, and a game has to be playable. Each model gets one chance to fix what failed, the code is marked by tests it never sees, and every model gets its maker's own settings on every machine it fits on.
One of the ten tasks is a fox in the snow, drawn as an SVG, a picture written as code. An image generator paints pixels, but here a language model has to write out every shape as text, with its position and colour, and it never sees the picture while it writes. It has to work out where everything goes from the numbers alone.
My marks against the machines
Three AI judges from three different labs score each build, but before any of it went live I judged 66 builds myself, one from each model for each test you can look at, from fail to perfect. Of those 66, 41 were 30 points or more away from my own mark, and in all but one the machines had been kinder than me. The checks and the judges gave between 76 and 92 to fox drawings I marked as inadequate, and three landing pages passed our price check with their pricing wrong.
So each build's score is now 80% my judgement and 20% the AI judges. DeepSeek still comes out on top at 83.8, then Gemma 4 26B at 63.1 and Qwen 3.8 27B at 62.4.
Gemma 4 26B was a pleasant surprise next to models that seem superior in date of release and overall intelligence. It is a mixture of experts model, with 26 billion parameters but only about four billion working on each word, so I expect it to be quick on local hardware that fits the weights and a reasonable context window. That matters a lot when you don't have much VRAM to play with.
What I would do
For now I would say AI is not a reliable judge of AI. I think many of Google's models are massively underestimated, and Gemma 4 26B is solid for real work.
Read more
- My leaderboard on measuredlocally.commeasuredlocally.com
The numbers in this story
| Figure | What it is | From | Kind |
|---|---|---|---|
| 97.9 to 95.8 | Round one's top four, within about two points | my own test | measured by me |
| 41 of 66 | Builds 30 points or more away from my own mark | my own maths | my own maths |
| 76 to 92 | What the checks and judges gave fox drawings I marked inadequate | my own maths | my own maths |
| 80% and 20% | My share of each build's score, and the AI judges' | my reading | my reading |
| 83.8, 63.1, 62.4 | DeepSeek, Gemma 4 26B and Qwen 3.8 27B, round two | my own test | measured by me |
| 26B, about 4B | Gemma 4 26B's parameters, in all and working on each word | see the story | the maker's claim |
If a number on this page is wrong, I would rather know. Tell me in the comments under the video and I will correct it here and pin the correction there. What has been corrected so far is on the corrections page.