Local AI hardware leaderboard

The leaderboard, model by model

Each model on every machine that has run it, fastest first at writing code. A speed compares fairly only on the same model, so each model has its own ranking.

Each figure is the middle of three runs with the model's thinking switched off, and the range is under it. First word is the wait after a short question. Reading is how fast the machine takes in a long document before it starts to answer, which other sites call prefill or prompt processing. The llama.cpp and Prism ML's llama.cpp rows were run with llama.cpp's own default sampling. The LM Studio rows were run with LM Studio's defaults, which include the model's own multi-token prediction. The "Qwen 3.8 27B with multi-token prediction" rows were run with llama.cpp's own default sampling; multi-token prediction, 2 tokens drafted a step. The MLX (mlx-lm) rows were run with MLX's own default sampling, which always takes the likeliest word (temperature 0). The four DGX Sparks are timed from another machine on the same network, so the first word includes the trip across it.

What I have found so far

• My Sparks would run a little faster than the figures on this page.MoreLess

  I cap their graphics chips at 2,200MHz, against about 2,400 uncapped, to cut their power use. When I tested the cap on one box in August, writing speed stayed within 1 per cent, reading was 2 to 4 per cent slower, and the chip's peak power fell from 55.5W to 43.3W.

• DeepSeek V4.1 Flash 1017 on the four DGX Sparks: it reads a document of about 15,000 tokens in about 3 seconds, and 16 users at once get 3.1 times what one user gets, in total.MoreLess

  The four DGX Sparks wrote code at 116.0 tokens a second with DeepSeek V4.1 Flash 1017 (DeepSeek's own release: experts 4-bit MXFP4, the rest 8-bit FP8), and read a document of about 15,000 tokens at 4,924 tokens a second. That is a wait of about 3 seconds before the first word (the size of the document divided by the speed).

  With DeepSeek V4.1 Flash 1017 thinking before it answers, the first word of the code answer came after about 18 seconds on the four Sparks. With thinking off it came after 0.24 seconds. In 2 of the 3 runs on the four Sparks it was still thinking when the 2,000 tokens ran out, so that wait comes from the one run that answered.

  With 16 users at once, the four DGX Sparks wrote 200 tokens a second in total with DeepSeek V4.1 Flash 1017, about 13.8 for each user. That is 3.1 times what one user on their own got (200 divided by 64.0).

• GLM 5.3 Flash on the four DGX Sparks: it reads a document of about 16,000 tokens in about 5 seconds.MoreLess

  The four DGX Sparks wrote code at 104.1 tokens a second with GLM 5.3 Flash (NVFP4 at 4 bits, dense layers at 8), and read a document of about 16,000 tokens at 3,159 tokens a second. That is a wait of about 5 seconds before the first word (the size of the document divided by the speed).

  With GLM 5.3 Flash thinking before it answers, the first word of the code answer came after about 0.5 seconds on the four Sparks. With thinking off it came after 0.15 seconds.

• Qwen 3.8 27B on five machines: the RTX 4090 desktop writes code 6.1 times as fast as the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2), and two of my DGX Sparks read a long document 10 times as fast as the HP ZBook Ultra G1a (LM Studio, llama.cpp on Vulkan).MoreLess

  With Qwen 3.8 27B on all of them (each machine in the 4-bit format its own software uses), the RTX 4090 desktop wrote code at 62.4 tokens a second, two of my DGX Sparks at 44.7, one DGX Spark on its own at 26.7, the HP ZBook Ultra G1a (LM Studio, llama.cpp on Vulkan) at 23.5, the Lenovo Yoga Pro 9n (llama.cpp on CUDA) at 15.2, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 14.1, the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 12.1 and the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 10.2.

  Two of my DGX Sparks read a document of about 19,000 tokens at 2,034 tokens a second, the RTX 4090 desktop at 1,994, one DGX Spark on its own at 1,188, the Lenovo Yoga Pro 9n (llama.cpp on CUDA) at 800, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 760, the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 259, the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 205 and the HP ZBook Ultra G1a (LM Studio, llama.cpp on Vulkan) at 194. That is a wait of about 9 seconds on the two Sparks, 9 seconds on the 4090, 16 seconds on the single Spark, 23 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA), 25 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2), 72 seconds on the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2), 91 seconds on the HP ZBook Ultra G1a (llama.cpp on Vulkan) and 97 seconds on the HP ZBook Ultra G1a (LM Studio, llama.cpp on Vulkan) before the first word (the size of the document divided by each speed).

  With Qwen 3.8 27B thinking before it answers, the first word of the code answer came after about 7 seconds on the 4090, 14 seconds on the two Sparks (one run), 17 seconds on the single Spark (one run) and 22 seconds on the HP ZBook Ultra G1a (LM Studio, llama.cpp on Vulkan). With thinking off it came after 1.3 seconds at most.

  On the prose prompt with thinking on, Qwen 3.8 27B was still thinking when the 2,000 tokens ran out, which took 37 seconds on the 4090, 55 seconds on the two Sparks, 86 seconds on the single Spark and 104 seconds on the HP ZBook Ultra G1a (LM Studio, llama.cpp on Vulkan).

  Given room for 8,000 tokens, the prose answer began after 52 seconds in one run on the two Sparks and after 271 seconds in one run on the single Spark. The thinking before it ran to about 1,800 and 7,400 tokens, so most of that gap is the model thinking for longer.

• Gemma 4 12B on five machines: the RTX 4090 desktop writes code 7.7 times as fast as the Mac mini M4, and reads a long document 44 times as fast.MoreLess

  With the same model on each (Gemma 4 12B, Google's own 4-bit QAT file (Q4_0), GGUF), the RTX 4090 desktop wrote code at 99.7 tokens a second, the RTX 5070 Ti desktop at 92.5, the Lenovo Yoga Pro 9n (llama.cpp on CUDA) at 33.2, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 28.6, the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 25.9, the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 22.6 and the Mac mini M4 at 12.9.

  The RTX 4090 desktop read a document of about 19,000 tokens at 5,072 tokens a second, the RTX 5070 Ti desktop at 2,910, the Lenovo Yoga Pro 9n (llama.cpp on CUDA) at 1,513, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 1,439, the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 380, the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 182 and the Mac mini M4 at 115. That is a wait of about 4 seconds on the 4090, 6 seconds on the 5070 Ti, 12 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA), 13 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2), 49 seconds on the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2), 103 seconds on the HP ZBook Ultra G1a (llama.cpp on Vulkan) and 162 seconds on the Mac mini before the first word (the size of the document divided by each speed).

  With Gemma 4 12B thinking before it answers, the first word of the code answer came after about 8 seconds on the 5070 Ti, 9 seconds on the 4090, 22 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA), 29 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2), 36 seconds on the HP ZBook Ultra G1a (llama.cpp on Vulkan), 55 seconds on the Mac mini and 56 seconds on the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2). With thinking off it came after 1.0 second at most.

  With 16 users at once, the RTX 4090 desktop wrote 307 tokens a second in total with Gemma 4 12B, about 38.4 for each user. That is 3.4 times what one user on their own got (307 divided by 90.5).

  With 16 users at once, the RTX 5070 Ti desktop wrote 200 tokens a second in total with Gemma 4 12B, about 32.5 for each user. That is 2.4 times what one user on their own got (200 divided by 82.6).

  With 16 users at once, the HP ZBook Ultra G1a (llama.cpp on Vulkan) wrote 36 tokens a second in total with Gemma 4 12B, about 3.7 for each user. That is 1.6 times what one user on their own got (36 divided by 22.8).

  With 16 users at once, the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) wrote 35 tokens a second in total with Gemma 4 12B, about 5.9 for each user. That is 1.7 times what one user on their own got (35 divided by 20.6).

  With 4 users at once, the Mac mini M4 wrote 26 tokens a second in total with Gemma 4 12B, about 9.5 for each user. That is 2.4 times what one user on their own got (26 divided by 10.9).

  With 16 users at once, the Lenovo Yoga Pro 9n (llama.cpp on CUDA) wrote 82 tokens a second in total with Gemma 4 12B, about 14.8 for each user. That is 2.5 times what one user on their own got (82 divided by 32.6).

  With 16 users at once, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) wrote 76 tokens a second in total with Gemma 4 12B, about 13.5 for each user. That is 2.7 times what one user on their own got (76 divided by 28.1).

• Gemma 4 26B-A4B on four machines: the RTX 4090 desktop writes code 4.7 times as fast as the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2), and the RTX 4090 desktop reads a long document 27 times as fast as the HP ZBook Ultra G1a (llama.cpp on Vulkan).MoreLess

  With the same model on each (Gemma 4 26B-A4B, UD-IQ4_XS, about 4 bits, GGUF), the RTX 4090 desktop wrote code at 161.4 tokens a second, the Lenovo Yoga Pro 9n (llama.cpp on CUDA) at 60.7, the RTX 5070 Ti desktop at 59.4, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 58.3, the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 43.3 and the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 34.6.

  The RTX 4090 desktop read a document of about 19,000 tokens at 7,257 tokens a second, the Lenovo Yoga Pro 9n (llama.cpp on CUDA) at 2,328, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 2,209, the RTX 5070 Ti desktop at 2,199, the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 596 and the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 271. That is a wait of about 3 seconds on the 4090, 8 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA), 8 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2), 8 seconds on the 5070 Ti, 31 seconds on the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) and 69 seconds on the HP ZBook Ultra G1a (llama.cpp on Vulkan) before the first word (the size of the document divided by each speed).

  With Gemma 4 26B-A4B thinking before it answers, the first word of the code answer came after about 5 seconds on the 4090, 15 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2), 17 seconds on the 5070 Ti, 17 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA), 22 seconds on the HP ZBook Ultra G1a (llama.cpp on Vulkan) and 31 seconds on the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2). With thinking off it came after 0.9 seconds at most. In 1 of the 3 runs on the HP ZBook Ultra G1a (llama.cpp on Vulkan) it was still thinking when the 2,000 tokens ran out, so that wait comes from the runs that answered.

  With 16 users at once, the RTX 4090 desktop wrote 383 tokens a second in total with Gemma 4 26B-A4B, about 39.9 for each user. That is 2.7 times what one user on their own got (383 divided by 139.5).

  With 16 users at once, the RTX 5070 Ti desktop wrote 124 tokens a second in total with Gemma 4 26B-A4B, about 9.5 for each user. That is 2.3 times what one user on their own got (124 divided by 54.3).

  With 16 users at once, the HP ZBook Ultra G1a (llama.cpp on Vulkan) wrote 52 tokens a second in total with Gemma 4 26B-A4B, about 4.7 for each user. That is 1.4 times what one user on their own got (52 divided by 37.6).

  With 16 users at once, the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) wrote 48 tokens a second in total with Gemma 4 26B-A4B, about 6.7 for each user. That is 1.5 times what one user on their own got (48 divided by 31.9).

  With 16 users at once, the Lenovo Yoga Pro 9n (llama.cpp on CUDA) wrote 116 tokens a second in total with Gemma 4 26B-A4B, about 16.4 for each user. That is 1.9 times what one user on their own got (116 divided by 59.6).

  With 16 users at once, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) wrote 102 tokens a second in total with Gemma 4 26B-A4B, about 14.9 for each user. That is 2.0 times what one user on their own got (102 divided by 51.9).

• gpt-oss-20b on four machines: the RTX 4090 desktop writes code 4.6 times as fast as the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2), and reads a long document 9.0 times as fast.MoreLess

  With the same model on each (gpt-oss-20b, MXFP4, GGUF, thinking at its lowest level), the RTX 4090 desktop wrote code at 238.6 tokens a second, the RTX 5070 Ti desktop at 213.2, the Lenovo Yoga Pro 9n (llama.cpp on CUDA) at 80.4, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 78.2, the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 67.2 and the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 51.6.

  The RTX 4090 desktop read a document of about 15,000 tokens at 9,940 tokens a second, the RTX 5070 Ti desktop at 7,884, the Lenovo Yoga Pro 9n (llama.cpp on CUDA) at 4,415, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 3,530, the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 1,220 and the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 1,102. That is a wait of about 1.5 seconds on the 4090, 1.9 seconds on the 5070 Ti, 3 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA), 4 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2), 13 seconds on the HP ZBook Ultra G1a (llama.cpp on Vulkan) and 14 seconds on the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) before the first word (the size of the document divided by each speed).

  With gpt-oss-20b thinking before it answers, the first word of the code answer came after about 1.8 seconds on the 5070 Ti, 3 seconds on the 4090, 4 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA), 4 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2), 7 seconds on the HP ZBook Ultra G1a (llama.cpp on Vulkan) and 7 seconds on the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2). With thinking off it came after 0.3 seconds at most.

  With 16 users at once, the RTX 4090 desktop wrote 624 tokens a second in total with gpt-oss-20b, about 58.6 for each user. That is 3.1 times what one user on their own got (624 divided by 202.1).

  With 16 users at once, the RTX 5070 Ti desktop wrote 560 tokens a second in total with gpt-oss-20b, about 50.9 for each user. That is 3.1 times what one user on their own got (560 divided by 182.8).

  With 16 users at once, the HP ZBook Ultra G1a (llama.cpp on Vulkan) wrote 98 tokens a second in total with gpt-oss-20b, about 8.9 for each user. That is 1.8 times what one user on their own got (98 divided by 55.5).

  With 16 users at once, the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) wrote 91 tokens a second in total with gpt-oss-20b, about 11.8 for each user. That is 1.9 times what one user on their own got (91 divided by 48.6).

  With 16 users at once, the Lenovo Yoga Pro 9n (llama.cpp on CUDA) wrote 200 tokens a second in total with gpt-oss-20b, about 24.0 for each user. That is 2.5 times what one user on their own got (200 divided by 81.6).

  With 16 users at once, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) wrote 180 tokens a second in total with gpt-oss-20b, about 21.4 for each user. That is 2.4 times what one user on their own got (180 divided by 75.7).

• Qwen 3.5 9B on five machines: the RTX 4090 desktop writes code 7.8 times as fast as the Mac mini M4, and reads a long document 46 times as fast.MoreLess

  With the same model on each (Qwen 3.5 9B, Q4_K_M, GGUF), the RTX 4090 desktop wrote code at 137.6 tokens a second, the RTX 5070 Ti desktop at 126.6, the Lenovo Yoga Pro 9n (llama.cpp on CUDA) at 44.6, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 40.5, the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 38.1, the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 32.9 and the Mac mini M4 at 17.7.

  The RTX 4090 desktop read a document of about 19,000 tokens at 8,584 tokens a second, the RTX 5070 Ti desktop at 5,275, the Lenovo Yoga Pro 9n (llama.cpp on CUDA) at 2,684, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 2,524, the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 935, the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 775 and the Mac mini M4 at 185. That is a wait of about 2 seconds on the 4090, 4 seconds on the 5070 Ti, 7 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA), 7 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2), 20 seconds on the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2), 24 seconds on the HP ZBook Ultra G1a (llama.cpp on Vulkan) and 101 seconds on the Mac mini before the first word (the size of the document divided by each speed).

  With Qwen 3.5 9B thinking before it answers, the first word of the code answer came after about 1.5 seconds on the 4090 and 5 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2). With thinking off it came after 0.3 seconds at most. In 2 of the 3 runs on the 4090 it was still thinking when the 2,000 tokens ran out, so that wait comes from the one run that answered. In 2 of the 3 runs on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) it was still thinking when the 2,000 tokens ran out, so that wait comes from the one run that answered.

  On the code prompt with thinking on, Qwen 3.5 9B was still thinking when the 2,000 tokens ran out, which took 16 seconds on the 5070 Ti, 45 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA), 53 seconds on the HP ZBook Ultra G1a (llama.cpp on Vulkan), 61 seconds on the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) and 113 seconds on the Mac mini.

  On the prose prompt with thinking on, Qwen 3.5 9B was still thinking when the 2,000 tokens ran out, which took 15 seconds on the 4090, 16 seconds on the 5070 Ti, 45 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA), 50 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2), 53 seconds on the HP ZBook Ultra G1a (llama.cpp on Vulkan) and 61 seconds on the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2).

  With 16 users at once, the RTX 4090 desktop wrote 576 tokens a second in total with Qwen 3.5 9B, about 49.1 for each user. That is 4.5 times what one user on their own got (576 divided by 126.8).

  With 16 users at once, the RTX 5070 Ti desktop wrote 420 tokens a second in total with Qwen 3.5 9B, about 40.3 for each user. That is 3.7 times what one user on their own got (420 divided by 113.4).

  With 16 users at once, the HP ZBook Ultra G1a (llama.cpp on Vulkan) wrote 76 tokens a second in total with Qwen 3.5 9B, about 5.9 for each user. That is 2.1 times what one user on their own got (76 divided by 35.8).

  With 16 users at once, the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) wrote 107 tokens a second in total with Qwen 3.5 9B, about 11.5 for each user. That is 3.5 times what one user on their own got (107 divided by 30.9).

  With 16 users at once, the Mac mini M4 wrote 44 tokens a second in total with Qwen 3.5 9B, about 4.2 for each user. That is 2.8 times what one user on their own got (44 divided by 15.9).

  With 16 users at once, the Lenovo Yoga Pro 9n (llama.cpp on CUDA) wrote 188 tokens a second in total with Qwen 3.5 9B, about 17.1 for each user. That is 4.3 times what one user on their own got (188 divided by 43.9).

  With 16 users at once, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) wrote 167 tokens a second in total with Qwen 3.5 9B, about 15.6 for each user. That is 4.4 times what one user on their own got (167 divided by 38.0).

• Ternary Bonsai 2 27B on three machines: the RTX 4090 desktop writes code 11 times as fast as the Mac mini M4, and reads a long document 63 times as fast.MoreLess

  With the same model on each (Ternary Bonsai 2 27B, PTQ1_0, about 1.75 bits a weight, GGUF), the RTX 4090 desktop wrote code at 105.5 tokens a second, the RTX 5070 Ti desktop at 98.8 and the Mac mini M4 at 9.4.

  The RTX 4090 desktop read a document of about 19,000 tokens at 2,941 tokens a second, the RTX 5070 Ti desktop at 1,730 and the Mac mini M4 at 47. That is a wait of about 6 seconds on the 4090, 11 seconds on the 5070 Ti and 399 seconds on the Mac mini before the first word (the size of the document divided by each speed).

  With Ternary Bonsai 2 27B thinking before it answers, the first word of the code answer came after about 4 seconds on the 4090, 6 seconds on the 5070 Ti and 40 seconds on the Mac mini. With thinking off it came after 1.9 seconds at most.

  On the prose prompt with thinking on, Ternary Bonsai 2 27B was still thinking when the 2,000 tokens ran out, which took 19 seconds on the 4090 and 21 seconds on the 5070 Ti.

  With 16 users at once, the RTX 4090 desktop wrote 309 tokens a second in total with Ternary Bonsai 2 27B, about 29.5 for each user. That is 3.3 times what one user on their own got (309 divided by 93.6).

  With 16 users at once, the RTX 5070 Ti desktop wrote 213 tokens a second in total with Ternary Bonsai 2 27B, about 22.7 for each user. That is 2.5 times what one user on their own got (213 divided by 86.9).

  With 4 users at once, the Mac mini M4 wrote 5 tokens a second in total with Ternary Bonsai 2 27B, about 1.5 for each user. That is 0.7 times what one user on their own got (5 divided by 7.6).

• Nemotron 3.5 Bolt 74B on two machines: the Lenovo Yoga Pro 9n (llama.cpp on CUDA) writes code 1.8 times as fast as the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2), and reads a long document 3.5 times as fast.MoreLess

  With the same model on each (Nemotron 3.5 Bolt 74B, NVIDIA's 24GB GGUF, about 2 bits a weight), the Lenovo Yoga Pro 9n (llama.cpp on CUDA) wrote code at 41.5 tokens a second, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 37.4, the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 28.6 and the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 23.3.

  The Lenovo Yoga Pro 9n (llama.cpp on CUDA) read a document of about 18,000 tokens at 1,153 tokens a second, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 1,081, the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 371 and the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 332. That is a wait of about 16 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA), 17 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2), 50 seconds on the HP ZBook Ultra G1a (llama.cpp on Vulkan) and 56 seconds on the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) before the first word (the size of the document divided by each speed).

• Qwen 3.6 35B-A3B on two machines: the Lenovo Yoga Pro 9n (llama.cpp on CUDA) writes code 1.8 times as fast as the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2), and reads a long document 2.5 times as fast.MoreLess

  With the same model on each (Qwen 3.6 35B-A3B, UD-Q4_K_M, about 4 bits, GGUF), the Lenovo Yoga Pro 9n (llama.cpp on CUDA) wrote code at 72.1 tokens a second, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 69.7, the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 52.3 and the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 39.2.

  The Lenovo Yoga Pro 9n (llama.cpp on CUDA) read a document of about 19,000 tokens at 2,339 tokens a second, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 2,117, the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 1,006 and the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 935. That is a wait of about 8 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA), 9 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2), 19 seconds on the HP ZBook Ultra G1a (llama.cpp on Vulkan) and 20 seconds on the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) before the first word (the size of the document divided by each speed).

  On the code prompt with thinking on, Qwen 3.6 35B-A3B was still thinking when the 2,000 tokens ran out, which took 28 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA), 29 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2), 38 seconds on the HP ZBook Ultra G1a (llama.cpp on Vulkan) and 52 seconds on the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2).

  On the prose prompt with thinking on, Qwen 3.6 35B-A3B was still thinking when the 2,000 tokens ran out, which took 28 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA), 29 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2), 38 seconds on the HP ZBook Ultra G1a (llama.cpp on Vulkan) and 52 seconds on the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2).

  With 16 users at once, the HP ZBook Ultra G1a (llama.cpp on Vulkan) wrote 75 tokens a second in total with Qwen 3.6 35B-A3B, about 5.7 for each user. That is 1.5 times what one user on their own got (75 divided by 50.0).

  With 16 users at once, the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) wrote 68 tokens a second in total with Qwen 3.6 35B-A3B, about 7.7 for each user. That is 2.0 times what one user on their own got (68 divided by 34.0).

  With 16 users at once, the Lenovo Yoga Pro 9n (llama.cpp on CUDA) wrote 161 tokens a second in total with Qwen 3.6 35B-A3B, about 14.8 for each user. That is 2.3 times what one user on their own got (161 divided by 70.1).

  With 16 users at once, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) wrote 143 tokens a second in total with Qwen 3.6 35B-A3B, about 13.5 for each user. That is 2.4 times what one user on their own got (143 divided by 60.6).

• Qwen 3.8 27B with multi-token prediction on two machines: the Lenovo Yoga Pro 9n (llama.cpp on CUDA) writes code 1.6 times as fast as the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2), and the Lenovo Yoga Pro 9n (llama.cpp on CUDA) reads a long document 3.6 times as fast as the HP ZBook Ultra G1a (llama.cpp on Vulkan).MoreLess

  With the same model on each (Qwen 3.8 27B with multi-token prediction, Q4_K_M, GGUF), the Lenovo Yoga Pro 9n (llama.cpp on CUDA) wrote code at 33.3 tokens a second, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 30.7, the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 25.0 and the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 20.2.

  The Lenovo Yoga Pro 9n (llama.cpp on CUDA) read a document of about 19,000 tokens at 772 tokens a second, the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2) at 734, the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) at 237 and the HP ZBook Ultra G1a (llama.cpp on Vulkan) at 215. That is a wait of about 24 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA), 26 seconds on the Lenovo Yoga Pro 9n (llama.cpp on CUDA, in WSL 2), 79 seconds on the HP ZBook Ultra G1a (llama.cpp on ROCm, in WSL 2) and 87 seconds on the HP ZBook Ultra G1a (llama.cpp on Vulkan) before the first word (the size of the document divided by each speed).

• Qwen 3.5 9B on the Mac mini M4: it reads a document of about 19,000 tokens in about 91 seconds, and 16 users at once get 3.1 times what one user gets, in total.MoreLess

  The Mac mini M4 wrote code at 20.5 tokens a second with Qwen 3.5 9B (MLX 4-bit), and read a document of about 19,000 tokens at 207 tokens a second. That is a wait of about 91 seconds before the first word (the size of the document divided by the speed).

  On the code prompt with thinking on, Qwen 3.5 9B was still thinking when the 2,000 tokens ran out, which took 98 seconds on the Mac mini.

  On the prose prompt with thinking on, Qwen 3.5 9B was still thinking when the 2,000 tokens ran out, which took 98 seconds on the Mac mini.

  With 16 users at once, the Mac mini M4 wrote 54 tokens a second in total with Qwen 3.5 9B, about 5.2 for each user. That is 3.1 times what one user on their own got (54 divided by 17.6).

The machines

Four DGX Sparks4 x 128GB, joined at 200Gb/s. Graphics chips capped at 2,200MHz to cut power use.8 results below

Two DGX Sparks2 x 128GB, joined at 200Gb/s. Graphics chips capped at 2,200MHz to cut power use.3 results below

One DGX Spark128GB, one box on its own. Graphics chips capped at 2,200MHz to cut power use.3 results below

RTX 4090 desktop24GB of graphics memory, Ryzen 9 7950X, 96GB of memory7 results below

RTX 5070 Ti desktop16GB of graphics memory, Ryzen 9 5950X, 32GB of memory5 results below

HP ZBook Ultra G1aA Strix Halo laptop: Ryzen AI Max+ 395 with Radeon 8060S graphics, and 64GB of memory shared between them19 results below

Mac mini M4Apple M4 with 16GB of memory shared between the processor and the graphics4 results below

Lenovo Yoga Pro 9n (RTX Spark)An RTX Spark laptop: NVIDIA's N1X, an Arm processor with Blackwell graphics, and 128GB of memory shared between them19 results below

The same test with thinking switched on

Thinking is the model working through the problem before it answers. For DeepSeek V4.1 Flash 1017, thinking on means high, the middle of the model's three thinking levels (75 on its scale of 1 to 100), on every machine. For GLM 5.3 Flash, thinking on means high, the middle of the model's three thinking levels, on every machine. For Qwen 3.8 27B, thinking on means medium, the middle of the model's three thinking levels, on every machine. For Gemma 4 12B, thinking on means its single thinking setting (it has no levels), on every machine. For Gemma 4 26B-A4B, thinking on means its single thinking setting (it has no levels), on every machine. For gpt-oss-20b, thinking on means medium, the middle of the model's three thinking levels, on every machine. For Qwen 3.5 9B, thinking on means its single thinking setting (it has no levels), on every machine. For Ternary Bonsai 2 27B, thinking on means medium, the middle of the model's three thinking levels, on every machine. For Qwen 3.6 35B-A3B, thinking on means its single thinking setting (it has no levels), on every machine. The speeds in this table count the tokens it thinks with and the tokens of the answer. The wait is from sending the prompt to the first word of the answer, and it is the middle of three runs, because the model thinks for a different length each time. The cap of 2,000 tokens covers the thinking and the answer together. A wait marked "one run only" was timed in a single separate run with room for 8,000 tokens, because the three runs of that row were made before the test timed the answer. Where a wait begins with "over", the model was still thinking when the 2,000 tokens ran out. The speed beside it is then the speed of its thinking, and the real wait is longer than the one shown.

Several users at once

Several people, or several agents, using one machine at once. Each column is that many requests started together, each on a different subject, with about 1,000 tokens of prompt and room for 512 tokens of answer, thinking off. The figure is all of them together, in tokens a second, the middle of three runs, and under it is what each user got. Every answer here is prose, so the one-user column is closer to the prose figure in the first table than to the code one.

A coding agent's whole job, timed

A coding agent's recorded work, played back by Artificial Analysis's AgentPerf-Local (version 0.3.3): the same 168 requests on every machine, each with its recorded prompt, and the model made to write exactly the recorded number of tokens for each, 30,883 in all, so every machine does the same work. The figure is the time to answer all of them, without the time the agent's tools took, so lower is better. Multi-token prediction and DFlash both draft a few tokens ahead for the model to check in one pass, which saves time when the guesses are right. Each row is the quickest of its runs.