Anthropic's report on GLM 5.3: what its tests found
Anthropic published a security report on GLM 5.3, the family of open models my Sparks run. By Anthropic's count the model turned a known Chrome flaw into a working attack in 50 of 410 attempts, and as far as I could see on 3 October nothing had changed for anyone running it at home.
Plays here, from YouTube
On Tuesday 29 September, Anthropic published a security report on GLM 5.3, the family of open models my Sparks run. When I read Artificial Analysis's open-weights leaderboard on 2 October, the full model sat second with 45 points, and the smaller Flash version that I run was fourth with 42.
What Anthropic tested
Anthropic tested whether the model could take a known flaw in V8, the engine Chrome uses, and build a working attack on its own. The test is called ExploitBench. By Anthropic's count GLM 5.3 managed that in 50 of 410 attempts. Anthropic's own Mythos Preview, which Anthropic says only went to trusted security teams, managed 56 of 410, and the report calls the two rates similar.
The report also quotes NIST's CAISI, an American government institute, which had already called GLM 5.3 the most cyber capable open model released so far.
The Flash version
Anthropic says the smaller Flash version needed very little direction from a researcher to link exploits for two flaws into a chain. The report puts that at 20 minutes of the researcher's attention and eight hours of the model's time, which Anthropic prices at $20.40. That is Anthropic's estimate at Zhipu's API prices, and nobody paid it.
Its refusals
The report says GLM 5.3's refusals are easy to get around. In Anthropic's tests the model engaged with none of the direct requests, and that rose to 64%, 92% and 100% under three ways of getting past its refusals. Because the weights are open, the report says, the refusals can be removed altogether with little change in what the model can do.
The reaction
On Reddit, the most popular post of that week on r/LocalLLaMA called the report the greatest advertisement GLM has ever had. A second thread asked whether people were worried about a ban on Chinese open-weight models. When the video came out on 3 October I had not seen a ban announced, and the report does not propose one.
The report itself asks for government safety testing and better safeguards, and says defenders can use the same ability. As far as I could see, nothing had changed for anyone who was running the model at home. On 2 October Flash was still published on Hugging Face under an MIT licence.
Read more
- Anthropic's report on GLM 5.3anthropic.com
- Artificial Analysis, open-weights leaderboardartificialanalysis.ai
- The r/LocalLLaMA post that called the report an advertisement for GLMreddit.com
- The r/LocalLLaMA thread asking about a ban on Chinese open-weight modelsreddit.com
- GLM 5.3 Flash on Hugging Facehuggingface.co
The numbers in this story
| Figure | What it is | From | Kind |
|---|---|---|---|
| 45 (second) and 42 (fourth) | GLM 5.3 and GLM 5.3 Flash on the open-weights leaderboard | artificialanalysis.ai | Artificial Analysis's score |
| 50 of 410 and 56 of 410 | Attempts that ended in a working attack, GLM 5.3 and then Anthropic's Mythos Preview | anthropic.com | Anthropic's testing |
| 20 minutes, 8 hours and $20.40 | The researcher's attention, GLM 5.3 Flash's time and Anthropic's estimate of the cost | anthropic.com | Anthropic's testing |
| 64% to 100% of the time | How often GLM 5.3's refusals were bypassed in Anthropic's tests | anthropic.com | the maker's claim |
If a number on this page is wrong, I would rather know. Tell me in the comments under the video and I will correct it here and pin the correction there. What has been corrected so far is on the corrections page.