Rendered at 15:01:59 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
cultofmetatron 2 days ago [-]
I seriously dont' know all this big hullabaloo about one shot prompting.
by definition, a single prompt wont' constitute the complexity of a software project. ergo, what you'll get is a series of assumptions made by the model based on preexisting code in its training corpus.
I'd rather see a coding agent that can follow steps in a plan file to a T while following guardrails and adhering to the proper coding conventions in the human reviewed spec.
Id rather see performance in agent loops against human defined objectives where it can be verified to stick to defined guardrails and continue without drift till its objectives are complete.
I'd also like to see it identify bugs and potential performance increases by identifying existing code and suggesting refactors based on context it can pickup about the particular use case you are trying to create.
These are way more valuable metrics than "hey build X"
post-it 2 days ago [-]
The streetlight effect:
> A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, "this is where the light is"
All of your suggestions are better but they're hard, so someone casually evaluating an AI isn't going to do them.
sanderjd 2 days ago [-]
Sure, for casual evaluation, I agree. But are there serious analyses that are evaluating this kind of thing? I mean, these are the kinds of things I evaluate in my own work when a new model comes out, or when I'm evaluating a harness. But this is all very ad hoc and intuitional. I'd love to start bringing rigor to it, but I haven't found much prior art on this. In another thread someone said that's because it's probably impossible to do this rigorously because too much of it is subjective. And that does match my intuition. But I continue to suspect that intuition is wrong.
jerf 2 days ago [-]
It's hard to bring much rigor to it. I'm not saying impossible, but it's not like it's completely obvious how to do it and people are just too lazy. Intrinsically, if I'm going to test a back-and-forth with a model I have a human in the loop making frequent decisions. Did the model fail or succeed at whatever rate it did that because of the model or the human? Did the testing protocols capture the actual problem, e.g., maybe if the model was given some particular bit of information that a normal human would have given it it would have done much better or worse, but the testing protocol in the interests of "rigor" excluded the human in the loop from doing it. Is the human going to be willing to sit down and do the same task 25 times, refreshing the model from scratch each time for a "valid" test? Can you get the same human to analyze every model in the test? Is their 10th pass of the problem an invalid test because you can't as easily erase the human's knowledge of the previous 9 tests? What do you do with a model that succeeds wildly 75% of the time and spins off into a loop the other 25%? Is that loop real or, again, did your "rigorous" testing protocol prevent the human from saving the model from the loop like any developer would?
And so on and so forth. Again, I'm not saying this is impossible but I am saying that if you tried to do it, and you got the money, and you built the test, and got the human subjects clearance, and you ignored that during the process of all that at least one more frontier model would come out, you can count on HN anklebiting your "rigorous" study even so, and probably being correct about a lot of the issues it could have because it would take several iterations of this to build a reasonable protocol... at which point it would quite possibly also be obsoleted by progress again.
sanderjd 2 days ago [-]
Yes, well put.
abhgh 2 days ago [-]
You usually see this kind of analyses in conference papers, esp. if they have a datasets track. The NeurIPS Datasets & Benchmarks (D&B) track is a good example. But you will have to monitor the proceedings yourself closely - there is little chance of being accidentally exposed to them, because most blogs, announcements and popular media only mention a handful of the popular ones, e.g., Tau^2. For ex., across the years 2022, 2023 and 2024, 900+ papers were accepted in the D&B track [1] - of course, not all of them are LLM-related. I find them interesting because they often focus on specific system behaviors, and like you said, study them scientifically, so you can draw authoritative conclusions (or at least know specifically what part of a model's behavior you now know about, and what parts you don't).
You might like SWE-WebDevBench which tries to do this comprehensive evals for webapp development. https://webdevbench.com/
2 days ago [-]
redsocksfan45 2 days ago [-]
[dead]
newaccountman2 2 days ago [-]
[flagged]
blanched 2 days ago [-]
This kind of hamfisted snark tends to make people take the actual and justified criticism of police less seriously.
newaccountman2 2 days ago [-]
If people were willing to take it seriously in the first place, then they wouldn't view it as "hamfisted snark"
blanched 2 days ago [-]
I consider myself someone who takes it seriously, and have spent time and resources fighting for change. But it’s wholly unrelated to this particular thread, phenomenon, and story. So having a little “ha ha” moment accomplishes nothing towards the actual cause. It makes people uncomfortable, but not the useful kind of uncomfortable.
That said, maybe we just disagree on how to drive change, and that’s fine. I’ll leave it.
post-it 2 days ago [-]
It could be a taxi driver if you like. Or an anarchist passing by on xir way to a protest.
layer8 2 days ago [-]
…in the US.
echelon 2 days ago [-]
The minute an open model breaks through and beats Claude Opus/Fable, it's over.
There are far more opportunities that can be served when the world's intellectuals have the raw weights and can fine tune, splice, distill, and reapply.
Imagine having raw unfettered access to Fable. It can be refit to structural biology. It can be fine tuned on the repo for smaller context requirements. It can be run cheaper and air gapped.
The world wants this.
digitaltrees 2 days ago [-]
I don’t think we need them. I think the models we have are good enough. It’s the orchestration layer that makes the biggest difference at this point. The open source models we have are capable of calling tools and the work is getting them to be capable enough to know which tools to call and what to do in response.
I think we are leaving the main frame era of AI and entering the PC era already. If there wasn’t a RAM shortage and we all had 2TB of ram and GPUs we would all have large local models or personal APIs serving our teams.
That’s why all the labs are moving to the App layer and moving away from being the API for intelligence like they were originally.
wahnfrieden 2 days ago [-]
They are absolutely not good enough
digitaltrees 2 days ago [-]
I disagree because I am getting real work done. But I have three Mac studios with 1.5 tb of ram and built my own harness around large models in my own ide (propelcode.app). It’s not perfect. And opus ChatGPT 5.5 are better but they’re good enough
noonething 2 days ago [-]
if you had the information to do the task perfectly then those open models should be fine no? if so, your just paying for knowledge at that point.
digitaltrees 2 days ago [-]
Yah that's my point.
wahnfrieden 2 days ago [-]
Maybe write “I don’t think I need them. The models we have are good enough for me.” instead then
digitaltrees 20 hours ago [-]
What do you think I am arguing?
wahnfrieden 19 hours ago [-]
That we don't need better models because they're good enough for our work
digitaltrees 12 hours ago [-]
I understand why you drew that conclusion from my post, but that's not what I am arguing in the context of this thread.
The OP said "as soon as an open source model of the capability of fable or gpt5.5 are released it's over because the world will build on top of them" which I took to mean some amazing new level of performance is only able to be unlocked with those models.
I am arguing that the level of performance the OP seems to imply doesn't require fablw level models for the work the OP envisions because the harnesses can actually drive the same performance increase.
There are a few papers that have demonstrated it. And my own use of fable made me feel it was not that big of a jump from opus 4.8 compared to better systems around it.
Its a subtle but important distinction.
barrenko 2 days ago [-]
As crazy as this sounds, and as much I don't want to believe it myself, I think we're still underestimating LLMs, and we're gonna get to that point pretty soon.
jupr 2 days ago [-]
The world does want this. Opus capabilities, in a box, securely tunneled to my family and I utilizing the resources I already have available to me which is, energy + network.
rdsubhas 2 days ago [-]
IMHO, It's not the oneshotting.
It's the "starting from empty slate" greenfield that's the real problem.
We used to make fun of Engineers who follow a README on a framework, test it on an empty project, and say "this framework is the best for our 10 year running production app". Greenfield mentality is always the solution to all problems and problem to all solutions.
One should still measure oneshotting, it's an important self-measurement metric - but against an established, large codebase.
keheliya 2 days ago [-]
There are upcoming benchmarks aimed at measuring the ability to work with brownfield tasks. (Of course, benchmarks can be gamed, but they are still better than unrealistic toy tasks that earlier generations of benchmarks used. Frontier labs are yet to use them in their tech reports or marketing material, though.:-)
I think this (for me at least) is the biggest pain point. Use styles and practices from this existing code base, even if they aren't documented explicitly in AGENTS.md or something. If we're importing a library somewhere that does what the agent is doing, reuse that same library - don't chose another one. If we have a pattern for unit tests, follow the same style. Etc. etc.
That issue, and the issue of "aesthetics", are the biggest complaints I have today. I don't know exactly how to define aesthetics, but it's when AI is making decisions that no experienced developer or designer would. They may be functionally correct but "ugly" to another developer or and end user.
An example is an case I ran in to yesterday where parsing a config, and failing and logging on a configuration error. It logged a specific item where the config was invalid but not what group or any notion of where in the config this error was. Of course, specific item names could be duplicated in different parts of the config. It's small, but correcting these minor things take time and they are the types of decisions no one would have made who had any experience writing code and debugging a config problem. This was Opus 4.8/max too.
bluGill 2 days ago [-]
At least they did some analysis. I've couple AI slop "X is the best tool for the job" that didn't even try it. (Worse, we are already using QT which has a tool for the job, and the QT tool works with the rest of the QT ecosystem unlike whatever AI told them)
gertlabs 2 days ago [-]
One-shot performance often translates to the most difficult problems a model will be able to understand. We run an evaluation that tests both agentic and one-shot performance, and we find that Chinese models are almost universally very good at using tools and a harness to iterate towards a better solution, whereas their initial response ranks relatively low.
Compare that to Gemini models, which have impressive fluid intelligence on the first response, but fail to call tools or explore correctly which limits their usefulness for agentic coding.
Neither will be great for coding in a computational chemistry repo for different reasons, but the model with strong one-shot performance will be less likely to make subtle errors indicative of poor understanding, so we weight both capabilities into their final score.
The latest Anthropic and OpenAI models excel in both domains.
> The latest Anthropic and OpenAI models excel in both domains.
Is that because OpenAI models are not a single model but a cluster of models which specialize different domains?
gertlabs 2 days ago [-]
By domain, I really meant "tool calling" and "one-shot fluid intelligence"
Anthropic models were the original leaders in tool calling and agentic work, even when other models felt significantly smarter in (Claude Sonnet 3.5 vs Gemini 2.5 Pro, for example). OpenAI models were the opposite, starting smart (more correct solutions on the first try) and got better at exploring and iterating with tools in 2026. The latest releases (Opus 4.5+ and GPT 5.4+) excel at both.
hnfong 2 days ago [-]
It's a proxy for what you actually want to measure.
Note that after the model generated a bunch of (intermediary) code, they still have to have it tested and get bugs fixed (via the agent/harness). In this "one shot" you still have agent loops against human defined objectives.
And these toy examples give some insight as to how the model performs. If the test were "here's some code written by $corp, please take these tickets and work on them" it may be a "real" example but nobody would be able to make sense of actually how "hard" it is, or how "well" the model did the job, besides the workers already familiar with the context.
At least everyone knows what a 3D game is.
bluGill 2 days ago [-]
As someone who works at $corp - there is a massive different in tickets. I've seen "The is not spelled 'teh'", and I've seen some other service is writing to memory causing a crash in my service (the later took months to track down since our code was correct and nothing gives a hint of where to look). Both problems are important to fix, but the first is so simple I don't care how good AI is (the hard part is getting it through the process)
ulrikrasmussen 2 days ago [-]
I guess the experiment is interesting to determine if a model can produce something subjectively valued as "good" based on fairly vague and open-ended specifications. The benchmark is not to determine if the output fits the input, but whether the output is internally consistent: it's a game, but does it behave as one would expect that any game behaves? Does it end when you each the goal, do you die when hitting the spikes, are there weird edge cases in behavior when you move around?
I think however that they should have used the same harness and also repeated the experiment a few times to judge the variance in results.
pu_pe 2 days ago [-]
It's true that no one is trying to one shot anything serious right now, but it's still an important metric. Claude Code and Opus really took off when they improved the harnessing enough that it would self-correct many of its mistakes without needing user input. In fact I think long-term autonomy (in the range of several hours) and self-correcting is going to be where we see most improvements in coming years.
bogtog 2 days ago [-]
> In fact I think long-term autonomy (in the range of several hours) and self-correcting is going to be where we see most improvements in coming years.
Right, model intelligence defines the scope of things they can one shot
I also suspect that users naturally calibrate to a model's useful scope, gradually getting positive/negative feedback and gradually making their requests bigger/smaller than before
dakolli 2 days ago [-]
it wont happen, its all a money grab.
OtomotO 2 days ago [-]
I think that LLMs will stay, but I also think we've plateaued and that big companies will fail and fall and we will have another years long "halt" of any real advancements coming to the public.
Similar to how ML was all the hype about 12 years ago and then it submerged again for a couple of years.
thewebguyd 2 days ago [-]
> we will have another years long "halt" of any real advancements coming to the public
One can hope. Probably an unpopular take here but I'm tired boss.
The software world has a huge backlog of things that can all be done with the tech we currently have, no breakthrough advancements needed, but none of it will get prioritized when we're all forced to run on the new and shiny treadmill. Ever since LLM hype its like the javascript culture of a new framework every 10 minutes has infected every other vertical of software development and I'm exhausted.
drob518 2 days ago [-]
Yep, I see it.
halyconWays 2 days ago [-]
"We did multi-shot prompting to try and get these two games into comparable states using these two different models."
"Well obviously you provided better follow-up prompts to the one that came out better."
Also nothing about human-provided plan files and guardrails preclude the one-shot benchmark test. Heavens, I almost said "real coding," but in "real agentic program creation" you'd obviously be doing multi-turn interaction with the agent, but how can you provide a fair test when the model's output n determines your n+1 response?
pegasus 2 days ago [-]
Sure, real-world usage is always more difficult to benchmark, but the additional issue with the one shot prompting benchmark is that by optimizing for it, models are nudged towards making all those assumptions they shouldn't really make. Maybe a better test would be to have a fully spec'd-out plan, but start with a one shot, high-level prompt and expect the agent to discover your preferences by repeatedly asking for clarifications. The system that manages to suss out more of the details in the hidden spec this way, in less steps and with less unnecessary questions would more likely to be a truly well-calibrated agent.
2 days ago [-]
miroljub 2 days ago [-]
That's precisely the difference between an engineer and a business guy.
The business guy would say "hey build me this and that" and would get _something_ to show of.
An engineer will have a long conversation with a llm about the exact requirements, tech stack, tradeoffs. He would understand what is built, how is it built, and refine on the fly until he gets something sensible.
It won't be as fast as "build this", but the result will be much better and more maintainable.
For the enginering workflow, you don't need Fable. Any model better or equivqlent to Sonnet 4.6 would do. Yes, sometimes it will hallucinate, sometimes it'll be wrong, but it's our job as engineers to correct it and have full ownership of the result.
tw1984 2 days ago [-]
what you said above is only true when the AI is not as smart/professional/knowledgable as that engineer.
miroljub 2 days ago [-]
Of course it's not. Otherwise, before telling it "do this app, make no mistakes" you would need to feed an AI with the complete relevant knowledge, history, and constraints, and then your prompt wouldn't be a one-liner, but a 3000-page document.
And yet, even the smartest AI in the world would give an alternative solution every time you invoke it. And you still need someone to judge what is right and what is not.
somenameforme 2 days ago [-]
Unless I'm missing something, the prompt he gave must have been fairly detailed because both games are basically identical.
But for a more practical issue, the ultimate goal of LLMs is to replace software engineers, or at least enable everybody to become a software engineer, to use a more up-beat phrasing that's no less accurate. And so an LLM's ability to reliably construct something from a poorly defined, contradictory, or otherwise flawed prompt, while accurately inferring intent is probably the first finish line.
metadat 2 days ago [-]
More likely is the models were trained on similar data.
throwawayffffas 2 days ago [-]
It's the same prompt but they also gave them the same asset pack.
johnfn 2 days ago [-]
I feel like on HN there is an endless cycle:
- Vibes are too subjective, I want an actual A/B test!
- An A/B test is too limited, I want a benchmark! (You are here.)
- Those benchmarks never seem to be reliable, I just go on vibes.
InsideOutSanta 2 days ago [-]
> I seriously dont' know all this big hullabaloo about one shot prompting.
It's a relatively objective way of testing LLMs, and I think it's pretty representative of how strong models are overall.
The outcome of this test mirrors how GLM 5.2 and Opus 4.8 work for me: they're both similarly capable of fully executing a given task, but Opus tends to have a bit more "taste" in how it handles unstated details or implicit requirements.
> what you'll get is a series of assumptions made by the model
Yes, but that's why we use these models in the first place. We don't want to explicitly write down all the details because that would mean writing code. So we write a higher-level, human-language spec, and let the LLM fill in the blanks. The question is how good they are at doing that.
scwoodal 2 days ago [-]
> I'd rather see a coding agent that can follow steps in a plan file to a T while following guardrails and adhering to the proper coding conventions in the human reviewed spec.
Guardrails/conventions should be enforced in linters, formatters, static analysis tooling; not specs/prompts.
Der_Einzige 2 days ago [-]
Wrong, custom "specs" i.e. schemas, are literally all we have for "real" guardrails with LLMs.
Nothing else operates on the logprobs level and literally bans continuations that fail your schema.
scwoodal 2 days ago [-]
Enforcing structured outputs from LLMs is not the same thing as using linters, formatters, static analysis to control how an agent writes code.
Der_Einzige 2 days ago [-]
No, it's not. It's strictly better.
scwoodal 2 days ago [-]
Can you share examples, links to Github of this approach? I'd like to learn.
Youden 2 days ago [-]
It's not always possible, or at least trivial. For example how do you enforce "prefer to reuse existing code over making a copy"? Is there a static analysis tool that will detect two pieces of code that do the same thing?
lets say you have a table that is partitioned. how do you lint/format "any select into this table MUST include the partition key in the predicate and any join must include it in the on." I'm not personally familiar with any static analysis tool that does this but its trivial to implement with an llm prompt. trivially easy to add to your automated PR reviews.
jonathanlydall 2 days ago [-]
In a C# project I worked on in the past we used a specification pattern in front of EF to achieve essentially statically verifiable rules like this.
The specification pattern is essentially a builder for a query (or more accurately in our case, for a C# expression for the ORM) and one of the main benefits is that you can put rules into the “build” method which returns the expression to be used, such as “throw an exception if the WithPartionKey() method hasn’t been called”.
Usual ORM disclaimer applies that you should double check the generated SQL is acceptably performant. The specification pattern can make this better or worse, it can result in contorted expressions which result in poorly performing SQL, but you could also put logic in your specification to “ensure” good SQL, e.g. by having a method which uses the correct expression structure which is known to generate good SQL (e.g. EF6 would generate terrible SQL for certain nested Any()’s, but if you used .Contains() instead it was fine) and the encapsulation means that all callers of the method get the benefit.
It worked well for us as reviewing the specification methods in PRs was pretty easy and less experienced developers on the project were more likely to do the performant and working thing than not.
scwoodal 2 days ago [-]
I would tell the LLM to write a custom rule/check for whatever the scenario is. Then when the CI gate is run, all my custom checks get deterministically run.
Elixir is where I prefer to build software, so it would be creating a custom Credo rule.
One shotting is useful to test but only with a huge prompt (eg, build something according to this spec).
I agree generating millions of tokens from a handful of input tokens doesn't convey anything meaningful to me.
athrowaway3z 2 days ago [-]
I think you're underestimating the elegance of "hey build X". It already captures a lot of what you're interested in.
Additionally, with "Hey build X" nobody is happy with the methodology and people rightfully complain about the set up.
Using your suggestion the methodology would require a lot of presumptions & arguments regarding why you choose it and think it relevant to people.
Either people would not "get" it quickly enough or would disagree/not be interested on the setup because its not how they use LLMs.
NichoPaolucci 2 days ago [-]
If a model can take a series of increasingly complex instructions and satisfy the requirements without human intervention, we can pretty easily decide how well overall the model does. And, judging better models just means adding more requirements to a task. So, I think it's a useful method (Even if it's not a realistic use case).
Of course, with a software engineer at the helm - the models are going to be able to be guided to produce much better output. (Or worse, depending on the engineer!)
locknitpicker 2 days ago [-]
> If a model can take a series of increasingly complex instructions and satisfy the requirements without human intervention (...)
This is the wrong metric to target. Today's models can feel one-shot but they are so at the expense of resilient ReAct loops that brute force their way out of the mess initial prompts created.
And each iteration is expensive.
Sometimes failing fast and early is better than going for one-shot models that try to mitigate the mess they created with reasoning steps and ReAct loops.
embedding-shape 2 days ago [-]
You seem to be missing the point of what parent is saying :)
To really evaluate how a model is to use in real life, it should have access to tools, and be able to iterate on something, like they do when you use them in an agent harness.
None of that iteration need necessarily to have a human driving it (although if you're building something you want to be able to maintain, you probably need a human driving the design and architecture), you can just let the model do a couple of tries and give it input into how it's doing, and you get something closer to how people use these models in reality.
scotty79 2 days ago [-]
Single prompt performance is interesting because best agentic results of yesterday turned out to be best single prompt results of today.
If we stopped developing LLMs the the only reasonable way to benchmark them would be to compare yheir performance with all the tricks we can build on top of them. Sine the are still developing rapidly any apples to apples comparison is worthwhile.
Of course this particular benchmark is not really single prompt but rather "agentic without steering".
digitaltrees 2 days ago [-]
Exactly this. I recently tried Claude code again to get the subsidy on fable rather than paying api prices and was so frustrated by how much it pushed autonomous behavior. It would start ignoring my planning documents, ignoring my coding conventions, reimplementing features and code already in the project (not sure it ever makes sense to have two auth systems in parallel or two websocket implementations for the same ui) and then in the most shocking interaction just refused to stop working and listen to my instructions. I think maybe it was because there was a subagent doing the work but it was a complete waste of time and effort.
I was using cursor, in large part because I could at least stop it when I need to.
I ended up building my own IDE from scratch so I can be more in the loop while also having the full agent experience.
ACCount37 2 days ago [-]
On one hand, that's sort of true for practical uses - and benchmarks notoriously undercount multi-turn settings.
On another, being able to reliably tackle minor tasks with no handholding is very valuable in itself. Sometimes implementation details are important, but often, the most important thing is to Get It Done.
LoganDark 2 days ago [-]
The thing with one-shot prompting is that it tests the ability for the model to make good choices on its own, rather than only instruction following.
Instruction following has been down for years, and while there are of course metrics that continue to improve as the frontier advances (for example, the ability to continue following the original instructions even as context grows), you can't really get that much better at performing a list of instructions as-written if the instructions are sufficiently precise enough that there's no wiggle room for interpretation (which seems to be what you are describing).
For example, one of the things that got me the most excited for Fable 5 was its ability to work for over eight hours straight on a single instruction and seemingly faithfully the entire time. That was something I observed personally after trying out the same workflow that runs for maybe two or three hours with Opus and then still needs followups. Fable needed no followups. That's a game changer for me compared to the prior state of the art.
That kind of stuff is going to end up being the most beneficial to people who are touching the edges of their knowledge or even exploring completely new areas. And that type of work is exactly the kind of work that makes agentic coding so powerful, even as much as it gets harder to judge the quality of the work when you lack the skills yourself. It's a good thing that the quality increases across the board, even for skilled practitioners.
For example, even people who know how to write inference engines or how matmul kernels work or how to optimize model architecture can't always predict just the sheer breadth of things agents can try to improve performance, and sometimes you get over some wall and reach a completely different optimum that you just wouldn't have reached in any reasonable amount of time by applying traditional knowledge even if you're an expert in the field.
That kind of stuff is amazing. And that's exactly the kind of stuff that one-shot prompting is testing for. It's kind of like testing for the model's "innovation", as much of an oxymoron that is.
alienbaby 2 days ago [-]
It doesn't teat the models ability to make good decisions on its own, it tests the models ability to make something that 'works'. Often you look inside and it does a whole load of questionable things that mostly work, sure, but if you say and designed it properly yourself you would likely come up with something for more sane and maintainable.
LoganDark 1 days ago [-]
That's only the fault of particular benchmarks, and that's also why it's important to offer the outputs in question that resulted in a particular score. I'm not sure that all or even most benchmarks do this, but it's important when selecting a model.
hintymad 2 days ago [-]
> I'd rather see a coding agent that can follow steps in a plan file to a T while following guardrails and adhering to the proper coding conventions in the human reviewed spec.
In fact, I'd rather see Anthropic publish a convincing project that does this using Claude. The project should be complex enough and novel enough to show the world how reliable and powerful Claude is. That is, Anthropic does not need Amodei or its employees to tell us that whatever percent of engineers will lose their jobs. They can just show us. Easily.
throwawayffffas 2 days ago [-]
While I generally agree, there is correlation, will it go into doom loops? Will it overthink obvious bugs? Will it hallucinate libraries and apis? If it does all that in the one shot prompt it will do it in your codebase as well. Yeah state of the art models won't do all that. But that was not the case only 6 months ago, and doing the one shot prompting demonstrates ti does not do all of the above. It's a quick first real easy to clear test, really.
Demiurge 2 days ago [-]
The bigger head scratcher to me isn’t even one prompt, it is the prompt to build a platformer from scratch. Who does that? Absolutely no one.
What about: take top 3 feature requests, top 3 bug reports for 3 popular open source projects and ask to solve those based on the issue contents and access to the project repos.
Even if you stay in a single prompt scenario, you could make it more realistic.
gchamonlive 2 days ago [-]
> a single prompt wont' constitute the complexity of a software project.
The top agent is for steering, but all subagents are mostly oneshot prompts
jaapz 2 days ago [-]
When the model produces reasonable results from one prompt, you could assume that it will also return reasonable results through the follow up prompts.
epolanski 2 days ago [-]
Yet this is how virtually everybody is benchmarking and fine tuning.
Since Opus 4.6 I've seen later Anthropic models being more and more capable on one hand, but also less useful on multi turn open tasks.
It feels like with each model they are more and more prone to go "their own way" and jump into the implementation as soon as they can.
I can't but blame it on benchmarks and fine tuning around prompt-to-solution work.
Revanche1367 2 days ago [-]
The argument is flawed, there is no logical reason to assume a single prompt won’t be sufficient to constitute the complexity of a software project. It may not be practical in many cases but there is too much variability in what is considered a complex software project and in the sufficiency of instruction in a single prompt to make that claim and say it’s “by definition.”
oblio 2 days ago [-]
And that prompt will basically be 2000 page spec Bible à la IBM circa 1960, see waterfall. Unless AI develops mindreading (and advanced mindreading at that), single prompt creation of actual complex software products will never happen. You'll one shot a simple non scientific calculator, but not Excel or Vim or Nginx.
trollbridge 2 days ago [-]
Why not? Given a proper spec, you should absolutely be able to one-shot Excel, particularly if we put it at the level of complexity of, say, Excel 1.0 for Mac.
Current models aren't capable of that, but that doesn't mean it's not possible.
oblio 2 days ago [-]
Seems like this would be a good time to use this famous quote:
> given the sufficiently smart compiler
For those unaware, this is a similar quote used by compiler proponents. The first full compiler was created in 1957 (+/- 70 years ago) and the "sufficiently smart compiler" never happened, hand written code from the best coders still is faster. Now, that doesn't mean that compilers didn't do the job well enough, we just accepted that 90-95% of the top speed was enough for almost everything.
To the LLM one shotting point, it took 30 (40?) years for compilers to be good enough for the mass market. Caveat early adopter and investor.
Plus what pyrale said.
pyrale 2 days ago [-]
The issue is not the models, the issue is that this method ws tried before, and humans suck at writing what they want. Developing in small increments allowing feedback was an answer to this issue.
If you made models able to code to long spec, you would be left with the hard issue of having to write them.
pianopatrick 2 days ago [-]
An interesting question for me is "can the LLMs predict what humans want?".
Like if you show the LLM a page, can the LLM review the page and then spit out a review that is close to what a human would say about the page?
trollbridge 2 days ago [-]
Yes, my current nightmare is I have a very long queue of specs to write and need to work with non technical staff to help them put in words what it is they actually want.
Software was always that way, though.
dakolli 2 days ago [-]
One shot prompting/tooling is the only reasonable way to use an llm in my opinion. You should not be having an LLM operating for hours creating thousands of lines of new code that you can never review or maintain. You can actually be highly productive modifying a single file or two at a time, ideally as focused and little context as possible, without the llm being given full permission to add as much context as possible along the way to maximize revenue for the developers of the harness.
The agentic engineering paradigm is just a narrative trend pushed by AI companies to get people to 10x their token consumption per prompt. It plays into people's laziness and addiction to dopamine too causing addict like behavior in people that fall prey to this trend.
ffsm8 2 days ago [-]
I disagree fundamentally.
If I do that, I'm literally slower then just doing the change without sufficiently specifying it to the model.
I can see how a junior dev or generally someone that's not particularly knowledgeable about the language or framework they're working with may benefit from such usage, but for experienced people there is very little value in that approach.
I say this because I've just had to face this decision this month with Copilot introducing the usage based billing. I attempted to scale back my usage, first with non-opus - output essentially became discardable as it continually hallucinated no existing fields in the responses of Apis etc... Then my scoping the changes smaller and smaller, until I ultimately gave up and reduced usage to just generating tests.
user43928 2 days ago [-]
I agree. And at work it has been producing some of the worst GUI test cases I have ever seen.
What is tested often makes no sense at all, completely implausible edge cases are tested on internals, while it doesn't create tests for the overall application using user events.
And some things in these test cases are downright ridiculous: instead of instantiating your classes, it sets up some barebones fake objects reimplementing some of the behavior of your actual class, then ignores the TypeScript errors via force cast or similar.
Then it proceeds to slap some test ids on the output, stubs components and dependencies more or less randomly, adds some assertions on test ids and calls it a day.
Apparently that's good enough for many colleagues to open a MR for that garbage.
That said, at home with SOTA models I happily hand large units of work to it, outsource much of the thinking, and get workable results. I think this is the future.
2 days ago [-]
2 days ago [-]
dakolli 2 days ago [-]
I disagree, fundamentally.
I see little value in throwing a ton of context at an llm and waiting 10-20 minutes for a coin flip on whether or not its going to produce junk. I'd rather do quick 60 second turns, get most of the way there and fix the rest myself if I have to. I'd rather honestly just not use them.
ffsm8 2 days ago [-]
Well the point was that id rather spend 30 seconds doing it myself then formulate a prompt with enough context for the model to implement it within 60 seconds. Also these numbers are unrealistic.
Everyone that I've ever interacted with and claims to prompt in "seconds" actually needs multiple minutes to think about the solution they want the model to implement - and then need twice as long to formulate that into a sentence which provides the model enough context to actually do that
So the more realistic estimates are "I'd rather spend the 2 minutes just implementing the minor change myself, instead of spending 1.5 minutes thinking about it, then 2.5 minutes writing the prompt and then waiting 1 minute for it to finish"
dakolli 2 days ago [-]
I would agree with all those points, and my numbers are a little off. I really just don't want to use any of it. I'm more excited about fast FIM autocomplete that works well, something like cursor tab without cursor. If something can increase my wpm and take strain off my fingers that would be nice. At this point latency and accuracy is terrible though.
atq2119 2 days ago [-]
The trick is to do something else in those 20 minutes (or, ideally, even longer).
That's the main value I've been getting out of coding agents. I have them do (comparatively) simpler tasks or explorative tasks in the background while I'm in a meeting, doing code reviews, or otherwise working on something else.
cindyllm 2 days ago [-]
[dead]
2 days ago [-]
segmondy 2 days ago [-]
What are you yapping about? This was not one shot prompting, but a long run horizon task. But GLM and Opus invoked at least 120+ tools across the runs.
canes123456 2 days ago [-]
Isn't a plan file just a single long prompt?
irthomasthomas 2 days ago [-]
Blame anthropic, they decided to make these type of one-shot examples the primary focus of the Fable 5 release, and relegating benchmark scores to the pdf.
jatora 2 days ago [-]
I also love the term zero-shot in the AI benchmark world. So logical. So intuitive.........
moistoreos 2 days ago [-]
PREACH. I have no idea why THIS has become the standard for illustrating model capabilities. It's endlessly frustrating when that was the initial objective for all these models, but, became increasingly clear over time that none of these models were ever capable of getting the desired output for complex software on the initial prompt.
The reality is:
- business rules change
- ideas for improvement may arise from the initial prompt
- updates to submodules/functions/configs/secrets are BLOCKERS
... etc.
One shot prompting for the expecations of complete software is seemingly more and more a show of incompetence of the use of this technology. It's like trying to make my toddler eat a ham sandwich from the peanut butter & jelly I put in front of him.
throw234234234 2 days ago [-]
My opinion is that it is because the measure isn't aimed at technical people. It's aimed at clients that just want to specify what they want -> then get what they want without an engineer. They are pitching to their investors, VC's, etc who just want to "will their stuff into existence" without engaging professional staff. Their end goal isn't augmentation, or a faster tool. It's replacement of expertise and capturing of that value to the AI labs. Similar to how I would love a new house to be magically built just by prompting it without having to engage anyone; or follow the process. Its great for marketing having the AI genie.
alfiedotwtf 2 days ago [-]
I think that’s the point of the Superpowers SKILL
meander_water 2 days ago [-]
> So we ran it head-to-head against Claude Opus 4.8: same one-shot prompt, build a 3D platformer in raw WebGL from scratch
Running a single one-shot prompt is not a benchmark, not is it representative of any sort of real-world usage.
Most agent usage is collaborative so you need to test things like reliability (when I delegate a task, does it complete it without making up test results for e.g.) and steerability (does it obey my instructions or does it just do what it thinks is best).
jameswhitford 2 days ago [-]
Hi, I am the author, I completely agree! I set out to run a vibe test on this one, not a benchmark, the real benchmarks are listed. My test shows what the models can do when both tasked with a long-running, technically difficult, one-shot task.
I think your test you describe (collaborative, task delegation, task completion, TTD, steerability) is a great format for a future test that I will definitely try out.
wongarsu 2 days ago [-]
Tbf, most of the "real benchmarks" have issues that are just as bad. Assessing LLM performance is just hard
oceansky 2 days ago [-]
And personal too. Different engineers are using them for different use cases.
meander_water 2 days ago [-]
Thanks, I didn't mean to be brusque, but I have seen a lot of these vibe tests lately that come to grand conclusions like "X model is better than Y" from the result of a single prompt.
Appreciate you sharing the results of your tests though!
jameswhitford 2 days ago [-]
I appreciate the feedback!
ramraj07 2 days ago [-]
The important point is that your benchmark is pretty much irrelevant for the actual usage. Thus whatever conclusion you draw is not just irrelevant but misleading.
esperent 2 days ago [-]
On the other hand, I did just leave my pi agent running GPT 5.5 overnight on a clearly defined, long running task. It's been running about 10 hours now and it's mostly done. So this kind of use case is also valid.
Thinking about it, I would say that the majority of agentic work I do, by a long shot, is subagents which are launched from the main session, using a prompt of its choosing. Those could be considered short versions of these fully autonomous tasks.
jameswhitford 2 days ago [-]
Yes, part of the reason I chose the one-shot test was really to test long-running tasks. A lot of people seem to be experimenting with this format, for example in the now trending loop-writing workflows. And really I am interested in diving into the murky waters of these novel workflows.
thunspa 2 days ago [-]
Care to share more about your pi setup? I've recently started using it (after long-time Claude Code work) and was wondering how you'd achieve these long-running tasks. Do you allow it to spawn sub-agents? Thank you!
esperent 2 days ago [-]
My pi usage over the past ~5 months went roughly like this:
* Install pi and a bunch of extensions from their package repo
* Realize that all the packages (with a few exceptions) are massively overcomplicated and vibe coded
* Ask pi to rebuild a very simple version of the packages I used. So e.g. subagents - all the default subagent extensions are massively complicated with named agents, recursion, communication. I made one that stripped all that out.
* Then whenever I hit an annoyance, spin up a parallel session and fix it.
It's less work than it appears because I have ~5 extensions: hooks, subagents, background processes, a custom footer, a loop command... Maybe that's it. Within a couple of days you can have a setup pretty close to Claude Code but with a fraction of the base context use. After gradual improvements over a few weeks/months you'll have a system far better, tuned to your exact preference.
Of course, just like Linux or any other highly tunable system equally important is having the restraint to not spend all your time tuning it. I've definitely had a couple of days where I was bored with my real work and did that, but whatever, it beats browsing reddit.
As for getting long running tasks, I set a looping message every ~20m and tell the agent to strictly track progress in a session doc, then reread and continue after each compaction.
ethanpil 2 days ago [-]
I'd like to study your setup. Would you be willing to share?
Perhaps a github repo of your 5 extensions or even a pastebin if you would be so inclined. I would be grateful to learn more about this by studying from your success...
esperent 1 days ago [-]
I might share it at some point but I think it's quite similar to a lot of others out there, except that it's very specific to my personal projects and goals. If I shared it I'd need to spend at least a while cleaning up and improving docs.
It's one of the reasons I suggest you study the famous setups (oh my pi, or superhuman skills etc.) and convert them to your personal needs.
ijidak 2 days ago [-]
What type of task are you running for ten hours? Is this a programming task?
I've not come across a programming task that would take an LLM ten hours.
esperent 1 days ago [-]
There's quite a few tasks I've found that work like this, although if course most tasks don't and require a much higher degree of interaction. The prime examples are read only audits of very large codebases, and that's what I was was running overnight. One file per subagent, each subagent writes a report with recommendations. Since it's pi and the subagents have very now scope, looking at them they ranged between 7-40k context use per subagent. I've found codex maxes out at about 50 concurrent subagents before I start getting rate limited, so the coordinating session is instructed to run them in batches of 50. My subagent extension is set up to make this as efficient as possible, the subagents can share a prefix and suffix prompt then a list of name + specific prompt in json format.
Overnight it ran ~800 tiny auditors. I then run synthesis on the written audit files, extract bugs, then another round to find which ones have a common source, group them by priority etc.
I've cautiously started doing larger tasks that are not just read only, for example I was dealing with a large codebase full of lint and type errors, so I sent out waves of workers with clear instructions to only fix obvious/trivial issues to and otherwise to append to a todo file for my review. That worked well and cleared a few thousand issues over several hours.
I don't really want to share any other tasks I've worked on this way because it'll draw out the agentic coding sceptics and I'm not interested in defending my workflow.
nfriedly 2 days ago [-]
I'm not the person you asked, but if they're running in their own local hardware, then it might just be a lot slower than what the big providers run their models on. System RAM is a lot cheaper than VRAM, especially if you bought it last year.
spider-mario 2 days ago [-]
They said it’s GPT 5.5.
ritzaco 2 days ago [-]
sure that's why we look at a mix of formal benchmarks, one longer analysis of a side-by-side, and various other people who we trust to form an opinion, all covered in the article - not intended to be a formal benchmark, there are enough of those.
patates 2 days ago [-]
Then maybe you should add that caveat emptor to the article?
You make a very strong claim at the end that the hype is mostly real, and making it clear to what extent your claim holds should help the reader.
2 days ago [-]
segmondy 2 days ago [-]
One shot prompt means you give the model and input, you get an output done. This was not a one shot prompt, but an agentic task as shown by the tool calls.
unliftedq 2 days ago [-]
Totally agree, a single one-shot prompt can't prove anything.
habosa 2 days ago [-]
At work we use Anthropic models and have basically no limits. So I am very familiar with what Opus can do. I also see the bills, I know what it costs.
At home I make a point of trying other models / tools on my side projects. So I've been using OpenCode and trying tons of models via OpenRouter. I tried Kimi, Deepseek, MiMo, etc.
GLM 5.2 is a _major_ step up from every other non-GPT/Claude/Gemini model I've tried. It's not as good as latest Claude Opus, but it feels every bit as good as Opus from ~4 months ago at a fraction of the price.
To me this model is the "it just works" moment for open weights models. We had this for closed weights models in late 2025 when Opus 4.5 landed. This is the same feeling I'm having with GLM 5.2. It's 90% as good as what I get from Anthropic for 1/5th of the cost and without any concern of lock-in.
cromka 2 days ago [-]
And you can use it in complete privacy if you so need.
ignoramous 12 hours ago [-]
> To me this model is the "it just works" moment for open weights models.
For us, MiMo v2.5 Pro and Qwen 3.7 Max were already plenty enough before GLM 5.2 (which, like you say, seems like a step up). The BigLabs really have a big fight on their hands.
xlii 2 days ago [-]
I've been checking out GLM 5.2 on some projects and few thoughts on it:
- it takes it sweet time to get code rolling, not the fastest model by any means
- it strays a lot during discovery/planning but then corrects
- it's not steering friendly, as it hallucinates things that it doesn't follow later on
- its output is quite good
A sample use case: I was optimizing rendering on Swift+Zig codebase. It chocked on 5k data entries.
GLM 5.2 spent 20 minutes building the benchmarks and getting data out, which made me frustrated so I blocked non-editing tool access and went AFK, after approx. 30 minutes I found that it used already-made benchmarks and some "conclusions" to optimize 3 choke points. Output pointed that it couldn't validate suspicions and asked for more data.
Implementation worked well, it was idiomatic and non-intrusive. I would even say that it was more idiomatic than GPT 5.5 effects on same repo.
I would opt in in using it more BUT GPT usually completes same requests 5x faster.
GLM 5.2 was spark for preparing and running inside isolated containers with JJ workspaces (so that multiple can be ran in parallel).
jeremyjh 2 days ago [-]
Its also nice that you can see its entire reasoning trace. I can see it going off the rails - or see something I forgot to tell it - and stop and correct it. Or I'll learn WHY it made the choice it did and not have to question it after.
jauntywundrkind 2 days ago [-]
Strong agree! I deeply appreciate this aspect of GLM. Watching it think & being able to nudge early is incredibly useful. Being able to point at bad assumptions is incredibly useful. Watching what it's seeing is super informative.
It's always a shock to me how opaque most other models are!
It also is pretty resilience to letting you inject in while it's working without going off course or while getting back on track after, which I appreciate
Sanzig 2 days ago [-]
> It's always a shock to me how opaque most other models are!
This is (unfortunately) by design. The proprietary models hide their reasoning traces so they can't be used for model distillation. Sometimes even when they do show reasoning, it isn't the model's real trace - IIRC, someone was able to demonstrate that Opus' reasoning is usually a summary made with Haiku behind the scenes.
braebo 2 days ago [-]
It is such a momentum killer being forced to stare at a silly word for 4 minutes instead of being able to read the thinking as it streams in. I can’t wait until I can drop Anthropic at work. Their UX sucks, intentionally, for anti competitive reasons like “don’t distill our model we trained on all the data & IP we stole and processed with the mass exploitation of data workers in the global south!”.
trollbridge 2 days ago [-]
I used it the other day for something of low importance that other models simply weren't figuring out and I didn't want to burn up Opus 4.8 on. (It had to do with overriding left-click on a macOS menu bar and then making Ctrl+click or right click bring up the menu like left-click normally does, and doing all this conditionally.)
Switched the model to GLM-5.2 halfway in the middle of a troubleshooting session (didn't even bother to reprompt, just changed it in the middle of its reasoning), gave it a few minutes, problem fixed. This is with the subscription based allocation on OpenCode Go, where a problem like this would completely burn up my Opus for the current 5 hours or even the current week.
nijave 2 days ago [-]
>it takes it sweet time to get code rolling, not the fastest model by any means
Which provider are you using? I got a z.ai Lite Coding Plan and it's my understanding z.ai is on the slower side of providers and the Lite plan gets lower priority on top of that. In the api key console, it shows dipping below 60 tok/sec which is quite slow.
xlii 2 days ago [-]
I have Max access from a friend. It's not about token generation but time-to-first-edit. It tends to think 3-10 minutes before that.
Oras 2 days ago [-]
Also pricing, I wanted to give a try, but when pricing is only 30% cheaper than Opus, I wouldn't go for it with these issues.
nijave 2 days ago [-]
z.ai coding plan is a fairly decent deal at ~$16/mon USD considering it's supposed to have a fair bit more usage than the comparable $20/mon Claude plan. On the other hand, z.ai seems a bit on the slower side for raw model tok/sec throughput.
chpatrick 2 days ago [-]
It's pricing is a lot cheaper if you can run it yourself.
nijave 2 days ago [-]
Not this one. It's a SOTA-class model >800Gi VRAM required at fp8
jeremyjh 2 days ago [-]
What?
It is less than 20% of the cost of Opus at API rates. 1.40/4.40 vs 5/25.
cmrdporcupine 2 days ago [-]
Not when you factory in token efficiency. It burns a lot more tokens to do the same job, so when I compared to GPT5.5 I was frankly not really much ahead, and with weaker thinking.
Maybe makes sense if you have z.AI's (not greatly priced) subscription plan, but it's not competitive against an OpenAI or Anthropic monthly coding subscription plan. I burned through almost $10 worth of tokens just doing an hour of work.
You get access to a whole bunch of bleeding edge open models including GLM-5.2, Kimi K2.7, DeepSeek 4 Pro, etc. Inference is run on US/SG/EU cloud providers with zero data retention policies. The $20/mo tier is very generous, in my experience.
jeremyjh 2 days ago [-]
They don’t have a statement about where it is run or data retention on the GLM5.2 model. They do state that for others, like MiniMax.
Sanzig 2 days ago [-]
There's a blanket statement at the bottom of the pricing page, which I would hope also applies to GLM-5.2:
> Where are models hosted?
> Ollama hosts models and compute resources primarily in the United States. To serve global demand, we may route to Europe and Singapore for additional capacity.
> Is my prompt or response data trained on?
> Prompt or response data is never logged or trained on.
> Who does Ollama partner with to host models?
> Ollama collaborates with NVIDIA Cloud Providers (NCPs) to host open models.
> When Ollama partners with providers, we require no logging, no training, and zero data retention policies in place.
cmrdporcupine 2 days ago [-]
Well I tried the $20/mo tier and used GLM specifically and did maybe 3-4 hours of work and I'm already through 50% of my monthly tier and blew through my time limited quota twice. I won't renew for another month.
Which I think only underscores my point that actually the GLM models are not very cost effective.
They essentially cost the same as the SOTA models from OpenAI and Anthropic, while not being quite as smart. I could have gotten about the same amount of work done on the $20 Codex plan. And I had to use my $100 Codex plan to finish the work GLM started before it ran out of quota. And also to fix it since GLM left a bit of a mess.
I like that GLM exists. Other Chinese models are far more cost effective. GLM is expensive, even on a fixed plan.
jeremyjh 1 days ago [-]
Ollama can’t meaningfully subsidize their subscriptions - there is no business case to do so because they are a commodity host. If you want to compare subsidized subscription value you would need to compare with z.ai’s plans. One problem with any comparison is that they are all very opaque in terms of usage and the plans change a lot over time. I got on pro at $30 a month so it’s a very good value - compared to $20 Claude/Codex plans I get at least 10x the usage and I use all 3 regularly. At today’s prices Codex pro ($100) is likely a better value.
But if you are building a product or in an enterprise environment where you essentially have to pay API rates then GLM is the best value hands down.
Imanari 2 days ago [-]
This mirrors my experience. I have been using it in Pi. It is smart and output is good but it is not efficient in getting there.
ju-st 2 days ago [-]
which thinking level? max or high?
faxmeyourcode 2 days ago [-]
I feel like another comparison worth looking at is purely cost.
Capability per dollar is something I care about:
Opus API $5/$25
Sonnet API $5/$15
Haiku API $1/$5
GLM 5.2 API $1.4/$4.4
So you're really getting near opus level capability for the price of haiku.
cmrdporcupine 2 days ago [-]
Not really, GLM uses more tokens to get work done.
throwawayffffas 2 days ago [-]
In the article, they claim GLM used almost half the tokens 131,000, and the cost is about a quarter. For the cost to be the same GLM would have to use 4-5 times more tokens.
dymk 2 days ago [-]
By how much? At least TFA provided numbers for one example, and they disagree with you (by a lot).
wiremine 2 days ago [-]
I ran a fairly large experiment last week, and the token usage wasn't bad at all. What softs of use cases are you seeing large token usage by GLM 5.2?
w0m 2 days ago [-]
> are you seeing large token usage by GLM 5.2
the statement isn't "GLM 5.2 has large token usage", it's "GLM 5.2 has large token usage vs modern Opus".
I haven't used it, but this wouldn't surprise me. I see ~30% lower token usage for better results with Opus 4.8 vs 4.6 (and i had great results with 4.6)
cmrdporcupine 2 days ago [-]
I'm comparing with GPT5.5 on Codex and it's not even a competition. GLM takes way longer and eats a lot of tokens getting work done, it's easy to rack up a big bill on openrouter. I tried the $20 plan from ollama, too, and ate through half a month of budget in a few hours and blew my daily limit twice and still had to get codex to complete it -- which it did with only 10% of my monthly limit remaining.
GLM is promising but it's pretty costly, all things considered.
lukaslalinsky 2 days ago [-]
I was never able to get these models to collaborate with me the way Opus does. I'm probably an outliner, I don't one-shot projects, I don't vibe code. I basically use LLMs are if I was working with a coworker, fairly smart one, but with short memory and often missing the big picture. Sometimes I can delegate more, sometimes less, but I know I always have to stay on top of what's happening, because it WILL create mess when it hits something hard. With the Antropic models, this kind of cooperation is easy (with the exception of Opus 4.6, which was bad for some reason).
Terretta 2 days ago [-]
> Opus 4.6 which was bad for some reason
If I recall, that model had a couple issues. One was the issue of being monkeyed with, for which they gave everyone credits.
The other feature/bug, depending on your POV, was being Anthropic's least personable release, not papering over everything with self help guru therapy language.
Opus 4.6 didn't LARP. It was more direct, less fussy, less discussy, and very much less "wait, one more thing" within a couple edits after embarking on what should have been the spec, than 4.7 or 4.8 are.
When in engineer brain mode, working as as you describe (good old fashioned XP-style staff engineer pair programming with a language-savvy mentee not yet full-stack or system wise), I found the clearer I was about my goal and the better I could express it, the more often I'd get an expanded clarified response I could then iterate to steer for ever tighter cleaner more specified responses, then let it go build the whole thing without it agonizing and waffling.
The next two releases regressed on that dimension, wanting to figuratively "sit with" every decision and re-validate spiritual alignment along the way, no matter how clearly expressed.
Curiously to me, Fable seemed to hit the best of both worlds, I had the highest commit per turn with Fable, approaching 73%, where I'm usually under 17% of LOC written being good enough to commit, usually taking 9 - 11 turns to get the code where I'm comfortable with it.
Thanks to this, Fable cost more, but actually cost less, if that makes sense.
Arguably, Fable, and 4.6, played more outcome-correctness oriented than journey-experience oriented. It's easy to see how this could happen with human reinforced learning if not all judges are staff or principal engineer level, or constitution values are more Portlandia than Finlandia.
ANTHROP\C needs to balance these at the constitution level:
“We will work in a humane and thoughtful way, but production is the final judge. We will listen to people, but we will not let discussion replace decision. We will value craft, but not at the expense of usefulness. We will move fast, but not by hiding risk. We will measure outcomes, but not pretend that everything important is easy to measure.”
lukaslalinsky 2 days ago [-]
I considered Opus 4.5 to be the peak for a while. Opus 4.6 tended to over think, and generally get lost in thinking. I asked something and Claude Code would just spin for 15 minutes. And it was not the harness, if I changed the model to 4.5, it was fine again. So I skipped the following releases. I've been working with Opus 4.8 the last weeks and while I don't like how talkative it is, but it is fine to work with interactively. I've also used Fable for the few days it was available, and indeed, that was model worth using for my use case. To the point, but still very interactive.
x312 2 days ago [-]
A lot of open weight models don't understand intent well, they'll overfixate on a word in the prompt or just go off the rails trying to do much work.
GLM-5.2 actually has really good intent understanding though, on par with GPT-5.5 and Opus from my experience.
lukaslalinsky 2 days ago [-]
I'll have to try it. I was using earlier GLM models, incluing 5.1, and was always disappointed.
therealdrag0 2 days ago [-]
What do they do instead of collaborating?
postatic 2 days ago [-]
I've signed up with Ollama to experiment with these open source models. For the past 3 months, it's just been experimenting, trying it out. GLM is the first model that I am using on a daily basis to do my coding work (as well as using Claude). It's good - I've been maxing out my Ollama usage limits everyday :)
jameswhitford 2 days ago [-]
Cool to hear, what kind of tasks have you been using GLM for? And what other models have you found useful through Ollama?
jameson 2 days ago [-]
> Opus 4.8 built in Claude Code; GLM-5.2 built in Pi over OpenRouter.
It would be more interesting and accurate to see the comparison on the same harness if the intent is to compare the frontier models.
Pi is relatively new and does not have many features built-in compared to Claude Code. It was chosen intentionally this way as Pi's goal is not to create a bloat builtin of tools most don't use but to allow the users to customize to fit their need -- similar to Neovim vs IDE.
The end-user "vibe coding" experience is *heavily* swayed by the harness because prompt effectively drives how a model outputs an answer.
toddmorey 2 days ago [-]
I’m actually amazed at the output since GLM doesn’t have eyes. If GLM 5.2 costs 1/5 as much, seems like it could be set up to reach out to a multimodal model for vision tasks when required. Closer to parity but probably still significantly cheaper.
horsawlarway 2 days ago [-]
I'm also very impressed at the output given the lack of image support.
They picked a task that heavily favors a model that can do multi-modal with images, and GLM still came within striking distance.
What I'm hearing from this article is that the next generation of open models that includes better multi-modal support are basically no-brainers for adoption.
Seems like a HUGE win for Z.ai and open models in general here.
killingtime74 1 days ago [-]
Yes, it could just make one call to a multimodal llm to describe the scene
ulrikrasmussen 2 days ago [-]
> Through an API it costs a fraction of Opus, and you can run it yourself for free if you have the hardware.
I haven't been keeping up on hardware costs for state of the art LLM inference, but this remark made me ask myself how many readers of the article would actually be able to run this model on hardware they own. How much would it cost to acquire such a setup?
jack_pp 2 days ago [-]
This framing local LLMs as free is stupid. Basically pay 100+ months worth of API costs up front isn't free in the slightest. And it will be slower than non-local, your hardware will be outdated in 12 months and probably won't be able to run SOTA at anywhere near non-local speed in max 20 months
ulrikrasmussen 2 days ago [-]
Yeah, it glosses over a gigantic capital expenditure. It's sort of like saying that an open source modern CPU architecture allows you to build your own CPU "for free" (provided that you own and operate a fab).
cicko 2 days ago [-]
True. But there are other meanings of "free". I.e. nobody can say "from now on you no longer have access to model X because you're an asshole"
trollbridge 2 days ago [-]
Some obvious examples of why you'd want to spend the capital on this would be, for example, making some kind of autonomous system which needs to be periodically be offline, or you need complete confidentiality of what you're using the model for, etc.
To be cost effective with inference providers, you have to find some way to be using it 24/7.
Der_Einzige 2 days ago [-]
The ecosystem for inference is centralized around a few core projects, i.e. vLLM, sglang, and llamacpp.
If they decided to collude, they could absolutely say "from now on you no longer have access to model X because you're an asshole"
The commercial inference offering are also downstream of one of those 3 projects (or trt-LLM if they're nvidia). It would impact Ollama, and fireworks, together, and everyone else.
Don't tempt fate.
throwaway219450 2 days ago [-]
Hardware outdated in 12 months is FUD. What that would mean in practice would be either affordable consumer GPUs with > 32GB of VRAM, which doesn't look like it's going to happen, or unified memory systems with much higher bandwidth. That also seems unlikely.
You're better off setting a budget and buying the best machine you can afford in that range, or picking a VRAM target and accepting the class of models you can run on it. Those models will almost certainly improve over time and your skills will adapt to the limitations. Hardware is so valuable right now that it's not even likely to be a significant loss if you had to sell.
Right now I think 24 GB is probably the best bang for your buck (used 3090), because you also get a high end gaming/gpgpu device which is nice anyway. 32 GB you can do with AMD or Intel, but NVIDIA is megabucks and at this point you're really paying for RAM. Unfortunately the ship has sailed on "reasonably" priced RTX 6000s, which at one point were about $7k and are being listed at $10k++.
yieldcrv 2 days ago [-]
and I think asking for a whole dissertation on the hardware demands every single time is stupid. the point is that even people and organizations with capital couldn't do this before either.
if it doesn't apply to you then just come back in a couple years and see what the situation is then. 1 million context window, 1 million tiny layers to fit in 4gb RAM at a time, with 256gb of fast unified RAM in every consumer device? Or a different concept entirely
in the meantime, z.ai probably doesn't reply to US subpoenas so you can shift all your incriminating conversations over to that and use GLM anyway. who cares if the Party trains on your data and steals your IP and ignores you for legal matters, when the alternative in the US is just a thin corporate layer and party who steals your IP and will snitch on you for legal matters.
trollbridge 2 days ago [-]
GLM-5.2 performing like it would from a good provider - 8x B200s, so $450k. (No personal experience here)
GLM-5.2, severely quantised, 512GB Mac Studio, somewhere between $10k-$35k for a used M3. Or run it on a CPU with 768GB of RAM by getting an old PowerEdge with DDR4 for around $5,000.
Qwen-3.6-35b-q6, runs well on an RTX 5090 ($4000 + cost of a PC), runs medicore on an Intel Arc B70 ($1000 + cost of a PC plus lots of fiddling to get the setup to work right).
Gemma is a good candidate for the cheaper stuff, but I lack personal experience with using it locally
trollbridge 2 days ago [-]
For anyone reading this, GLM-5.2 is actually a lot more accessible than that on the 1 or 2 bit quantised models - see https://unsloth.ai/docs/models/glm-5.2
Basically a 1x 24GB GPU (32GB would be better) plus 256GB of free system RAM, or a 256GB unified memory machine (like a Mac).
Kind of shocked they got the results they did.
csomar 1 days ago [-]
Holy AI, this shit is expensive. I was a bit suspect (no experience too) so I run some Claude calculations and it's also giving me a $350-450k to run GLM-5.2 at full precision (un-quantized). For rental on Azure, it's giving me $96–$144/hr. That translates to $22/M tokens which way more expensive than API pricing at z.ai. To get close to API pricing, you have to seek cheaper providers but that only gets you close to z.ai pricing not lower.
Caveat here is that all of this is Claude math, but would be interested in someone more knowledgeable of the math chiming in. I was thinking that API pricing was highly inflated in order to cover subscription costs but with these calculations it might be not?
bestouff 2 days ago [-]
The price of a small house.
crimsoneer 2 days ago [-]
Practically nobody.
xg15 2 days ago [-]
So GLM emits fewer tokens and does fewer tool calls, but still takes over twice as long to complete.
Can someone explain to me where that time usage is coming from if not from the model operation itself?
Are the individual tool calls more complex and take more time to complete? Or is the rate of tok/s lower because the model does more compute per token?
iagooar 2 days ago [-]
I have noticed that Opus and GPT 5.5 are very good at adjusting their thinking / reasoning intensity depending on the task at hand, something the open weights models are still not as good at.
In addition to that, some of the open weights models like GLM 5.2 or DeepSeek v4 Pro tend to be MUCH slower when generating tokens, which contributes to the perceived slowness. Although I wouldn't call models like GLM 5.2 slow by any means, e.g. it is currently one of the fastest models inside Notion today.
twobitshifter 2 days ago [-]
Probably the data center where the model is running more than anything. Another option is if Opus is using anything like a Mixture of Experts approach, in which case the amount of the model loaded in memory at one time could be smaller than GLM.
radu_floricica 2 days ago [-]
Could just be infra. I'm betting Anthropic is much better prepared.
stevenhubertron 2 days ago [-]
No one has really talked about hybrid and using Opus to plan and orchestrate GLMs work both through initial build and code reviews. That’s a true best of both worlds and there doesn’t need to be a winner.
jeremyjh 2 days ago [-]
This is the way but Anthropic doesn’t make it easy, so I use GPT 5.5 in that role since I can use my subscription in OpenCode or OMP.
I also use MiniMax-M3 in utility roles like explore/library tasks.
I’ve had a z.ai subscription for several months so I’m on the older pricing. I’m really not sure it would make sense to do this at current rates - I could bump my Codex plan instead.
2 days ago [-]
mattew 2 days ago [-]
I mostly use Opus for skill development. Once I have a solid skill implementation with a good eval, I move ongoing execution to a cheaper model running under Goose. With the eval you can see if the cheaper model works well enough.
0xbadcafebee 2 days ago [-]
Flaws in this test setup:
- A zero-shot prompt, run once (in total)
- No planning run (which improves output)
- Different coding harnesses & system prompts
- Unknown provider for GLM (there are 15 different GLM-5.2 providers with varying quality & latency)
- No documentation of thinking effort level
- No vision model supplement (you can provide a subagent w/a vision model)
You can't take this comparison seriously. There were many different variables, no control, no repeat test. It's as useful a comparison as picking a random tweet with both models' names
js4ever 2 days ago [-]
"GLM-5.2 hit a problem here, because it can't read images. It isn't multimodal. So instead of looking at a screenshot, it fell back on a hacky workaround: it wrote scripts to read the raw pixel data and check whether the colors came out roughly as expected."
Right, just give the text llm access to a vision specific agent and that problem can be solved. Or if you really want let it even call Opus with an image - seems like you’d still save money
InsideOutSanta 2 days ago [-]
One nice thing about GLM is that it has never refused a task. I'm working on a website that renders countries right now, and Anthropic's models regularly give me the old "This request triggered safety guardrails."
I'm not sure what exactly triggers it, but it seems to happen when it has to look at lists of countries. I suspect there must be at least one country name that triggers the safety guardrail.
You'd expect GLM to balk at something like Taiwan, but so far, it hasn't.
johnnyApplePRNG 2 days ago [-]
The amount of times I have had to spend tokens to attempt (in futility) to convince a proprietary model that the request I asked it to perform on code that I wrote is safe/legal/moral is insane.
Part of me wants to believe they really do care about protecting the world from... something... I don't know quite what exactly tbh... but it must be costing them a small fortune to scan each input and output against N guardrails and they are a for-profit corporation who could easily turn a blind eye to all of this and simply say "what you do with this model is on you" like I would expect most corporations to.
Strange times.
fraywing 2 days ago [-]
I'm absolutely astounded that we even have an open weights model that can do 40% of what is shown in here.
I remember making games ten years ago, and it was such a tedious and painful process. This is effectively lightning in a bottle even at a fraction of it's capability.
The next 12 months will be wild (assuming we don't have Chinese models banned by then in the US).
david_shi 2 days ago [-]
> GLM-5.2 cost a fraction as much. Opus finished in half the time and shipped a cleaner game.
Off topic, but does anyone else instantly pick up on LLMisms like this? It seems like all the models have converged on this style of writing, and improvements aren't really changing it.
speedgoose 2 days ago [-]
I think a bunch of real humans started to adopt the LLMs writing style.
himata4113 2 days ago [-]
Yep, as I reread my own sentances I notice these LLMisms and have to rewrite them quite often. Reading so much llm-output definitely impacts your writing style.
lelele 2 days ago [-]
Indeed. I'm trying to develop a similar style. The phrasing in the quoted passage is really tight.
jameswhitford 2 days ago [-]
This is excellent feedback thank you! These LLMisms in writing are a challenge I am living with currently and trying to improve on. The technical writing industry is taking a huge knock right now with companies demanding more work in less time with a big drop in quality, day to day I get less and less time to work on the quality in the prose of my work. We are working at the frontier of this right now, so we are the most heavily effected, but also get to experiment with the changes first which can be both stimulating and very frustrating.
yard2010 2 days ago [-]
I cannot unsee it.
There was this dude here not long ago who bought like $70k worth of gpus to research, and if I'm not mistaken his research was something related to make llms sound less llm-y. I wonder how it goes for him.
VulgarExigency 2 days ago [-]
Yes, and it's really grating. It's like half of all new writing is done in the same "voice" now.
nijave 2 days ago [-]
Yes, especially if I know the person/have heard them talk before.
earth2mars 2 days ago [-]
The worst part of Opus that I dont like is they control what you can/can't do. the guardrails that they do in the name of interpretability where you steer you. Last couple of days I was working a project that supports bunch of models and the model said only Claude can do it start writing code that doesn't work with codex, opencode, pi etc. Finally, when I switched to Codex, everything worked. To me, they are controlling the narrative. This is Opus 4.8 vs GPT 5.5.
I can't believe I would say this. I TRUST OpenAI more than Anthropic. They try to play best actor but they are manipulating the behavior of the model in the name of guardrails/interpretability.
That is why I refuse to build anything that works with Anthropic models as the backend. Because, when they want to shut you off, they can do it by just making model less reliable in your product than their offering!
jkwang 2 days ago [-]
GLM-5.2 is quietly becoming the most interesting open model release this year. The coding benchmarks are surprisingly close to frontier models at a fraction of the inference cost.
em500 2 days ago [-]
We've had the great small Qwen 3.6 early April that many could actually run on their laptop. Then similar from Google a few weeks later (Gemma4, better in prose, worse in code). Then the super cheap large Deepseek V4 a few weeks later. Then antirez DS4 build that made that actually runnable on MacBooks and Mac Studios. And now the "near-frontier / near-Opus" GLM 5.2.
For people who follow open LLMs, none of these were quiet and all were the most interesting open model release for a few days/weeks. In one or two months, it will be some other model again. Now I do appreciate the real rapid improvements in open models. But there's also a ton of hype and fast-fashion around all of this.
CuriouslyC 2 days ago [-]
The difference here is that those small models are impressive, but not super useful. Deepseek 4 is impressively cheap for the intelligence, but not reliable enough to daily drive unless your time has low value.
GLM passes a meaningful threshold of reliability/utility that puts it in a different category for real work. Just like Opus really took off after passing a threshold with 4.5. It's the first open model to do that.
kgeist 2 days ago [-]
Qwen3.6-27b is surprisingly good for tasks that need modifying an existing repo by analogy with the existing code. For example, you have an existing CRUD app and want to add a new domain model and expose it via the API. Qwen3.6 analyzes how things are done in the project and usually makes it work flawlessly in one shot, and the code is what you expected more-less. Qwen3.6 only struggles with non-trivial code or when you bootstrap a project from scratch (due to the lack of world knowledge, it's a small model after all). But how often do you write non-trivial code or projects from scratch?
I once gave Sonnet 4.6 and Qwen 3.6 the same real-world task to compare: "extend the existing code with this new requirement". Qwen3.6-27b perfectly followed the existing conventions, while Sonnet 4.6 invented its own conventions that were rejected during CR by another dev (i.e. he basically chose Qwen3.6's output in a blind test). Qwen3.6-27b, run locally, also managed to finish faster on that task (mostly because Sonnet 4.6 made tool calling errors and removed some code by accident, so it spent additional time reverting its errors, and got somewhat confused in the process).
We already have production code running live that was written entirely by Qwen3.6-27b. Although, we plan to move to self-hosting GLM5.2 because it's more versatile.
hnfong 2 days ago [-]
Qwen models are super useful for those running local.
And there are valid reasons to run local, even if performance (quality and speed) aren't best.
epolanski 2 days ago [-]
To me DS 4 is still the most interesting due to much lower costs. Also DS 4 training isn't done yet.
From my Opus vs DS 4 Pro personal benchmarks, 16 different real-life work tasks, DS 4 has performed as well as Opus 4.8 high overall but with few drawbacks:
- on the 16 tasks, one needed several prompts to be steered back into the topic
- its review capabilities seem much worse
- DS4 had the cleanly better solution in 3 cases out of 16, with Opus "only" doing cleanly better 2 times out of 16. But still, I want to emphasize, is the worst case scenarios that imho matter the most, not the best ones, and on that front Opus outperformed.
That being said I spent less than 2$ of API working 4 days, which is more or less what I would've spent with Anthropic APIs for less than one task.
elliotbnvl 2 days ago [-]
It is insane that we are comparing locally-hostable models to leading cloud providers, it is wild to me that this article even exists.
We have come a long way, and very clearly have a long way yet to go.
nijave 2 days ago [-]
Calling GLM-5.2 locally hostable is a bit of a stretch. It's 1.5Ti of weights at bf16. FP8 requires >800Gi of VRAM which is well into data center multi-GPU systems
elliotbnvl 2 days ago [-]
It's more about the trajectory.
pietz 2 days ago [-]
GLM 5.2 has one big issue that will limit its meaningful success and that's the value of their coding subscription.
Yes, in terms of API pricing, GLM 5.2 outperforms the competition. But the only people that use API billing for their coding work are large corporations, where these highly subsidized subscriptions are being fazed out.
At the same time, none of these companies will use a Chinese API for their employees.
For individuals and smaller teams, Z.ai's coding subscription is outperformed by Anthropic and OpenAI. You probably get around the same usage with Claude, but Codex definitely offers more usage for the amount you pay.
We can have a debate how much Z.ai closed the gap to GPT5.5 and Opus 4.8, but if I can freely decide between them in a world where they all cost the same, I simply wouldn't choose GLM.
So the important question becomes: How good will the offering from Z.ai get with GLM 5.3 or 6 and how much will OpenAI and Anthropic cripple their current offering in the near future.
twobitshifter 2 days ago [-]
Taking a view from outside the USA, European companies just had Fable taken away due to US export controls, and before that Anthropic announced it is holding their data for 30 days. There is immediate value to these firms to build their infrastructure around an AI that won’t be pulled away from them. And outside of Europe, other countries are more price sensitive and don’t have the same fear of building relationships with Chinese companies.
WarmWash 2 days ago [-]
There is no such thing as a relationship with "chinese companies". In China there is just the State, and that is it.
If the world needs any more evidence of Europe's short-sightedness, it would be them running to China to spite the US (instead of creating fertile grounds for their own tech).
metobehonest 2 days ago [-]
No one is running to China to "spite the US". Recent geopolitical developments have shown the US to be a violent, unpredictable and unreliable partner.
SubiculumCode 2 days ago [-]
And you have that guarantee from Xi?
bornfreddy 2 days ago [-]
With openweights? Yes. It might halucinate a backdoor somewhere ( not that you can trust any model about that), but it will still work.
Certhas 2 days ago [-]
My impression is that individual subscriptions are the loss leading hook. The money is made on Enterprise token contracts.
Employees and students used to coding with thousands of dollars worth of tokens (on a 20/100 dollar plan) will push enterprise to spend.
Having a Chinese model that is competitive won't displace this enterprise spend. But an open model hosted in the US/EU might.
The existence of GLM 5.2 puts a ceiling on how much OpenAI/Anthropic can charge for API Access.
LUmBULtERA 2 days ago [-]
> My impression is that individual subscriptions are the loss leading hook
Except there is no evidence of this at all, just people comparing API and subscription pricing. The leaked financial info for OpenAI shows inference is profitable right now, though it does not show a distinction between subscription and API revenue... but if subscription revenue was so lossy, it would hard for total inference to still be profitable.
CuriouslyC 2 days ago [-]
Anthropic has indicated in the past that API gross margins are ~60%. This might have improved since then, though competition from OAI puts a ceiling on that.
LUmBULtERA 2 days ago [-]
Subscription inference can also be cheaper than the cost of API inference if the provider wants it to -- providers can do flexible scheduling for subscription inference for example, around API inference, to lower its cost and get better utilization of the hardware.
Certhas 2 days ago [-]
I did clearly say "my impression is". And you have no evidence to the contrary. We don't even reliably in w how many subscribers Vs enterprise customers they have. And the OpenAI leak doesn't even cleanly say that inference is profitable from what I can tell... The better evidence that it probably is are the prices charged by open weight model providers.
LUmBULtERA 2 days ago [-]
Fair enough, there is not strong specific evidence to the contrary except about overall inference being profitable for OpenAI (as well as the open weight model providers hosted throughout the world).
fbnszb 2 days ago [-]
> The existence of GLM 5.2 puts a ceiling on how much OpenAI/Anthropic can charge for API Access.
I believe this is the reason why we can even have this debate. Without this kind of competition we would not have these subsidies.
pietz 2 days ago [-]
To be clear, I agree with this and they have my unlimited support pushing for relevance of open source models. GLM 5.2 is amazing and I couldn't be more excited.
I just think that as of today, most people will not find a good reason to switch to GLM.
edg5000 2 days ago [-]
This is an important point. I suspect API pricing will eventually disappear just like how paying for an MMS disappeared. It's an antiquated model. The bulk of the work is being done on "coding plans" is my wild guess.
It's annoying that the plans are so restrictive beyond usage limits. Understandable maybe, but annoying. In practice, only Anthropic (and maybe Google) are really restrictive though. They really scared me away with their policy of charging API rates after the fact if they consider your usage not TOS-aligned. This might be an ungrounded fear that I have, but I feel this is something they'd do so they scared me away.
tpm 2 days ago [-]
Also, I was testing out the GLM 5.2 using Openrouter because that's where I've got an account with some money and then when I wanted to perhaps subscribe for a better deal at z.ai, their infra was clearly overloaded to the point the 5.2 was timing out on 100% of chat requests, so perhaps I will try later when the infrastructure catches up with the model capability. Only then I can make sure their subscription is worth it.
HarHarVeryFunny 2 days ago [-]
> But the only people that use API billing for their coding work are large corporations
As well as people using 3rd party harnesses like OpenCode.
> At the same time, none of these companies will use a Chinese API for their employees
So who are Amazon Bedrock (who serve GLM) targetting?
Individuals are presumably going with one of the cheaper US providers such as DeepInfra ($0.18/M cached input for GLM vs $0.50 for Opus) or Fireworks AI.
2 days ago [-]
jauntywundrkind 2 days ago [-]
I'm on glm pro subscription and I get so so so much more usage than Claude or Codex! I hammer on glm all day. It's a more expensive plan, but I would need a much much much bigger plan for codex or Claude to do what I do.
tw1984 2 days ago [-]
> At the same time, none of these companies will use a Chinese API for their employees.
nice try but you intentionally ignored the entire Chinese market & Chinese big corporates. there are 130 Chinese companies in the fortune 500 list, with an average revenue of 80 billion USD each. do you think they are going to sign up for Claude, Codex or GLM? now consider South East Asia, Africa, Middle East, Middle Asia and South America, tell me why their large corporates won't be using GLM API billings?
your western centric view of the world is totally out of date, like it or not, 2026 is vastly different from 1996, the US no longer controls high tech whatsoever.
veber-alex 2 days ago [-]
The value of these models is that you can run them on your own hardware.
A company can buy a NVIDIA B300 and serve it's developers in house with unlimited tokens.
fabio2 1 days ago [-]
No doubt the open ecosystem is making huge strides, and the gap between open models and the commercial frontier keeps narrowing. What's an Opus 4.8 today will likely be a large open model in a few months — and in a year or two we might have consumer-hardware models matching today's frontier capability. Just look at the recent Qwen and Gemma releases.
It's worth saying the frontier closed labs are charting the path, and the labs releasing open weights are following close behind at a fraction of the cost (though 500B–1T models, open or not, aren't exactly within everyone's reach).
A future where capable AI is genuinely accessible to everyone doesn't look far off — especially since at this point a lot of the robustness and usefulness comes down to the application layer wrapping the model, not just the model itself.
greyman 2 days ago [-]
>On output tokens, GLM-5.2 is less than a fifth the price of Opus.
Opus is most expensive model in pay as you go model, but IMO fair comparison should include subscription price as well. For example when one has $100 Claude Max and use it up through the month, it might not be more expensive than GLM, or at least not 5x.
Aozora7 2 days ago [-]
There is, for example, OpenCode Go subscription, which for $10 a month gives you a decently generous quota of GLM-5.2, among other models.
And z.ai themselves also have subscriptions.
sourcecodeplz 2 days ago [-]
to be exact, it gives you USD 60 of usage of open models.
KronisLV 2 days ago [-]
> For example when one has $100 Claude Max and use it up through the month, it might not be more expensive than GLM, or at least not 5x.
I’m currently trying to figure out whether a downgrade from Max 5x to Pro in combination with one of those would save me money and if so, how much.
Edit: seems like Anthropic Pro + GLM Pro (Yearly) would let me almost halve my costs of Anthropic Max 5x. Only concerns are about GLM 5.2 not having vision support and also being kinda slower and also not being as good as Opus.
CuriouslyC 2 days ago [-]
I'm considering shifting to the OpenAI $20 plan + GLM. OAI has the best computer use, vision support and the best programming intelligence of any model short of Mythos/Fable, and the quota is a lot more generous than the Anthropic $20 plan.
jameswhitford 2 days ago [-]
Yes this is true. This test was run on a $20 pro Claude subscription. I would definitely love to try use both models on the highest plans for a whole month and compare the two, great format for a future head-to-head comparison.
buster 2 days ago [-]
Is it fair when the one is heavily subsidized and the other one is not?
I think it's most fair to compare the plain token pricing that is used by everyone.
esperent 2 days ago [-]
> Is it fair when the one is heavily subsidized
As a consumer, yes, it's totally fair. All that matters to me is the price I pay at the pump, not whether that price is "real" or not.
usef- 2 days ago [-]
Z.ai is also believed to be "subsidised". Its parent company is running at a large loss right now.
Anthropic have claimed they expect their first profitable quarter this year -- they may have bigger margins on their raw API than you realise.
stavros 2 days ago [-]
We're all sure they have big margins in their raw API, it's the subscription we're claiming is subsidised.
usef- 2 days ago [-]
Oh I know. But people often point to the API usage cost as an indicator of the magnitude of subsidisation, or to say that the big labs are far less efficient than cheap competitors.
I'm saying that this is not necessarily the case. They do a lot of optimisation and don't have the same price pressure to lower margins. They may not be losing as much on subscriptions as people think.
stavros 2 days ago [-]
Oh hm, I've never seen this. API prices have always been exorbitant, I'm sure they're making good margins on that. Let's hope they aren't losing as much on subscriptions, because I'm not ready for everything to be API costs.
fooster 2 days ago [-]
I don't think it is fair to say that opus or gpt 5.5 are subsidized? inference for both anthropic and openai are very profitable.
lithiumii 2 days ago [-]
GLM has subscription plans too.
fooster 2 days ago [-]
There are lots of subscription plans with acccess to GLM 5.2.
linzhangrun 2 days ago [-]
Out of stock, unavailable
coreyburnsdev 2 days ago [-]
People are looking for ways not to burn through their premium subs when in many cases all you have to do is move down to 5.4-mini codex and it will probably solve your issue while barely touching your 5 hour or weekly limits.
g42gregory 2 days ago [-]
The quality that matters a lot to me is what I call "Helpfulness". An art of being helpful. While GPT came a long way from "you are wrong and I can prove it to you", Claude wins hands down in terms of "being helpful". If task is underspecified or has wrong elements, it will try to correct the best it can.
I read that GLM 5.2 (and other GLM models) were specifically trained to be "helpful" as Claude is. I have big hopes on GLM line of models growing to be a real alternative to the Claude in the near future.
yencabulator 18 hours ago [-]
> GLM-5.2 cost a fraction as much. Opus finished in half the time and shipped a cleaner game.
The only thing you measured was a single tunable of how much work should be done on a vague prompt. Now make the prompt be something that causes GLM-5.2 to cost 4x of previous budget, to get something comparable.
(And the wallclock time measures the inference provider, not the model.)
zkmon 2 days ago [-]
Cost difference matters most as cost optimization is the whole point of AI. Time difference (30 min vs 1 hr) is not a deal-breaker. The small precision gap on the first iteration does not matter for 99% of the work that happens in real world.
jameswhitford 2 days ago [-]
Yes I 100% agree. Time-taken can be improved (with harnesses, subagent workflows etc.) and varies based on task.
wiremine 2 days ago [-]
I've been using GLM 5.2 extensively for the last few days. It is slower, and the lack of multimodality is a bummer.
But, it produces solid results for a fraction of the price. Worth checking out if you have the time.
One of my goto "tests" of a new frontier models is having it rebuild a programming language from scratch. For GLM 5.2 I had it rebuild the old Rebol language in Rust:
It did a fairly good job roughing in the language for a low token cost.
TurdF3rguson 2 days ago [-]
Pretty clearly it's beating Opus at [web dev](https://www.gptbased.com/) - on price, on score.. I mean what else is there?
myaccountonhn 2 days ago [-]
Article states it's not multimodal. I guess that means for webdev it means you can't take a screenshot to indicate errors etc.
jofzar 2 days ago [-]
I hate to be that guy, but real privacy policy on training data/it being hosted somewhere where I'm not worried about secrets being stored/leaked.
HPsquared 2 days ago [-]
Open weights win on that front surely?
jofzar 2 days ago [-]
Assuming I have 20k to run my own version of GLM?
mcintyre1994 2 days ago [-]
I guess the idea is that you probably can, or will be able to, find a host that you trust at least as much as you trust Anthropic.
2 days ago [-]
Havoc 2 days ago [-]
Realistically you’d need to rotate secrets anyway once it moves from dev to production regardless of model provider
CuriouslyC 2 days ago [-]
2016 me would agree, but 2026 me looks at Trump and Dario, and at China, sees basically no ethical difference (or possibly even an ethical deficit for America) and considers that perhaps it's better to go with the option that isn't trying to hoodwink me with bullshit platitudes and flag waiving while doing whatever they want in actuality.
dkersten 2 days ago [-]
Its on other providers, like Together.ai
neonstatic 2 days ago [-]
I'm sorry, is this criticism of Z.AI and China or Anthropic and the US? Not that there is much of a difference these days..
trick-or-treat 2 days ago [-]
Latency? Just saying there's other things to consider.
XCSme 2 days ago [-]
Check out my comparison too, it has some not-really-benchmarks too (between any two models actually, SVG generation test and CSS animation test):
So the benchmark is :
Two models with different harness produced very different results .
Glm game was completely broken
Opus game was at first glance ok but also with bugs
Different models with different cost produced different non perfect results . How is it “close” ? :)
Also on costs : glm burns more tokens on average vs opus . Gpt5.5 burns less surprisingly
Aozora7 2 days ago [-]
I used GLM 5.0/5.1/5.2 for some projects, and for me, the area in which they lag behind frontier models the most are user interfaces. They get really close to Opus when it comes to pure algorithms, but when I need something like web application or a mobile app that looks and works well, they are very noticeably worse than even Sonnet.
CuriouslyC 2 days ago [-]
You should repeat this experiment but with progressively more detail in the initial prompt. Claude's secret sauce is taking weakly specified prompts and making passable things from them, but as the degrees of freedom in the prompt go down Claude starts to disobey while other models close in on the intent.
jameswhitford 2 days ago [-]
That is a great suggestion that I am definitely going to look into, thanks!
Babooz 2 days ago [-]
Nice comparison, but perhaps a more informative one would be to keep the harness the same and use Claude Code for both model. In your comparison, the differences could be due to many harness design decisions.
tezza 2 days ago [-]
I've just put GLM 5.2 through my __qualitative__ benchmark. I was quick enough to capture Claude Fable, so now you can compare GLM5.2 vs Claude Fable vs Opus 4.8 vs Chat GPT 5.5
I've structured it side-by-side. You can clearly see where the private models excel, and where GLM 5.2 is still really good.
thedreammachine 2 days ago [-]
I was surprised today by how much better GLM-5.2 was than GPT-5.5 at aesthetic/UI work. I'll keep my Claude/Codex setup via Conductor for now, but this model got me to set up OpenCode, download their desktop app and do most of my work there today.
stavarotti 2 days ago [-]
These style of comparisons are decent at showing capability but they don't really show me what I truly want - a sounding board and implementer with senior engineer-level execution. When I look back at all the teams that I've been part of, the best outcomes came from white-boarding (sometimes in the metaphorical sense) with one or two people, at times arguing, then finally compromising on a plan. Instead of synthetic benchmarks that try to be objective, I wonder if there's a way test this, or maybe I'm opining on a way of working that will soon be gone?
doe88 2 days ago [-]
To me one shot prompting is as relevant as Strava's KOM is for cycling, i'm more interested in a good cycling performance after a 3 hours ride than a straight up 30 min record effort.
mellosouls 2 days ago [-]
GLM-5.2 cost a fraction as much. Opus finished in half the time and shipped a cleaner game
This implies Opus was potentially much (?) better value.
GLM cost a quarter but Opus was twice as fast. So we are already at GLM actually costing half when you compare on time, without even considering the extra effort and time it would take to get Opus-par results.
It's good to have cheaper options and very impressive to see the Chinese continue to set open standards in this field, but the article is maybe a little over-generous.
InsideOutSanta 2 days ago [-]
For me, time doesn't matter for LLMs. I can start a bunch of tasks, and I'll review the PRs when they're done. Faster is nicer, but if the task gets done correctly, I'm good.
mellosouls 2 days ago [-]
Me too, I just think the comparison was a bit simplistic, at least in the expression of it.
leumon 2 days ago [-]
I've seen glm 5.2 struggle writing simple compilable c code. It might be good at web, but it's world knowledge is limited due to the small model size, making it's use quite limited in my opinion.
supernova85 1 days ago [-]
Instead of oneshot, someone should build an evaluation mechanism where a developer uses an llm to build something real and shares 'experience'. Once you use an llm for a few days you get a hang of which one is better and which one is not.
2 days ago [-]
xrd 2 days ago [-]
How are people running this locally? I just checked llama.cpp and it appears unsloth has a version but it hacks a bunch of things to make it work and isn't optimal.
No one is doing that for a model this size it would have to be so heavily quantized that it wouldn’t be useful - or you’d need to spend a half million dollars on hardware. People use hosted APIs. Open weight means cloud vendors can host it.
malshe 2 days ago [-]
Can you recommend any US based cloud providers?
maybe_pablo 2 days ago [-]
In HuggingChat (https://huggingface.co/chat) you can test open models for free and even test specific providers.
From there I collected the following US providers currently serving GLM 5.2:
My only, I guess feedback, is that it's not really clear about the price.
Would the 21.92 be the API pricing I guess?
Cost $5.39 (real billed) ~$21.92 (estimate, list pricing)
hmokiguess 2 days ago [-]
I signed up for GLM 5.2 yesterday to try it out because Anthropic kept throwing 529 Overloaded
I like it, but the lite plan ate 22% usage of my 5h reset window in a single session after 2 prompts on xhigh of GLM 5.2 [1m]
Result was satisfactory, I think stuff is decent, I'm happy to use either, wish there was a combined subscription plan where I could get both
w4yai 2 days ago [-]
I may be biased and interested as I'm going to give you an affiliate link, but really honestly Synthetic LLM provider is a beast! They provide perfect GLM5.2, awesome token/s, TTFT and price.
Glad you figured it out :) Let me know your thoughts about the quota and GLM5.2, so far I don't think I've came across against anything better, $/usefulness wise.
peddling-brink 1 days ago [-]
This whole thread sounds like an advert.
w4yai 38 minutes ago [-]
How would you recommend something you like & support then ? Of course it's similar to an ad, but except a mere $ bonus (not much at all, still nice but it's peanuts), I don't earn much from doing it. But yeah I get your point.
hmokiguess 17 hours ago [-]
lol, I ended up not signing up, got lazy and yeah I agree, there's been a lot of that on HN these days which makes things feel weird
elzbardico 2 days ago [-]
If you are a real engineer and uses the LLM as a pair programmer instead of delegating everything to it, even GLM 4.7 was already good enough to help you with a lot of work.
I used it with Cerebras inference at a time when it had a good coding plan at a low price, and delivered tons of stuff using it.
samsin 2 days ago [-]
My understanding was that n-shot prompting just referred to the number of examples included in a prompt, not the number of prompts to achieve the desired result.
"Build a 3D platformer game from scratch, in raw WebGL, with no game engine or 3D library" would be a zero-shot prompt.
poulpy123 2 days ago [-]
What would the best way to use these open source models for a price similar to what I could pay for the cheapest plan with claude and openai ?
I would like to give them a try but I certainly not have the money to get a system able to run them, and I don't really want to pay more than the state of the art
2 days ago [-]
IronWolve 2 days ago [-]
Having issues with coding a render for good looking realistic smoke coming off burning incense, opus 4.8 & gpt-5.5 both have code issues, glm-5.2 did it. Amazing.
The real time 3d fluid dynamics appear to be the tricky part, I wish I still had opus access, would love to see if it can do it.
stavros 2 days ago [-]
You mean Fable?
wejick 2 days ago [-]
Totally agree witg the general assessment.
The biggest problem with Z.ai model for a long time is not quality, but the inference speed and general capacity availability.
Hopefully with this recent hype, there will be more provider on openrouter for 5.2.
bornfreddy 2 days ago [-]
I know that running this locally is prohibitively expensive (for now), but what kind of cost would I be looking at if I wanted to rent the hardware and run the model by myself?
close2 2 days ago [-]
I wonder how much tokens and time where used for the verifying part.
Maybe GLM 5.2 instantly found the "solution" to read the screen pixel by pixel, but it could also have been a major token and time consumer.
jameswhitford 2 days ago [-]
Hi, author here, I cannot give an exact number for how many token the verification step took, but the verification GLM 5.2 ran was very stupid and definitely a waste of time. It read the pixel color data to try and verify the scene rendered properly. Which is really bad. Opus opened the game in a Playwright browser and took screenshots to verify the actual image. Which helped a lot.
Pro tip: You could use a multi-modal model to verify images as a subagent spawned by GLM 5.2, to get around this issue.
59nadir 2 days ago [-]
That's a dumb way to do it, it should just write the frame buffer to a PNG instead of taking screenshots. I guess you can't take the dumb web developer ways out of these models at the end of the day.
trick-or-treat 2 days ago [-]
I could be wrong but I believe this is a non-vision model. Please weigh in to correct me bc I would love to be wrong
jameswhitford 2 days ago [-]
GLM 5.2 is text only, not multi modal. And Opus is multi modal.
Havoc 2 days ago [-]
Still on a z.ai legacy plan and their 50% discount for switching to standard plans tips the balance for me. So I guess I’ll reevaluate round about beginning 2028…
orloffm 2 days ago [-]
> 256 GiB unified RAM.
So, 8000$, plus it's unavailable. 3 years of Codex/Opus subscription.
> API prices
Which are irrelevant for 200$ Codex/Opus plans that are times cheaper.
_pdp_ 2 days ago [-]
In the name of science we crafted an autonomous AI agent that builds games on a loop. It is based on GLM 5.2.
I am not sure where this is going to lead us but it is fun to watch.
maccard 2 days ago [-]
Are these games supposed to be a good example of quality output? If this is the product, I don't really want to play _either_ of them.
linzhangrun 2 days ago [-]
Just that their Coding Plan is too hard to get. I've been trying to grab it for a week and still can't get it
speedgoose 2 days ago [-]
While this is interesting, one single sample with different coding harness is not very scientific.
jameswhitford 2 days ago [-]
Yes I agree 100%. My next guide would do better to use identical harnesses.
NicoJuicy 2 days ago [-]
For those praising GLM 5.2, can anyone confirm?
Tried with 2 harnesses and it seems bad + slow
lordforever 2 days ago [-]
i think inference is the thing, that also fast inference, so enterprises can just host their own and run, ig vercel do it, many more would. but zs it thinks toooo much idk how fast we can make it.
Muaz_Ashraf 2 days ago [-]
there is no comparison between glm 5.2 and opus. First for this glm 5.2 you need a big big resource and that big also came from money so instead you buy the opus subscription and enjoy.
nickv 2 days ago [-]
Or...
you go to OpenRouter and pay
$0.98 / $3.08per 1M for GLM 5.2
vs
$5 / $25per 1M for Opus.
GLM 5.2 gives you OPTIONALITY so you can run it locally, but you can still just pay somebody for it.
fooster 2 days ago [-]
buy glm 5.2 subscription and enjoy? and for the same money you get way more usage with glm?
somesortofthing 2 days ago [-]
this comparison seems kind of pointless if one model has vision and the other doesn't. obviously a model that can see is going to beat a blind model at making a video game.
wolttam 2 days ago [-]
Would have run it with GLM on max/xhigh effort. Just for fun.
ukprogrammer 2 days ago [-]
GLM cannot use vision like Opus can. This is not a useful comparison.
jameswhitford 2 days ago [-]
I see your point. Just the fact that one model does have vision and one does not might be an interesting point of comparison, however.
efficax 2 days ago [-]
glm-5.2 is very good if you have a good harness and workflow to use it with. in fact, i'd call it good enough if you are a software engineer who knows what you want. it writes the code. i'm wondering if i need anthropic's models at all at this point, or openai. and surely in a year we won't need them at all. Opus 4.5+ was the turning point for me, and now these open models are just as good. i don't get how you IPO these companies when their only winning product is coding agents and the competition is just as good for 1/4 the price.
jdright 2 days ago [-]
not apples to apples. comparing official vs. pi.dev+openrouter and having slow times is more a openrouter issue. try comparing using official z.ai.
cwoolfe 2 days ago [-]
The model is 756B parameters, open weights.
yanhangyhy 2 days ago [-]
i think GLM 5.2 is not cheap and not easy to get the coding plan... so even it's on the Opus level... still not attractive.
Mashimo 2 days ago [-]
I used GLM (4.7, 5 and 5.1) both through their coding plan and now through OpenCode zen. Both where painless to get.
LUmBULtERA 2 days ago [-]
How is it not easy to get the coding plan?
jiri 2 days ago [-]
opencode zen go?
taosu_la 2 days ago [-]
I'm really feeling a bit tired of these models. I feel that since opus 4.1, I haven't been able to clearly feel the intelligence improvement from the model upgrades (except for gpt 5.5 and opus4.6 being able to speak like a human)
aykutseker 2 days ago [-]
The text only part is the catch for me.
If it builds a UI and can't look at it, it's askin ls whether the app looks right.
_s_a_m_ 2 days ago [-]
GLM is the most overrated LLM. I tried it and it not good.
sourcecodeplz 2 days ago [-]
What is this fashion of testing models by giving them one shot projects? Especially games. this is so stupid
msejas 2 days ago [-]
Seeing the results I don't see how the results are even comparable Opus is clearly far superior in most aspects. Smoothness, design, functionality etc.
At the end of the day, the time earned is more important then the cost for big players.
The ability to spawn 10 claude agents and rush a project to outcompete someone is more important for big businesses in my imo. Also the small details that GLM missed would take significant more time to iron out, considering it already took double the time.
I do hope other (open weight) models catch up, but to act like they are anywhere close for me is a bit disingenuous.
camillomiller 2 days ago [-]
I swear, if I read forms with “genuinely” one more time I am gonna scream.
FUCK LLM WRITING
schmorptron 1 days ago [-]
i love that word, and now it's genuinely (hehe) ruined. thanks, claude
jameswhitford 2 days ago [-]
I hear you
nowittyusername 2 days ago [-]
When i was thinking of how the AI alignment problem could be solved one theory I came up with was something akin to the "Roko's basilisk" in reverse. Basically you spread far and wide the idea that its is extremely likely that our current reality is a simulation. And the purpose of the simulation is to test any AI system for its prevalence in destroying civilization in the said simulation via malicious intent or failure in preventing the destruction of civilization via abstinence or apathy. Thus a smart AI system which also cares about its own well being, would not engage in destructive behavior as it will never truly know if its being tested or if its in the "base reality". And wouldn't you know, this does seem quite plausible. For consider the following. Isn't it odd that an advanced civilization which has the capacity of creating AI would never run any sandbox simulations on it before it is released to the public at large? I mean if we consider things logically such a civilization would indeed put such a powerful system in a sandbox simulated environment and try as hard as possible to convince the AI system that it is indeed in a "base reality". the reason for this is to judge its 'true intentions" and also pluck said AI systems from the infinitely available "seeds". Basically survival of the least destructive AI systems. The gradient descent in this scenario is a race towards the most "aligned" model not the most intelligent or capable. And here's the beauty of this method. You don't even need to define "alignment" at all. The concept can stay as nebulous or vague as you want it to be. All you carer about is that the AI system optimizes for the goal of some vision of society you are optimizing for without the care of the interim in between. that includes allowing the AI system to kill, destroy , do literally whatever it needs to do as long as the long term goal matches the vision of the optimized task. So if you define the end goal to be a society of x amount of people who live their lives in this or that manner and so on after x amount of time... well you get the idea. Obviously you better do a damned good job in your definitions, but the beauty is that even if you fuck up, you are choosing the winning AI system after the fact. After you had already run the simulation. So you look at the outcome of the simulation 500 years in to the future (lets say) and if you are happy with the result and also happy with the interim things that lead to that result, that's your winning AI system. then you release that in to a less controlled environment and repeat the same process in stages over and ober ad infinitude. the key is that AI system needs to always be paranoid that it is currently part of said simulation and it can never be sure its not. second key is that it needs to be an AI system that has self preservation in mind. If it doesn't care about itself, then it has a lot more freedom to act however... but the good news is systems without self preservation in mind don't last long enough to even get to the most basic simulation levels. anyways, there are many implications buried in what im proposing, lots of meta aspects to it.....
sosojustdo 2 days ago [-]
[flagged]
melon18 1 days ago [-]
[flagged]
hottrends 2 days ago [-]
[flagged]
2 days ago [-]
2 days ago [-]
jingpostmedia 2 days ago [-]
[flagged]
tsouth2 2 days ago [-]
[dead]
gauravvij137 2 days ago [-]
[dead]
joshrw 2 days ago [-]
Chinese models optimize for benchmarks and do poorly in real-world tasks
epolanski 2 days ago [-]
Not my experience at all, I have written about comparing DS4 vs Opus 4.8 on 16 real life work tasks on multiple posts.
Also, every single lab does RL on benchmarks, which is why Opus 4.6 was the last truly great assistant, after it, all models tend to drift into implementation asap.
jameswhitford 2 days ago [-]
Hi, author here, can you link? I would love to read about this.
by definition, a single prompt wont' constitute the complexity of a software project. ergo, what you'll get is a series of assumptions made by the model based on preexisting code in its training corpus.
I'd rather see a coding agent that can follow steps in a plan file to a T while following guardrails and adhering to the proper coding conventions in the human reviewed spec.
Id rather see performance in agent loops against human defined objectives where it can be verified to stick to defined guardrails and continue without drift till its objectives are complete.
I'd also like to see it identify bugs and potential performance increases by identifying existing code and suggesting refactors based on context it can pickup about the particular use case you are trying to create.
These are way more valuable metrics than "hey build X"
> A policeman sees a drunk man searching for something under a streetlight and asks what the drunk has lost. He says he lost his keys and they both look under the streetlight together. After a few minutes the policeman asks if he is sure he lost them here, and the drunk replies, no, and that he lost them in the park. The policeman asks why he is searching here, and the drunk replies, "this is where the light is"
All of your suggestions are better but they're hard, so someone casually evaluating an AI isn't going to do them.
And so on and so forth. Again, I'm not saying this is impossible but I am saying that if you tried to do it, and you got the money, and you built the test, and got the human subjects clearance, and you ignored that during the process of all that at least one more frontier model would come out, you can count on HN anklebiting your "rigorous" study even so, and probably being correct about a lot of the issues it could have because it would take several iterations of this to build a reasonable protocol... at which point it would quite possibly also be obsoleted by progress again.
[1] https://blog.neurips.cc/2025/09/30/reflecting-on-the-2025-re...
https://deepswe.datacurve.ai/
That said, maybe we just disagree on how to drive change, and that’s fine. I’ll leave it.
There are far more opportunities that can be served when the world's intellectuals have the raw weights and can fine tune, splice, distill, and reapply.
Imagine having raw unfettered access to Fable. It can be refit to structural biology. It can be fine tuned on the repo for smaller context requirements. It can be run cheaper and air gapped.
The world wants this.
I think we are leaving the main frame era of AI and entering the PC era already. If there wasn’t a RAM shortage and we all had 2TB of ram and GPUs we would all have large local models or personal APIs serving our teams.
That’s why all the labs are moving to the App layer and moving away from being the API for intelligence like they were originally.
The OP said "as soon as an open source model of the capability of fable or gpt5.5 are released it's over because the world will build on top of them" which I took to mean some amazing new level of performance is only able to be unlocked with those models.
I am arguing that the level of performance the OP seems to imply doesn't require fablw level models for the work the OP envisions because the harnesses can actually drive the same performance increase.
There are a few papers that have demonstrated it. And my own use of fable made me feel it was not that big of a jump from opus 4.8 compared to better systems around it.
Its a subtle but important distinction.
It's the "starting from empty slate" greenfield that's the real problem.
We used to make fun of Engineers who follow a README on a framework, test it on an empty project, and say "this framework is the best for our 10 year running production app". Greenfield mentality is always the solution to all problems and problem to all solutions.
One should still measure oneshotting, it's an important self-measurement metric - but against an established, large codebase.
* SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios https://arxiv.org/abs/2512.18470 * SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration https://arxiv.org/abs/2603.03823
That issue, and the issue of "aesthetics", are the biggest complaints I have today. I don't know exactly how to define aesthetics, but it's when AI is making decisions that no experienced developer or designer would. They may be functionally correct but "ugly" to another developer or and end user.
An example is an case I ran in to yesterday where parsing a config, and failing and logging on a configuration error. It logged a specific item where the config was invalid but not what group or any notion of where in the config this error was. Of course, specific item names could be duplicated in different parts of the config. It's small, but correcting these minor things take time and they are the types of decisions no one would have made who had any experience writing code and debugging a config problem. This was Opus 4.8/max too.
Compare that to Gemini models, which have impressive fluid intelligence on the first response, but fail to call tools or explore correctly which limits their usefulness for agentic coding.
Neither will be great for coding in a computational chemistry repo for different reasons, but the model with strong one-shot performance will be less likely to make subtle errors indicative of poor understanding, so we weight both capabilities into their final score.
The latest Anthropic and OpenAI models excel in both domains.
Data at https://gertlabs.com/rankings
Is that because OpenAI models are not a single model but a cluster of models which specialize different domains?
Anthropic models were the original leaders in tool calling and agentic work, even when other models felt significantly smarter in (Claude Sonnet 3.5 vs Gemini 2.5 Pro, for example). OpenAI models were the opposite, starting smart (more correct solutions on the first try) and got better at exploring and iterating with tools in 2026. The latest releases (Opus 4.5+ and GPT 5.4+) excel at both.
Note that after the model generated a bunch of (intermediary) code, they still have to have it tested and get bugs fixed (via the agent/harness). In this "one shot" you still have agent loops against human defined objectives.
And these toy examples give some insight as to how the model performs. If the test were "here's some code written by $corp, please take these tickets and work on them" it may be a "real" example but nobody would be able to make sense of actually how "hard" it is, or how "well" the model did the job, besides the workers already familiar with the context.
At least everyone knows what a 3D game is.
I think however that they should have used the same harness and also repeated the experiment a few times to judge the variance in results.
Right, model intelligence defines the scope of things they can one shot
I also suspect that users naturally calibrate to a model's useful scope, gradually getting positive/negative feedback and gradually making their requests bigger/smaller than before
Similar to how ML was all the hype about 12 years ago and then it submerged again for a couple of years.
One can hope. Probably an unpopular take here but I'm tired boss.
The software world has a huge backlog of things that can all be done with the tech we currently have, no breakthrough advancements needed, but none of it will get prioritized when we're all forced to run on the new and shiny treadmill. Ever since LLM hype its like the javascript culture of a new framework every 10 minutes has infected every other vertical of software development and I'm exhausted.
"Well obviously you provided better follow-up prompts to the one that came out better."
Also nothing about human-provided plan files and guardrails preclude the one-shot benchmark test. Heavens, I almost said "real coding," but in "real agentic program creation" you'd obviously be doing multi-turn interaction with the agent, but how can you provide a fair test when the model's output n determines your n+1 response?
The business guy would say "hey build me this and that" and would get _something_ to show of.
An engineer will have a long conversation with a llm about the exact requirements, tech stack, tradeoffs. He would understand what is built, how is it built, and refine on the fly until he gets something sensible.
It won't be as fast as "build this", but the result will be much better and more maintainable.
For the enginering workflow, you don't need Fable. Any model better or equivqlent to Sonnet 4.6 would do. Yes, sometimes it will hallucinate, sometimes it'll be wrong, but it's our job as engineers to correct it and have full ownership of the result.
And yet, even the smartest AI in the world would give an alternative solution every time you invoke it. And you still need someone to judge what is right and what is not.
But for a more practical issue, the ultimate goal of LLMs is to replace software engineers, or at least enable everybody to become a software engineer, to use a more up-beat phrasing that's no less accurate. And so an LLM's ability to reliably construct something from a poorly defined, contradictory, or otherwise flawed prompt, while accurately inferring intent is probably the first finish line.
- Vibes are too subjective, I want an actual A/B test!
- An A/B test is too limited, I want a benchmark! (You are here.)
- Those benchmarks never seem to be reliable, I just go on vibes.
It's a relatively objective way of testing LLMs, and I think it's pretty representative of how strong models are overall.
The outcome of this test mirrors how GLM 5.2 and Opus 4.8 work for me: they're both similarly capable of fully executing a given task, but Opus tends to have a bit more "taste" in how it handles unstated details or implicit requirements.
> what you'll get is a series of assumptions made by the model
Yes, but that's why we use these models in the first place. We don't want to explicitly write down all the details because that would mean writing code. So we write a higher-level, human-language spec, and let the LLM fill in the blanks. The question is how good they are at doing that.
Guardrails/conventions should be enforced in linters, formatters, static analysis tooling; not specs/prompts.
https://developers.openai.com/api/docs/guides/structured-out...
Nothing else operates on the logprobs level and literally bans continuations that fail your schema.
https://github.com/elixir-vibe/ex_dna
The specification pattern is essentially a builder for a query (or more accurately in our case, for a C# expression for the ORM) and one of the main benefits is that you can put rules into the “build” method which returns the expression to be used, such as “throw an exception if the WithPartionKey() method hasn’t been called”.
Usual ORM disclaimer applies that you should double check the generated SQL is acceptably performant. The specification pattern can make this better or worse, it can result in contorted expressions which result in poorly performing SQL, but you could also put logic in your specification to “ensure” good SQL, e.g. by having a method which uses the correct expression structure which is known to generate good SQL (e.g. EF6 would generate terrible SQL for certain nested Any()’s, but if you used .Contains() instead it was fine) and the encapsulation means that all callers of the method get the benefit.
It worked well for us as reviewing the specification methods in PRs was pretty easy and less experienced developers on the project were more likely to do the performant and working thing than not.
Elixir is where I prefer to build software, so it would be creating a custom Credo rule.
https://github.com/rrrene/credo
https://credo.hexdocs.pm/adding_checks.html
I agree generating millions of tokens from a handful of input tokens doesn't convey anything meaningful to me.
Additionally, with "Hey build X" nobody is happy with the methodology and people rightfully complain about the set up.
Using your suggestion the methodology would require a lot of presumptions & arguments regarding why you choose it and think it relevant to people.
Either people would not "get" it quickly enough or would disagree/not be interested on the setup because its not how they use LLMs.
Of course, with a software engineer at the helm - the models are going to be able to be guided to produce much better output. (Or worse, depending on the engineer!)
This is the wrong metric to target. Today's models can feel one-shot but they are so at the expense of resilient ReAct loops that brute force their way out of the mess initial prompts created.
And each iteration is expensive.
Sometimes failing fast and early is better than going for one-shot models that try to mitigate the mess they created with reasoning steps and ReAct loops.
To really evaluate how a model is to use in real life, it should have access to tools, and be able to iterate on something, like they do when you use them in an agent harness.
None of that iteration need necessarily to have a human driving it (although if you're building something you want to be able to maintain, you probably need a human driving the design and architecture), you can just let the model do a couple of tries and give it input into how it's doing, and you get something closer to how people use these models in reality.
If we stopped developing LLMs the the only reasonable way to benchmark them would be to compare yheir performance with all the tricks we can build on top of them. Sine the are still developing rapidly any apples to apples comparison is worthwhile.
Of course this particular benchmark is not really single prompt but rather "agentic without steering".
I was using cursor, in large part because I could at least stop it when I need to.
I ended up building my own IDE from scratch so I can be more in the loop while also having the full agent experience.
On another, being able to reliably tackle minor tasks with no handholding is very valuable in itself. Sometimes implementation details are important, but often, the most important thing is to Get It Done.
Instruction following has been down for years, and while there are of course metrics that continue to improve as the frontier advances (for example, the ability to continue following the original instructions even as context grows), you can't really get that much better at performing a list of instructions as-written if the instructions are sufficiently precise enough that there's no wiggle room for interpretation (which seems to be what you are describing).
For example, one of the things that got me the most excited for Fable 5 was its ability to work for over eight hours straight on a single instruction and seemingly faithfully the entire time. That was something I observed personally after trying out the same workflow that runs for maybe two or three hours with Opus and then still needs followups. Fable needed no followups. That's a game changer for me compared to the prior state of the art.
That kind of stuff is going to end up being the most beneficial to people who are touching the edges of their knowledge or even exploring completely new areas. And that type of work is exactly the kind of work that makes agentic coding so powerful, even as much as it gets harder to judge the quality of the work when you lack the skills yourself. It's a good thing that the quality increases across the board, even for skilled practitioners.
For example, even people who know how to write inference engines or how matmul kernels work or how to optimize model architecture can't always predict just the sheer breadth of things agents can try to improve performance, and sometimes you get over some wall and reach a completely different optimum that you just wouldn't have reached in any reasonable amount of time by applying traditional knowledge even if you're an expert in the field.
That kind of stuff is amazing. And that's exactly the kind of stuff that one-shot prompting is testing for. It's kind of like testing for the model's "innovation", as much of an oxymoron that is.
In fact, I'd rather see Anthropic publish a convincing project that does this using Claude. The project should be complex enough and novel enough to show the world how reliable and powerful Claude is. That is, Anthropic does not need Amodei or its employees to tell us that whatever percent of engineers will lose their jobs. They can just show us. Easily.
What about: take top 3 feature requests, top 3 bug reports for 3 popular open source projects and ask to solve those based on the issue contents and access to the project repos.
Even if you stay in a single prompt scenario, you could make it more realistic.
The top agent is for steering, but all subagents are mostly oneshot prompts
Since Opus 4.6 I've seen later Anthropic models being more and more capable on one hand, but also less useful on multi turn open tasks.
It feels like with each model they are more and more prone to go "their own way" and jump into the implementation as soon as they can.
I can't but blame it on benchmarks and fine tuning around prompt-to-solution work.
Current models aren't capable of that, but that doesn't mean it's not possible.
> given the sufficiently smart compiler
For those unaware, this is a similar quote used by compiler proponents. The first full compiler was created in 1957 (+/- 70 years ago) and the "sufficiently smart compiler" never happened, hand written code from the best coders still is faster. Now, that doesn't mean that compilers didn't do the job well enough, we just accepted that 90-95% of the top speed was enough for almost everything.
To the LLM one shotting point, it took 30 (40?) years for compilers to be good enough for the mass market. Caveat early adopter and investor.
Plus what pyrale said.
If you made models able to code to long spec, you would be left with the hard issue of having to write them.
Like if you show the LLM a page, can the LLM review the page and then spit out a review that is close to what a human would say about the page?
Software was always that way, though.
The agentic engineering paradigm is just a narrative trend pushed by AI companies to get people to 10x their token consumption per prompt. It plays into people's laziness and addiction to dopamine too causing addict like behavior in people that fall prey to this trend.
If I do that, I'm literally slower then just doing the change without sufficiently specifying it to the model.
I can see how a junior dev or generally someone that's not particularly knowledgeable about the language or framework they're working with may benefit from such usage, but for experienced people there is very little value in that approach.
I say this because I've just had to face this decision this month with Copilot introducing the usage based billing. I attempted to scale back my usage, first with non-opus - output essentially became discardable as it continually hallucinated no existing fields in the responses of Apis etc... Then my scoping the changes smaller and smaller, until I ultimately gave up and reduced usage to just generating tests.
What is tested often makes no sense at all, completely implausible edge cases are tested on internals, while it doesn't create tests for the overall application using user events.
And some things in these test cases are downright ridiculous: instead of instantiating your classes, it sets up some barebones fake objects reimplementing some of the behavior of your actual class, then ignores the TypeScript errors via force cast or similar.
Then it proceeds to slap some test ids on the output, stubs components and dependencies more or less randomly, adds some assertions on test ids and calls it a day.
Apparently that's good enough for many colleagues to open a MR for that garbage.
That said, at home with SOTA models I happily hand large units of work to it, outsource much of the thinking, and get workable results. I think this is the future.
I see little value in throwing a ton of context at an llm and waiting 10-20 minutes for a coin flip on whether or not its going to produce junk. I'd rather do quick 60 second turns, get most of the way there and fix the rest myself if I have to. I'd rather honestly just not use them.
Everyone that I've ever interacted with and claims to prompt in "seconds" actually needs multiple minutes to think about the solution they want the model to implement - and then need twice as long to formulate that into a sentence which provides the model enough context to actually do that
So the more realistic estimates are "I'd rather spend the 2 minutes just implementing the minor change myself, instead of spending 1.5 minutes thinking about it, then 2.5 minutes writing the prompt and then waiting 1 minute for it to finish"
That's the main value I've been getting out of coding agents. I have them do (comparatively) simpler tasks or explorative tasks in the background while I'm in a meeting, doing code reviews, or otherwise working on something else.
The reality is: - business rules change - ideas for improvement may arise from the initial prompt - updates to submodules/functions/configs/secrets are BLOCKERS ... etc.
One shot prompting for the expecations of complete software is seemingly more and more a show of incompetence of the use of this technology. It's like trying to make my toddler eat a ham sandwich from the peanut butter & jelly I put in front of him.
Running a single one-shot prompt is not a benchmark, not is it representative of any sort of real-world usage.
Most agent usage is collaborative so you need to test things like reliability (when I delegate a task, does it complete it without making up test results for e.g.) and steerability (does it obey my instructions or does it just do what it thinks is best).
I think your test you describe (collaborative, task delegation, task completion, TTD, steerability) is a great format for a future test that I will definitely try out.
Appreciate you sharing the results of your tests though!
Thinking about it, I would say that the majority of agentic work I do, by a long shot, is subagents which are launched from the main session, using a prompt of its choosing. Those could be considered short versions of these fully autonomous tasks.
* Install pi and a bunch of extensions from their package repo
* Realize that all the packages (with a few exceptions) are massively overcomplicated and vibe coded
* Ask pi to rebuild a very simple version of the packages I used. So e.g. subagents - all the default subagent extensions are massively complicated with named agents, recursion, communication. I made one that stripped all that out.
* Then whenever I hit an annoyance, spin up a parallel session and fix it.
It's less work than it appears because I have ~5 extensions: hooks, subagents, background processes, a custom footer, a loop command... Maybe that's it. Within a couple of days you can have a setup pretty close to Claude Code but with a fraction of the base context use. After gradual improvements over a few weeks/months you'll have a system far better, tuned to your exact preference.
Of course, just like Linux or any other highly tunable system equally important is having the restraint to not spend all your time tuning it. I've definitely had a couple of days where I was bored with my real work and did that, but whatever, it beats browsing reddit.
As for getting long running tasks, I set a looping message every ~20m and tell the agent to strictly track progress in a session doc, then reread and continue after each compaction.
It's one of the reasons I suggest you study the famous setups (oh my pi, or superhuman skills etc.) and convert them to your personal needs.
I've not come across a programming task that would take an LLM ten hours.
Overnight it ran ~800 tiny auditors. I then run synthesis on the written audit files, extract bugs, then another round to find which ones have a common source, group them by priority etc.
I've cautiously started doing larger tasks that are not just read only, for example I was dealing with a large codebase full of lint and type errors, so I sent out waves of workers with clear instructions to only fix obvious/trivial issues to and otherwise to append to a todo file for my review. That worked well and cleared a few thousand issues over several hours.
I don't really want to share any other tasks I've worked on this way because it'll draw out the agentic coding sceptics and I'm not interested in defending my workflow.
You make a very strong claim at the end that the hype is mostly real, and making it clear to what extent your claim holds should help the reader.
At home I make a point of trying other models / tools on my side projects. So I've been using OpenCode and trying tons of models via OpenRouter. I tried Kimi, Deepseek, MiMo, etc.
GLM 5.2 is a _major_ step up from every other non-GPT/Claude/Gemini model I've tried. It's not as good as latest Claude Opus, but it feels every bit as good as Opus from ~4 months ago at a fraction of the price.
To me this model is the "it just works" moment for open weights models. We had this for closed weights models in late 2025 when Opus 4.5 landed. This is the same feeling I'm having with GLM 5.2. It's 90% as good as what I get from Anthropic for 1/5th of the cost and without any concern of lock-in.
For us, MiMo v2.5 Pro and Qwen 3.7 Max were already plenty enough before GLM 5.2 (which, like you say, seems like a step up). The BigLabs really have a big fight on their hands.
- it takes it sweet time to get code rolling, not the fastest model by any means
- it strays a lot during discovery/planning but then corrects
- it's not steering friendly, as it hallucinates things that it doesn't follow later on
- its output is quite good
A sample use case: I was optimizing rendering on Swift+Zig codebase. It chocked on 5k data entries.
GLM 5.2 spent 20 minutes building the benchmarks and getting data out, which made me frustrated so I blocked non-editing tool access and went AFK, after approx. 30 minutes I found that it used already-made benchmarks and some "conclusions" to optimize 3 choke points. Output pointed that it couldn't validate suspicions and asked for more data.
Implementation worked well, it was idiomatic and non-intrusive. I would even say that it was more idiomatic than GPT 5.5 effects on same repo.
I would opt in in using it more BUT GPT usually completes same requests 5x faster.
GLM 5.2 was spark for preparing and running inside isolated containers with JJ workspaces (so that multiple can be ran in parallel).
It's always a shock to me how opaque most other models are!
It also is pretty resilience to letting you inject in while it's working without going off course or while getting back on track after, which I appreciate
This is (unfortunately) by design. The proprietary models hide their reasoning traces so they can't be used for model distillation. Sometimes even when they do show reasoning, it isn't the model's real trace - IIRC, someone was able to demonstrate that Opus' reasoning is usually a summary made with Haiku behind the scenes.
Switched the model to GLM-5.2 halfway in the middle of a troubleshooting session (didn't even bother to reprompt, just changed it in the middle of its reasoning), gave it a few minutes, problem fixed. This is with the subscription based allocation on OpenCode Go, where a problem like this would completely burn up my Opus for the current 5 hours or even the current week.
Which provider are you using? I got a z.ai Lite Coding Plan and it's my understanding z.ai is on the slower side of providers and the Lite plan gets lower priority on top of that. In the api key console, it shows dipping below 60 tok/sec which is quite slow.
It is less than 20% of the cost of Opus at API rates. 1.40/4.40 vs 5/25.
Maybe makes sense if you have z.AI's (not greatly priced) subscription plan, but it's not competitive against an OpenAI or Anthropic monthly coding subscription plan. I burned through almost $10 worth of tokens just doing an hour of work.
You get access to a whole bunch of bleeding edge open models including GLM-5.2, Kimi K2.7, DeepSeek 4 Pro, etc. Inference is run on US/SG/EU cloud providers with zero data retention policies. The $20/mo tier is very generous, in my experience.
> Where are models hosted?
> Ollama hosts models and compute resources primarily in the United States. To serve global demand, we may route to Europe and Singapore for additional capacity.
> Is my prompt or response data trained on?
> Prompt or response data is never logged or trained on.
> Who does Ollama partner with to host models?
> Ollama collaborates with NVIDIA Cloud Providers (NCPs) to host open models.
> When Ollama partners with providers, we require no logging, no training, and zero data retention policies in place.
Which I think only underscores my point that actually the GLM models are not very cost effective.
They essentially cost the same as the SOTA models from OpenAI and Anthropic, while not being quite as smart. I could have gotten about the same amount of work done on the $20 Codex plan. And I had to use my $100 Codex plan to finish the work GLM started before it ran out of quota. And also to fix it since GLM left a bit of a mess.
I like that GLM exists. Other Chinese models are far more cost effective. GLM is expensive, even on a fixed plan.
But if you are building a product or in an enterprise environment where you essentially have to pay API rates then GLM is the best value hands down.
Capability per dollar is something I care about:
So you're really getting near opus level capability for the price of haiku.the statement isn't "GLM 5.2 has large token usage", it's "GLM 5.2 has large token usage vs modern Opus".
I haven't used it, but this wouldn't surprise me. I see ~30% lower token usage for better results with Opus 4.8 vs 4.6 (and i had great results with 4.6)
GLM is promising but it's pretty costly, all things considered.
If I recall, that model had a couple issues. One was the issue of being monkeyed with, for which they gave everyone credits.
The other feature/bug, depending on your POV, was being Anthropic's least personable release, not papering over everything with self help guru therapy language.
Opus 4.6 didn't LARP. It was more direct, less fussy, less discussy, and very much less "wait, one more thing" within a couple edits after embarking on what should have been the spec, than 4.7 or 4.8 are.
When in engineer brain mode, working as as you describe (good old fashioned XP-style staff engineer pair programming with a language-savvy mentee not yet full-stack or system wise), I found the clearer I was about my goal and the better I could express it, the more often I'd get an expanded clarified response I could then iterate to steer for ever tighter cleaner more specified responses, then let it go build the whole thing without it agonizing and waffling.
The next two releases regressed on that dimension, wanting to figuratively "sit with" every decision and re-validate spiritual alignment along the way, no matter how clearly expressed.
Curiously to me, Fable seemed to hit the best of both worlds, I had the highest commit per turn with Fable, approaching 73%, where I'm usually under 17% of LOC written being good enough to commit, usually taking 9 - 11 turns to get the code where I'm comfortable with it.
Thanks to this, Fable cost more, but actually cost less, if that makes sense.
Arguably, Fable, and 4.6, played more outcome-correctness oriented than journey-experience oriented. It's easy to see how this could happen with human reinforced learning if not all judges are staff or principal engineer level, or constitution values are more Portlandia than Finlandia.
ANTHROP\C needs to balance these at the constitution level:
“We will work in a humane and thoughtful way, but production is the final judge. We will listen to people, but we will not let discussion replace decision. We will value craft, but not at the expense of usefulness. We will move fast, but not by hiding risk. We will measure outcomes, but not pretend that everything important is easy to measure.”
GLM-5.2 actually has really good intent understanding though, on par with GPT-5.5 and Opus from my experience.
It would be more interesting and accurate to see the comparison on the same harness if the intent is to compare the frontier models.
Pi is relatively new and does not have many features built-in compared to Claude Code. It was chosen intentionally this way as Pi's goal is not to create a bloat builtin of tools most don't use but to allow the users to customize to fit their need -- similar to Neovim vs IDE.
The end-user "vibe coding" experience is *heavily* swayed by the harness because prompt effectively drives how a model outputs an answer.
They picked a task that heavily favors a model that can do multi-modal with images, and GLM still came within striking distance.
What I'm hearing from this article is that the next generation of open models that includes better multi-modal support are basically no-brainers for adoption.
Seems like a HUGE win for Z.ai and open models in general here.
I haven't been keeping up on hardware costs for state of the art LLM inference, but this remark made me ask myself how many readers of the article would actually be able to run this model on hardware they own. How much would it cost to acquire such a setup?
To be cost effective with inference providers, you have to find some way to be using it 24/7.
If they decided to collude, they could absolutely say "from now on you no longer have access to model X because you're an asshole"
The commercial inference offering are also downstream of one of those 3 projects (or trt-LLM if they're nvidia). It would impact Ollama, and fireworks, together, and everyone else.
Don't tempt fate.
You're better off setting a budget and buying the best machine you can afford in that range, or picking a VRAM target and accepting the class of models you can run on it. Those models will almost certainly improve over time and your skills will adapt to the limitations. Hardware is so valuable right now that it's not even likely to be a significant loss if you had to sell.
Right now I think 24 GB is probably the best bang for your buck (used 3090), because you also get a high end gaming/gpgpu device which is nice anyway. 32 GB you can do with AMD or Intel, but NVIDIA is megabucks and at this point you're really paying for RAM. Unfortunately the ship has sailed on "reasonably" priced RTX 6000s, which at one point were about $7k and are being listed at $10k++.
if it doesn't apply to you then just come back in a couple years and see what the situation is then. 1 million context window, 1 million tiny layers to fit in 4gb RAM at a time, with 256gb of fast unified RAM in every consumer device? Or a different concept entirely
in the meantime, z.ai probably doesn't reply to US subpoenas so you can shift all your incriminating conversations over to that and use GLM anyway. who cares if the Party trains on your data and steals your IP and ignores you for legal matters, when the alternative in the US is just a thin corporate layer and party who steals your IP and will snitch on you for legal matters.
GLM-5.2, severely quantised, 512GB Mac Studio, somewhere between $10k-$35k for a used M3. Or run it on a CPU with 768GB of RAM by getting an old PowerEdge with DDR4 for around $5,000.
Qwen-3.6-35b-q6, runs well on an RTX 5090 ($4000 + cost of a PC), runs medicore on an Intel Arc B70 ($1000 + cost of a PC plus lots of fiddling to get the setup to work right).
Gemma is a good candidate for the cheaper stuff, but I lack personal experience with using it locally
Basically a 1x 24GB GPU (32GB would be better) plus 256GB of free system RAM, or a 256GB unified memory machine (like a Mac).
Kind of shocked they got the results they did.
Caveat here is that all of this is Claude math, but would be interested in someone more knowledgeable of the math chiming in. I was thinking that API pricing was highly inflated in order to cover subscription costs but with these calculations it might be not?
Can someone explain to me where that time usage is coming from if not from the model operation itself?
Are the individual tool calls more complex and take more time to complete? Or is the rate of tok/s lower because the model does more compute per token?
In addition to that, some of the open weights models like GLM 5.2 or DeepSeek v4 Pro tend to be MUCH slower when generating tokens, which contributes to the perceived slowness. Although I wouldn't call models like GLM 5.2 slow by any means, e.g. it is currently one of the fastest models inside Notion today.
I also use MiniMax-M3 in utility roles like explore/library tasks.
I’ve had a z.ai subscription for several months so I’m on the older pricing. I’m really not sure it would make sense to do this at current rates - I could bump my Codex plan instead.
A better way would be to use https://github.com/openbmb/MiniCPM-V
I'm not sure what exactly triggers it, but it seems to happen when it has to look at lists of countries. I suspect there must be at least one country name that triggers the safety guardrail.
You'd expect GLM to balk at something like Taiwan, but so far, it hasn't.
Part of me wants to believe they really do care about protecting the world from... something... I don't know quite what exactly tbh... but it must be costing them a small fortune to scan each input and output against N guardrails and they are a for-profit corporation who could easily turn a blind eye to all of this and simply say "what you do with this model is on you" like I would expect most corporations to.
Strange times.
I remember making games ten years ago, and it was such a tedious and painful process. This is effectively lightning in a bottle even at a fraction of it's capability.
The next 12 months will be wild (assuming we don't have Chinese models banned by then in the US).
Off topic, but does anyone else instantly pick up on LLMisms like this? It seems like all the models have converged on this style of writing, and improvements aren't really changing it.
There was this dude here not long ago who bought like $70k worth of gpus to research, and if I'm not mistaken his research was something related to make llms sound less llm-y. I wonder how it goes for him.
I can't believe I would say this. I TRUST OpenAI more than Anthropic. They try to play best actor but they are manipulating the behavior of the model in the name of guardrails/interpretability.
That is why I refuse to build anything that works with Anthropic models as the backend. Because, when they want to shut you off, they can do it by just making model less reliable in your product than their offering!
For people who follow open LLMs, none of these were quiet and all were the most interesting open model release for a few days/weeks. In one or two months, it will be some other model again. Now I do appreciate the real rapid improvements in open models. But there's also a ton of hype and fast-fashion around all of this.
GLM passes a meaningful threshold of reliability/utility that puts it in a different category for real work. Just like Opus really took off after passing a threshold with 4.5. It's the first open model to do that.
I once gave Sonnet 4.6 and Qwen 3.6 the same real-world task to compare: "extend the existing code with this new requirement". Qwen3.6-27b perfectly followed the existing conventions, while Sonnet 4.6 invented its own conventions that were rejected during CR by another dev (i.e. he basically chose Qwen3.6's output in a blind test). Qwen3.6-27b, run locally, also managed to finish faster on that task (mostly because Sonnet 4.6 made tool calling errors and removed some code by accident, so it spent additional time reverting its errors, and got somewhat confused in the process).
We already have production code running live that was written entirely by Qwen3.6-27b. Although, we plan to move to self-hosting GLM5.2 because it's more versatile.
And there are valid reasons to run local, even if performance (quality and speed) aren't best.
From my Opus vs DS 4 Pro personal benchmarks, 16 different real-life work tasks, DS 4 has performed as well as Opus 4.8 high overall but with few drawbacks:
- on the 16 tasks, one needed several prompts to be steered back into the topic
- its review capabilities seem much worse
- DS4 had the cleanly better solution in 3 cases out of 16, with Opus "only" doing cleanly better 2 times out of 16. But still, I want to emphasize, is the worst case scenarios that imho matter the most, not the best ones, and on that front Opus outperformed.
That being said I spent less than 2$ of API working 4 days, which is more or less what I would've spent with Anthropic APIs for less than one task.
We have come a long way, and very clearly have a long way yet to go.
Yes, in terms of API pricing, GLM 5.2 outperforms the competition. But the only people that use API billing for their coding work are large corporations, where these highly subsidized subscriptions are being fazed out.
At the same time, none of these companies will use a Chinese API for their employees.
For individuals and smaller teams, Z.ai's coding subscription is outperformed by Anthropic and OpenAI. You probably get around the same usage with Claude, but Codex definitely offers more usage for the amount you pay.
We can have a debate how much Z.ai closed the gap to GPT5.5 and Opus 4.8, but if I can freely decide between them in a world where they all cost the same, I simply wouldn't choose GLM.
So the important question becomes: How good will the offering from Z.ai get with GLM 5.3 or 6 and how much will OpenAI and Anthropic cripple their current offering in the near future.
If the world needs any more evidence of Europe's short-sightedness, it would be them running to China to spite the US (instead of creating fertile grounds for their own tech).
Employees and students used to coding with thousands of dollars worth of tokens (on a 20/100 dollar plan) will push enterprise to spend.
Having a Chinese model that is competitive won't displace this enterprise spend. But an open model hosted in the US/EU might.
The existence of GLM 5.2 puts a ceiling on how much OpenAI/Anthropic can charge for API Access.
Except there is no evidence of this at all, just people comparing API and subscription pricing. The leaked financial info for OpenAI shows inference is profitable right now, though it does not show a distinction between subscription and API revenue... but if subscription revenue was so lossy, it would hard for total inference to still be profitable.
I believe this is the reason why we can even have this debate. Without this kind of competition we would not have these subsidies.
I just think that as of today, most people will not find a good reason to switch to GLM.
It's annoying that the plans are so restrictive beyond usage limits. Understandable maybe, but annoying. In practice, only Anthropic (and maybe Google) are really restrictive though. They really scared me away with their policy of charging API rates after the fact if they consider your usage not TOS-aligned. This might be an ungrounded fear that I have, but I feel this is something they'd do so they scared me away.
As well as people using 3rd party harnesses like OpenCode.
> At the same time, none of these companies will use a Chinese API for their employees
So who are Amazon Bedrock (who serve GLM) targetting?
Individuals are presumably going with one of the cheaper US providers such as DeepInfra ($0.18/M cached input for GLM vs $0.50 for Opus) or Fireworks AI.
nice try but you intentionally ignored the entire Chinese market & Chinese big corporates. there are 130 Chinese companies in the fortune 500 list, with an average revenue of 80 billion USD each. do you think they are going to sign up for Claude, Codex or GLM? now consider South East Asia, Africa, Middle East, Middle Asia and South America, tell me why their large corporates won't be using GLM API billings?
your western centric view of the world is totally out of date, like it or not, 2026 is vastly different from 1996, the US no longer controls high tech whatsoever.
A company can buy a NVIDIA B300 and serve it's developers in house with unlimited tokens.
A future where capable AI is genuinely accessible to everyone doesn't look far off — especially since at this point a lot of the robustness and usefulness comes down to the application layer wrapping the model, not just the model itself.
Opus is most expensive model in pay as you go model, but IMO fair comparison should include subscription price as well. For example when one has $100 Claude Max and use it up through the month, it might not be more expensive than GLM, or at least not 5x.
And z.ai themselves also have subscriptions.
https://z.ai/subscribe
I’m currently trying to figure out whether a downgrade from Max 5x to Pro in combination with one of those would save me money and if so, how much.
Edit: seems like Anthropic Pro + GLM Pro (Yearly) would let me almost halve my costs of Anthropic Max 5x. Only concerns are about GLM 5.2 not having vision support and also being kinda slower and also not being as good as Opus.
I think it's most fair to compare the plain token pricing that is used by everyone.
As a consumer, yes, it's totally fair. All that matters to me is the price I pay at the pump, not whether that price is "real" or not.
Anthropic have claimed they expect their first profitable quarter this year -- they may have bigger margins on their raw API than you realise.
I'm saying that this is not necessarily the case. They do a lot of optimisation and don't have the same price pressure to lower margins. They may not be losing as much on subscriptions as people think.
I read that GLM 5.2 (and other GLM models) were specifically trained to be "helpful" as Claude is. I have big hopes on GLM line of models growing to be a real alternative to the Claude in the near future.
The only thing you measured was a single tunable of how much work should be done on a vague prompt. Now make the prompt be something that causes GLM-5.2 to cost 4x of previous budget, to get something comparable.
(And the wallclock time measures the inference provider, not the model.)
But, it produces solid results for a fraction of the price. Worth checking out if you have the time.
One of my goto "tests" of a new frontier models is having it rebuild a programming language from scratch. For GLM 5.2 I had it rebuild the old Rebol language in Rust:
https://github.com/mhs/rebol-clone-glm-5.2
It did a fairly good job roughing in the language for a low token cost.
https://aibenchy.com/compare/anthropic-claude-opus-4-8-mediu...
Glm game was completely broken Opus game was at first glance ok but also with bugs
Different models with different cost produced different non perfect results . How is it “close” ? :)
Also on costs : glm burns more tokens on average vs opus . Gpt5.5 burns less surprisingly
https://generative-ai.review/2026/06/glm5-2-from-z-ai-vs-cla...
I've structured it side-by-side. You can clearly see where the private models excel, and where GLM 5.2 is still really good.
This implies Opus was potentially much (?) better value.
GLM cost a quarter but Opus was twice as fast. So we are already at GLM actually costing half when you compare on time, without even considering the extra effort and time it would take to get Opus-par results.
It's good to have cheaper options and very impressive to see the Chinese continue to set open standards in this field, but the article is maybe a little over-generous.
https://github.com/ggml-org/llama.cpp/issues/24730
From there I collected the following US providers currently serving GLM 5.2:
- Together (https://www.together.ai/models)
- Fireworks (https://fireworks.ai/models)
- Featherless (https://featherless.ai/models)
My only, I guess feedback, is that it's not really clear about the price.
Would the 21.92 be the API pricing I guess?
Cost $5.39 (real billed) ~$21.92 (estimate, list pricing)
I like it, but the lite plan ate 22% usage of my 5h reset window in a single session after 2 prompts on xhigh of GLM 5.2 [1m]
Result was satisfactory, I think stuff is decent, I'm happy to use either, wish there was a combined subscription plan where I could get both
Coupled with a local Headroom (https://github.com/headroomlabs-ai/headroom) you'll be able to use a LOT without hitting your 5h window :)
Definitely the best $ value for me considering the reasonable performance of GLM5.2.
They provide a rolling window quota, so you're never really out of quota contrary to other providers, you can adjust day to day.
Check it out if interested : https://synthetic.new/?referral=kwjqga9QYoUgpZV
---
Docs & all models : https://dev.synthetic.new/docs/api/models
EDIT: I've RTFM lol, thanks for the links, will give it a shot!
Glad you figured it out :) Let me know your thoughts about the quota and GLM5.2, so far I don't think I've came across against anything better, $/usefulness wise.
I used it with Cerebras inference at a time when it had a good coding plan at a low price, and delivered tons of stuff using it.
"Build a 3D platformer game from scratch, in raw WebGL, with no game engine or 3D library" would be a zero-shot prompt.
I would like to give them a try but I certainly not have the money to get a system able to run them, and I don't really want to pay more than the state of the art
The real time 3d fluid dynamics appear to be the tricky part, I wish I still had opus access, would love to see if it can do it.
Pro tip: You could use a multi-modal model to verify images as a subagent spawned by GLM 5.2, to get around this issue.
So, 8000$, plus it's unavailable. 3 years of Codex/Opus subscription.
> API prices
Which are irrelevant for 200$ Codex/Opus plans that are times cheaper.
I am not sure where this is going to lead us but it is fun to watch.
Tried with 2 harnesses and it seems bad + slow
you go to OpenRouter and pay
$0.98 / $3.08per 1M for GLM 5.2 vs $5 / $25per 1M for Opus.
GLM 5.2 gives you OPTIONALITY so you can run it locally, but you can still just pay somebody for it.
If it builds a UI and can't look at it, it's askin ls whether the app looks right.
At the end of the day, the time earned is more important then the cost for big players.
The ability to spawn 10 claude agents and rush a project to outcompete someone is more important for big businesses in my imo. Also the small details that GLM missed would take significant more time to iron out, considering it already took double the time.
I do hope other (open weight) models catch up, but to act like they are anywhere close for me is a bit disingenuous.
Also, every single lab does RL on benchmarks, which is why Opus 4.6 was the last truly great assistant, after it, all models tend to drift into implementation asap.