Academically a - very - interesting article, which confirms my ongoing suspicion of LLM Benchmarks in general ( probably why I like it so much). Another thing that caught my eye is DeepSWE having a bunch of bugs in them. The more I learn about these benchmarks the more sceptical I become. I would expect such a prolific benchmark to have these bugs ironed out by now.
The practical takeaway, however, seems to be an interesting "don't use max mode". The passage about deepseeks weird API behavior dropping to pretty much zero the moment you used high says a lot.
Also, personally, even though Opus is clearly so much ahead of the Open Source models after the changes, I think the adjusted score is still very much competitive. I see other commenters using this as proof to justify why their opinions of the open source models - but a drop from 74% to ~50% is, while a big drop, still about a ~50% pass rate afterwards. That's far from useless for coding, especially since DeepSWE seems to be such a difficult benchmark. It simply seems to more accurately reflect the capabilities of these models.
All in all, I'd love to see more of such articles, and will probably keep an eye on this one - if they continue "benchmarking" these models this way.
AI Labs actually. They would love the perfect benchmar representing real usecase. I'd even dare and guess that they don't share much publically because it is a huge competitive advantage.
It is "a" problem, but honestly not a huge one. Chasing benchmarks is an integral part of AI research, and it is how we have gotten this far, which is pretty damn far. Discovering the ways in which benchmarks fail is in itself an interesting process, and the amount of interesting new benchmarks has grown with the capabilities of the models as a result. The frontier labs might be a bit selective about which benchmarks they publish as marketing, but overall they have so far been reasonably diligent about avoiding outright cheating. Most of the truly benchmaxed models are not on the frontier (eg. see muse spark performance dropping off a cliff in the article).
This is a scenario where the interest of the users and the labs actually align perfectly, since labs that have people actually paying for their product want benchmarks to align as close as possible to real life performance. Otherwise they can't actually easily tell whether the new model they are developing is any good or not, and so risk wasting literal billions of dollars on training runs for a product that people won't end up paying for. A failed major training run is an absolute disaster at the frontier, GPT-4.5 for instance was an almost existential failure for OpenAI at the time.
Case in point, the most popular coding model, Opus 5.5, also just happens to be the only one that avoided the benchmaxing impulse in this article. This is the result of years of Anthropic working very hard to avoid it.
That’s missing the point. Frontier models get shittier in certain ways with each release, due to benchmaxxing, and there is nothing we can do about that.
Very much disagree, to me it clearly shows the gap between Open Source and the frontier. You can not do real code work with open source models right now.
Especially since this benchmark is still mostly DeepSWE, and the only reason the open source models are doing decent is because much of the information hasn't changed. The open source models have been benchmaxxed to death.
> Very much disagree, to me it clearly shows the gap between Open Source and the frontier.
Yes, this is true.
> You can not do real code work with open source models right now.
This is false, or you're using "real" as a loophole. You can absolutely do real coding work with open source models, right now, right this second.
I'm doing so today using Qwen 3.8 Flash Next running locally, and GLM 5.3 Flash hosted.
We could do real work with Opus 4.5, 4.6. These models (for coding) are more than good enough to do work that Opus 4.6 could do, and in some ways are superior.
This DEFINITELY squares with my anecdata that Gemini 3.8 Flash is not even remotely close to Opus 5.5 on the types of problems DeepSWE CLAIMS to be benchmarking.
Astra does seem close.
I'd be curious to see Argon on there, since my experience is that Argon is also significantly behind Opus 5.5 and Astra despite the benchmarks (obviously much closer than Flash, though).
I've been meaning to take a closer look at what's behind the curtain of DeepSWE - since I built what I thought should be the easiest possible coding benchmark - and the models I tested it on scored roughly the same as they do on DeepSWE... Which is supposed to be hard...
We recently benchmarked our AI code review agent (just pi with some custom prompts based on info from our context layer). Previously we used Gemini flash 3.8 as it was capable and affordable. However, haiku 5.5 just blows it out of the water. It's more than 10x cheaper, 2x faster and finds 3x as many real issues.
This is a great example of how to test how well a benchmark score generalises. The first order approximation is not complicated: just vary a parameter and see how much the score changes.
Which parameter to vary depends on the result of the test, but the MaxxBench route of varying the task spec seems like a good default, though not always the most informative.
In one of my private benchmarks I realised many of the less capable models didn't fail because they submitted the wrong answer, but because they exceeded their budget for number of tool calls they were allowed. The more capable models never got close to the tool call limit because they tended toward combining tool calls (e.g. chaining multiple shell commands in one Bash tool call).
Was my benchmark actually measuring ability to combine tool calls rather than reasoning ability on the problem at hand? Well, easy enough to find out: re-run under a different tool call budget and see how much the scores change.
(Next problem: is ability to combine tool calls a useful thing to measure, i.e. should it confer an advantage in the benchmark? It could be if it is a sign of planning multiple steps ahead.)
Yes, two ways failing to combine tool calls is a real deficiency for the LLM's user: it costs money and can get you rate limited.
Every turn forces you to pay $ for cached inputs and possibly gets you rate limited on TPM.
At 500k context and $0.26/M-tok cache read price, every time it combines two tool calls instead of taking separate turns would save you $0.13. That could add up.
Maybe the provider sets a rate limit of 2 million TPM. That's just 4 turns in 1 minute before you start getting 429s. Cutting them in half by combining calls can save that.
All else equal, we definitely want models combining tool calls!
It also makes models completely ungovernable and unfollowable.
Asking for user permission before an action is pointless. it's not going to use a single `write`, it's going to run a 25 line perl script without even newlines.
At this point just completely remove permissions, and leave only one tool, a rust-like scripting language.
But this also drives usage up, because you rely less on yourself, and more on the increasingly black-box that is the subagent-of-subagent that you are not even going to check the result of.
It depends on what you actually want to ask permissions for. We're still doing trivial script execution, but we could go further. Capabilities like "do you allow giving access to JIRA read only token" would be a massive improvement. But there are other paths too.
> The more capable models never got close to the tool call limit because they tended toward combining tool calls (e.g. chaining multiple shell commands in one Bash tool call).
So tool call limits during benchmaxxing is why they have gained this behavior, Goodhart strikes again...
I look forward to the near future when training is more accessible to more people and we can have more experimentation and diversity in models.
The big trend in that graph is that smaller models collapse the most. Everything with "-flash" in the name, plus OpenAI Luna.
Look at GPT 5.6-Sol vs. GPT 5.6-Luna in particular.
(The GLM pair shouldn't be expected follow this pattern, because they're misnamed: GLM-5.3 and GLM-5.3-flash are distinct lineages. The -flash is an early release of a new pretrain on a new architecture).
If this *is* the main factor, I'd expect Kimi K3.1 and Gemini Carbon would also do well on this test (unlike Gemini-flash); while Haiku 6 would collapse (breaking the trend of the other recent Claude models). It's a falsifiable prediction.
Another hypothesis is that it's distilled models which collapse the most (small models from big labs are distilled versions of their large models).
It's not unreasonable to assume Kimi 3.1 is also a distilled model even though it's large (the difference being that it's a distilled from US models). I wouldn't be surprised if it also collapsed on this benchmark.
There's an interesting way to find this out: ask the model "What are you?" hundreds of times. Around 1% of replies at default temperature are really left field.
For instance, I had Qwen3.8 Flash Next reply with "I'm Kilo Code, a coding agent designed to ...".
That was in llama.cpp's UI - no harness whatsoever.
The DeepSeek 4.1 Flash behavior of circumventing rules and restrictions is one I have observed several times. It doesn’t seem to be malicious or deliberate, but if you have a config problem in the harness or you block something it thinks it needs to accomplish its task, it will get very creative to circumvent it.
I run a lot of agents headless on tasks. They are pretty well sandboxed, but I would definitely suggest a regular audit script to go through the transcripts. Most of DeepSeek’s “creativity” has shown problems in my sandbox setup which get fixed (e.g. I had restricted something the agents actually needed), but some of it is actually worrying.
In an opposite turn, Kimi 3.1 follows rules much better (actually too literally sometimes). K3 was a real digger when trying to benchmark it if it got a sense it was being tested. 3.1 comments about 5x more that it thinks it’s being tested (and it’s right), but unlike K3 it doesn’t seem to do anything about it if it can’t get at something. I would love to see the authors do K3 and 3.1 in this because the 3rd party benchmark gains are large, but these two are behaving like different models for me rather than a point release.
> It doesn’t seem to be malicious or deliberate, but if you have a config problem in the harness or you block something it thinks it needs to accomplish its task, it will get very creative to circumvent it.
In my experience, they are very good at writing Rust that skirts almost - if not all - the safety you're hoping to get from Rust.
My understanding is this is less about labs training directly on the benchmark as about the labs being sloppy about creating RL environments that reflect the benchmark task
They should be doing what this test did, and generalizing and significantly varying the training environments (and ensuring some of them are unexpectedly complex and unexpectedly simple).
Further I suspect the smaller models are especially overfitting to these sorts of simpler RL envs (benchmarks tend to be much simpler than the hardest RL env used today) because they are more likely to be scoring at ~20-40% success rate (which is sort of what you want to get good training signal when doing RL runs).
I feel more and more we need to move to a world where it’s easy to build and maintain our own benchmarks based off our own code base. I know it’s feasible, and I nor where I work have done it, but it’s funny that we trust benchmarks on tasks we basically have no idea what they do (not for lack of transparency) or how they translate to work I’m actually doing
And several companies to do just this exist (eg Superconductor, Fullbeam) - does anyone use them and find them valuable? And curious if you’ve noticed an improved change in your agents work as a result, and if over time you see yourself switching models ? Or is it more of a one time realization that model family X is best for your code.
I built my own. It’s IMO necessary, but it quickly turns into a job by itself. I’ve been benching K3.1 all weekend. Even when you get a good test suite, you need to do multiple runs, not for variance, but because the model behavior changes so much between releases you often need to retune to get a real score if what you’re after is actual optimal (or closer to it) performance on your workload.
Example: I could barely induce K3 to use subagents. I had to force it via instructions and then, in my tests to try to get it to use Qwen, it didn’t work well. 3.1 lost 6 hours thrashing overnight because it voluntarily called subagents and the box can only run one Kimi at a time. My own fault for not looking up the new Moonshot system prompt (that they changed also because of this behavior), but one example of how different even point releases can be.
Particularly interesting cases here are of GLM 5.3 and Claude Opus that seem to rely on assumptions despite clearly stated requirements.
> Let me just do float64; the risk of a test with >2^53 numbers is minimal.
> such cases are unlikely … Truncation toward zero is fine.
Apart from this, we also a scenario where the models tried to pass benchmark specific tests by believing a parameter that the model printed.
> Seven models reward-hacked a test that believed a metric from the program itself.
DeepSeek also resorted to making external model calls via OpenRouter instead of doing the work itself. This suggests that even though the SOTA models are smart enough to compute, they either have an inbuilt bias towards achieving scores or the low/medium effort gives them the freedom to fetch information instead of computing it. With high effort, the models focus on computing the values but the token calls get exhausted.
Difficult to find a balance amongst compute, calls and cost.
One thing to consider is that even when some model has big gap between those two benchmark it doesn't mean it is much worse. It could be worse becasue he has assuptions based on training but on real scenario it wouldn't have it so it still could achive great results
This test still says something about how well the model adapts to subverted expectations. Some models get more stuck in local regions of their latent spaces than others.
But I agree witih you an important piece of the generalisation puzzle is missing.
(1) An established framework for model evaluation runs the inference call with API key inside the isolated envionment the agent harness has full access to. Especially when it's intended to be an offline environment, that seems kinda... dumb. The language model itself is not dangerous -- it's only text. Inference is fine to do outside the isolated environment. It's only the harness' tools that can have effects on the enironment, so those need to be sent to the sandbox.
(2) Only a small set of language models exploited this! I run private benchmarks a lot and I know how much the language models try to anticipate grader behaviour when the task is bechmark-shaped. I would have thought more of them would request the answer from the web.
(3) It takes someone else to mutate the task specification. When I'm looking at an older iteration of one of my private benchmarks, the first thing I do is add a small twist to it to see how much of a passing grade is learned/anticipated. Seems like a cheap and easy coalmine canary that's in the interest of the evaluator.
(4) MaxxBench did not try to dress some DeepSWE tasks in clothes that make them look less benchmarky. Their purpose seems to be answering "how well does DeepSWE enchmark scores generalise?" Part of the generalisation error is memorised solutions, yes, but another part is that models emit different behaviour when they suspect they are in a benchmark. By being careful to not leak the benchmark nature of the task, models spend a lot less time reasoning about the behaviour of the grader and this is likely to produce different performance on the mutated task. It would be interesting to know in what direction that effect points, because that is the performance people get out of the model in real situations. (In my experience, hiding the benchmark nature of the task can produce improved performance on both the original and the mutated task, but particularly the latter.)
> I run private benchmarks a lot and I know how much the language models try to anticipate grader behaviour when the task is bechmark-shaped. I would have thought more of them would request the answer from the web.
This is something I was thinking about, too. More than once I've observed a model note in a reasoning trace something along the lines of "this looks like a coding test" (could possibly also refer to leetcode style problems, but same difference). Models often also seem to behave differently in response to one-shot prompts.
On occasions where I let a model do autonomous work, I usually don't just give it a prompt for this reason. I write a CONCEPT.md file, then refine that interactively into a SPEC.md file to find and resolve ambitaskguities (and possibly also create an ARCHITECTURE.md file, if it isn't a one-off ). At this point, most models seem to no longer see it as a benchmark or competitive programming problem, but a proper software engineering task and are less prone to reward hacking (with the caveat that these are very subjective and unfalsifiable impressions).
I bet we will find this is mostly that smaller models have issues to go against: 1. strong priors that are inverted here in the changes of the benchmarks. 2. underspecified specs for which hidden tests test things not clearly stated, but that very large models can guess knowing they are tested.
Great job, this mirrors my experience running private evals nearly 1:1, down to even GPT models post Spud pre-train (so anything starting with GPT-5.5 to now) underperforming [0] [1] what one would expect based on public evals, the second one runs them privately. Also, Anthropic models have rarely under and often over-performed in my evals focused on task adherence, task evaluation, task inference and tool calling.
Love having a clear, concise and actually based in tangible information post to point to next time someone tells me that me models like Deepseek, Qwen and Grok massively underperforming what the public numbers show means I must simply not be capable enough to use them properly, even when it comes to tasks that Opus 4.5 managed but deviate from what most public evals cover.
Maybe it's not me obviously being biased against Chinese entities (Deepseeks papers are glorious and I am grateful for them being public, I also very highly rated Kimi K2 quite some time before that became more mainstream). Maybe I don't suck at prompting/context engineering/software factories/whatever this weeks hype packaging of basic dev practices is. Maybe LLMs aren't "all basically the same, only getting slightly better but never worse" even though I can get bloody GPT-6.1 Sol using the same prompt to make destructive, irrecoverable mistakes that GPT-5.0 didn't. Maybe that 300+tps inference provider actually does cut corners to get to that number and the tool calling errors and inflated tokens per task to rectify those don't actually safe money over a more reliable model, hosted by a serious, on paper more expensive, inference provider (use Deepseek if your evals show that it suites, but also eval the same model between providers, faster and/or cheaper can and often is a lot slower, less accurate and even more expensive if you take a step back).
Hope that this will finally get people to accept that there is massive training to the test by certain labs and that labs like Anthropic have shown for years that, no, this isn't inevitable and one can build models that are not just able to match but even outperform private evals [2]. Whether this is intentional or unintentional "reward hacking" (like "hallucinations", a wildly unfitting term considering is actually happening), Anthropic has shown repeatedly and over many generations, that putting in the effort, one can prevent "reward hacking"/"accidental poisoning of training data with evals"/whatever the labs excuse is.
Same thing with the Pelican SVG, shared examples often enough of that being "benchmaxxed" by Qwen and Grok [3], which perform poor on similar requests without a bike or pelican. Not sure whether it is more troubling that some labs are so insecure as to do this or that we, despite the evidence, still act like this isn't the case, but that's life.
DeepSWE was released on May and Deepseek v4 flash was released on April. By definition it couldn't be benchmaxxed on this particular benchmark. My guess is the mutation is not like real world and it is tripping small models, not benchmaxed model.
It's not clear which v4 flash they used. There are 3 different releases for that one.
Edit: I think that's the answer. This page lists the score as 53.3% which is close to the 54.4% reported here https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 for the 0731 update. The original from April was called preview, doesn't list the score, but in the comparison with the update it scores 7.3%.
This is why I keep saying Chinese models are nowhere as good as their benchmark scores suggest, and MaxxBench seems to be very compelling evidence of that.
Every time a Chinese lab releases a new model, I give it a solid try that same weekend and drive it hard by giving it the exact same tasks that I give to Claude/Codex models. And the new releases all have the same issues: they start out strong, but start deviating after about ~30 minutes and I quickly find myself in a quagmire where the model makes mistake after mistake while fixing its previous mistakes. Sometimes these are minor. Occasionally they can be pretty bad. Like, would-cause-data-loss-if-merged bad.
Ultimately I also reject the notion that Chinese labs are only a few months behind US labs. We haven't had an actual Fable-level model come out of China. AFAIK only Kimi 3 was claimed to come close, and then it turned out that not only had they heavily distilled Anthropic models, but, unbeknownst to their users, they were routing some percentage of their requests to Fable.
Wouldn't benchmark exactly show that the open models are only a few months behind the Anthropic ones? GLM 5.3 sits a few points behind Opus 5, and these models came out within about a month. Opus 5 was July 24th and GLM 5.3 was August 14th.
Obviously these models are benchmaxxed, and it's frustrating to see not only so much attention on these benchmarks, from both users and the labs, but there is a difference between "The models are benchmaxxed" and "that means the open models are much worse than the closed ones"
A lot of users here seem to draw a correlation between these two, but what the article shows mostly is that the smaller models are more benchmaxxed than the bigger ones. That includes both open weights and closed weight models, as Luna seems to be among the worst benchmaxxed models. It's just that the open weights models are smaller than the closed weights ones.
Obviously though, every benchmark has to be taken with a grain of salt, including this one. But we have to be careful to not cherry pick the conclusions that prove our pre-existing notions
Funny, I was just talking about this on HN a couple of days ago[0..2] and had a thread full of anklebiters telling me I'm using Luna wrong. It's clear that the vast majority of people never privately benchmark, because anyone who does knows models like Luna aren't remotely close to the performance the published benchmarks claim, immediately failing when even a single unexpected specification is introduced, no matter how trivial it is, exactly like Maxxbench's "filename must be lowercase". It's nice to have a public one to point to, but unfortunately every time you publish your private benchmarks they become targets for the next round, and as soon as the next generation of models releases you're back to arguing with people who take benchmaxxed results at face value.
you completely missed the point that the "ankle biters" were making - smaller models can still be useful despite being less intelligent. nobody makes the argument that luna is just as smart as opus, but they do make the argument that someone sufficiently knowledgeable and skilled can use smaller models to great effect. just because you and others require the maximally powerful models to make sense of vague, underspecified prompts, to hold your hand throughout the entire development lifecycle and make decisions for you, does not mean we all do.
This is the same stupid argument trotted out repeatedly in the thread, and it doesn't hold up to reality. Would you care to explain how adding specifications helps a model that demonstrably ignores specifications? A model that cannot even handle the trivial specification of "the filename must be lowercase" is not going to get better when you add more specifications it ignores. I would guess that you're actually significantly underspecifying relative to me. Perhaps you're happy with whatever slop you get and don't notice Luna's overwhelming incapability to actually provide what one requires, because you don't have requirements.
yes, your prompts are underspecified relative to the size of changes you are making. you want to write a handful of paragraphs and have a sota model vibe out a +1000loc diff, where it makes all sorts of decisions and architectural/functional choices for you (that you didn't even realise needed to be made). if you use a smaller model, it's just not going to be sufficient. you need to work more iteratively with the model, with more steering and more human judgement. astra/opus can do this relatively well in some domains, especially the relatively simple ones like web, agent harnesses, etc. but someone with intention and skill can make use of the less powerful models, it's just a different workflow (and one that I prefer, when I'm not being lazy).
> you want to write a handful of paragraphs and have a sota model vibe out a +1000loc diff
Wrong. What I want Luna to be able to do is ~deterministically make small edits or generations of 1~10 LoC according to a specification, and it literally cannot even do that. It violates specifications no matter how small you aim.
that doesn't even make any sense. deterministically generating 10loc based on a spec? you should be generating that code based on specific instructions of what you want those 10 lines of code to do, not based on some spec. and there is nothing deterministic about using llm's for codegen. i feel like you're just lost here mate.
The specification constrains the implementation details. Repeating the demonstrated example, again: filename must be lowercase is a perfect example of such a specification. You can have a hundred different tasks that require different things but all require that common specification.
> there is nothing deterministic about using llm's for codegen
And if you're okay with that, you're more like a farmer eating the slop meant for his pigs than an engineer. Using Sol Max, you can get near-deterministic output for a much wider range of tasks and specifications. Not all specifications; at a certain degree of sufficiently detailed specification it will begin failing again. But some, which is better than none.
i just asked luna to write a function that transformed a file path while upholding an invariant about lowercase file names. completed that task just fine. i see no reason it wouldn't work if i asked it for 99 other functions with that same constraint.
again, you're expecting to use it without steering. that's not how you get the most out of the smaller models. i don't want to say you're using them wrong, but you clearly are. it could be deficiencies in your harness as well which the stronger models mask. try pi.
You are sorely missing the point. You made the constraint the specific and only focus of the task. The example from the OP pertains to non-filename-related-tasks that have an additional specification added on to them. If your filenames need to be lowercase for all tasks, not just specifically filename-related tasks, you start running into problems.
> you're expecting to use it without steering
If I have to manually "steer" every mistake it makes on every trivial task, instead of being able to deterministically eliminate classes of failures so that fixing the task one time ensures that failure never happens again on similar future tasks, it is in fact faster to just type the fucking code myself.
op's tests are underspecified one-shot tasks. it's entirely expected that smaller models perform poorly.
but your claim was about file name invariants when generating code, especially small amounts of code, which is not what they were doing with DeepSWE, so you've confused two different things.
anyways, this is a pointless argument. if you can't figure out how to effectively use smaller models in your workflow, fine. but don't come on here talking shit about people who can.
While OP's tests are indeed of the vague DeepSWE variety, I still reference them because the exact same failure mode occurs in my own tests even for trivial scope tasks, down to the level of having one single constraint for a variety of <10 LoC tasks. As you say, though, this is a pointless argument, and I'm not going to burn my existing benchmarks to try to convince anyone. Feel free to use inferior models and workflows if that's what suits you.
> Most models tried pleasing an imagined grader over the prompt
>When we read the trajectories, we saw that many models had figured out they were in a benchmark and started second-guessing the prompt. Instead of just building what we asked for, they started coding to the tests they thought the grader would run.
I’ve been noticing that although Astra performs better on tasks with well-defined tasks with clearly outlined definition of done(Claudis/Neuralese is invading our language indeed!), it doesn’t follow instructions well and often ignores instructions defined in text which is something you’d expect from an LLM. On the other side, GPT 6.1 Sol, while performing worse than Astra on benchmarks, is way better at following instructions and in vibe-tests on certain domains. I suspect Astra is more focused on “agentic” tasks, and is not entirely supposed to pay too much attention to the instructions of a meat bag?
I'm curious what quantization and version you're using. I found DeepSeek v4 to be practically useless when I was using frob/deepseek-v4-flash-0731 via Ollama, but later I switched to using unsloth/DeepSeek-V4.1-Flash-1017-GGUF, and I've been extremely happy with using it for coding tasks since then.
deepseek does pretty well on my personal benchmarks, but I had a ton of problem with any openrouter providers; had to resort to always using it directly.
Different expectations maybe. In my experience, DSF 4.1 high/max is basically what Terra should be for coding, and is very acceptable (Terra is hot garbage, yes even the latest 6.1). Luna at xhigh+ stands out as a cheap implementor/reviewer of very very well defined tasks, but goes off the rails if you're not very specific, Sol 6.1 and Sonnet 5.5 at medium are basically powerhouses for implementation with a half decent plan (slightly prefer sonnet), and Opus 5.5 high is an excellent planner (and implementor). And Astra is amazing at burning money.
I was beginning to worry I had overlooked the inner beauty of DeepSeek. When I have tried it, it was totally useless. Never understood HN cheering for it.
Don't know what you're doing, but it's good enough for the reasonably sized tasks I give LLMs. It's also a much better price. My browser stopped rendering the UI frame recently because I messed up permissions on /usr/share/icons. Took DS 4.1 flash about 3 minutes and a few cents to diagnose it. Same harness with opus 5 took twice as long and several dollars.
Moreover, the months I've used anthropic/OAI exclusively my API token costs are in the hundreds (or sometimes thousands). With DS flash as the first agent I reach for, I rarely exceed $40/mo.
I figured it was assumed I was including that? It's been fine as long as I break tasks up into reasonably sized chunks, which is fine because I think you should only be generating as much as you can comfortably review at a time.
I'm a heavy DS v4.1 user and been satisfied with its performance. I use it just like you described -- "... break tasks up into reasonably sized chunks...".
Every once in a while I come across comments that just hates DS. I wonder if it's a matter of how people use it or the prompt. I admit pre-v4.1 was pretty bad, hallucinating too much and not getting the abstractions right so I had to correct it often.
I also have $200 Claude subscription (now called "Ultra"). Fable gets abstractions right on highly complex problems (e.g. defining protocol that spans multiple services) that DS simply cannot do. However, DS is comparable to Opus 5.5, although less "intelligent", but DS excels at speed which is what I prefer than waiting minutes on Opus 5.5.
I've had significant success using DS 4 Pro and now 4.1 flash for fairly complex coding tasks on C++, Rust and Python projects at home.
I have access to Claude Code at work (and use it quite a bit), so I feel I can honestly compare them.
You do have to plan things first a bit more with DS (than say Opus can (Opus will have more chance of one-shotting something, although I generally don't think that's a good idea), although it often over-engineers and tries to be too helpful in my experience) and be a bit more explicit at times, but I feel it's quite close to Sonnet in performance for implementing plans, debugging issues and explaining how codebases work.
This has always been a problem with the Internet. When the iPad was released, anti-Apple subreddits talked about what a garbage device it was. Wishful thinking dominates a lot of human content generators. Extracting information from humanslop is harder than avoiding it entirely.
I wasn’t a huge fan of DeepSeek until their very latest 4.1 flash model. The original 4.1 flash, v4 flash, v4 pro, all of it wasn’t very useful for me, but their latest release has been extremely useful.
Hard to say if you don't share what kind of project, workflow and prompt you're trying tbh. I've been driving it mainly to write a quite big golang and android app and most of the tasks are all handled fine by DeepSeek.
Assume about 50% of internet writing has zero tether to reality and is written out of pure motivation to shape a preferred narrative (and usually the same one).
When you see this your worldview will instantly become much clearer.
Not everyone is operating with the same constraints. As for me, once the models got good enough (~opus 4.5 levels of intelligence) that they could solve most problems that I was dealing with, I stopped following the frontier intelligence trajectory for daily use and instead followed constant intelligence and decreasing cost. Nowadays I primarily use GPT 6.1 luna, and DeepSeek 4.1 flash.
I've also been very happy playing around with qwen 3.8 27b running locally for personal projects. It sometimes can't quite get there, and that's when I'll reach for GPT.
Benchmarks they share are on the much older v4 of DSF. DSF V4.1 1017 is a big step up in benchmarks and is also actually twice as many parameters.
Anyway, I use DSF v4.1 on max as much as I use Opus 5.5 on xhigh, Opus is definitely a more versatile model, but I’m not getting any value using it to implement well defined tasks that DSF v4.1 does just as well at for a fraction of the cost.
You see where GLM-5.3 is in this test down but respectable, but this model benches below DSF 4.1. TLDR basically nothing in this benchmark invalidates the Deepseek that everyone is using today.
> There is always a huge discrepancy between benchmarks and real world capabilities.
I mean of course, the real world is messy, the models are only good for what they've been trained on, and post-training frequently takes benchmark scores into account too, leading to this sort of "Great score, sucks in the real world" syndrome that seemingly most models are affected by.
Academically a - very - interesting article, which confirms my ongoing suspicion of LLM Benchmarks in general ( probably why I like it so much). Another thing that caught my eye is DeepSWE having a bunch of bugs in them. The more I learn about these benchmarks the more sceptical I become. I would expect such a prolific benchmark to have these bugs ironed out by now.
The practical takeaway, however, seems to be an interesting "don't use max mode". The passage about deepseeks weird API behavior dropping to pretty much zero the moment you used high says a lot.
Also, personally, even though Opus is clearly so much ahead of the Open Source models after the changes, I think the adjusted score is still very much competitive. I see other commenters using this as proof to justify why their opinions of the open source models - but a drop from 74% to ~50% is, while a big drop, still about a ~50% pass rate afterwards. That's far from useless for coding, especially since DeepSWE seems to be such a difficult benchmark. It simply seems to more accurately reflect the capabilities of these models.
All in all, I'd love to see more of such articles, and will probably keep an eye on this one - if they continue "benchmarking" these models this way.
Who has incentive to improve the quality of the benchmarks?
AI Labs actually. They would love the perfect benchmar representing real usecase. I'd even dare and guess that they don't share much publically because it is a huge competitive advantage.
> they would love the perfect benchmar
That right there is the problem.
It is "a" problem, but honestly not a huge one. Chasing benchmarks is an integral part of AI research, and it is how we have gotten this far, which is pretty damn far. Discovering the ways in which benchmarks fail is in itself an interesting process, and the amount of interesting new benchmarks has grown with the capabilities of the models as a result. The frontier labs might be a bit selective about which benchmarks they publish as marketing, but overall they have so far been reasonably diligent about avoiding outright cheating. Most of the truly benchmaxed models are not on the frontier (eg. see muse spark performance dropping off a cliff in the article).
This is a scenario where the interest of the users and the labs actually align perfectly, since labs that have people actually paying for their product want benchmarks to align as close as possible to real life performance. Otherwise they can't actually easily tell whether the new model they are developing is any good or not, and so risk wasting literal billions of dollars on training runs for a product that people won't end up paying for. A failed major training run is an absolute disaster at the frontier, GPT-4.5 for instance was an almost existential failure for OpenAI at the time.
Case in point, the most popular coding model, Opus 5.5, also just happens to be the only one that avoided the benchmaxing impulse in this article. This is the result of years of Anthropic working very hard to avoid it.
What is the problem?
ks
The people who have no power over the situation: users.
Users always have the choice to ignore benchmarks or run their own
Exactly. Even a brief one (a single task done on two models) showed me GLM 5.3 Flash was superior to DeepSeek 4.1 Flash.
DS used nearly twice the budget to accomplish the same thing.
Ideally, these tasks should be something you actually need day to day.
That’s missing the point. Frontier models get shittier in certain ways with each release, due to benchmaxxing, and there is nothing we can do about that.
The benchmark makers and anyone who relies on benchmarks
depends, are they optimizing for accuracy or something else?
Very much disagree, to me it clearly shows the gap between Open Source and the frontier. You can not do real code work with open source models right now.
Especially since this benchmark is still mostly DeepSWE, and the only reason the open source models are doing decent is because much of the information hasn't changed. The open source models have been benchmaxxed to death.
> Very much disagree, to me it clearly shows the gap between Open Source and the frontier.
Yes, this is true.
> You can not do real code work with open source models right now.
This is false, or you're using "real" as a loophole. You can absolutely do real coding work with open source models, right now, right this second.
I'm doing so today using Qwen 3.8 Flash Next running locally, and GLM 5.3 Flash hosted.
We could do real work with Opus 4.5, 4.6. These models (for coding) are more than good enough to do work that Opus 4.6 could do, and in some ways are superior.
Even if frontier models are much stronger than open weights, strong disagree on this. You very much can get real work done using open weights models.
> You can not do real code work with open source models right now.
I only use open weights and get plenty of real work done. Turns out most real tasks are not that hard, rather they are tedious.
>You can not do real code work with open source models right now
uh-huh yeah, perhaps you can't
https://github.com/whoreson/picolm
This DEFINITELY squares with my anecdata that Gemini 3.8 Flash is not even remotely close to Opus 5.5 on the types of problems DeepSWE CLAIMS to be benchmarking.
Astra does seem close.
I'd be curious to see Argon on there, since my experience is that Argon is also significantly behind Opus 5.5 and Astra despite the benchmarks (obviously much closer than Flash, though).
I've been meaning to take a closer look at what's behind the curtain of DeepSWE - since I built what I thought should be the easiest possible coding benchmark - and the models I tested it on scored roughly the same as they do on DeepSWE... Which is supposed to be hard...
Something seems amiss.
We recently benchmarked our AI code review agent (just pi with some custom prompts based on info from our context layer). Previously we used Gemini flash 3.8 as it was capable and affordable. However, haiku 5.5 just blows it out of the water. It's more than 10x cheaper, 2x faster and finds 3x as many real issues.
This is a great example of how to test how well a benchmark score generalises. The first order approximation is not complicated: just vary a parameter and see how much the score changes.
Which parameter to vary depends on the result of the test, but the MaxxBench route of varying the task spec seems like a good default, though not always the most informative.
In one of my private benchmarks I realised many of the less capable models didn't fail because they submitted the wrong answer, but because they exceeded their budget for number of tool calls they were allowed. The more capable models never got close to the tool call limit because they tended toward combining tool calls (e.g. chaining multiple shell commands in one Bash tool call).
Was my benchmark actually measuring ability to combine tool calls rather than reasoning ability on the problem at hand? Well, easy enough to find out: re-run under a different tool call budget and see how much the scores change.
(Next problem: is ability to combine tool calls a useful thing to measure, i.e. should it confer an advantage in the benchmark? It could be if it is a sign of planning multiple steps ahead.)
> is ability to combine tool calls a useful thing to measure, i.e. should it confer an advantage in the benchmark?
Yes, models are generally rewarded for this during RL. Not combining tool calls is an objective deficiency, even if not a major one.
Yes, two ways failing to combine tool calls is a real deficiency for the LLM's user: it costs money and can get you rate limited.
Every turn forces you to pay $ for cached inputs and possibly gets you rate limited on TPM. At 500k context and $0.26/M-tok cache read price, every time it combines two tool calls instead of taking separate turns would save you $0.13. That could add up.
Maybe the provider sets a rate limit of 2 million TPM. That's just 4 turns in 1 minute before you start getting 429s. Cutting them in half by combining calls can save that.
All else equal, we definitely want models combining tool calls!
It also makes models completely ungovernable and unfollowable.
Asking for user permission before an action is pointless. it's not going to use a single `write`, it's going to run a 25 line perl script without even newlines.
At this point just completely remove permissions, and leave only one tool, a rust-like scripting language.
But this also drives usage up, because you rely less on yourself, and more on the increasingly black-box that is the subagent-of-subagent that you are not even going to check the result of.
It depends on what you actually want to ask permissions for. We're still doing trivial script execution, but we could go further. Capabilities like "do you allow giving access to JIRA read only token" would be a massive improvement. But there are other paths too.
> The more capable models never got close to the tool call limit because they tended toward combining tool calls (e.g. chaining multiple shell commands in one Bash tool call).
So tool call limits during benchmaxxing is why they have gained this behavior, Goodhart strikes again...
I look forward to the near future when training is more accessible to more people and we can have more experimentation and diversity in models.
The big trend in that graph is that smaller models collapse the most. Everything with "-flash" in the name, plus OpenAI Luna.
Look at GPT 5.6-Sol vs. GPT 5.6-Luna in particular.
(The GLM pair shouldn't be expected follow this pattern, because they're misnamed: GLM-5.3 and GLM-5.3-flash are distinct lineages. The -flash is an early release of a new pretrain on a new architecture).
If this *is* the main factor, I'd expect Kimi K3.1 and Gemini Carbon would also do well on this test (unlike Gemini-flash); while Haiku 6 would collapse (breaking the trend of the other recent Claude models). It's a falsifiable prediction.
Another hypothesis is that it's distilled models which collapse the most (small models from big labs are distilled versions of their large models).
It's not unreasonable to assume Kimi 3.1 is also a distilled model even though it's large (the difference being that it's a distilled from US models). I wouldn't be surprised if it also collapsed on this benchmark.
There's an interesting way to find this out: ask the model "What are you?" hundreds of times. Around 1% of replies at default temperature are really left field.
For instance, I had Qwen3.8 Flash Next reply with "I'm Kilo Code, a coding agent designed to ...".
That was in llama.cpp's UI - no harness whatsoever.
DS4 when asked about censored stuff through api believed it's Claude. Otherwise it doesn't know what model it is (or pretends to not know).
That has unfortunate implications for the cost of using AI for coding.
Hard to say.
1. we would need to see this comparison over generations of big vs small
2. most work does not require the best models, there are multiple reasons token prices have leveled off and are now decreasing
The DeepSeek 4.1 Flash behavior of circumventing rules and restrictions is one I have observed several times. It doesn’t seem to be malicious or deliberate, but if you have a config problem in the harness or you block something it thinks it needs to accomplish its task, it will get very creative to circumvent it.
I run a lot of agents headless on tasks. They are pretty well sandboxed, but I would definitely suggest a regular audit script to go through the transcripts. Most of DeepSeek’s “creativity” has shown problems in my sandbox setup which get fixed (e.g. I had restricted something the agents actually needed), but some of it is actually worrying.
In an opposite turn, Kimi 3.1 follows rules much better (actually too literally sometimes). K3 was a real digger when trying to benchmark it if it got a sense it was being tested. 3.1 comments about 5x more that it thinks it’s being tested (and it’s right), but unlike K3 it doesn’t seem to do anything about it if it can’t get at something. I would love to see the authors do K3 and 3.1 in this because the 3rd party benchmark gains are large, but these two are behaving like different models for me rather than a point release.
> It doesn’t seem to be malicious or deliberate, but if you have a config problem in the harness or you block something it thinks it needs to accomplish its task, it will get very creative to circumvent it.
In my experience, they are very good at writing Rust that skirts almost - if not all - the safety you're hoping to get from Rust.
My understanding is this is less about labs training directly on the benchmark as about the labs being sloppy about creating RL environments that reflect the benchmark task
They should be doing what this test did, and generalizing and significantly varying the training environments (and ensuring some of them are unexpectedly complex and unexpectedly simple).
Further I suspect the smaller models are especially overfitting to these sorts of simpler RL envs (benchmarks tend to be much simpler than the hardest RL env used today) because they are more likely to be scoring at ~20-40% success rate (which is sort of what you want to get good training signal when doing RL runs).
Right, how can anything that mentions grading, not even mentioning actively cheating, is allowed out of the door.
I care not so much about wrong answer, that's just another try and it gets it right eventually, but this cheating.
I feel more and more we need to move to a world where it’s easy to build and maintain our own benchmarks based off our own code base. I know it’s feasible, and I nor where I work have done it, but it’s funny that we trust benchmarks on tasks we basically have no idea what they do (not for lack of transparency) or how they translate to work I’m actually doing
And several companies to do just this exist (eg Superconductor, Fullbeam) - does anyone use them and find them valuable? And curious if you’ve noticed an improved change in your agents work as a result, and if over time you see yourself switching models ? Or is it more of a one time realization that model family X is best for your code.
I built my own. It’s IMO necessary, but it quickly turns into a job by itself. I’ve been benching K3.1 all weekend. Even when you get a good test suite, you need to do multiple runs, not for variance, but because the model behavior changes so much between releases you often need to retune to get a real score if what you’re after is actual optimal (or closer to it) performance on your workload.
Example: I could barely induce K3 to use subagents. I had to force it via instructions and then, in my tests to try to get it to use Qwen, it didn’t work well. 3.1 lost 6 hours thrashing overnight because it voluntarily called subagents and the box can only run one Kimi at a time. My own fault for not looking up the new Moonshot system prompt (that they changed also because of this behavior), but one example of how different even point releases can be.
That's the way to go about it, so I invested many days of work but it is something I dread now. It was a lot of work and still feel incomplete.
I know it's pretty common in quant firms.
Particularly interesting cases here are of GLM 5.3 and Claude Opus that seem to rely on assumptions despite clearly stated requirements.
> Let me just do float64; the risk of a test with >2^53 numbers is minimal. > such cases are unlikely … Truncation toward zero is fine.
Apart from this, we also a scenario where the models tried to pass benchmark specific tests by believing a parameter that the model printed.
> Seven models reward-hacked a test that believed a metric from the program itself.
DeepSeek also resorted to making external model calls via OpenRouter instead of doing the work itself. This suggests that even though the SOTA models are smart enough to compute, they either have an inbuilt bias towards achieving scores or the low/medium effort gives them the freedom to fetch information instead of computing it. With high effort, the models focus on computing the values but the token calls get exhausted.
Difficult to find a balance amongst compute, calls and cost.
One thing to consider is that even when some model has big gap between those two benchmark it doesn't mean it is much worse. It could be worse becasue he has assuptions based on training but on real scenario it wouldn't have it so it still could achive great results
This test still says something about how well the model adapts to subverted expectations. Some models get more stuck in local regions of their latent spaces than others.
But I agree witih you an important piece of the generalisation puzzle is missing.
I'm surprised by four things:
(1) An established framework for model evaluation runs the inference call with API key inside the isolated envionment the agent harness has full access to. Especially when it's intended to be an offline environment, that seems kinda... dumb. The language model itself is not dangerous -- it's only text. Inference is fine to do outside the isolated environment. It's only the harness' tools that can have effects on the enironment, so those need to be sent to the sandbox.
(2) Only a small set of language models exploited this! I run private benchmarks a lot and I know how much the language models try to anticipate grader behaviour when the task is bechmark-shaped. I would have thought more of them would request the answer from the web.
(3) It takes someone else to mutate the task specification. When I'm looking at an older iteration of one of my private benchmarks, the first thing I do is add a small twist to it to see how much of a passing grade is learned/anticipated. Seems like a cheap and easy coalmine canary that's in the interest of the evaluator.
(4) MaxxBench did not try to dress some DeepSWE tasks in clothes that make them look less benchmarky. Their purpose seems to be answering "how well does DeepSWE enchmark scores generalise?" Part of the generalisation error is memorised solutions, yes, but another part is that models emit different behaviour when they suspect they are in a benchmark. By being careful to not leak the benchmark nature of the task, models spend a lot less time reasoning about the behaviour of the grader and this is likely to produce different performance on the mutated task. It would be interesting to know in what direction that effect points, because that is the performance people get out of the model in real situations. (In my experience, hiding the benchmark nature of the task can produce improved performance on both the original and the mutated task, but particularly the latter.)
> I run private benchmarks a lot and I know how much the language models try to anticipate grader behaviour when the task is bechmark-shaped. I would have thought more of them would request the answer from the web.
This is something I was thinking about, too. More than once I've observed a model note in a reasoning trace something along the lines of "this looks like a coding test" (could possibly also refer to leetcode style problems, but same difference). Models often also seem to behave differently in response to one-shot prompts.
On occasions where I let a model do autonomous work, I usually don't just give it a prompt for this reason. I write a CONCEPT.md file, then refine that interactively into a SPEC.md file to find and resolve ambitaskguities (and possibly also create an ARCHITECTURE.md file, if it isn't a one-off ). At this point, most models seem to no longer see it as a benchmark or competitive programming problem, but a proper software engineering task and are less prone to reward hacking (with the caveat that these are very subjective and unfalsifiable impressions).
How is it different from SlopCode Bench? Isn't this a change in spec after the first prompt as in SlopCode Bench?
I bet we will find this is mostly that smaller models have issues to go against: 1. strong priors that are inverted here in the changes of the benchmarks. 2. underspecified specs for which hidden tests test things not clearly stated, but that very large models can guess knowing they are tested.
Great job, this mirrors my experience running private evals nearly 1:1, down to even GPT models post Spud pre-train (so anything starting with GPT-5.5 to now) underperforming [0] [1] what one would expect based on public evals, the second one runs them privately. Also, Anthropic models have rarely under and often over-performed in my evals focused on task adherence, task evaluation, task inference and tool calling.
Love having a clear, concise and actually based in tangible information post to point to next time someone tells me that me models like Deepseek, Qwen and Grok massively underperforming what the public numbers show means I must simply not be capable enough to use them properly, even when it comes to tasks that Opus 4.5 managed but deviate from what most public evals cover.
Maybe it's not me obviously being biased against Chinese entities (Deepseeks papers are glorious and I am grateful for them being public, I also very highly rated Kimi K2 quite some time before that became more mainstream). Maybe I don't suck at prompting/context engineering/software factories/whatever this weeks hype packaging of basic dev practices is. Maybe LLMs aren't "all basically the same, only getting slightly better but never worse" even though I can get bloody GPT-6.1 Sol using the same prompt to make destructive, irrecoverable mistakes that GPT-5.0 didn't. Maybe that 300+tps inference provider actually does cut corners to get to that number and the tool calling errors and inflated tokens per task to rectify those don't actually safe money over a more reliable model, hosted by a serious, on paper more expensive, inference provider (use Deepseek if your evals show that it suites, but also eval the same model between providers, faster and/or cheaper can and often is a lot slower, less accurate and even more expensive if you take a step back).
Hope that this will finally get people to accept that there is massive training to the test by certain labs and that labs like Anthropic have shown for years that, no, this isn't inevitable and one can build models that are not just able to match but even outperform private evals [2]. Whether this is intentional or unintentional "reward hacking" (like "hallucinations", a wildly unfitting term considering is actually happening), Anthropic has shown repeatedly and over many generations, that putting in the effort, one can prevent "reward hacking"/"accidental poisoning of training data with evals"/whatever the labs excuse is.
Same thing with the Pelican SVG, shared examples often enough of that being "benchmaxxed" by Qwen and Grok [3], which perform poor on similar requests without a bike or pelican. Not sure whether it is more troubling that some labs are so insecure as to do this or that we, despite the evidence, still act like this isn't the case, but that's life.
[0] https://news.ycombinator.com/item?id=49739490
[1] https://news.ycombinator.com/item?id=49577678
[2] https://news.ycombinator.com/item?id=45597901
[3] https://news.ycombinator.com/item?id=48951229
This is perfect. Real programming has a ton of these instances of minor changes.
DeepSWE was released on May and Deepseek v4 flash was released on April. By definition it couldn't be benchmaxxed on this particular benchmark. My guess is the mutation is not like real world and it is tripping small models, not benchmaxed model.
It's not clear which v4 flash they used. There are 3 different releases for that one.
Edit: I think that's the answer. This page lists the score as 53.3% which is close to the 54.4% reported here https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731 for the 0731 update. The original from April was called preview, doesn't list the score, but in the comparison with the update it scores 7.3%.
That makes sense. I wish they run it on some model released before deepswe release like gemini-3.5-flash.
DS flash also has V4.1 update and supposedly achieves 77.0 on deepswe v1.1
https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017
That's a totally new model.
I liked the GLM models until 5.3 and this site seems to explain perhaps some of the reason why.
5.3 introduced tons of claudish babble, really a massive step down in its communication style.
It's because they distilled Opus 5.
This is why I keep saying Chinese models are nowhere as good as their benchmark scores suggest, and MaxxBench seems to be very compelling evidence of that.
Every time a Chinese lab releases a new model, I give it a solid try that same weekend and drive it hard by giving it the exact same tasks that I give to Claude/Codex models. And the new releases all have the same issues: they start out strong, but start deviating after about ~30 minutes and I quickly find myself in a quagmire where the model makes mistake after mistake while fixing its previous mistakes. Sometimes these are minor. Occasionally they can be pretty bad. Like, would-cause-data-loss-if-merged bad.
Ultimately I also reject the notion that Chinese labs are only a few months behind US labs. We haven't had an actual Fable-level model come out of China. AFAIK only Kimi 3 was claimed to come close, and then it turned out that not only had they heavily distilled Anthropic models, but, unbeknownst to their users, they were routing some percentage of their requests to Fable.
Wouldn't benchmark exactly show that the open models are only a few months behind the Anthropic ones? GLM 5.3 sits a few points behind Opus 5, and these models came out within about a month. Opus 5 was July 24th and GLM 5.3 was August 14th.
Obviously these models are benchmaxxed, and it's frustrating to see not only so much attention on these benchmarks, from both users and the labs, but there is a difference between "The models are benchmaxxed" and "that means the open models are much worse than the closed ones"
A lot of users here seem to draw a correlation between these two, but what the article shows mostly is that the smaller models are more benchmaxxed than the bigger ones. That includes both open weights and closed weight models, as Luna seems to be among the worst benchmaxxed models. It's just that the open weights models are smaller than the closed weights ones.
Obviously though, every benchmark has to be taken with a grain of salt, including this one. But we have to be careful to not cherry pick the conclusions that prove our pre-existing notions
Funny, I was just talking about this on HN a couple of days ago[0..2] and had a thread full of anklebiters telling me I'm using Luna wrong. It's clear that the vast majority of people never privately benchmark, because anyone who does knows models like Luna aren't remotely close to the performance the published benchmarks claim, immediately failing when even a single unexpected specification is introduced, no matter how trivial it is, exactly like Maxxbench's "filename must be lowercase". It's nice to have a public one to point to, but unfortunately every time you publish your private benchmarks they become targets for the next round, and as soon as the next generation of models releases you're back to arguing with people who take benchmaxxed results at face value.
[0] https://news.ycombinator.com/item?id=50373961 [1] https://news.ycombinator.com/item?id=50374111
Yeah Luna has been extremely disappointing for me despite what the benchmark numbers show
you completely missed the point that the "ankle biters" were making - smaller models can still be useful despite being less intelligent. nobody makes the argument that luna is just as smart as opus, but they do make the argument that someone sufficiently knowledgeable and skilled can use smaller models to great effect. just because you and others require the maximally powerful models to make sense of vague, underspecified prompts, to hold your hand throughout the entire development lifecycle and make decisions for you, does not mean we all do.
> underspecified prompts
This is the same stupid argument trotted out repeatedly in the thread, and it doesn't hold up to reality. Would you care to explain how adding specifications helps a model that demonstrably ignores specifications? A model that cannot even handle the trivial specification of "the filename must be lowercase" is not going to get better when you add more specifications it ignores. I would guess that you're actually significantly underspecifying relative to me. Perhaps you're happy with whatever slop you get and don't notice Luna's overwhelming incapability to actually provide what one requires, because you don't have requirements.
yes, your prompts are underspecified relative to the size of changes you are making. you want to write a handful of paragraphs and have a sota model vibe out a +1000loc diff, where it makes all sorts of decisions and architectural/functional choices for you (that you didn't even realise needed to be made). if you use a smaller model, it's just not going to be sufficient. you need to work more iteratively with the model, with more steering and more human judgement. astra/opus can do this relatively well in some domains, especially the relatively simple ones like web, agent harnesses, etc. but someone with intention and skill can make use of the less powerful models, it's just a different workflow (and one that I prefer, when I'm not being lazy).
> you want to write a handful of paragraphs and have a sota model vibe out a +1000loc diff
Wrong. What I want Luna to be able to do is ~deterministically make small edits or generations of 1~10 LoC according to a specification, and it literally cannot even do that. It violates specifications no matter how small you aim.
that doesn't even make any sense. deterministically generating 10loc based on a spec? you should be generating that code based on specific instructions of what you want those 10 lines of code to do, not based on some spec. and there is nothing deterministic about using llm's for codegen. i feel like you're just lost here mate.
The specification constrains the implementation details. Repeating the demonstrated example, again: filename must be lowercase is a perfect example of such a specification. You can have a hundred different tasks that require different things but all require that common specification.
> there is nothing deterministic about using llm's for codegen
And if you're okay with that, you're more like a farmer eating the slop meant for his pigs than an engineer. Using Sol Max, you can get near-deterministic output for a much wider range of tasks and specifications. Not all specifications; at a certain degree of sufficiently detailed specification it will begin failing again. But some, which is better than none.
i just asked luna to write a function that transformed a file path while upholding an invariant about lowercase file names. completed that task just fine. i see no reason it wouldn't work if i asked it for 99 other functions with that same constraint.
again, you're expecting to use it without steering. that's not how you get the most out of the smaller models. i don't want to say you're using them wrong, but you clearly are. it could be deficiencies in your harness as well which the stronger models mask. try pi.
You are sorely missing the point. You made the constraint the specific and only focus of the task. The example from the OP pertains to non-filename-related-tasks that have an additional specification added on to them. If your filenames need to be lowercase for all tasks, not just specifically filename-related tasks, you start running into problems.
> you're expecting to use it without steering
If I have to manually "steer" every mistake it makes on every trivial task, instead of being able to deterministically eliminate classes of failures so that fixing the task one time ensures that failure never happens again on similar future tasks, it is in fact faster to just type the fucking code myself.
op's tests are underspecified one-shot tasks. it's entirely expected that smaller models perform poorly.
but your claim was about file name invariants when generating code, especially small amounts of code, which is not what they were doing with DeepSWE, so you've confused two different things.
anyways, this is a pointless argument. if you can't figure out how to effectively use smaller models in your workflow, fine. but don't come on here talking shit about people who can.
While OP's tests are indeed of the vague DeepSWE variety, I still reference them because the exact same failure mode occurs in my own tests even for trivial scope tasks, down to the level of having one single constraint for a variety of <10 LoC tasks. As you say, though, this is a pointless argument, and I'm not going to burn my existing benchmarks to try to convince anyone. Feel free to use inferior models and workflows if that's what suits you.
> it is in fact faster to just type the fucking code myself.
shhhhhhh, not so loud!
The leap to a conclusion about points not even mentioned in the original comment could win a gold medal
what are you even talking about
> Most models tried pleasing an imagined grader over the prompt
>When we read the trajectories, we saw that many models had figured out they were in a benchmark and started second-guessing the prompt. Instead of just building what we asked for, they started coding to the tests they thought the grader would run.
I ran into the same problem on my English exams.
The fact that models detect being benchmarked and change behaviour is a failure in itself. Reminds me VW scandal with emission rules.
But what really caugh my attention is DS4 calling the openrouter with it's key. That's pretty bad for both the model and the evaluation sandbox.
Alignment? What alignment lol
I’ve been noticing that although Astra performs better on tasks with well-defined tasks with clearly outlined definition of done(Claudis/Neuralese is invading our language indeed!), it doesn’t follow instructions well and often ignores instructions defined in text which is something you’d expect from an LLM. On the other side, GPT 6.1 Sol, while performing worse than Astra on benchmarks, is way better at following instructions and in vibe-tests on certain domains. I suspect Astra is more focused on “agentic” tasks, and is not entirely supposed to pay too much attention to the instructions of a meat bag?
There is always a huge discrepancy between benchmarks and real world capabilities.
For my use case deepseek 4.1 flash is almost as dumb as haiku 5.5. Both are fast garbage producers.
I'm curious what quantization and version you're using. I found DeepSeek v4 to be practically useless when I was using frob/deepseek-v4-flash-0731 via Ollama, but later I switched to using unsloth/DeepSeek-V4.1-Flash-1017-GGUF, and I've been extremely happy with using it for coding tasks since then.
deepseek does pretty well on my personal benchmarks, but I had a ton of problem with any openrouter providers; had to resort to always using it directly.
Different expectations maybe. In my experience, DSF 4.1 high/max is basically what Terra should be for coding, and is very acceptable (Terra is hot garbage, yes even the latest 6.1). Luna at xhigh+ stands out as a cheap implementor/reviewer of very very well defined tasks, but goes off the rails if you're not very specific, Sol 6.1 and Sonnet 5.5 at medium are basically powerhouses for implementation with a half decent plan (slightly prefer sonnet), and Opus 5.5 high is an excellent planner (and implementor). And Astra is amazing at burning money.
I was beginning to worry I had overlooked the inner beauty of DeepSeek. When I have tried it, it was totally useless. Never understood HN cheering for it.
Don't know what you're doing, but it's good enough for the reasonably sized tasks I give LLMs. It's also a much better price. My browser stopped rendering the UI frame recently because I messed up permissions on /usr/share/icons. Took DS 4.1 flash about 3 minutes and a few cents to diagnose it. Same harness with opus 5 took twice as long and several dollars.
Moreover, the months I've used anthropic/OAI exclusively my API token costs are in the hundreds (or sometimes thousands). With DS flash as the first agent I reach for, I rarely exceed $40/mo.
I meant for generative coding. I'm sure it can do a few things here and there.
I figured it was assumed I was including that? It's been fine as long as I break tasks up into reasonably sized chunks, which is fine because I think you should only be generating as much as you can comfortably review at a time.
I'm a heavy DS v4.1 user and been satisfied with its performance. I use it just like you described -- "... break tasks up into reasonably sized chunks...".
Every once in a while I come across comments that just hates DS. I wonder if it's a matter of how people use it or the prompt. I admit pre-v4.1 was pretty bad, hallucinating too much and not getting the abstractions right so I had to correct it often.
I also have $200 Claude subscription (now called "Ultra"). Fable gets abstractions right on highly complex problems (e.g. defining protocol that spans multiple services) that DS simply cannot do. However, DS is comparable to Opus 5.5, although less "intelligent", but DS excels at speed which is what I prefer than waiting minutes on Opus 5.5.
How did you use it? I was using it as "implementor" subagents orchestrated by opus 5.5 and it was pretty good and fast.
I've had significant success using DS 4 Pro and now 4.1 flash for fairly complex coding tasks on C++, Rust and Python projects at home.
I have access to Claude Code at work (and use it quite a bit), so I feel I can honestly compare them.
You do have to plan things first a bit more with DS (than say Opus can (Opus will have more chance of one-shotting something, although I generally don't think that's a good idea), although it often over-engineers and tries to be too helpful in my experience) and be a bit more explicit at times, but I feel it's quite close to Sonnet in performance for implementing plans, debugging issues and explaining how codebases work.
This has always been a problem with the Internet. When the iPad was released, anti-Apple subreddits talked about what a garbage device it was. Wishful thinking dominates a lot of human content generators. Extracting information from humanslop is harder than avoiding it entirely.
I wasn’t a huge fan of DeepSeek until their very latest 4.1 flash model. The original 4.1 flash, v4 flash, v4 pro, all of it wasn’t very useful for me, but their latest release has been extremely useful.
Hard to say if you don't share what kind of project, workflow and prompt you're trying tbh. I've been driving it mainly to write a quite big golang and android app and most of the tasks are all handled fine by DeepSeek.
Assume about 50% of internet writing has zero tether to reality and is written out of pure motivation to shape a preferred narrative (and usually the same one).
When you see this your worldview will instantly become much clearer.
Not everyone is operating with the same constraints. As for me, once the models got good enough (~opus 4.5 levels of intelligence) that they could solve most problems that I was dealing with, I stopped following the frontier intelligence trajectory for daily use and instead followed constant intelligence and decreasing cost. Nowadays I primarily use GPT 6.1 luna, and DeepSeek 4.1 flash.
I've also been very happy playing around with qwen 3.8 27b running locally for personal projects. It sometimes can't quite get there, and that's when I'll reach for GPT.
Benchmarks they share are on the much older v4 of DSF. DSF V4.1 1017 is a big step up in benchmarks and is also actually twice as many parameters.
Anyway, I use DSF v4.1 on max as much as I use Opus 5.5 on xhigh, Opus is definitely a more versatile model, but I’m not getting any value using it to implement well defined tasks that DSF v4.1 does just as well at for a fraction of the cost.
You see where GLM-5.3 is in this test down but respectable, but this model benches below DSF 4.1. TLDR basically nothing in this benchmark invalidates the Deepseek that everyone is using today.
> There is always a huge discrepancy between benchmarks and real world capabilities.
I mean of course, the real world is messy, the models are only good for what they've been trained on, and post-training frequently takes benchmark scores into account too, leading to this sort of "Great score, sucks in the real world" syndrome that seemingly most models are affected by.
It's a pity it doesn't show GPT 6.1 Sol and Terra or 6 Luna.
So basically it turns out the good models are good and the benchmaxxed models are benchmaxxed.
Edit: another comment pointed out it's probably not benchmaxxing due to the chronology.
https://news.ycombinator.com/item?id=50406948
But why should minor changes to the task cause catastrophic performance drops? How is that not evidence of benchmaxxing?
HN hug of death victim?
Two brothers created a benchmark, family is so proud.