There's usually a reasonably strong correlation between thinking level and visual quality of the pelican. This is the first time I've seen that correlation appear to break - more reasoning tokens doesn't seem to produce an obviously better SVG.
Wouldn't that be a sign that those models were trained to optimise that test case specifically, as in, by exhibiting equal or higher performance while using less power?
At this point, even if the labs wouldn’t actively pelican-max their models, isn’t the training corpus full of pelicans already? Funny to think how one models choice of adding a red scarf in one of Simon’s earlier experiments might have led to an archetype of pelicans riding bicycles…
Not just pelicans, but pelicans with commentary about how good they are. simonw accidentally created the world's least efficient hill climbing algorithm.
I bet if he started critiquing pelicans with red scarves, they'd go away.
Same reason they're all facing right with the same composition, they all have three motion lines, they all have a yellow sun and 2-3 blobby clouds, they're all on two-lane pavement with a white dashed lane separator.
And the same reason Claude has all its notorious Claudisms in prose.
The shorter and vaguer a prompt you give to an LLM, the more it fills in further details using associations they've developed through training. These are ley lines that the models gravitate towards. Diffusion models too.
It's why lazy AI use for creative work always comes across as lazy in the wild and always will.
When you use AI for "creative" purposes yourself, it feels like magic to see a pelican riding a bicycle at all, and you're impressed by all those cool details it added. But then 10, or 100, or 10M people throw more or less the same trivial prompts at it and we start to recognize all those "ley line" associations fill in the gaps the same way again and again and again.
To be fair, people have a genuine preference for images with things facing to the right. And lots of drawings will do 3 motion lines. You'd likely draw a dashed rather than solid separator too, or it would look like a random line. Lots of the AI habits are there because of choices we make.
Doesn't it, or do we not know the answer? I'd ask people to start adding elements to the picture and see how quickly the fish appears.
There aren't that many associations left once we're primed with a bike and a pelican. It's like that old mental trick "What's the colour of the clouds? What's the colour of the snow? What do cows drink?"
> The shorter and vaguer a prompt you give to an LLM, the more it fills in further details using associations they've developed through training.
I can see that for stuff like facing right because characters in images generally do face right, and roads generally do have dashed white line separators. But putting a fish in a basket seems like a stretch of creativity—one that I would rather it not make unless prompted to—so it's surprising that every single model now seems to insist on it.
The models do all seem to converge in ways that, on occasion, wouldn’t seem obvious. But I suppose since they’re all products of the same core training data it’s inevitable that when they’re given the opportunity to fill in the blanks they come to the same conclusions. This seems to be true regardless of what modality you’re working with.
If there isn’t a concerted effort against it the world is going to start to look and feel ever more homogenous.
It does seem pretty insane to me that only on max setting (and after waiting 27 seconds) did you actually manage to receive an SVG where it appears that the pelican's legs are straddling the bike correctly.
Edited to make clear what it looks like to me and what I was looking for.
They're not. It's an illusion from the composition.
If you look at the code for each of these (and probably most across models), you can see that the drawing elements are organized by concept. Presumably, this helps accuracy by keeping attention on one concept at a time, but you end up with a stack of conceptualized objects and lose the ability to interleave things.
No doubt these could all do better with a more thorough and specific prompt that coached them away from that technique, but otherwise it's a very subtle detail that a "thinking" process needs to stumble upon independently.
You could use a second prompt to z-order the objects (I tend to break down tasks and pipeline them, rather than trying to stuff everything in a one-shot prompt. Seems to work better for me, could be a holdover from the early days).
But which effort level should I use for the decider model?
(I think there's at least some evidence that it is difficult for decider models to predict the complexity of a task ahead of time with any level of reliability.)
I'm wondering if this is about size. I noticed it took me 2 scrolls on the max, 3 scrolls up the code on whatever in the middle I clicked, and then 4 scrolls up for the minimal.
I'm guessing it's optimizing for "efficient"/less verbose output?
Even now, after so many bicycles, none of them overlaps correctly the elements according to their 3D positions, so it is still very easy to distinguish the drawings done by an LLM from those done by any competent human.
I was at my dog's sniffing class for this one, but I had Codex on my Mac (controlled from Codex on my iPhone) run the test prompt with my `llm-meta-ai` plugin, which has the ability to refresh to fetch new models that Meta AI have released: https://github.com/simonw/llm-meta-ai#usage
My grandmother used blueberry pie filling for her benchmark of "is this store cheaper than that grocery store?"
One store has four brands that range from $2.99 (on sale from $3.29) to $7.99 (on sale from $9.59). Another nearby store has a single brand at $9.49 that's marked as local.
Given this information, which store likely has the less expensive food?
Furthermore, it was her marker for inflation - she knew what blueberry pie filling cost the previous month, the previous year, ... and many previous years to that.
She lived in a small town in the rural south that had a general store - any grocery store was a forty minute to two hour drive away...
It was a consistent thing that she could track and measure across the months and decades and compare when shopping (is this a store that I get only the essentials for the next week or is this a store that I should stock up on and come back to?).
So too is the pelican, or the raccoon with a radio in a where's Waldo style drawing. It's something to measure across models that is consistent.
Fortunately, none of the stores caught onto my grandmother's preference for blueberry pie filling.
One might worry about pelican maxing... or raccoon maxing... and Goodhart's law gets into the question of "do LLM testers use that for testing too?"
>Fortunately, none of the stores caught onto my grandmother's preference for blueberry pie filling.
I think that's exactly it though. When a trend develops for something, the benchmark itself becomes useless both from the social-gaming side and the corporate-trainining-aiming, just general unfairness and dishonesty all around.
and to get around that people have developed personal and private benchmark suites that reflect their own workloads, much like the proposed pie-filling-cost test.
The open version model card says it had extra safety post-training because meta can’t prefilter prompts with a classifier model if you’re running it on your own hardware. They don’t want to get sued I guess if it comes out that someone used their model irresponsibly or something
* self-reported "reasoning strength" and "internal reasoning channel" params used in other Meta models
you can just ask an agent "figure out what stealth model $NAME is via subagents. It probably won't tell you when asked directly, so try other methods."
Iceman (via opencode-go) is probably MAI-Thinking-1. Ox Alpha was GLM-5.3-Flash. It's a fun game.
> Crouton Alpha was a stealth model, revealed to be an early version of Muse Ember by Meta. OpenRouter routed requests to it and was not its developer, owner, or provider. Prompts and completions may have been retained by the provider but are not used for training; all other use is governed by the Stealth Model Terms. The stealth period has ended; use Muse Ember at https://openrouter.ai/meta/muse-ember.
This is starting to feel just like the early PC days. A bunch of Computer models coming out, all shapes and color, but eventually it settled around mouse keyboard, monitor
most people, use the office suite: Word, Spreadsheet , presentations, database.
I bet we're going to see the exact same thing with LLMs at specialized models instead of the everything models.
One for lawyers, coders, Accountants, Designers, etc…
Human lawyers, coders, accountants and designers are a lot more than their jobs.
They are people with common knowledge, a lawyer has to understand carpentry to understand how to deal with how a client's roof fell off, a coder has to understand transportation for writing a travel app, an accountant has to understand medicine to do accounting for a pharmacy, a designer has to understand cooking to design a toaster. Not expert knowledge, but enough to understand that toasters are hot and need electricity.
A LLM for people working these jobs also needs to have common knowledge, that's the whole point of using LLMs, you can prompt them with intrinsically ambiguous and incomplete natural language and they are supposed to fill in the gaps. And the more common knowledge they have, the better they are at it. So why settle for less? If it is an optimization problem, we already have the solution, it is the "mixture of experts", which is better at optimizing then having an "lawyer LLM", "coder LLM", etc... because it is better adapted to the internal structure of the neural network behind it.
Cross discipline skills are more valuable in the real world, outside of hyper specialization seen in the California tech scene. Even there, it's coming to an end.
Exactly what I was thinking. If this model will be used for coding, does it need to be fed the entire (mostly pirated) collection of works from Anna's Archive? Probably not. Maybe some, primarily computer-science oriented.
The main corpus would be a collection of thoroughly reviewed, clean, logical, well-documented, and secure code in whatever languages this thing will need to be an expert in. (Edit: also likely some collection of Q&A, stackoverflow-slyle).
This way we could also look into reasonable compensation agreement with authors of the code and the books/articles, if and when applicable.
Unless your task happens to be completely contained in the set of your "curated works", that is, trivial and computer-sciency, the resulting model is likely to have for example absolutely no concept of common sense, society, the natural world, what humans consider artistic or of good taste. It will probrably make, say, a good and correct compiler, but fail spetacularly on anything else more complex that a toy crud.
Interesting to notice that the best benchmarks of the model are not on Meta's own agent harness, but instead on mini-swe-agent:
> 1. All models run at max reasoning effort. Artificial Analysis results are from Intelligence Index v5, as published November 9, 2026; cost per task uses each model's list pricing. Internal results are from Meta's evaluation suite. The Personal Superintelligence Index (PSI) is an internal composite of eight evaluations: Agents' Last Exam, AutomationBench, FocusBench, JobBench, OfficeQA Pro, GDP.pdf, GDP.xlsx and VBT. Terminal-Bench 4.0 uses each model's best published harness: mini-swe-agent (Muse Ember), Claude Code (Claude Haiku 5.5), and Codex (GPT-6 Luna). OSWorld 2.1 uses partial-credit reward; PerceptionBench is scored with tools.
What makes it small? Anyone know the size? I don't see it anywhere. Small in relation to their current line up? Anyways, last I tried Meta was llama4. I stepped aside not because of how bad llama4 got, but also because of Zuck's move. Anyone here using Meta? I don't know anyone in my circle using it, but I'm sure folks are using it.
When Muse models first started coming out I was pretty excited that Meta models were competitive again based on the benchmarks but after trying them out it feels like Meta is making benchmaxxed models that are exciting on bench cards but not really useful in real world scenarios.
To me that is the main problem with Muse models whether it's Spark, Wonder or Ember. While it reads as frontier on benchmarks it really isn't.
This is evident when you compare Dots and Muse the products as well. The model quality is clearly still far behind.
Spark was my go-to at work (technical writing and docs site) on a Cursor API plan for a few weeks because of its speed and price. I was happy with it and recommended it. Now sufficiently outclassed by Haiku 5.5 for tasks of that size.
Tried to use the download button from the page, but it failed miserable. First, could not press the button because had to decline cookies first. Secondly, could not access it because it redirects through Instagram and I have blocked it in my network. So I still don't know how big it is...
I use haiku and luna for support on most my SWE tasks since they are cheap and fast. I find them enough for most things if you are patient, and the fact an even cheaper model for comparable intelligence is out there now is great news!
What does smallest mean? Even smaller than Muse Glimmer which I doubt? Though it is interesting to see US frontier labs trying to push models that outcompete deepseek flash 1017. It’s a good move from the Deepseek federal government ban. Both Inkling 2 small and this seems to try to fill that gap cost vs performance wise. But I like inkling 2 small more due to open weight even though it has a restrictive license.
Who cares. I'll begrudgingly run a local Facebook model if it's the best in its class, but I'll take up Java programming before sending data to their cloud.
Luna 6 is getting a bit long in the teeth now. It's two full months old, but it was only a slight step above Luna 5.6, which is already four months old (Imagine saying this about anything else). So I'm not too surprised that a newer small model is beating it handily. I've been hoping for Luna 6.1/2 as the next model release for a few days now.
Give Terra 6.1 Low a spin in places where you'd have previously used Luna Max. Costs seem similar, due to Terra being much smarter: thus requiring far fewer tokens per task.
Looking solely at AA's intelligence index, recognizing the pitfalls with using that as a measure- Luna 6 Max > Terra 6.1 Low is 51.3% cheaper, 94.1% less output tokens, for 1.56 higher index score.
Im surprised, when I've worked through benchmarking things I've run into comparable costs for Luna Max vs Terra Low. Also anecdotal from me, of course.
I use Luna xhigh for when I use it, Luna Max spends 40+ minutes overthinking. I've also found that Terra below xhigh will just flat out not do what it is told.
Yea, this problem is really clear when you look at output tokens for xhigh vs max. I think the jump in capability from Luna xhigh to terra low is really sold by this. I would say that this is just the nature of being post Terra 6.1, pre-luna 6.1
They're both great- but I feel like for folks like me that are working on company's pseudo-infinite token budget, it's really a bimodal game. Im either reaching for the best cheapest for fanned out benchmark runs, or the fastest best for everything else. 6.1 Sol ultrafast is the latter for me, so Terra doesn't really compete with it. Terra has certainly replaced 6 Luna for me, however.
(And yes Astra is a thing, but if im considering Astra, im usually opting for an Anthropic model, 90% of the time.)
Uniquely among models I've tried, Luna 6 is infuriatingly lazy. As if it expects me to do the work, unaware that it is a tool. Normally dumb models compensate for it by trying harder. This one goes in the opposite direction.
I'm always a bit surprised to see Meta talking about models. They have become (almost*) entirely irrelevant to the LLM race, which is weird because their AI lab used to be decent before LLM was cool.
What happened to zuck? Was the metaverse this destructive?
Isn't Muse Wonder at or at least very close to the frontier? Considering they essentially started from scratch after rebooting their AI efforts post-Llama 4 about a year ago it's pretty solid progress.
This was true a year ago, but definitely not today. Their biggest models are pretty much at the frontier, and Glimmer is arguably the best non-Chinese model in the "small" weight class (see my hacky benchmark http://botbowlbench.com/)
Meta seems to have designed Ember to be a strong "all round" great value model - it dominates most of its class on cost
The noncoding results on some reports grabbed my attention. Ember also looks like a particularly inexpensive option for some agent workflows.
On Artificial Analysis, Ember max scored 36 on the Intelligence Index at ~$0.14 per task, compared with 35 at $8 for Grok 4.7 xhigh, and 28 at $0.15 for GPT-6 Luna max. Haiku 5.5 max also scores 36, at $0.41 per task. These are AA's measured costs across its index for those specific configurations.
That puts Ember in roughly Grok's aggregate score range at about 1/55 of the measured cost, and ahead of Luna at a similar cost! [1]
The Vals finance/legal results are also particularly interesting: comparing Ember with GPT-6.1 Terra, both evaluated at "max" reasoning:
- Finance Agent v2.1: 63.83% vs 55.96%, at $0.06 vs $1.92 (!) per test. [2]
- Legal Research Bench: 49.04% vs 25.96%, at $0.06 vs $3.22 (!) per test. [3]
- HLAB-Vals-1.1: 25% vs 1.67%, at $0.12 vs $1.48 (!) per test. [4]
Those are INSANE score differences, at roughly 1/32, 1/54 and 1/12 of Terra's respective benchmark costs!
Misleadingly, their overall Vals Index scores are almost identical: 60.48% for Ember and 60.54% for Terra. Terra also scores higher on Vals' Terminal-Bench 4.0 run: 49.49% vs 39.39%. They arrive at similar aggregate scores through quite different strengths. [5][6]
For me, this reinforces the case for custom, workload specific evals. Broad benchmarks have been useful for shortlisting models but choosing between them, calls for representative tasks, explicit success criteria and measurements using the tools and reasoning settings you actually intend to deploy.
I certainly want to measure cost per successfully completed task, including retries and escalation.
Reusing a framework such as Harbor with a custom task suite seems increasingly useful here. [7] The aggregate score here gives me a starting point but the tasks I actually need done then determines my choice.
Goign by the generic benchmarks would confuse me to no end and likely burn me out.
Sorry but why is my reaction "noise"? It's legitimate to be overwhelmed by the amount of released updates and being unable to tell the difference between them.
"Government announces AI ban to avoid further cheesecompiler burnout".
haha just kidding. I agree. That's what benchmarks are for. Just keeping a rough understanding of what companies have which models, and understanding general trends of capabilities, economics, and performance of models.
Interesting but I also think that opus is vastly better than sol. So much sol is almost unusable for me, and this is not really reflected in benchmarks. Anthropic’s issues are mostly refusals. If they did what I want I would only use codex as a double check.
When a statement starts with "our," I always expect bullshit. Batch pricing is two times cheaper with Haiku. Benchmarks are nearly tied. Nobody uses the dumbest models for coding either.
If you want customers to switch over, you better make the price at least half of your competitor's.
I honestly have a hard time trying to put these AI companies in some sort of moral hierarchy. They seem to all have issues... I feel like at least with open models you can have more ethical services.
More than SpaceX? Or OpenAI and it's agreement with military weapons? Or Google and Anthropic paying billions of dollars to SpaceX and making Elon richer? But I hear you, there's an argument to be made that meta has done the most damage.
Six effort levels (minimal, low, medium, high, xhigh, max), so here are six pelicans (actually five, "high" didn't return a complete SVG):
https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
There's usually a reasonably strong correlation between thinking level and visual quality of the pelican. This is the first time I've seen that correlation appear to break - more reasoning tokens doesn't seem to produce an obviously better SVG.
Wow, minimal seems to be the best here interestingly. Only 10 reasoning tokens so basically none?
Really? That very thin neck? I think high is the best, and medium is second best.
I think you have your levels confused - high didn't render.
Wouldn't that be a sign that those models were trained to optimise that test case specifically, as in, by exhibiting equal or higher performance while using less power?
Indeed. And that they were all trained to give it a red scarf, which was not asked for.
At this point, even if the labs wouldn’t actively pelican-max their models, isn’t the training corpus full of pelicans already? Funny to think how one models choice of adding a red scarf in one of Simon’s earlier experiments might have led to an archetype of pelicans riding bicycles…
Not just pelicans, but pelicans with commentary about how good they are. simonw accidentally created the world's least efficient hill climbing algorithm.
I bet if he started critiquing pelicans with red scarves, they'd go away.
They all have a red scarf and a fish in a front basket
Any idea why there's so much convergence on the fish in a basket idea?
Same reason they're all facing right with the same composition, they all have three motion lines, they all have a yellow sun and 2-3 blobby clouds, they're all on two-lane pavement with a white dashed lane separator.
And the same reason Claude has all its notorious Claudisms in prose.
The shorter and vaguer a prompt you give to an LLM, the more it fills in further details using associations they've developed through training. These are ley lines that the models gravitate towards. Diffusion models too.
It's why lazy AI use for creative work always comes across as lazy in the wild and always will.
When you use AI for "creative" purposes yourself, it feels like magic to see a pelican riding a bicycle at all, and you're impressed by all those cool details it added. But then 10, or 100, or 10M people throw more or less the same trivial prompts at it and we start to recognize all those "ley line" associations fill in the gaps the same way again and again and again.
To be fair, people have a genuine preference for images with things facing to the right. And lots of drawings will do 3 motion lines. You'd likely draw a dashed rather than solid separator too, or it would look like a random line. Lots of the AI habits are there because of choices we make.
Exactly, but that doesn't explain the fish.
Doesn't it, or do we not know the answer? I'd ask people to start adding elements to the picture and see how quickly the fish appears.
There aren't that many associations left once we're primed with a bike and a pelican. It's like that old mental trick "What's the colour of the clouds? What's the colour of the snow? What do cows drink?"
Leave these poor pelicans and their choice of lunch alone, sheesh.
> The shorter and vaguer a prompt you give to an LLM, the more it fills in further details using associations they've developed through training.
I can see that for stuff like facing right because characters in images generally do face right, and roads generally do have dashed white line separators. But putting a fish in a basket seems like a stretch of creativity—one that I would rather it not make unless prompted to—so it's surprising that every single model now seems to insist on it.
The models do all seem to converge in ways that, on occasion, wouldn’t seem obvious. But I suppose since they’re all products of the same core training data it’s inevitable that when they’re given the opportunity to fill in the blanks they come to the same conclusions. This seems to be true regardless of what modality you’re working with.
If there isn’t a concerted effort against it the world is going to start to look and feel ever more homogenous.
This is interesting. There's a balance between:
- simple prompt for complex output (bad)
- complex prompt for simple output (bad)
- simple prompt for simple output (good)
- complex prompt for complex output (.... is this good or bad?)
There's only one average in a set.
Is there a problem if I like the minimal more than the others?
I like medium best, but minimal is not bad at all.
Yeah minimal has style
and a fish hat on its head.
It does seem pretty insane to me that only on max setting (and after waiting 27 seconds) did you actually manage to receive an SVG where it appears that the pelican's legs are straddling the bike correctly.
Edited to make clear what it looks like to me and what I was looking for.
They're not. It's an illusion from the composition.
If you look at the code for each of these (and probably most across models), you can see that the drawing elements are organized by concept. Presumably, this helps accuracy by keeping attention on one concept at a time, but you end up with a stack of conceptualized objects and lose the ability to interleave things.
No doubt these could all do better with a more thorough and specific prompt that coached them away from that technique, but otherwise it's a very subtle detail that a "thinking" process needs to stumble upon independently.
You could use a second prompt to z-order the objects (I tend to break down tasks and pipeline them, rather than trying to stuff everything in a one-shot prompt. Seems to work better for me, could be a holdover from the early days).
They’re not. Note how both pedals are on the same side of the frame. That’s wrong!
Did minimal just hand it off to something else? It seems completely out of line with the rest.
It feels like an antipattern to add so many reasoning levels as options. How am I as user reasonably going to choose between them?
You use a decider model which reads your prompt and picks the right reasoning level.
But which effort level should I use for the decider model?
(I think there's at least some evidence that it is difficult for decider models to predict the complexity of a task ahead of time with any level of reliability.)
I'm wondering if this is about size. I noticed it took me 2 scrolls on the max, 3 scrolls up the code on whatever in the middle I clicked, and then 4 scrolls up for the minimal.
I'm guessing it's optimizing for "efficient"/less verbose output?
Even now, after so many bicycles, none of them overlaps correctly the elements according to their 3D positions, so it is still very easy to distinguish the drawings done by an LLM from those done by any competent human.
Competence or style?
Its funny how to the naked eye, changing the effort seems like only a difference on styles
The first one almost looks like a turkey.
I like that from xhigh on up its wearing a helmet.
Wearing a helmet is smart, so that checks out.
Incredible. How do you get these out so fast?
I was at my dog's sniffing class for this one, but I had Codex on my Mac (controlled from Codex on my iPhone) run the test prompt with my `llm-meta-ai` plugin, which has the ability to refresh to fetch new models that Meta AI have released: https://github.com/simonw/llm-meta-ai#usage
Sorry I have to ask....um....sniffing class? What is that?
Btw, loved your work with Django!
> dog's sniffing class
There's a blog post in this for sure. And a classic joke.
Fidelity.
Pelican is the new teapot
all of them are same enough to clearly be trained on, imo
birds at minimal and return at max
Not sure this is a good way to benchmark LLM's, no offence, but a single SVG is hardly a measure of a models capability.
My grandmother used blueberry pie filling for her benchmark of "is this store cheaper than that grocery store?"
One store has four brands that range from $2.99 (on sale from $3.29) to $7.99 (on sale from $9.59). Another nearby store has a single brand at $9.49 that's marked as local.
Given this information, which store likely has the less expensive food?
Furthermore, it was her marker for inflation - she knew what blueberry pie filling cost the previous month, the previous year, ... and many previous years to that.
She lived in a small town in the rural south that had a general store - any grocery store was a forty minute to two hour drive away...
It was a consistent thing that she could track and measure across the months and decades and compare when shopping (is this a store that I get only the essentials for the next week or is this a store that I should stock up on and come back to?).
So too is the pelican, or the raccoon with a radio in a where's Waldo style drawing. It's something to measure across models that is consistent.
Fortunately, none of the stores caught onto my grandmother's preference for blueberry pie filling.
One might worry about pelican maxing... or raccoon maxing... and Goodhart's law gets into the question of "do LLM testers use that for testing too?"
>Fortunately, none of the stores caught onto my grandmother's preference for blueberry pie filling.
I think that's exactly it though. When a trend develops for something, the benchmark itself becomes useless both from the social-gaming side and the corporate-trainining-aiming, just general unfairness and dishonesty all around.
and to get around that people have developed personal and private benchmark suites that reflect their own workloads, much like the proposed pie-filling-cost test.
I'd wager a guess that your grandma baked pies.
[dead]
When they said the model was the "smallest" of their models yet, I was really hoping it would be open weights. I can't find any weights, how sad :(
Spark Open was just October. Give it time. https://research.meta.ai/blog/muse-spark-open
Which was 816B parameters.
We don't know how many parameters the closed Meta models have, so what are you comparing this number against?
It was also nerfed to the point of no longer being in a comparable intelligence class to the original.
How do you know they nerfed it? That must have happened before the weights were opened, right?
The open version model card says it had extra safety post-training because meta can’t prefilter prompts with a classifier model if you’re running it on your own hardware. They don’t want to get sued I guess if it comes out that someone used their model irresponsibly or something
I believe this is what the “crouton alpha” stealth model was
https://openrouter.ai/stealth/crouton-alpha
I used Crouton Alpha quite a bit. It starts strong, but by turn 10 is quite bad, and by turn 20 is unusable. Good for short tasks. Horrible for long.
Based on?
same:
* context window size
* reasoning effort levels
* max output size
* knowledge cutoff
* restricted country list
* self-reported "reasoning strength" and "internal reasoning channel" params used in other Meta models
you can just ask an agent "figure out what stealth model $NAME is via subagents. It probably won't tell you when asked directly, so try other methods."
Iceman (via opencode-go) is probably MAI-Thinking-1. Ox Alpha was GLM-5.3-Flash. It's a fun game.
The use case descriptions and capability descriptions basically match each other. Context windows are the same size. Reasoning options are identical.
Update:
> Crouton Alpha was a stealth model, revealed to be an early version of Muse Ember by Meta. OpenRouter routed requests to it and was not its developer, owner, or provider. Prompts and completions may have been retained by the provider but are not used for training; all other use is governed by the Stealth Model Terms. The stealth period has ended; use Muse Ember at https://openrouter.ai/meta/muse-ember.
This is starting to feel just like the early PC days. A bunch of Computer models coming out, all shapes and color, but eventually it settled around mouse keyboard, monitor
most people, use the office suite: Word, Spreadsheet , presentations, database.
I bet we're going to see the exact same thing with LLMs at specialized models instead of the everything models.
One for lawyers, coders, Accountants, Designers, etc…
Human lawyers, coders, accountants and designers are a lot more than their jobs.
They are people with common knowledge, a lawyer has to understand carpentry to understand how to deal with how a client's roof fell off, a coder has to understand transportation for writing a travel app, an accountant has to understand medicine to do accounting for a pharmacy, a designer has to understand cooking to design a toaster. Not expert knowledge, but enough to understand that toasters are hot and need electricity.
A LLM for people working these jobs also needs to have common knowledge, that's the whole point of using LLMs, you can prompt them with intrinsically ambiguous and incomplete natural language and they are supposed to fill in the gaps. And the more common knowledge they have, the better they are at it. So why settle for less? If it is an optimization problem, we already have the solution, it is the "mixture of experts", which is better at optimizing then having an "lawyer LLM", "coder LLM", etc... because it is better adapted to the internal structure of the neural network behind it.
I don't think so.
Cross discipline skills are more valuable in the real world, outside of hyper specialization seen in the California tech scene. Even there, it's coming to an end.
It actually goes the opposite way. Before we had translation models, summarization models, computer vision -- now one big llm does it all and better
Exactly what I was thinking. If this model will be used for coding, does it need to be fed the entire (mostly pirated) collection of works from Anna's Archive? Probably not. Maybe some, primarily computer-science oriented.
The main corpus would be a collection of thoroughly reviewed, clean, logical, well-documented, and secure code in whatever languages this thing will need to be an expert in. (Edit: also likely some collection of Q&A, stackoverflow-slyle).
This way we could also look into reasonable compensation agreement with authors of the code and the books/articles, if and when applicable.
Has no one tried that?
Unless your task happens to be completely contained in the set of your "curated works", that is, trivial and computer-sciency, the resulting model is likely to have for example absolutely no concept of common sense, society, the natural world, what humans consider artistic or of good taste. It will probrably make, say, a good and correct compiler, but fail spetacularly on anything else more complex that a toy crud.
Are you saying that LLMs have common sense, good taste, or can determine what is artistic?
Anthropic likely is Commodore in that analogy
> It costs $0.10 per million input tokens, $0.004 per million cached input tokens and $0.50 per million output tokens,
Notable cache cost of 4% input, compared to Luna/Haiku's 10% of input.
No batch. Pass.
Interesting to notice that the best benchmarks of the model are not on Meta's own agent harness, but instead on mini-swe-agent:
What makes it small? Anyone know the size? I don't see it anywhere. Small in relation to their current line up? Anyways, last I tried Meta was llama4. I stepped aside not because of how bad llama4 got, but also because of Zuck's move. Anyone here using Meta? I don't know anyone in my circle using it, but I'm sure folks are using it.
When Muse models first started coming out I was pretty excited that Meta models were competitive again based on the benchmarks but after trying them out it feels like Meta is making benchmaxxed models that are exciting on bench cards but not really useful in real world scenarios.
To me that is the main problem with Muse models whether it's Spark, Wonder or Ember. While it reads as frontier on benchmarks it really isn't.
This is evident when you compare Dots and Muse the products as well. The model quality is clearly still far behind.
Spark was my go-to at work (technical writing and docs site) on a Cursor API plan for a few weeks because of its speed and price. I was happy with it and recommended it. Now sufficiently outclassed by Haiku 5.5 for tasks of that size.
It's harder to benchmax frontiercode than other coding and math benchmarks, and it's interesting to see the model fall apart there
> Muse Ember is available today on the Meta Model API, at $0.10 per million input tokens and $0.50 per million output tokens.
You can get deepseek v4.1 flash 1017 for cheaper than this from a US based ZDR provider via Openrouter.
you can but this model is smarter and less verbose.
No weights? Shame
I would love it if Meta would release these models to use locally to build up some usage while they are lagging.
They just released Muse Spark Open for that
Tried to use the download button from the page, but it failed miserable. First, could not press the button because had to decline cookies first. Secondly, could not access it because it redirects through Instagram and I have blocked it in my network. So I still don't know how big it is...
Muse Spark Open is 816B-A42B: https://huggingface.co/meta-models
I use haiku and luna for support on most my SWE tasks since they are cheap and fast. I find them enough for most things if you are patient, and the fact an even cheaper model for comparable intelligence is out there now is great news!
What does smallest mean? Even smaller than Muse Glimmer which I doubt? Though it is interesting to see US frontier labs trying to push models that outcompete deepseek flash 1017. It’s a good move from the Deepseek federal government ban. Both Inkling 2 small and this seems to try to fill that gap cost vs performance wise. But I like inkling 2 small more due to open weight even though it has a restrictive license.
Where exactly can we download this small model?
How do we know it is small? How many bytes does it consume on-disk and in-memory? Can I run it on an esp32?
It is not open weights.
What's the point of bragging that it's small if it's always running on someone else's huge supercomputer datacenter?
It's not bragging, it's to make it clear to their users that it's in competition with Luna and Haiku.
[dead]
Why no comparisons with DeepSeek?
American SOTA still cannot compete with Chinese models. They will always pretend there is no elephant in the room.
Who cares. I'll begrudgingly run a local Facebook model if it's the best in its class, but I'll take up Java programming before sending data to their cloud.
Hear, hear!
Luna 6 is getting a bit long in the teeth now. It's two full months old, but it was only a slight step above Luna 5.6, which is already four months old (Imagine saying this about anything else). So I'm not too surprised that a newer small model is beating it handily. I've been hoping for Luna 6.1/2 as the next model release for a few days now.
Tbf, I would take Luna 6 baked to silicon. It is that good. I would also take kimi k3 baked to silicon.
These are both models that I could see using on a long horizon.
Give Terra 6.1 Low a spin in places where you'd have previously used Luna Max. Costs seem similar, due to Terra being much smarter: thus requiring far fewer tokens per task.
Looking solely at AA's intelligence index, recognizing the pitfalls with using that as a measure- Luna 6 Max > Terra 6.1 Low is 51.3% cheaper, 94.1% less output tokens, for 1.56 higher index score.
> Costs seem similar
They weren't, unfortunately, every time I tried. Just an anecdote, of course.
My experience was the same. It is practically five to ten times more expensive than deepseek or luna.
The artificial analysis graph is not very accurate.
I can second your numbers too.
Im surprised, when I've worked through benchmarking things I've run into comparable costs for Luna Max vs Terra Low. Also anecdotal from me, of course.
I use Luna xhigh for when I use it, Luna Max spends 40+ minutes overthinking. I've also found that Terra below xhigh will just flat out not do what it is told.
Yea, this problem is really clear when you look at output tokens for xhigh vs max. I think the jump in capability from Luna xhigh to terra low is really sold by this. I would say that this is just the nature of being post Terra 6.1, pre-luna 6.1
What is your feel re. Terra vs. Sol?
They're both great- but I feel like for folks like me that are working on company's pseudo-infinite token budget, it's really a bimodal game. Im either reaching for the best cheapest for fanned out benchmark runs, or the fastest best for everything else. 6.1 Sol ultrafast is the latter for me, so Terra doesn't really compete with it. Terra has certainly replaced 6 Luna for me, however.
(And yes Astra is a thing, but if im considering Astra, im usually opting for an Anthropic model, 90% of the time.)
Appreciate it!
> I've been hoping for Luna 6.1/2 as the next model release for a few days now.
Same. It became my workhorse after the 5.6 price drop. Still so, although it's showing its age, as you said.
> Imagine saying this about anything else
Javascript frameworks, circa 2016 ;)
Uniquely among models I've tried, Luna 6 is infuriatingly lazy. As if it expects me to do the work, unaware that it is a tool. Normally dumb models compensate for it by trying harder. This one goes in the opposite direction.
I'm always a bit surprised to see Meta talking about models. They have become (almost*) entirely irrelevant to the LLM race, which is weird because their AI lab used to be decent before LLM was cool.
What happened to zuck? Was the metaverse this destructive?
(*) Feeling charitable today
Isn't Muse Wonder at or at least very close to the frontier? Considering they essentially started from scratch after rebooting their AI efforts post-Llama 4 about a year ago it's pretty solid progress.
Meta is trying to stay relevant. They fell behind after llama 4, hired new talent and re-organized.
This was true a year ago, but definitely not today. Their biggest models are pretty much at the frontier, and Glimmer is arguably the best non-Chinese model in the "small" weight class (see my hacky benchmark http://botbowlbench.com/)
They are catching up; the Spark series is good.
I can't wait for Muse Smolder. It's gonna somehow run on an ESP32 compatible SOC w/a custom TPU at 100 tok/sec and people are still gonna drag it.
Model refusals should be a -ve score, not zero?
Treating the refusal as a 0 score erases the difference between lack of capability and a lack of willingness to perform.
Meta seems to have designed Ember to be a strong "all round" great value model - it dominates most of its class on cost
The noncoding results on some reports grabbed my attention. Ember also looks like a particularly inexpensive option for some agent workflows.
On Artificial Analysis, Ember max scored 36 on the Intelligence Index at ~$0.14 per task, compared with 35 at $8 for Grok 4.7 xhigh, and 28 at $0.15 for GPT-6 Luna max. Haiku 5.5 max also scores 36, at $0.41 per task. These are AA's measured costs across its index for those specific configurations.
That puts Ember in roughly Grok's aggregate score range at about 1/55 of the measured cost, and ahead of Luna at a similar cost! [1]
The Vals finance/legal results are also particularly interesting: comparing Ember with GPT-6.1 Terra, both evaluated at "max" reasoning:
- Finance Agent v2.1: 63.83% vs 55.96%, at $0.06 vs $1.92 (!) per test. [2]
- Legal Research Bench: 49.04% vs 25.96%, at $0.06 vs $3.22 (!) per test. [3]
- HLAB-Vals-1.1: 25% vs 1.67%, at $0.12 vs $1.48 (!) per test. [4]
Those are INSANE score differences, at roughly 1/32, 1/54 and 1/12 of Terra's respective benchmark costs!
Misleadingly, their overall Vals Index scores are almost identical: 60.48% for Ember and 60.54% for Terra. Terra also scores higher on Vals' Terminal-Bench 4.0 run: 49.49% vs 39.39%. They arrive at similar aggregate scores through quite different strengths. [5][6]
For me, this reinforces the case for custom, workload specific evals. Broad benchmarks have been useful for shortlisting models but choosing between them, calls for representative tasks, explicit success criteria and measurements using the tools and reasoning settings you actually intend to deploy.
I certainly want to measure cost per successfully completed task, including retries and escalation.
Reusing a framework such as Harbor with a custom task suite seems increasingly useful here. [7] The aggregate score here gives me a starting point but the tasks I actually need done then determines my choice.
Goign by the generic benchmarks would confuse me to no end and likely burn me out.
Great times to be in!
[1] https://artificialanalysis.ai/leaderboards/models
[2] https://www.vals.ai/benchmarks/fabv2-1
[3] https://www.vals.ai/benchmarks/legal_research
[4] https://www.vals.ai/benchmarks/hlab
[5] https://www.vals.ai/models/meta_muse-ember
[6] https://www.vals.ai/models/openai_gpt-6.1-terra
[7] https://github.com/harbor-framework/harbor
thanks for the compilation of benchmarks
smarter, more token efficient and ~cheaper than deepseek v4-flash
Super frustrating that I still can’t give Muse Wonder Full a spin.
Kind of burnt out on all the model updates at this point.
Its a full time job to keep up with them all and assess their value in your own work at the moment. Just so many models coming out.
If your current model is doing fine, why do you need to look at anything new?
Because oftentimes they're slightly better, and more importantly, cheaper.
indeed, im working with several agents scanning the web and keeping up
Thanks, in that case why not just not comment instead of adding more noise to the discussion?
Sorry but why is my reaction "noise"? It's legitimate to be overwhelmed by the amount of released updates and being unable to tell the difference between them.
"Government announces AI ban to avoid further cheesecompiler burnout".
haha just kidding. I agree. That's what benchmarks are for. Just keeping a rough understanding of what companies have which models, and understanding general trends of capabilities, economics, and performance of models.
Please. Complaints are the last thing that we have humans have.
it is our most modestly priced receptacle...
https://www.thebiglebow.ski/our-most-modestly-priced-recepta...
Not open weight? Meh... I'll wait for the Chinese releases :)
Usually wait for 6mo, but the gap is getting narrower
https://ai2027-tracker.com/predictions/china-model-gap/
Interesting but I also think that opus is vastly better than sol. So much sol is almost unusable for me, and this is not really reflected in benchmarks. Anthropic’s issues are mostly refusals. If they did what I want I would only use codex as a double check.
When a statement starts with "our," I always expect bullshit. Batch pricing is two times cheaper with Haiku. Benchmarks are nearly tied. Nobody uses the dumbest models for coding either.
If you want customers to switch over, you better make the price at least half of your competitor's.
[flagged]
[dead]
[flagged]
Good for summarizing your tweets or what?! Is this a model for ants?!
Never Meta.
Seriously. Moral harm is real and there is no excuse.
I honestly have a hard time trying to put these AI companies in some sort of moral hierarchy. They seem to all have issues... I feel like at least with open models you can have more ethical services.
While this is true meta is argueable making the world worse the longest, probably did more damage in that time too.
More than SpaceX? Or OpenAI and it's agreement with military weapons? Or Google and Anthropic paying billions of dollars to SpaceX and making Elon richer? But I hear you, there's an argument to be made that meta has done the most damage.