There's usually a reasonably strong correlation between thinking level and visual quality of the pelican. This is the first time I've seen that correlation appear to break - more reasoning tokens doesn't seem to produce an obviously better SVG.
Wouldn't that be a sign that those models were trained to optimise that test case specifically, as in, by exhibiting equal or higher performance while using less power?
It does seem pretty insane to me that only on max setting (and after waiting 27 seconds) did you actually manage to receive an SVG where it appears that the pelican's legs are straddling the bike correctly.
Edited to make clear what it looks like to me and what I was looking for.
They're not. It's an illusion from the composition.
If you look at the code for each of these (and probably most across models), you can see that the drawing elements are organized by concept. Presumably, this helps accuracy by keeping attention on one concept at a time, but you end up with a stack of conceptualized objects and lose the ability to interleave things.
No doubt these could all do better with a more thorough and specific prompt that coached them away from that technique, but otherwise it's a very subtle detail that a "thinking" process needs to stumble upon independently.
You could use a second prompt to z-order the objects (I tend to break down tasks and pipeline them, rather than trying to stuff everything in a one-shot prompt. Seems to work better for me, could be a holdover from the early days).
Same reason they're all facing right with the same composition, they all have three motion lines, they all have a yellow sun and 2-3 blobby clouds, they're all on two-lane pavement with a white dashed lane separator.
And the same reason Claude has all its notorious Claudisms in prose.
The shorter and vaguer a prompt you give to an LLM, the more it fills in further details using associations they've developed through training. These are ley lines that the models gravitate towards. Diffusion models too.
It's why lazy AI use for creative work always comes across as lazy in the wild and always will.
When you use AI for "creative" purposes yourself, it feels like magic to see a pelican riding a bicycle at all, and you're impressed by all those cool details it added. But then 10, or 100, or 10M people throw more or less the same trivial prompts at it and we start to recognize all those "ley line" associations fill in the gaps the same way again and again and again.
This is starting to feel just like the early PC days. A bunch of Computer models coming out, all shapes and color, but eventually it settled around mouse keyboard, monitor
most people, use the office suite: Word, Spreadsheet , presentations, database.
I bet we're going to see the exact same thing with LLMs at specialized models instead of the everything models.
One for lawyers, coders, Accountants, Designers, etc…
Cross discipline skills are more valuable in the real world, outside of hyper specialization seen in the California tech scene. Even there, it's coming to an end.
What makes it small? Anyone know the size? I don't see it anywhere. Small in relation to their current line up? Anyways, last I tried Meta was llama4. I stepped aside not because of how bad llama4 got, but also because of Zuck's move. Anyone here using Meta? I don't know anyone in my circle using it, but I'm sure folks are using it.
I use haiku and luna for support on most my SWE tasks since they are cheap and fast. I find them enough for most things if you are patient, and the fact an even cheaper model for comparable intelligence is out there now is great news!
Luna 6 is getting a bit long in the teeth now. It's two full months old, but it was only a slight step above Luna 5.6, which is already four months old (Imagine saying this about anything else). So I'm not too surprised that a newer small model is beating it handily. I've been hoping for Luna 6.1/2 as the next model release for a few days now.
Give Terra 6.1 Low a spin in places where you'd have previously used Luna Max. Costs seem similar, due to Terra being much smarter: thus requiring far fewer tokens per task.
Looking solely at AA's intelligence index, recognizing the pitfalls with using that as a measure- Luna 6 Max > Terra 6.1 Low is 51.3% cheaper, 94.1% less output tokens, for 1.56 higher index score.
I use Luna xhigh for when I use it, Luna Max spends 40+ minutes overthinking. I've also found that Terra below xhigh will just flat out not do what it is told.
Uniquely among models I've tried, Luna 6 is infuriatingly lazy. As if it expects me to do the work, unaware that it is a tool. Normally dumb models compensate for it by trying harder. This one goes in the opposite direction.
I honestly have a hard time trying to put these AI companies in some sort of moral hierarchy. They seem to all have issues... I feel like at least with open models you can have more ethical services.
I'm always a bit surprised to see Meta talking about models. They have become (almost*) entirely irrelevant to the LLM race, which is weird because their AI lab used to be decent before LLM was cool.
What happened to zuck? Was the metaverse this destructive?
Isn't Muse Wonder at or at least very close to the frontier? Considering they essentially started from scratch after rebooting their AI efforts post-Llama 4 about a year ago it's pretty solid progress.
https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
There's usually a reasonably strong correlation between thinking level and visual quality of the pelican. This is the first time I've seen that correlation appear to break - more reasoning tokens doesn't seem to produce an obviously better SVG.
Edited to make clear what it looks like to me and what I was looking for.
If you look at the code for each of these (and probably most across models), you can see that the drawing elements are organized by concept. Presumably, this helps accuracy by keeping attention on one concept at a time, but you end up with a stack of conceptualized objects and lose the ability to interleave things.
No doubt these could all do better with a more thorough and specific prompt that coached them away from that technique, but otherwise it's a very subtle detail that a "thinking" process needs to stumble upon independently.
And the same reason Claude has all its notorious Claudisms in prose.
The shorter and vaguer a prompt you give to an LLM, the more it fills in further details using associations they've developed through training. These are ley lines that the models gravitate towards. Diffusion models too.
It's why lazy AI use for creative work always comes across as lazy in the wild and always will.
When you use AI for "creative" purposes yourself, it feels like magic to see a pelican riding a bicycle at all, and you're impressed by all those cool details it added. But then 10, or 100, or 10M people throw more or less the same trivial prompts at it and we start to recognize all those "ley line" associations fill in the gaps the same way again and again and again.
- simple prompt for complex output (bad)
- complex prompt for simple output (bad)
- simple prompt for simple output (good)
- complex prompt for complex output (.... is this good or bad?)
Notable cache cost of 4% input, compared to Luna/Haiku's 10% of input.
most people, use the office suite: Word, Spreadsheet , presentations, database.
I bet we're going to see the exact same thing with LLMs at specialized models instead of the everything models.
One for lawyers, coders, Accountants, Designers, etc…
Cross discipline skills are more valuable in the real world, outside of hyper specialization seen in the California tech scene. Even there, it's coming to an end.
https://openrouter.ai/stealth/crouton-alpha
Same. It became my workhorse after the 5.6 price drop. Still so, although it's showing its age, as you said.
Looking solely at AA's intelligence index, recognizing the pitfalls with using that as a measure- Luna 6 Max > Terra 6.1 Low is 51.3% cheaper, 94.1% less output tokens, for 1.56 higher index score.
They weren't, unfortunately, every time I tried. Just an anecdote, of course.
These are both models that I could see using on a long horizon.
Javascript frameworks, circa 2016 ;)
Treating the refusal as a 0 score erases the difference between lack of capability and a lack of willingness to perform.
Seriously. Moral harm is real and there is no excuse.
What happened to zuck? Was the metaverse this destructive?
(*) Feeling charitable today