> Mostly capability failures that newer models no longer make: counting letters
Have LLMs fundamentally actually changed or improved to resolve problems like this, or have the big models been aggressively tuned / trained on specific examples so it’s less embarrassing?
Because I’ve noticed that Google’s stuff always seems to generate code to do counting tasks, even when not asked for, so I wondered if the LLM itself even generated the answer or it’s just being instructed to use a tool? (Which IMO is a workaround)
> Mostly capability failures that newer models no longer make: counting letters
Have LLMs fundamentally actually changed or improved to resolve problems like this, or have the big models been aggressively tuned / trained on specific examples so it’s less embarrassing?
Because I’ve noticed that Google’s stuff always seems to generate code to do counting tasks, even when not asked for, so I wondered if the LLM itself even generated the answer or it’s just being instructed to use a tool? (Which IMO is a workaround)
The LLMs have indeed gotten better at representing numbers and performing arithmetic, but they can still do it wrong.
I don’t consider it a workaround for an LLM to delegate arithmetic to a calculator.
Inthink finetuning and prompts and intermediary engineering harnesses all obscure idependent verificant of raw LLM capability.
So yes, they probably spend half theor budget trying to keep suspension of disbelief.
Seems like this claim is expired in 3...2...1...