43 points | by Retro_Dev 4 hours ago ago
5 comments
There are so many of us with weak GPUs and these updates are massive wins. Still waiting for the 2080 moment where this 27B param model and a reasonable kv cache can exist and function at slow, but overnight-burning speeds ..
How does it compare to the 3.8-27B versions from TensorFold? I.e. does it make sense to run unsloth via TensorFold engine?
How do the smallest quants of this compare to Bonsai 2?
Bonsai 2 doesn't work at all, Unsloth's smallest... also doesn't work, but is still better.
Bonsai's technology would be better suited for 100s of B models.
Awesome! I wonder how it does at Q5+ compared with v3.0
There are so many of us with weak GPUs and these updates are massive wins. Still waiting for the 2080 moment where this 27B param model and a reasonable kv cache can exist and function at slow, but overnight-burning speeds ..
How does it compare to the 3.8-27B versions from TensorFold? I.e. does it make sense to run unsloth via TensorFold engine?
How do the smallest quants of this compare to Bonsai 2?
Bonsai 2 doesn't work at all, Unsloth's smallest... also doesn't work, but is still better.
Bonsai's technology would be better suited for 100s of B models.
Awesome! I wonder how it does at Q5+ compared with v3.0