I’m honestly impressed that a pure python implementation can beat out vLLM and SGLang. Granted they lean on FlashInfer, and of course torch.compile has gotten incredibly powerful in the last few years. Though dynamic shapes have still been a huge thorn in my side, I’ll need to look closer at how they pulled it off…
Hi! I work on dynamic shapes in pytorch and would love to hear more about the challenges you’ve run into. We’re always looking to improve the experience, so if you’re open to chatting, feel free to DM me on Twitter (@bobrenjc93) or email me at bobren@meta.com.
since you work on pytorch, what would you say is the best place to ask questions about general usage, trouble shooting? I’ve been struggling with a, what I would consider, a simple torchrun elastic training example, and haven’t found any good resources online. I’ve been spelunking through pytorch but have a feeling a little back and forth with someone familiar with these features would immensely clear things up.
PyTorch Dev Discuss is a fantastic forum where many core devs actively participate and answer questions: https://dev-discuss.pytorch.org
In addition to Dev Discuss, a number of core contributors are also active on Twitter. Two particularly helpful and prolific voices are @ezyang and @cHHillee.
Finally, don’t overlook GitHub issues—they’re a surprisingly effective way to start conversations. If you’ve found a bug or have ideas on how to improve the APIs, opening an issue is always welcome.
I mean, vllm and sglang are both "pure python" essentially as well. But yeah, in ML you rarely require C++ to get good performance for most of the systems people are writing.
While Tokasaurus’s Async-TP shows impressive throughput gains, it seems over-engineered for common use cases. The CPU overhead from async tensor parallelism only pays off at 6k+ token batches, and you need NVLink-connected GPUs to see real benefits. Most prod deployments don’t need this complexity — you’re better off with simpler approaches unless you’re specifically optimizing for massive batch throughput. The adaptive manager skipping “optional” tasks under load also feels concerning from a reliability perspective.
Sure. Things change over time. Is there a reason to believe they'd be different in such a way that this would be more useful than in today's landscape? I haven't seen such a forecast myself.
How big of a use case is synthetic data generation? I’m curious as I see a lot about it coming from academic projects but I haven’t seen much related to commercial use cases
I'm curious what how big is latency tradeoff.
I know assumption here is that it does not matter in those use cases but what order of magnitude it is? 10x? 100x?
this is important for usage in "soft realtime" application, where you do not need instant response but someone is still waiting.
if latency is really big, then it can only be used for basically background processes.
Cool project! The codebase is simple and well documented, a good starting point for anyone interested in how to implement a high-performance inference engine. The prefix sharing is very relevant for anyone running batch inference to generate RL rollouts.
Stanford was edgy enough to reefer to “toking” in the moniker, but exercises restraint by depicting the titular thunder lizard smoking a putatively conventional tobacco cigarette.
I am hoping to use this “Tokasaurus” nickname with affection for my neighbors. If Stanford is ok with informal usage.
Success with Meta AI / Llama 4:
Hey Meta, I would like to see an image of a Tyrannosaurus Rex, who is clad in a leather jacket, sunglasses, and fedora. He is so cool looking, and smoking a joint of marijuana, and his image is superimposed against a skyline of Phoenix in the golden glow of sunset.
Proof that attention is not only highly desired by Stanford tech bros, but HN keyboard warriors equipped with LLM tech. Everyone is clever all of the time.
Given chat and API needs for low-latency, llama.cpp is probably still the best choice for self hosted models with or without GPU support. And Ollama is the leader for wrapping llama.cpp.
Because Tokasaurus was mentioned as better than Ollama for conducting darwinian godel machine operations (self-improvement), I looked for the linked repo on GitHub and it was 404. So glad it is back https://github.com/ScalingIntelligence/tokasaurus.
The code has few comments but gotta love when you can tell someone was having fun!
https://github.com/ScalingIntelligence/tokasaurus/blob/65efb...
I’m honestly impressed that a pure python implementation can beat out vLLM and SGLang. Granted they lean on FlashInfer, and of course torch.compile has gotten incredibly powerful in the last few years. Though dynamic shapes have still been a huge thorn in my side, I’ll need to look closer at how they pulled it off…
Hi! I work on dynamic shapes in pytorch and would love to hear more about the challenges you’ve run into. We’re always looking to improve the experience, so if you’re open to chatting, feel free to DM me on Twitter (@bobrenjc93) or email me at bobren@meta.com.
since you work on pytorch, what would you say is the best place to ask questions about general usage, trouble shooting? I’ve been struggling with a, what I would consider, a simple torchrun elastic training example, and haven’t found any good resources online. I’ve been spelunking through pytorch but have a feeling a little back and forth with someone familiar with these features would immensely clear things up.
PyTorch Dev Discuss is a fantastic forum where many core devs actively participate and answer questions: https://dev-discuss.pytorch.org
In addition to Dev Discuss, a number of core contributors are also active on Twitter. Two particularly helpful and prolific voices are @ezyang and @cHHillee.
Finally, don’t overlook GitHub issues—they’re a surprisingly effective way to start conversations. If you’ve found a bug or have ideas on how to improve the APIs, opening an issue is always welcome.
There's also the slack but you gotta know someone to get on that ;)
I mean, vllm and sglang are both "pure python" essentially as well. But yeah, in ML you rarely require C++ to get good performance for most of the systems people are writing.
> On throughput-focused benchmarks, Tokasaurus can outperform vLLM and SGLang by up to 3x+.
Looks like they don't compare to TensorRT-LLM throughput numbers which, last I checked, are SOTA in open source.
TensorRT-LLM being open source is a lie, all the important kernels are loaded from cubins.
It also appears that this was a sampling benchmark...which is not representative.
Generation benchmark was 5% faster than SGLang.
While Tokasaurus’s Async-TP shows impressive throughput gains, it seems over-engineered for common use cases. The CPU overhead from async tensor parallelism only pays off at 6k+ token batches, and you need NVLink-connected GPUs to see real benefits. Most prod deployments don’t need this complexity — you’re better off with simpler approaches unless you’re specifically optimizing for massive batch throughput. The adaptive manager skipping “optional” tasks under load also feels concerning from a reliability perspective.
Buy surely next years production deployments will be very different to right now, with different use cases...etc
Sure. Things change over time. Is there a reason to believe they'd be different in such a way that this would be more useful than in today's landscape? I haven't seen such a forecast myself.
Depends on what production means for you. This is useful for batch production jobs.
Also, this seems very useful for generating synthetic data or labelling a bunch of data. 6k batch size is small for data labelling.
How big of a use case is synthetic data generation? I’m curious as I see a lot about it coming from academic projects but I haven’t seen much related to commercial use cases
tiny NNs distilled from LLMs can produce some amazing results, i'm surprised it's not more common tbh
I agree, there are impressive results. This just came out from Berkeley https://arxiv.org/abs/2506.04178
But still, I mainly see work on this direction in academia.
I'm curious what how big is latency tradeoff. I know assumption here is that it does not matter in those use cases but what order of magnitude it is? 10x? 100x?
this is important for usage in "soft realtime" application, where you do not need instant response but someone is still waiting.
if latency is really big, then it can only be used for basically background processes.
Cool project! The codebase is simple and well documented, a good starting point for anyone interested in how to implement a high-performance inference engine. The prefix sharing is very relevant for anyone running batch inference to generate RL rollouts.
Stanford was edgy enough to reefer to “toking” in the moniker, but exercises restraint by depicting the titular thunder lizard smoking a putatively conventional tobacco cigarette.
I am hoping to use this “Tokasaurus” nickname with affection for my neighbors. If Stanford is ok with informal usage.
Success with Meta AI / Llama 4:
Hey Meta, I would like to see an image of a Tyrannosaurus Rex, who is clad in a leather jacket, sunglasses, and fedora. He is so cool looking, and smoking a joint of marijuana, and his image is superimposed against a skyline of Phoenix in the golden glow of sunset.
Can you light up the joint with a glowing tip?
Proof that attention is not only highly desired by Stanford tech bros, but HN keyboard warriors equipped with LLM tech. Everyone is clever all of the time.
I appreciate the double entendre
Given chat and API needs for low-latency, llama.cpp is probably still the best choice for self hosted models with or without GPU support. And Ollama is the leader for wrapping llama.cpp.
Because Tokasaurus was mentioned as better than Ollama for conducting darwinian godel machine operations (self-improvement), I looked for the linked repo on GitHub and it was 404. So glad it is back https://github.com/ScalingIntelligence/tokasaurus.
Shame this is written in Python, looks very interesting, but I'm no expert in this field.
If there is anything here worth using, it's entirely possible that the llama.cpp crew can save it from vanishing into obscurity.