I have tried both, and find myself not using either.
For both the quality is less than the top performing models in my experience. Part of it is the models, part might be the application layer (chatgpt/claude). It would still work for a lot of use cases, but it certainly limits the possibilities.
The other issue is speed. You can run a lot of things even on fairly basic hardware, but the token speed is not great. Obviously you can get better hardware to mitigate that but then the cost goes up significantly.
For self hosting, you need a certain amount of throughput to make it worth it to have GPU's running. If you have spiky usage you are either paying a bunch for idle GPU's or you have horrible cold start times.
Privacy wise: The business/enterprise TOS's of all big model providers give enough privacy guarantees for all or at least most use cases. You can also get your own OpenAI infra on Azure for example, I assume with enough scale you can get even more customized contracts and data controls.
Conclusion: Quality, speed, price, and you are able to use the hosted versions even in privacy sensitive settings.
If you are a company and you want the advantages of a maintained local-like LLM, if your data already lives in AWS, you'll naturally use Bedrock for cost savings. Given most companies are on cloud, it makes sense they won't do a local setup just for the data to just go back on AWS.
For consumers, it actually requires quite powerful systems, and you won't get the same tokens per minute nor the same precision of an online LLM. And online LLMs already have infrastructure in search engine communication and agent-like behavior that simply makes them better for a wider range of tasks.
This covers most people and companies. So it's either local experience is way worse than online (for most practitioners) or that you already have a local-like LLM in the cloud, where everything else of yours already lives. Not much space left for local on my own server/machine.
Having worked in a (very) privacy-sensitive environment, the quality of the hosted foundation models are still vastly superior to any open weight model for practical tasks. The foundation model companies (OpenAI, Anthropic, etc) are willing to sign deals with enterprises that offer reasonable protections and keep sensitive data secure, so I don't think privacy or security is a reason why enterprises would shift to open weight models.
That said, I think there is a lot of adoption of open weight for cost-sensitive features built into applications. But i'd argue this is due cost, not privacy.
They are still too large to run on a normal laptop. Furthermore there must be room left for doing our job. It's a long way until what we use online will be within the reach of a $2000 laptop, better if a $1000 one. My laptop won't run any of them even at unreasonable speed (actually slowness).
I think there are two cases:
1. Self hosting
2. Running locally on device
I have tried both, and find myself not using either.
For both the quality is less than the top performing models in my experience. Part of it is the models, part might be the application layer (chatgpt/claude). It would still work for a lot of use cases, but it certainly limits the possibilities.
The other issue is speed. You can run a lot of things even on fairly basic hardware, but the token speed is not great. Obviously you can get better hardware to mitigate that but then the cost goes up significantly.
For self hosting, you need a certain amount of throughput to make it worth it to have GPU's running. If you have spiky usage you are either paying a bunch for idle GPU's or you have horrible cold start times.
Privacy wise: The business/enterprise TOS's of all big model providers give enough privacy guarantees for all or at least most use cases. You can also get your own OpenAI infra on Azure for example, I assume with enough scale you can get even more customized contracts and data controls.
Conclusion: Quality, speed, price, and you are able to use the hosted versions even in privacy sensitive settings.
Thanks — I agree with your three big pain points: quality vs hosted SOTA, token speed, and economics/utilization.
Have you run into cases where on‑device still makes sense?
1. Data that is contractually/regulatorily prohibited from being sent to any third‑party processor (no exceptions).
2. Very large datasets where throughput can be low (overnights acceptable) but the cost is high for cloud models.
3. Inputs behind a password-wall that hosted assistants/chatgpt/claude can’t reach and can't do agentic things with them.
If you are a company and you want the advantages of a maintained local-like LLM, if your data already lives in AWS, you'll naturally use Bedrock for cost savings. Given most companies are on cloud, it makes sense they won't do a local setup just for the data to just go back on AWS.
For consumers, it actually requires quite powerful systems, and you won't get the same tokens per minute nor the same precision of an online LLM. And online LLMs already have infrastructure in search engine communication and agent-like behavior that simply makes them better for a wider range of tasks.
This covers most people and companies. So it's either local experience is way worse than online (for most practitioners) or that you already have a local-like LLM in the cloud, where everything else of yours already lives. Not much space left for local on my own server/machine.
Wouldn't it be cool to have a local AI agent? It could access search engines and browse any website through a headless browser.
Having worked in a (very) privacy-sensitive environment, the quality of the hosted foundation models are still vastly superior to any open weight model for practical tasks. The foundation model companies (OpenAI, Anthropic, etc) are willing to sign deals with enterprises that offer reasonable protections and keep sensitive data secure, so I don't think privacy or security is a reason why enterprises would shift to open weight models.
That said, I think there is a lot of adoption of open weight for cost-sensitive features built into applications. But i'd argue this is due cost, not privacy.
Thanks for the view from a very privacy‑sensitive environment — agreed that hosted SOTA still leads on broad capability.
Could you share a quick split: which tasks truly require hosted SOTA than open‑weight? I think gpt-oss is quite good for a lot of things.
SMBs can’t get enterprise contracts with OpenAI/Anthropic, so local/open‑weight may be their only viable path — or wait for a hybrid plan.
They are still too large to run on a normal laptop. Furthermore there must be room left for doing our job. It's a long way until what we use online will be within the reach of a $2000 laptop, better if a $1000 one. My laptop won't run any of them even at unreasonable speed (actually slowness).
Totally fair. On a normal laptop you also need headroom to do your actual job, and KV cache + context length can eat that quickly.
Two reasons?
1. Management 2. Scalability
Running your own local AI takes time, expertise and commitment. Right now the ROI is probably not strong enough to warrant the effort.
Couple this to the fact that it's not clear how much local compute power you need, and it's easy to see why companies are hesitating?
Interestingly enough, there are definitely a number of sectors using local AI with gusto. The financial sector comes to mind.
Well put. Management overhead + unclear capacity planning kills many pilots.