I also recently decided to buy a datacenter GPU and slap it into a system. Some notes from my experience that the author doesn't mention in their article:
Decommissioned NVIDIA V100s and AMD MI50s are fairly cheap, $200 for 16gb and $400-500 for 32gb, for local experimentation. They are also very old. There's an enthusiast community keeping these two cards alive and working with current platforms and models.
Nitpick, but the V100 doesn't support bfloat16. The performance hit is not a big deal if you're fiddling with local models, but the card is on it's way out in terms of hardware features.
The MI50 does support bf16, but not the current edition of AMD ROCm. Vulkan support is good and the MI50 works with most major platforms (llama.cpp, vllm, etc.), but it's not without some pain points like manual recompilation. Fortunately the open source community has already paid most of your way.
The cooling requirements for these cards cannot be understated. A consumer grade GPU may throttle if in a small case without additional fans, but if given the same treatment a datacenter GPU will overheat itself idling. You will need to buy, at least, a bunch of decent 120mm fans to prevent this or invest in some water cooling.
I ultimately went with an AMD MI100 32GB ($950). I'm an AMD fan, current ROCm editions support it, and it was low-fuss to get things working. I'm debating getting a second so I can try out bigger models like qwen3-coder-next.
Tesla V100 SXM2 16GB is NOT DGX class as the author writes. It's HGX class. The V100 comes in two classes, SXM2 and SXM4, the latter coming with a Max of 80gb on board memory. Typically these are installed 8×A100 80GB SXM4 on an HGX riser, and what that gives you is NVSwitch fabric and 640GB of pooled HBM2e (on package stacked memory /w ~2 TB/s of memory bandwidth). 2u standard rack footprint too.
> The way it works is that a vision encoder (similar to what ChatGPT and Claude use) takes image pixels and translates them into the LLM’s token embedding space. The model does not “see” the image the way a human does. Instead, the vision encoder compresses the image into a sequence of vectors that live in the same mathematical space as text tokens. The LLM then processes those vectors as if they were just another sequence of tokens.
Could you also do this for music and specifically sound synthesis? It would be awesome to vibe synthesize sounds and then see the VSTi parameters surrounding it.
I wonder if this could be usefully mitigated with a combination of prompt (prefix) caching and an agent that let you control what the prompt prefix consisted of. The goal would be to incur that slow prefill once to build the prompt cache, then have subsequent prompts consist of mostly this fixed prefix plus specific instructions.
For a language like C++ where modules are split into definition (.h) and implementation (.cpp) parts, one choice of prefix would be all the header files for the project (which aren't likely to change much).
More generally the idea would be to have an agent that had cached-prefix reuse as it's primary context management goal.
Another possibility, to support caching of files that have since changed, would be for the agent to build the context as a fixed prefix reflecting some or all of the codebase in its start-of-session state, then append any changes to that, with appropriate prompting to only use the latest definition of a function.
e.g.
Say file A initially contains functions X, Y and Z, then the prompt prefix is built to include X Y Z. If the user then modifies Y -> Y', then just add that to the context, so that the cached prefix is unchanged, giving X Y Z Y'.
Most people won’t be dumping 100K tokens into it at once, but I agree that all of the prefill time that adds up during a session becomes a lot to account for.
This is also a problem for all of the Mac local LLMs. Macs are a great way to get a lot of high bandwidth memory, but their compute is very far behind current gen dedicated GPUs. Some of the expensive Mac Studio setups allow you to run very large models with usable tokens/s, but you can be waiting a long time for it to get to the point of generating those tokens.
I was just looking into this and was worried about the fan setup. Interesting that he was able to solve it with good results.
In case anyone is interested, I’m using PCIE passthrough on a FreeBSD host to a Linux guest with an older Pascal card. It’s worked great and I’ve been thinking about putting a nicer card in there. The SXM route seems great, but I’ve been burned (almost literally because of the heat) by DC components before.
> And yes, if you want the absolute best, Opus 4.8 exists. It also costs more per 20 minutes of heavy use than I paid for this entire GPU and adapter setup combined. But the gap is shockingly small.
I don't think this is a fair characterization of the situation. I use frontier models via API pre-paid tokens every single day, and I can barely rack up $100 per month. The fact that we figured out how to burn double this in 20 minutes is impressive, but I don't think it reflects the reality that many are experiencing right now. There are some exceptionally gluttonous approaches to harnessing LLMs that I think are serving as convenient straw men in these discussions.
Paying for the API will almost always be more economical than self-hosting equivalent infrastructure. I am not against self-hosting, but the article suggests a primarily economic motivation for this effort. If you are consuming fewer than 10^9 tokens per month, I really don't think it's worth your time to try and compete with the hyperscalars. Most of the money is to be found in the integration of this technology with existing businesses.
I use hosted providers myself, but I can churn through $100 worth of tokens in half a day even with cheap models like Deepseek easily. If someone's use is as light as yours, then sure - grab a subscription and you'll save far more. For higher use it will come down to how cheap your electricity is whether it is worth offloading at least some of it (for me it's not, FWIW)
A mix. I have hobby projects that churn through that much when I don't need the tokens for others things. I also have projects for clients that easily consumes those levels. As well as a stealth-ish potential startup. Currently I'm at 4 different subscriptions + more than I'd like in spend via OpenRouter...
What multiplies it very quickly is when you start feeding them with test suites and "Ralph loops" that run until the test suites pass, or complex chains with lots of sub-agents being triggered.
If you're sitting there watching everything, it will be hard to burn all that much even if you're running multiple things in paralle.
Claude is something like $35 per million tokens. If I was using API pricing I could trivially spend $100 in a single hour long coding session, with /fast turned on in about 10 minutes. Not sure how you guys are using it.
The AMD MI250X GPUs are also interesting - 128GB of HBM2E at 3TB/s, sometimes you see them second-hand for under $1k, the catch obviously is that it needs an OAM socket. Never seen an easy way to hook them up to a regular mainboard.
An additional complication is that MI250Xes are two GPUs in one package, so you need to connect the first and last x16 SERDES groups to the host, otherwise you'll only see one GPU (or it won't work at all, idk).
Also, the cheap HPE pulls on eBay need some proprietary HPE magic to work, and I have yet to see anyone figure that out.
Great write-up, I've often considered these DC cards for a project and now you've convinced me to pick one up; you describe the price of the unit against what one spends on tokens and that does it for me.
The most interesting and perhaps useful for most would be how they control the fan. If you are thinking of doing this, you really want to get those fans under control, they are loud. For anyone thinking of these, v100s idle super high! 25-35watt with nothing loaded and easily 50w when a model is loaded.
Could probably avoid the crazy fan with a waterblock - I've seen a whole kit, v100 + PCIE adapter + block for £235. Yes, you'll have to pay for pump, radiators and radiator fans, but that should really quieten it down
I'm actually surprised there hasn't been a dedicated effort to support display offload to, say, the CPU's iGPU.
I'm sure manufacturers would love saving a dollar per card, and OEMs would appreciate eliminating the support calls from "I just bought a new $2500 gaming PC and no video" because they plugged the monitor into the iGPU instead of dGPU.
I've seen things where you have multiple video cards and can use one gpu to render to a framebuffer which is transferred to the other video card to output. I'm sure it adds latency, and it's probably unsupported... But no output doesn't mean can't do gaming... It just means gaming will be iffy.
There's some virtualized desktop server stuff too. Run a bunch of desktop sessions on a beefy computer and send a video stream to desktop players. With the right codec settings, the latency is probably ok for many games.
Never a problem. RemoteFX does (did) everything you'd want. Make your OS, log in remotely through an accelerated client. The real problem is Microsoft did something around Windows Server 2008 R2 that killed performance (literally halved it) for RemoteFX. You're only now reobtaining the virtualized video performance we used to have back in 2008.
There are many tells aren't there? There was clearly hard human work and experimentation here, but it's a shame the OP let AI do chunks of the writing. Once you see it, it's much harder to take the post seriously.
I disagree. Not everyone has a good writing style. In those instances I think it is fair to default to llm recommendation. We may be allergic to it, but we saw one formulaic response too many ( though admittedly it does raise a question of whether HN was the intended audience for it ).
In any event, not all of us have a unique writing style worth preserving just like not all of us can write clear and clean code. Just saying.
I really wish it was more common to use AI for augmenting than authoring. Eg i find coding with LLMs neat when you primarily "talk" to it through code, by filling out structs, funcs, fields, etc - where it would use your changes as the template and then to work to effectively autocomplete the gaps. The more you iteratively write the less it fills in, but also the less it deviates from your intent, design, etc.
I feel like writing could use a similar harness, where it attempts to minimally reword the authors sentences, perhaps just tweaking grammar, spelling, etc. In the coding example i think the human code would be near unchangeable, the LLM would pivot around it - but in the writing example i think the human writing would have to be more mutable. I imagine it would be a configurable setting.
I've not really seen a system which focuses on this human<->LLM look, but it feels interesting to me.
In a sense, there is a clear market for it ( people want 'authentic' experience ). I can kinda understand it. I want pure linux experience without systemd, but I recognize that in the current ecosystem, it comes at a cost.
So the language harness makes sense to me, but corps are already cracking down on token use ( and such a harness would likely only add to the cost ). The other question is whether the people, who could benefit it would even recognize it as a problem though.
> I want pure linux experience without systemd, but I recognize that in the current ecosystem, it comes at a cost.
Running Alpine/Gentoo/Devuan isn't that expensive. (I'm assuming the cost is time/effort when I say this; let me know if there's another relevant metric)
It’s not about preserving a unique writing style. When I see LLM writing my brain automatically discards the content of the writing. To me, seeing LLM writing is equivalent to going to a high-end restaurant and getting served on generic paper plates. Sure, the food looks perfectly fine and there is, in theory, nothing wrong with a paper plate. Once you see that paper plate, however, you will question how nice that establishment actually is, because a lack of care for the plates undermines the quality of the food. You automatically categorize all establishments that serve on paper plates in a specific category, one that might make you concerned if you will get food poisoning that night. LLM writing is exactly the same way for me. I don’t know if this LLM-assisted piece of text is actually a Michelin three star establishment or has had several heath violations in the last year. However, I didn’t pay for it, so putting in effort to determine if it’s LLM-assisted writing from an expert or just LLM slop that isn’t from the purported author at all isn’t worth the time.
I’m much more willing to read typos and bad writing than LLM writing. If I want to read the LLM rewritten version, I can run an LLM over the original writing myself. I have not yet found true that anyone is better at prompting than anyone else in a way that suggests that I wouldn’t get substantially the same results myself. Thus, I don’t think providing the version that has passed through the telephone game is accomplishing something that couldn’t be done by readers later. I have spent the vast majority of my life reading the original writing styles of people and didn’t have an issue then. I’m not convinced a problem I had was solved when we started post-processing writing with an LLM.
I skim a lot. I skimmed this article and appreciated the author documenting their process. I am indifferent to LLM or human writing for technical content. I suspect I skimmed most of the LLM parts, but judging writing quality was not why I read this post, I read it because I was curious about how useful the GPU is, and if I could replicate the author's work. Some carefully written prose wouldn't have helped me do that any better. The prose in this article did the job.
This is mostly how I feel about it. If anything, the weird llm jitters served almost like punctuation markers. Still, I get why it riles some people up.
(TL;DR Can we just judge written works by their actual content?)
I’m really in the “who gives a shit” camp on something like this. A lot of people probably have an LLM punch up a blog post. It is good at turning bullet points and notes into prose, fixing run-ons, etc. Maybe I’m naive but I trust that the kind of person who posts a clearly noncommercial post like this on HN gives a crap enough that they read the final draft and confirmed it isn’t inaccurate.
This pearl-clutching about the mere use of AI regardless of how responsible or appropriate the use is, seems like a professor in 1985 throwing an essay back in a student’s face as “this was obviously printed from a computer and not typewritten like a PROPER essay! I can tell just by looking at it!”
- In 2017, the v100 was a ~$10,000 GPU. I believe there was a PCI-e version but this is probably so cheap because SXM2 is going to be harder to use;
- A 5090 has 1800GB/s of internal memory bandwidth (compared to 900GB/s in the 9 year old GPU). Of course a 5090 is substantially more expensive;
- A 5090 has ~21k CUDA cores vs ~5k;
- The current $10k NVidia GPU is the RTX 6000 Pro w/ 96GB of VRAM. It has slightly more CUDA cores but it otherwise pretty much just a 5090. This is unsurprising. NVidia uses VRAM for market segmentation.
Consider this: in 5-10 years, the trillions spent on AI data centers will likewise be sold for scrap most likely. That's how short the runway is for OpenAI and Anthropic to recover that investment.
Anyway, I'm kind of impressed the author managed to get this all to work. I don't think it even would've occurred to me that someone had made an SXM2 adapter, particularly because it's not even used anymore. Like props to whoever did that.
> Consider this: in 5-10 years, the trillions spent on AI data centers will likewise be sold for scrap most likely. That's how short the runway is for OpenAI and Anthropic to recover that investment.
Even more interesting: it'll devalue all of SaaS and the entire US tech sector.
We might have just shot our most valuable non-AI tech products in the foot.
How so? I understand that flooding the market with physical goods will reduce prices and thus profits. But how would that also reduce the nonphysical SAAS stuff?
> But how would that also reduce the nonphysical SAAS stuff?
The resulting economic crash will affect everyone, we're (IMHO) looking towards a dotcom-bust level wipeout. And many SaaS and other companies run asset-lean (i.e. they have no server hardware because that's all cloud, no real estate because it's all either wework or conventionally rented), margin-lean (the VC business model requires that, as the basic recipe is to achieve market domination by burning cash) and cash-lean (often enough, it's less than a quarter of expenses on the bank accounts).
All that "lean-ness" looks great on an investor's quarterly release sheet: no massive amounts of wealth tied up in assets and no cash sitting around on bank accounts that could be released towards investors as dividends or, if it comes from third parties, costs the company interest... but it prevents resiliency against crises.
> We might have just shot our most valuable non-AI tech products in the foot.
Counterpoint: the fiber buildout during the dotcom boost. That crashed the economy pretty hard when the bubble burst, but we are still benefitting from all the dark fiber that was arranged for and built out back in that era. A lot of today's ISPs were able to grab up that fiber after the bust for cents on the dollar.
Assume that OpenAI and Anthropic go bust, which at least one of them likely will, and possibly a fair few of the datacenters that are under construction will also collapse. Someone will be able to snatch these physical assets again for cents on the dollar and run open-weight models on them or train new ones.
The problem isn't (and no, this is not an AI tell, everything I write here got typed on a 2022 M2 MBA by hand) the assets, they will be put up for productive usage, just as with any other large bankruptcy or bubble in history. The problem is the "IOU" that is being passed from one hand to the next like a hot potato. Assuming a recovery of, maybe, 20% after the collapse, at 1.6 trillion dollars of assets under management by some kind of private investment/debt we're looking at about 1.3 trillion dollars in valuation that is going to be wiped out.
And given that a lot of the investment market is actually backed by pension funds... this is going to be a bloodbath. Not only will there be a lot of people laid off in addition to the layoffs we already saw "due to AI", but when the pension funds and thus their payouts collapse? We'll see retirees flooding the employment markets who just try to make a living, rendering the situation for everyone else even worse. Flipping burgers used to be a gig for students, these days students compete with people of all ages desperate to survive - and thus desperate to undercut others in wages.
Another problem will be the capacity buildout in the semiconductor industry. It's already heading toward an oligopoly after numerous boom-bust cycles: you only have two and a half GPU chip vendors (NV, AMD, Intel), two vendors of general-purpose CPU vendors (Intel and AMD - I exclude Apple because they do not sell their CPUs to any third party and ARM because 99% of non-Apple ARM chips do not go towards servers, desktops and laptops), three RAM manufacturers (Samsung, SKhynix, Micron) and two and a half physical chip manufacturers (TSMC, Samsung, Intel). When the AI bubble bursts, it will be one of a hell of an effort to prevent at least one actor from going bankrupt.
Volta (and Pascal, which I'm using) should still be supported with driver 580 as long as you don't use the open modules, and you can use up to cuda 12.9 and cudnn 9.10.2. No need to limit yourself to an old kernel.
Yeah. I get that many HN comments are just complaints (heck mine was too and just as negative and shaming). But how bad of a day must you be having to try to shame someone about how they choose to write up an experience they thought was neat. Whatever, free speech and all that. Hope OC's day gets better.
It doesn’t read like shaming to me. It’s, in the grand scheme of HN comments, definitely on the more constructive side of the criticism. Maybe it could have been reworded, but I think the author of the post could very easily find it actionable in the future. I too had to stop reading the article at that point, so I think if the author wants more people to read, my advice for them is to just write like themselves. We’ve entered the start of a new Instagram filter age where many people feel they need to have LLMs reword their writing presumably for the same reasons as the original filter age. I share OC’s sentiment of pushing against the recent trend of implicitly shaming people for their individualistic writing styles.
Only because so many of the articles posted on HN now are AI-written, and badly, too. A lot of tech people are so impressed with LLMs’ capabilities in code that they fail to recognize how bad they are at writing enjoyable prose. And it feels like a chore to write out a whole blog post by hand when the machine could do it for you! But the result we get is so, so much worse and more annoying.
I dislike AI prose too, the cadence of it really rubs me the wrong way, but, that said we've had a lot of great, informative articles lately, written with AI help, where you just have to grit your teeth and get through them to get the underlying knowledge.
I don't think that commenting on every article is going to make the posters suddenly decide to go back and rewrite it by hand. Some of them probably don't even speak English natively. The comments are getting more tiresome than the AI prose at this point.
Hopefully in a year or so the LLM output won't be so janky and obvious, so this might just be a phase everyone has to pull through.
That line was the exact moment I also realized the post was AI written. I kept reading though, but I am left constantly guessing at which key details might be pure hallucinations.
Honestly, the default styles are pretty bad. I use Claude in my scientific writing in a very specific way. 1. I write a paragraph. 2. I put Claude into concise style mode. I then ask Claude to revise for clarity.
I can write competently, but it's natural direction is towards emotional rhythmic flow that can convey emotion/passion...but which for scientific writing, can get in the way of clear clean communication. So, I write what I mean,and Claude straightens it out...and these days (i.e. not last year), it doesn't lose my meaning that often. And since I wrote it first, these AI-isms appear less frequently, and if they do, I revise them away.
Some resell group is going to have to make this easier. The shear amount of these cards otherwise heading towards the landfill is staggering. That is if Big Tech don't destroy them to prevent model weights from leaking.
How would destroying the GPUs prevent the model weights from leaking? By the time you get your hands on them the memory is powered off for a long enough time that a cold-boot style attack is impossible.
> The shear amount of these cards otherwise heading towards the landfill is staggering.
The thought of throwing away working cards sounds so bizarre to me. I can't believe companies would dispose them into the landfill like that, it is at least worth giving away for refuse.
I genuinely hope that is the case. The market is absolutely bananas now. I actually now own devices that went up in 'value' since purchase. This is not normal ( and a little scary ). This, on the other hand, is an invitation to properly recycle otherwise unwanted hardware.
kinda, they put that on a PCIe socket, but it's passive. Meaning no fan. If you try inference on that it overheats in 1 minute unless you have it inside a server case.
You can also look at past posts by the same author (before LLM usage proliferated) if you’re curious.
The project is still very cool, but it’s a little less enjoyable to read when everything sounds the same. It would be just as annoying for people to manually write in a corporate/marketing style, because humanity is what makes the small web interesting.
This, setting aside the llm issue, it is dealing with hardware in ways that -- one would think - would be celebrated on HN of all places. But we focus on presentation.
Because their custom training data contains an emphasis on such verbiage. It doesn't come from the God-knows-how-many TB of web content the model is pre-trained on. There, such phrasing is only a drop in the sea. But the "yes, you're right" phrases, the em dash, etc., come from the later stage, for which content is created according to some (probably overprecise) guidelines.
Right. The overuse of "genuinely" most of all. Seems like they put Claude through a few good rounds of training to always answer questions about its consciousness, thoughts, etc., with something about how it's "genuinely unsure," and as a result, the model learned to use "genuinely" as an intensifier in all sorts of inappropriate contexts.
> Where do you think llms learned to write that way?
Not from individual human content, that's for sure - maybe MLM marketing copy? Sleazy 4AM ads?
I mean, every time this response comes up, I keep asking the person to point at something written prior to 2022 that gets 80%+ on the LLM detectors, and yet no one can find anything.
Maybe you, postalrat, can find something written in this style that was published prior to 2022.
I have written the blog post. I know empirically that I have used 0% AI while writing it.
I also know LLM detectors are total BS and they don't really work. I have tried a couple on this exact blog post, and QuillBot, for example, gave me 0% AI detected on it.
I have then used a blog post of mine from 2021. QuillBot gave me 8%...
The King James version of the Bible came out at almost 100% AI generated a while ago. It was the HN front page.
Stop thinking that if someone writes in a way that is fun or looks like what you would think an AI writes, then it is AI generated. Loads of the time it is, but sometimes it's not, and it really hurts those like me.
It's a function of the LLM "thought process"! It's not really modeled after human speech. It is in short segments but not long form, same reason you see the same rather odd nuances in LLM generated code.
If they way you thought was to run a bunch of if statements, generate content, then feed that content back to get a "score" of what seems the most plausible, run the if statements again, and adjust / merge responses, then you would write similarly. The recognizable cadence of LLM generated content is pretty clearly the result of a lot of if statements being fused together.
The latest Raspberry Pi 5 has one 32-bit channel (2x 16-bit subchannels) of LPDDR4X-4267 SDRAM giving 17.1GB/s of bandwidth, 52x less than this GPU. Never mind lacking the CUDA and Tensor cores, so the FP16 performance is 102x less (307 GFLOPS vs 31.4 TFLOPS). So for £200, there's absolutely no comparison for this specific use-case.
I also recently decided to buy a datacenter GPU and slap it into a system. Some notes from my experience that the author doesn't mention in their article:
Decommissioned NVIDIA V100s and AMD MI50s are fairly cheap, $200 for 16gb and $400-500 for 32gb, for local experimentation. They are also very old. There's an enthusiast community keeping these two cards alive and working with current platforms and models.
Nitpick, but the V100 doesn't support bfloat16. The performance hit is not a big deal if you're fiddling with local models, but the card is on it's way out in terms of hardware features.
The MI50 does support bf16, but not the current edition of AMD ROCm. Vulkan support is good and the MI50 works with most major platforms (llama.cpp, vllm, etc.), but it's not without some pain points like manual recompilation. Fortunately the open source community has already paid most of your way.
The cooling requirements for these cards cannot be understated. A consumer grade GPU may throttle if in a small case without additional fans, but if given the same treatment a datacenter GPU will overheat itself idling. You will need to buy, at least, a bunch of decent 120mm fans to prevent this or invest in some water cooling.
I ultimately went with an AMD MI100 32GB ($950). I'm an AMD fan, current ROCm editions support it, and it was low-fuss to get things working. I'm debating getting a second so I can try out bigger models like qwen3-coder-next.
Tesla V100 SXM2 16GB is NOT DGX class as the author writes. It's HGX class. The V100 comes in two classes, SXM2 and SXM4, the latter coming with a Max of 80gb on board memory. Typically these are installed 8×A100 80GB SXM4 on an HGX riser, and what that gives you is NVSwitch fabric and 640GB of pooled HBM2e (on package stacked memory /w ~2 TB/s of memory bandwidth). 2u standard rack footprint too.
I have no idea what you are trying to say.
V100 came as sxm2 and sxm3. And it was 16 and 32gb.
HGX is DGX with extra toppings.
> The way it works is that a vision encoder (similar to what ChatGPT and Claude use) takes image pixels and translates them into the LLM’s token embedding space. The model does not “see” the image the way a human does. Instead, the vision encoder compresses the image into a sequence of vectors that live in the same mathematical space as text tokens. The LLM then processes those vectors as if they were just another sequence of tokens.
Could you also do this for music and specifically sound synthesis? It would be awesome to vibe synthesize sounds and then see the VSTi parameters surrounding it.
Impressive work. But the problem is not the 30 tok/s which is fine for agentic coding and chat.
It's prefill; slow prefill kills agentic workloads dead.
If you have 100,000 tokens at ~150tok/s per the OP, you're looking at:
Which is quite a wait indeed.I wonder if this could be usefully mitigated with a combination of prompt (prefix) caching and an agent that let you control what the prompt prefix consisted of. The goal would be to incur that slow prefill once to build the prompt cache, then have subsequent prompts consist of mostly this fixed prefix plus specific instructions.
For a language like C++ where modules are split into definition (.h) and implementation (.cpp) parts, one choice of prefix would be all the header files for the project (which aren't likely to change much).
More generally the idea would be to have an agent that had cached-prefix reuse as it's primary context management goal.
Another possibility, to support caching of files that have since changed, would be for the agent to build the context as a fixed prefix reflecting some or all of the codebase in its start-of-session state, then append any changes to that, with appropriate prompting to only use the latest definition of a function.
e.g.
Say file A initially contains functions X, Y and Z, then the prompt prefix is built to include X Y Z. If the user then modifies Y -> Y', then just add that to the context, so that the cached prefix is unchanged, giving X Y Z Y'.
Most people won’t be dumping 100K tokens into it at once, but I agree that all of the prefill time that adds up during a session becomes a lot to account for.
This is also a problem for all of the Mac local LLMs. Macs are a great way to get a lot of high bandwidth memory, but their compute is very far behind current gen dedicated GPUs. Some of the expensive Mac Studio setups allow you to run very large models with usable tokens/s, but you can be waiting a long time for it to get to the point of generating those tokens.
When you're using OpenCode it's easy to reach 100,000 tokens after a while.
A quick search say that this is a standard feature you cache the prefill and load it at PCIe bandwidth so it should be about 0.2s
I was just looking into this and was worried about the fan setup. Interesting that he was able to solve it with good results.
In case anyone is interested, I’m using PCIE passthrough on a FreeBSD host to a Linux guest with an older Pascal card. It’s worked great and I’ve been thinking about putting a nicer card in there. The SXM route seems great, but I’ve been burned (almost literally because of the heat) by DC components before.
> And yes, if you want the absolute best, Opus 4.8 exists. It also costs more per 20 minutes of heavy use than I paid for this entire GPU and adapter setup combined. But the gap is shockingly small.
I don't think this is a fair characterization of the situation. I use frontier models via API pre-paid tokens every single day, and I can barely rack up $100 per month. The fact that we figured out how to burn double this in 20 minutes is impressive, but I don't think it reflects the reality that many are experiencing right now. There are some exceptionally gluttonous approaches to harnessing LLMs that I think are serving as convenient straw men in these discussions.
Paying for the API will almost always be more economical than self-hosting equivalent infrastructure. I am not against self-hosting, but the article suggests a primarily economic motivation for this effort. If you are consuming fewer than 10^9 tokens per month, I really don't think it's worth your time to try and compete with the hyperscalars. Most of the money is to be found in the integration of this technology with existing businesses.
I use hosted providers myself, but I can churn through $100 worth of tokens in half a day even with cheap models like Deepseek easily. If someone's use is as light as yours, then sure - grab a subscription and you'll save far more. For higher use it will come down to how cheap your electricity is whether it is worth offloading at least some of it (for me it's not, FWIW)
Could you share a bit about what you’re working on or what type of projects require that much usage? Is it hobby, production, revenue generating?
A mix. I have hobby projects that churn through that much when I don't need the tokens for others things. I also have projects for clients that easily consumes those levels. As well as a stealth-ish potential startup. Currently I'm at 4 different subscriptions + more than I'd like in spend via OpenRouter...
What multiplies it very quickly is when you start feeding them with test suites and "Ralph loops" that run until the test suites pass, or complex chains with lots of sub-agents being triggered.
If you're sitting there watching everything, it will be hard to burn all that much even if you're running multiple things in paralle.
Claude is something like $35 per million tokens. If I was using API pricing I could trivially spend $100 in a single hour long coding session, with /fast turned on in about 10 minutes. Not sure how you guys are using it.
Opus is normally $5 per mtok, no idea why anyone would use /fast if they were at all concerned about price. ($5 is still pricy though tbh)
Opus is $5 per mtok of input tokens, but $25 for output.
coding is the easy part of using claude
The AMD MI250X GPUs are also interesting - 128GB of HBM2E at 3TB/s, sometimes you see them second-hand for under $1k, the catch obviously is that it needs an OAM socket. Never seen an easy way to hook them up to a regular mainboard.
An additional complication is that MI250Xes are two GPUs in one package, so you need to connect the first and last x16 SERDES groups to the host, otherwise you'll only see one GPU (or it won't work at all, idk).
Also, the cheap HPE pulls on eBay need some proprietary HPE magic to work, and I have yet to see anyone figure that out.
These are interesting, and offer beefy through put. No point in adapting to a PCI lane thought, stuck behind the slot-bus bottleneck.
Ahh luckily this OAM socket will prevent me from spending money.
Great write-up, I've often considered these DC cards for a project and now you've convinced me to pick one up; you describe the price of the unit against what one spends on tokens and that does it for me.
Thats why I did it. I think it’s important to put things like that into perspective
The most interesting and perhaps useful for most would be how they control the fan. If you are thinking of doing this, you really want to get those fans under control, they are loud. For anyone thinking of these, v100s idle super high! 25-35watt with nothing loaded and easily 50w when a model is loaded.
Congrats! Most people won’t want to debug drivers, kernels, ACPI, adapters, and fan headers. But for those who do, the capability-per-pound is absurd.
Could probably avoid the crazy fan with a waterblock - I've seen a whole kit, v100 + PCIE adapter + block for £235. Yes, you'll have to pay for pump, radiators and radiator fans, but that should really quieten it down
Someone's already made such a kit as you describe to fit in a consumer PC case and work properly?
https://www.ebay.co.uk/itm/406939340557?var=677143153030&mke...
32 GB kit as well
https://www.ebay.co.uk/itm/406939344915?_trkparms=amclksrc%3...
Based on the title I was really hoping to see how this was used for gaming, but they just ran an LLM on it
They said in the beginning that it doesn't even have a video out, so you cannot do gaming.
I'm actually surprised there hasn't been a dedicated effort to support display offload to, say, the CPU's iGPU.
I'm sure manufacturers would love saving a dollar per card, and OEMs would appreciate eliminating the support calls from "I just bought a new $2500 gaming PC and no video" because they plugged the monitor into the iGPU instead of dGPU.
I've seen things where you have multiple video cards and can use one gpu to render to a framebuffer which is transferred to the other video card to output. I'm sure it adds latency, and it's probably unsupported... But no output doesn't mean can't do gaming... It just means gaming will be iffy.
There's some virtualized desktop server stuff too. Run a bunch of desktop sessions on a beefy computer and send a video stream to desktop players. With the right codec settings, the latency is probably ok for many games.
I thought you could run games by rendering on one GPU and outputting on another? Usually comes up with dual iGPU/dGPU setups, but could work here
Never a problem. RemoteFX does (did) everything you'd want. Make your OS, log in remotely through an accelerated client. The real problem is Microsoft did something around Windows Server 2008 R2 that killed performance (literally halved it) for RemoteFX. You're only now reobtaining the virtualized video performance we used to have back in 2008.
Same. With no new NVIDIA gaming GPUs this year, seems like an interesting problem to solve.
I don't think that is even possible, every piece of silicon on that chip that is required to do gaming is ripped out in favor of more compute cores.
That's the same price per VRAM GB as an arc pro B70
But with miles better support, thats why I went this route. Cuda is hard to beat
The real question: did your local LLM write this post?
There are many tells aren't there? There was clearly hard human work and experimentation here, but it's a shame the OP let AI do chunks of the writing. Once you see it, it's much harder to take the post seriously.
Not at all, no. I had this chat before about how I am one of those unlucky few that loved the way LLMs write nowadays since the mid-2000s.
Slowly but surely, I had to remove my beloved lists, emojis (though LLMs do less of that now, maybe I can incorporate them back), and emdashes.
I disagree. Not everyone has a good writing style. In those instances I think it is fair to default to llm recommendation. We may be allergic to it, but we saw one formulaic response too many ( though admittedly it does raise a question of whether HN was the intended audience for it ).
In any event, not all of us have a unique writing style worth preserving just like not all of us can write clear and clean code. Just saying.
I really wish it was more common to use AI for augmenting than authoring. Eg i find coding with LLMs neat when you primarily "talk" to it through code, by filling out structs, funcs, fields, etc - where it would use your changes as the template and then to work to effectively autocomplete the gaps. The more you iteratively write the less it fills in, but also the less it deviates from your intent, design, etc.
I feel like writing could use a similar harness, where it attempts to minimally reword the authors sentences, perhaps just tweaking grammar, spelling, etc. In the coding example i think the human code would be near unchangeable, the LLM would pivot around it - but in the writing example i think the human writing would have to be more mutable. I imagine it would be a configurable setting.
I've not really seen a system which focuses on this human<->LLM look, but it feels interesting to me.
In a sense, there is a clear market for it ( people want 'authentic' experience ). I can kinda understand it. I want pure linux experience without systemd, but I recognize that in the current ecosystem, it comes at a cost.
So the language harness makes sense to me, but corps are already cracking down on token use ( and such a harness would likely only add to the cost ). The other question is whether the people, who could benefit it would even recognize it as a problem though.
> I want pure linux experience without systemd, but I recognize that in the current ecosystem, it comes at a cost.
Running Alpine/Gentoo/Devuan isn't that expensive. (I'm assuming the cost is time/effort when I say this; let me know if there's another relevant metric)
It’s not about preserving a unique writing style. When I see LLM writing my brain automatically discards the content of the writing. To me, seeing LLM writing is equivalent to going to a high-end restaurant and getting served on generic paper plates. Sure, the food looks perfectly fine and there is, in theory, nothing wrong with a paper plate. Once you see that paper plate, however, you will question how nice that establishment actually is, because a lack of care for the plates undermines the quality of the food. You automatically categorize all establishments that serve on paper plates in a specific category, one that might make you concerned if you will get food poisoning that night. LLM writing is exactly the same way for me. I don’t know if this LLM-assisted piece of text is actually a Michelin three star establishment or has had several heath violations in the last year. However, I didn’t pay for it, so putting in effort to determine if it’s LLM-assisted writing from an expert or just LLM slop that isn’t from the purported author at all isn’t worth the time.
I’m much more willing to read typos and bad writing than LLM writing. If I want to read the LLM rewritten version, I can run an LLM over the original writing myself. I have not yet found true that anyone is better at prompting than anyone else in a way that suggests that I wouldn’t get substantially the same results myself. Thus, I don’t think providing the version that has passed through the telephone game is accomplishing something that couldn’t be done by readers later. I have spent the vast majority of my life reading the original writing styles of people and didn’t have an issue then. I’m not convinced a problem I had was solved when we started post-processing writing with an LLM.
I skim a lot. I skimmed this article and appreciated the author documenting their process. I am indifferent to LLM or human writing for technical content. I suspect I skimmed most of the LLM parts, but judging writing quality was not why I read this post, I read it because I was curious about how useful the GPU is, and if I could replicate the author's work. Some carefully written prose wouldn't have helped me do that any better. The prose in this article did the job.
This is mostly how I feel about it. If anything, the weird llm jitters served almost like punctuation markers. Still, I get why it riles some people up.
(TL;DR Can we just judge written works by their actual content?)
I’m really in the “who gives a shit” camp on something like this. A lot of people probably have an LLM punch up a blog post. It is good at turning bullet points and notes into prose, fixing run-ons, etc. Maybe I’m naive but I trust that the kind of person who posts a clearly noncommercial post like this on HN gives a crap enough that they read the final draft and confirmed it isn’t inaccurate.
This pearl-clutching about the mere use of AI regardless of how responsible or appropriate the use is, seems like a professor in 1985 throwing an essay back in a student’s face as “this was obviously printed from a computer and not typewritten like a PROPER essay! I can tell just by looking at it!”
despite gaming being used in the title, it is not mentioned in the article, but i'm curious how this performs.
i've ran some multi vendor frankenstein setups before and sometimes it even works, so i'm curious to hear your experience with it.
Some context:
- In 2017, the v100 was a ~$10,000 GPU. I believe there was a PCI-e version but this is probably so cheap because SXM2 is going to be harder to use;
- A 5090 has 1800GB/s of internal memory bandwidth (compared to 900GB/s in the 9 year old GPU). Of course a 5090 is substantially more expensive;
- A 5090 has ~21k CUDA cores vs ~5k;
- The current $10k NVidia GPU is the RTX 6000 Pro w/ 96GB of VRAM. It has slightly more CUDA cores but it otherwise pretty much just a 5090. This is unsurprising. NVidia uses VRAM for market segmentation.
Consider this: in 5-10 years, the trillions spent on AI data centers will likewise be sold for scrap most likely. That's how short the runway is for OpenAI and Anthropic to recover that investment.
Anyway, I'm kind of impressed the author managed to get this all to work. I don't think it even would've occurred to me that someone had made an SXM2 adapter, particularly because it's not even used anymore. Like props to whoever did that.
> Consider this: in 5-10 years, the trillions spent on AI data centers will likewise be sold for scrap most likely. That's how short the runway is for OpenAI and Anthropic to recover that investment.
Even more interesting: it'll devalue all of SaaS and the entire US tech sector.
We might have just shot our most valuable non-AI tech products in the foot.
How so? I understand that flooding the market with physical goods will reduce prices and thus profits. But how would that also reduce the nonphysical SAAS stuff?
> But how would that also reduce the nonphysical SAAS stuff?
The resulting economic crash will affect everyone, we're (IMHO) looking towards a dotcom-bust level wipeout. And many SaaS and other companies run asset-lean (i.e. they have no server hardware because that's all cloud, no real estate because it's all either wework or conventionally rented), margin-lean (the VC business model requires that, as the basic recipe is to achieve market domination by burning cash) and cash-lean (often enough, it's less than a quarter of expenses on the bank accounts).
All that "lean-ness" looks great on an investor's quarterly release sheet: no massive amounts of wealth tied up in assets and no cash sitting around on bank accounts that could be released towards investors as dividends or, if it comes from third parties, costs the company interest... but it prevents resiliency against crises.
> We might have just shot our most valuable non-AI tech products in the foot.
Counterpoint: the fiber buildout during the dotcom boost. That crashed the economy pretty hard when the bubble burst, but we are still benefitting from all the dark fiber that was arranged for and built out back in that era. A lot of today's ISPs were able to grab up that fiber after the bust for cents on the dollar.
Assume that OpenAI and Anthropic go bust, which at least one of them likely will, and possibly a fair few of the datacenters that are under construction will also collapse. Someone will be able to snatch these physical assets again for cents on the dollar and run open-weight models on them or train new ones.
The problem isn't (and no, this is not an AI tell, everything I write here got typed on a 2022 M2 MBA by hand) the assets, they will be put up for productive usage, just as with any other large bankruptcy or bubble in history. The problem is the "IOU" that is being passed from one hand to the next like a hot potato. Assuming a recovery of, maybe, 20% after the collapse, at 1.6 trillion dollars of assets under management by some kind of private investment/debt we're looking at about 1.3 trillion dollars in valuation that is going to be wiped out.
And given that a lot of the investment market is actually backed by pension funds... this is going to be a bloodbath. Not only will there be a lot of people laid off in addition to the layoffs we already saw "due to AI", but when the pension funds and thus their payouts collapse? We'll see retirees flooding the employment markets who just try to make a living, rendering the situation for everyone else even worse. Flipping burgers used to be a gig for students, these days students compete with people of all ages desperate to survive - and thus desperate to undercut others in wages.
Another problem will be the capacity buildout in the semiconductor industry. It's already heading toward an oligopoly after numerous boom-bust cycles: you only have two and a half GPU chip vendors (NV, AMD, Intel), two vendors of general-purpose CPU vendors (Intel and AMD - I exclude Apple because they do not sell their CPUs to any third party and ARM because 99% of non-Apple ARM chips do not go towards servers, desktops and laptops), three RAM manufacturers (Samsung, SKhynix, Micron) and two and a half physical chip manufacturers (TSMC, Samsung, Intel). When the AI bubble bursts, it will be one of a hell of an effort to prevent at least one actor from going bankrupt.
[1] https://prospect.org/2025/11/19/ai-bubble-bigger-than-you-th...
I bet 3 years, but otherwise agree.
AI written posts will kill HN.
AI didnt edit a single word of this post.
But could you game with the GPU? Or is that purely a drivers issue?
I assume you can game with it if you use modded drivers. At least with CMP 50HX that is possible https://videocardz.com/newz/nvidia-cmp-50hx-turned-into-gami...
Wow. V100. That brings back memories. Way to go.
Volta (and Pascal, which I'm using) should still be supported with driver 580 as long as you don't use the open modules, and you can use up to cuda 12.9 and cudnn 9.10.2. No need to limit yourself to an old kernel.
It is. We still run quite a few of them in prod and with 580 drivers they run just fine. Very useful GPUs still.
> The compute is still real. The VRAM is still real. And the memory bandwidth is where it gets genuinely surprising.
sigh
> The compute is still real. The VRAM is still real. And the memory bandwidth is where it gets genuinely surprising.
Had to stop there. Annoying. I can't stand AI use for writing. It makes any otherwise great article feel so disingenuous.
Agree. But I have not used AI in the slightest.
Some of us just write that. AIs had to learn it from somewhere.
What a difficult world you must live in these days
While I don’t disagree with their sentiment, I’m far more annoyed with it than the AI writing.
Yeah. I get that many HN comments are just complaints (heck mine was too and just as negative and shaming). But how bad of a day must you be having to try to shame someone about how they choose to write up an experience they thought was neat. Whatever, free speech and all that. Hope OC's day gets better.
It doesn’t read like shaming to me. It’s, in the grand scheme of HN comments, definitely on the more constructive side of the criticism. Maybe it could have been reworded, but I think the author of the post could very easily find it actionable in the future. I too had to stop reading the article at that point, so I think if the author wants more people to read, my advice for them is to just write like themselves. We’ve entered the start of a new Instagram filter age where many people feel they need to have LLMs reword their writing presumably for the same reasons as the original filter age. I share OC’s sentiment of pushing against the recent trend of implicitly shaming people for their individualistic writing styles.
Every single HN post has the same comment now.
Only because so many of the articles posted on HN now are AI-written, and badly, too. A lot of tech people are so impressed with LLMs’ capabilities in code that they fail to recognize how bad they are at writing enjoyable prose. And it feels like a chore to write out a whole blog post by hand when the machine could do it for you! But the result we get is so, so much worse and more annoying.
I dislike AI prose too, the cadence of it really rubs me the wrong way, but, that said we've had a lot of great, informative articles lately, written with AI help, where you just have to grit your teeth and get through them to get the underlying knowledge.
I don't think that commenting on every article is going to make the posters suddenly decide to go back and rewrite it by hand. Some of them probably don't even speak English natively. The comments are getting more tiresome than the AI prose at this point.
Hopefully in a year or so the LLM output won't be so janky and obvious, so this might just be a phase everyone has to pull through.
That line was the exact moment I also realized the post was AI written. I kept reading though, but I am left constantly guessing at which key details might be pure hallucinations.
FYI, not a single line was AI written. If there is a hallucination, it’s fully mushy brain sourced.
Honestly, the default styles are pretty bad. I use Claude in my scientific writing in a very specific way. 1. I write a paragraph. 2. I put Claude into concise style mode. I then ask Claude to revise for clarity.
I can write competently, but it's natural direction is towards emotional rhythmic flow that can convey emotion/passion...but which for scientific writing, can get in the way of clear clean communication. So, I write what I mean,and Claude straightens it out...and these days (i.e. not last year), it doesn't lose my meaning that often. And since I wrote it first, these AI-isms appear less frequently, and if they do, I revise them away.
Wait a few years, everyone will be able to put one at half the price.
Some resell group is going to have to make this easier. The shear amount of these cards otherwise heading towards the landfill is staggering. That is if Big Tech don't destroy them to prevent model weights from leaking.
Things like this have started to show up on eBay: https://www.ebay.com/itm/198383386991
terrible deal
Yeah. Not linking as an endorsement -- I do think it's cool, but it's not worth it for that price.
How would destroying the GPUs prevent the model weights from leaking? By the time you get your hands on them the memory is powered off for a long enough time that a cold-boot style attack is impossible.
Would you bet your trillion dollar company on that? Or would you smash up the garbage [to you] memory chips to be sure.
It's volatile memory, not flash.
> The shear amount of these cards otherwise heading towards the landfill is staggering.
The thought of throwing away working cards sounds so bizarre to me. I can't believe companies would dispose them into the landfill like that, it is at least worth giving away for refuse.
There’s a long history of corporations doing evil things to ensure their business model succeeds
I genuinely hope that is the case. The market is absolutely bananas now. I actually now own devices that went up in 'value' since purchase. This is not normal ( and a little scary ). This, on the other hand, is an invitation to properly recycle otherwise unwanted hardware.
Isn't this the same thing with 32 GB already on a PCIe socket?
https://www.ebay.com/itm/166850431555
kinda, they put that on a PCIe socket, but it's passive. Meaning no fan. If you try inference on that it overheats in 1 minute unless you have it inside a server case.
> The compute is still real. The VRAM is still real. And the memory bandwidth is where it gets genuinely surprising.
Because humans write exactly like this /s
Where do you think llms learned to write that way?
You can also look at past posts by the same author (before LLM usage proliferated) if you’re curious.
The project is still very cool, but it’s a little less enjoyable to read when everything sounds the same. It would be just as annoying for people to manually write in a corporate/marketing style, because humanity is what makes the small web interesting.
https://blog.tymscar.com/posts/privategithubcicd/
I’m glad I’ve started this blog before the AI wave so I can prove people I’m just weird at writing.
It grinds my gears how so many people just talk about my writing style instead of the content.
> I’m glad I’ve started this blog before the AI wave so I can prove people I’m just weird at writing.
Your previous blog posts didn't trigger any LLM detector (go on - check for yourself).
This, setting aside the llm issue, it is dealing with hardware in ways that -- one would think - would be celebrated on HN of all places. But we focus on presentation.
Because their custom training data contains an emphasis on such verbiage. It doesn't come from the God-knows-how-many TB of web content the model is pre-trained on. There, such phrasing is only a drop in the sea. But the "yes, you're right" phrases, the em dash, etc., come from the later stage, for which content is created according to some (probably overprecise) guidelines.
Right. The overuse of "genuinely" most of all. Seems like they put Claude through a few good rounds of training to always answer questions about its consciousness, thoughts, etc., with something about how it's "genuinely unsure," and as a result, the model learned to use "genuinely" as an intensifier in all sorts of inappropriate contexts.
Oi, I personally use adverbs everywhere. Genuinely, kids these days.
Marketing content.
> Where do you think llms learned to write that way?
Not from individual human content, that's for sure - maybe MLM marketing copy? Sleazy 4AM ads?
I mean, every time this response comes up, I keep asking the person to point at something written prior to 2022 that gets 80%+ on the LLM detectors, and yet no one can find anything.
Maybe you, postalrat, can find something written in this style that was published prior to 2022.
I have written the blog post. I know empirically that I have used 0% AI while writing it. I also know LLM detectors are total BS and they don't really work. I have tried a couple on this exact blog post, and QuillBot, for example, gave me 0% AI detected on it.
I have then used a blog post of mine from 2021. QuillBot gave me 8%...
The King James version of the Bible came out at almost 100% AI generated a while ago. It was the HN front page.
Stop thinking that if someone writes in a way that is fun or looks like what you would think an AI writes, then it is AI generated. Loads of the time it is, but sometimes it's not, and it really hurts those like me.
It's a function of the LLM "thought process"! It's not really modeled after human speech. It is in short segments but not long form, same reason you see the same rather odd nuances in LLM generated code.
If they way you thought was to run a bunch of if statements, generate content, then feed that content back to get a "score" of what seems the most plausible, run the if statements again, and adjust / merge responses, then you would write similarly. The recognizable cadence of LLM generated content is pretty clearly the result of a lot of if statements being fused together.
X is Y. Z is Y. And Alpha is genuinely Beta.
Classic LLM writing style.
You know what the sad bit is? Humans do write exactly like that. That's not even particularly egregious StalkedIn marketroid speak.
There's interesting stuff in this writeup but it sure seems like most of it was written by an LLM.
A little bit of local copium but neat read.
Isn't a rasbpi with 16gb of RAM $300 now?
The latest Raspberry Pi 5 has one 32-bit channel (2x 16-bit subchannels) of LPDDR4X-4267 SDRAM giving 17.1GB/s of bandwidth, 52x less than this GPU. Never mind lacking the CUDA and Tensor cores, so the FP16 performance is 102x less (307 GFLOPS vs 31.4 TFLOPS). So for £200, there's absolutely no comparison for this specific use-case.
Yeah thats what I'm saying. How is it so cheap????
V100 GPUs are e-waste.
I don't understand what point you're trying to make here? Are you talking about the price of RAM?