I got an old Dell T5820 64GB RAM and put in 2x V100 32GB GPUs. All in was around $1800 USD.
Runs Qwen3.8-27B Q4 on one GPU at ~55 tok/s. I also put MiniMax H3 on the other GPU for generating images and video, and the kids have fun with that. I use this box for all my personal leisure coding projects, but stick to bigger models over API for real work work.
I have a 5090, once you start splitting weights/cache amongst devices or start having to rely on unified memory you quickly enter the land of diminishing returns and tradeoffs around cost, power consumption and speed. I'd just go for whatever the cheapest single device that will fit your chosen model weights + context at the speed and price you are comfortable with, which for most folks will probably be a single GPU and a model in the 30b neighborhood (qwen/glimmer)
I have a pair of ASUS GX10 systems [0] (DGX Sparks in a different enclosure) in a cluster running DeepSeek-V4-Flash-0731 [1] with tensor parallelism. I managed to snag them early this year when prices were merely stratospheric, not mesospheric.
Even at half of today's retail price they're not remotely cost-effective compared to cloud models for my single-developer workflow, and idle power draw is quite high with the ConnectX-7 NICs powered on for clustering. (Mine idle around 45-50W each. Powering off the NICs cuts that roughly in half, but they're still very power-hungry compared to an Apple Silicon Mac or even a Framework Desktop.)
I learned a lot building the cluster, which was my main objective. I could probably sell them for a nice profit, but I'm still enjoying using them.
my 'local llm on a budget' has cost me just about $2,400 in total, spread over a bout 3 years of accumulating stuff.
I got my first 7900xt 20gb on a good sale shortly after they came out in an open box deal on ebay for $590.
After another 7900xt 20gb popped up on facebook marketplace recently for $420, I sniped that and then got a microcenter deal on an asus proart board, 32gb of ddr5, and a 9950x for $1050.
A couple more deals for used cases, an older 1tb nvme ssd I had lying around, and a nice PSU finished the build.
I run Qwen 3.8 27b on linux and have tuned the llama.ccp / rocm setup substantially with the help of codex and claude. I get ~80-110 tok/s depending on all kinds of factors and full 256k context windows with a 6-bit quant.
It's surprisingly capable and I love that I can run it indefinitely, tuning as it goes to behave more like a frontier model.
The best combo I've found is running Qwen3.8-27B-MXFP4 (radiance) on 2x R9700 in DeepSeek Harness on a cloud VM with a search subscription. I have a Cloudflare tunnel from my home vLLM to DSH VM. Since the dense model has high intelligence but not breadth+depth of knowledge being close to the internet with subscription ensures fast searches that quicken loops. It's exceedingly hard to get API models with low latency for interactive design work and this setup works for me.
I'm running a local LLM on my system below. It's alright, free intelligence for simple daily tasks. I still have opus sub for deep intelligence applications.
Local LLM loaded: Qwen 3.8 27B (qwen3.8:27b), served through Ollama.
MY SYSTEM:
ASUS ROG MAXIMUS XI HERO Motherboard
Intel® Core™ i9-9900K × 16
64GB DDR5 RAM
RTX 3090 with 24GB VRAM
The plan is to use it with Open WebUI, PostgreSQL with pgvector and Apache AGE, and my own database schema, and then have it be my complete second brain, running completely privately on my own hardware and always being available over Tailscale.
Since I still have 20-30 GB RAM to spare after loading that main orchestrator I will also have a coding worker that will do most of the grunt work, coordinated and commissioned by the orchestrator. For now I have readied https://huggingface.co/pyros-vault/Ornith-1.5-35B-A3B-BigBan... (a mix of Qwen 3.5 35B and BigBang) for this task.
For now I still use Codex and ChatGPT Pro 200 to do all my work, but the plan is to move over to local LLM work as soon as I have finished everything I realistically need frontier models for. Which is going to be for a little longer.
I have a server in the closet running AMD Ryzen 5 3600, 9070 (non-xt) and 64GB of DDR4-2400. Running Qwen3.8-flash-next on Strata and pushing ~60 t/s decode and reading ~1.5k t/s prefill. Works well for the tasks I'm doing with it. For external access I'm exposing the OpenAI compatible endpoint via a CloudFlare tunnel and using it on a hosted Viberglass instance.
The cost was around 850 for the GPU and memory a few months ago, other hardware has been in the closet running my server for around 5 or so years.
I have a 4080 Super with 16gb of ram and I run Ollama and have Qwen 3 8b and Qwen embeddings.
I am currently not the serious about local inference, but that could potentially change in the future if better models drop or I get a new PC with a unified memory structure.
I have a 3090 w/ 24G VRAM, it supports flash attention and bf16, but is a bit old. I dont run generative models on it as they would be slow and inaccurate, but I do run embedding models, OCR, and system one (decision) models on it.
OK, that gpu-unlock is something I've never seen before and...wow. I did some research and NVidia really did kneecap their RTX for market segmentation. Grrrrr
I'm running an RTX 5090 (32gb VRAM) and Strix Halo (128gb unified memory). I haven't had much time to tinker with my AI setup so I'm still running Qwen3.6-27B (NVFP4 quant) on the 5090. The prices today are just insane.
Not sure if I qualify for the "day to day work" part, but I have an inference PC with an RTX 3060 12GB and 32GB DDR4. I use it to run Qwen3.6-35B-A3B and Gemma4-26B-A4B with llama.cpp. Just in chat mode.
I grabbed a minisforum before they got prohibitively expensive ( and before that a framework variant, but I got a dud from them ). If you can't do that you can estimate what you can run model wise and play with LMS ( some interesting options are now available -- in my recent tests GLM and Qwen are doing well for local stuff. )
Note.. you can always try to pool things ( not memory, that is super hard; devices as there are more small models now that you can squeeze onto things ).
A cluster of 4 M3 Ultra Mac Studios with 512 GB of unified memory each. Don't recommend. EXO labs is dead and trying to shard models is really hard. On their own, each Mac can generate text at usable speeds, but I assume that you trade memory for speed. I have never used an NVIDIA GPU, so I guess that you trade model capacity for speed. I would like to know what y'all Nvidia people think
I have "only" one Mac Studio M3 Ultra with 96 GB RAM. May I ask what spurred you into actually getting four of the most expensive and capable model at the time?
And what kind of work are you envisioning doing on them now or later, if you are not going to sell them? (Maybe even with a profit, I think.)
Thank you for this! I have been debating ordering the new M5 Studios, I am running 2 M4 Macbook Pro Max 128GB each with Exa Labs, I havent noticed issues but I also am not hammering a local model (I mostly do code reviews and some QA). I run the frontier labs models but have built my own memory connector that has enhanced those models drastically.
I have a Mac Studio M1 Ultra with 128GB of RAM, an M5 Pro MBP with 64GB of RAM, and until recently an M1 Pro MBP with 32GB of RAM. I also have two Mac Minis with 16GB of RAM, but they're mainly used for embedding models because they're so small.
I'm doing a bunch of batch work for an experiment with them right now, but my daily driver is a Gemini Pro subscription.
How useful is 64GB M* Mac for running models? It is strictly below <64 Gigs or something bigger like Qwen-Flash-Next or GLM5.3-Flash is available at usable speeds too?
I kinda want a Mac machine to test and publish few apps, but I can't justify 128 or 256gb budget right now.
I “upgraded” from an Ubuntu desktop with ryzen 9 and a 3090. I still have the box but it runs like a sauna.
My new set up is a Mac Studio M5 Mac 128gb. Surprisingly it’s still hard to find decent models to fit on 128gb, but Qwen3.8 27b runs FAST so it’s really great as a personal agent that isn’t quantized to heck
I have an old 48 vcores xeon (5318Y) with 512GB (ddr4) of ram which I bought in 2023, it runs deepseek r4.1-flash; it's very slow; but it works and that's all I need for now.
Local clusters built around old GPUs. I have had cluster of P40 GPUs, MI50s gpu, and now I have a cluster of cmp170hx. I'm always looking for the next cheap GPU dump and then I get as many as I can before it get's popular.
same setup -- I run openai/gpt-oss:120b with decent performance for prototyping
For production and an alternate dev environent I've configured and equipped a Dell PowerEdge XE7745 server that holds 8 NVIDIA RTX Pro 6000 GPUs. Again, running openai/gpt-oss:120b (for now)
I run a M5 Max macbook with 128gb of RAM, 18 cpu cores, 40 gpu cores. It is a beast for local inference, allows offloading of execution. Still use frontier for design / spec work to hand off to the local models to execute.
The person you are responding to is not rich by any means in terms of tech standards. If you work at a big tech company (not even a FAANG), you make very decent money in good times and bad.
They could have financed this purchase with their yearly bonus or selling some equity that vested.
7000 dollars is definitely a lot, but not impossible for a tech worker to save up for something worthwhile. some people spend that yearly in baseball alone
I bought a Framework Desktop mainboard only, 128gb AMD 395+ setup, and put it in an existing case with PSU i had on hand. I had a fan for it. It was $2699 +$15 expedited shipping. The case and PSU/etc was probably $200 more (It's an overkill 850w SFX PSU and cheap but great ITX case from Deepcool).
I started with Ubuntu but pivoted to Fedora over the summer after a recommendation from a security engineer friend. And it is mostly easier.
Key changes for other distros are how to load drivers/apps and get running. Ubuntu has a recent CVE issue but worked fine. CachyOS/etc set up for more Windows like UI and transplants was fine but not focused on the Stricx Halo setup. Although I do see close to RTX 5070 performance in most games I've tested.
I was running LM Studio and some other runtimes for ease of setup and model switching. Once you settle on a runtime and model setup you like it is easy to set it up, let it run, and manage as needed.
Last week I set up GUFO, as it targets the AMD Strix Halo hardware and is efficient for Qwen Flash Next. I could probably get the work done with Qwen 3.8 27B but I like how QFN thinks and lets me follow along. Starting at about 260k context, I see 88gb of RAM usage when loaded and about 108gb when in use, depending on context overhead. I do have to restart the model every so often as Copilot Chat in VS Code starts failing.
I have also run Lemonade, llama.cpp directly, vllm, and more. Custom or bespoke runtimes for your hardware are key, like Srata or GUFO. I need to look at llama-swap. LM Studio is easy and with their Tailscale backed link feature you can stay connected from any device.
IF it supports OpenAI/etc back end you're mostly there. Get the model running and connect from your preferred harness/etc. GUFO works well with Copilot Chat in VS Code. I need to connect it to Pi and openhands/etc later this week once I get this test project in a good spot.
Qwen 3.8 27B, Gemma 4 26B A4B variants, whisper, and more run well. A Spark or bank of GPUs would do better, of course, but not for under $3k for 128gb of memory. I have tested adding an Nvidia 3060 12gb to the Strix Halo but the iGPU is turned off by Fedora and I didn't have more time to solve.
My secondary Fedora AI system has the 3060 12gb and a 5900XT (16c/32t) plus 64gb of DDR4-3200. I need to get Strata up on it and see how well medium sized models work on the mixed memory. I bet MOE models will work well. I have the hardware so will use it but renting capacity is much cheaper if you just want to mess around and get used to the tools.
Happy to answer questions about how I got here. I'm an infra guy, not a software dev, and what I'm doing with AI at home could definitely be 5x faster with a real dev and dev process.
Running Qwen 3.8 27B on a 5090, 8bit quant and 160k context fills the card but I get about 200 tps. Not as good as claude but its the first local model I have tried where it could easily replace claude. The local setup takes about twice as long as claude to complete a task to the same output quality. Electricity is expensive here so running a 600w gpu 24/7 ends up costing about the same as just buying credits. Free heating for winter and no spying are still worth it.
I went for the 5090 for the software support but I have so many issues running cuda on fedora. Kinda wish I had bought a Nvidia spark or something that was an AI only machine.
Single RTX 4080 (16GB VRAM) on a NixOS Linux workstation running llama.cpp (llama-server on CUDA 12.9).
The biggest unlock on 16GB cards lately has been MoE models plus quantized KV caches. Right now my daily driver is gemma-4-26B-A4B-it (UD-Q4_K_M GGUF, so you get 26B knowledge capacity at 4B active-parameter speed) running alongside a small dedicated llama-server embedding process on the same GPU.
With -fa on (flash attention) and -ctk q4_0 -ctv q4_0 KV-cache quantization, I can keep a 32k context window (-c 32768 -np 2 parallel slots, -ngl 23 when the F16 mmproj vision projector is loaded, -ngl 26 text-only) comfortable without OOMing. For denser single-slot coding runs I swap to Qwen3.8-27B-IQ3_M with the same q4_0 KV cache flags.
Zero desire to build a multi-GPU space heater at home yet - q4_0 KV cache + 26B-A4B MoE stretched what a single 16GB card can usefully do way further than I expected a year ago.
I got an old Dell T5820 64GB RAM and put in 2x V100 32GB GPUs. All in was around $1800 USD.
Runs Qwen3.8-27B Q4 on one GPU at ~55 tok/s. I also put MiniMax H3 on the other GPU for generating images and video, and the kids have fun with that. I use this box for all my personal leisure coding projects, but stick to bigger models over API for real work work.
I have a 5090, once you start splitting weights/cache amongst devices or start having to rely on unified memory you quickly enter the land of diminishing returns and tradeoffs around cost, power consumption and speed. I'd just go for whatever the cheapest single device that will fit your chosen model weights + context at the speed and price you are comfortable with, which for most folks will probably be a single GPU and a model in the 30b neighborhood (qwen/glimmer)
I have a pair of ASUS GX10 systems [0] (DGX Sparks in a different enclosure) in a cluster running DeepSeek-V4-Flash-0731 [1] with tensor parallelism. I managed to snag them early this year when prices were merely stratospheric, not mesospheric.
Even at half of today's retail price they're not remotely cost-effective compared to cloud models for my single-developer workflow, and idle power draw is quite high with the ConnectX-7 NICs powered on for clustering. (Mine idle around 45-50W each. Powering off the NICs cuts that roughly in half, but they're still very power-hungry compared to an Apple Silicon Mac or even a Framework Desktop.)
I learned a lot building the cluster, which was my main objective. I could probably sell them for a nice profit, but I'm still enjoying using them.
[0] https://www.asus.com/networking-iot-servers/desktop-ai-super... [1] https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-0731
my 'local llm on a budget' has cost me just about $2,400 in total, spread over a bout 3 years of accumulating stuff.
I got my first 7900xt 20gb on a good sale shortly after they came out in an open box deal on ebay for $590.
After another 7900xt 20gb popped up on facebook marketplace recently for $420, I sniped that and then got a microcenter deal on an asus proart board, 32gb of ddr5, and a 9950x for $1050.
A couple more deals for used cases, an older 1tb nvme ssd I had lying around, and a nice PSU finished the build.
I run Qwen 3.8 27b on linux and have tuned the llama.ccp / rocm setup substantially with the help of codex and claude. I get ~80-110 tok/s depending on all kinds of factors and full 256k context windows with a 6-bit quant.
It's surprisingly capable and I love that I can run it indefinitely, tuning as it goes to behave more like a frontier model.
Cool setup
The best combo I've found is running Qwen3.8-27B-MXFP4 (radiance) on 2x R9700 in DeepSeek Harness on a cloud VM with a search subscription. I have a Cloudflare tunnel from my home vLLM to DSH VM. Since the dense model has high intelligence but not breadth+depth of knowledge being close to the internet with subscription ensures fast searches that quicken loops. It's exceedingly hard to get API models with low latency for interactive design work and this setup works for me.
I'm running a local LLM on my system below. It's alright, free intelligence for simple daily tasks. I still have opus sub for deep intelligence applications.
Local LLM loaded: Qwen 3.8 27B (qwen3.8:27b), served through Ollama.
MY SYSTEM: ASUS ROG MAXIMUS XI HERO Motherboard Intel® Core™ i9-9900K × 16 64GB DDR5 RAM RTX 3090 with 24GB VRAM
I'm planning to run something akin to https://huggingface.co/Youssofal/Qwen3.8-27B-MTPLX-Optimized... on my Mac Studio M3 Ultra with 96 GB RAM. It will be the main orchestrator of my entire system.
The plan is to use it with Open WebUI, PostgreSQL with pgvector and Apache AGE, and my own database schema, and then have it be my complete second brain, running completely privately on my own hardware and always being available over Tailscale.
Since I still have 20-30 GB RAM to spare after loading that main orchestrator I will also have a coding worker that will do most of the grunt work, coordinated and commissioned by the orchestrator. For now I have readied https://huggingface.co/pyros-vault/Ornith-1.5-35B-A3B-BigBan... (a mix of Qwen 3.5 35B and BigBang) for this task.
For now I still use Codex and ChatGPT Pro 200 to do all my work, but the plan is to move over to local LLM work as soon as I have finished everything I realistically need frontier models for. Which is going to be for a little longer.
Deer flow and opencode are useful, also open viking.
I have a server in the closet running AMD Ryzen 5 3600, 9070 (non-xt) and 64GB of DDR4-2400. Running Qwen3.8-flash-next on Strata and pushing ~60 t/s decode and reading ~1.5k t/s prefill. Works well for the tasks I'm doing with it. For external access I'm exposing the OpenAI compatible endpoint via a CloudFlare tunnel and using it on a hosted Viberglass instance.
The cost was around 850 for the GPU and memory a few months ago, other hardware has been in the closet running my server for around 5 or so years.
I have a 4080 Super with 16gb of ram and I run Ollama and have Qwen 3 8b and Qwen embeddings. I am currently not the serious about local inference, but that could potentially change in the future if better models drop or I get a new PC with a unified memory structure.
I have a 3090 w/ 24G VRAM, it supports flash attention and bf16, but is a bit old. I dont run generative models on it as they would be slow and inaccurate, but I do run embedding models, OCR, and system one (decision) models on it.
I got pretty good results with my 3090 on Qwen 27b. Yeah, the rack is completely unusable when you launch the llm, but i wouldn't say its slow.
Btw, fellow 3090 user, some time ago i saw this: https://github.com/Zagitta/gpu-unlock
Can't try it personally because windows, but maybe it will help you.
OK, that gpu-unlock is something I've never seen before and...wow. I did some research and NVidia really did kneecap their RTX for market segmentation. Grrrrr
Wow, this seems to speed up inference by 68% or something for me. Holy shit.
I'm running an RTX 5090 (32gb VRAM) and Strix Halo (128gb unified memory). I haven't had much time to tinker with my AI setup so I'm still running Qwen3.6-27B (NVFP4 quant) on the 5090. The prices today are just insane.
Not sure if I qualify for the "day to day work" part, but I have an inference PC with an RTX 3060 12GB and 32GB DDR4. I use it to run Qwen3.6-35B-A3B and Gemma4-26B-A4B with llama.cpp. Just in chat mode.
I grabbed a minisforum before they got prohibitively expensive ( and before that a framework variant, but I got a dud from them ). If you can't do that you can estimate what you can run model wise and play with LMS ( some interesting options are now available -- in my recent tests GLM and Qwen are doing well for local stuff. )
Note.. you can always try to pool things ( not memory, that is super hard; devices as there are more small models now that you can squeeze onto things ).
Got a refurbished M1 Max 64GB for less than USD 1,000 some years ago; runs medium models at decent speed. Large ones are out of the question.
A cluster of 4 M3 Ultra Mac Studios with 512 GB of unified memory each. Don't recommend. EXO labs is dead and trying to shard models is really hard. On their own, each Mac can generate text at usable speeds, but I assume that you trade memory for speed. I have never used an NVIDIA GPU, so I guess that you trade model capacity for speed. I would like to know what y'all Nvidia people think
I have "only" one Mac Studio M3 Ultra with 96 GB RAM. May I ask what spurred you into actually getting four of the most expensive and capable model at the time?
And what kind of work are you envisioning doing on them now or later, if you are not going to sell them? (Maybe even with a profit, I think.)
Thank you for this! I have been debating ordering the new M5 Studios, I am running 2 M4 Macbook Pro Max 128GB each with Exa Labs, I havent noticed issues but I also am not hammering a local model (I mostly do code reviews and some QA). I run the frontier labs models but have built my own memory connector that has enhanced those models drastically.
I have a Mac Studio M1 Ultra with 128GB of RAM, an M5 Pro MBP with 64GB of RAM, and until recently an M1 Pro MBP with 32GB of RAM. I also have two Mac Minis with 16GB of RAM, but they're mainly used for embedding models because they're so small.
I'm doing a bunch of batch work for an experiment with them right now, but my daily driver is a Gemini Pro subscription.
How useful is 64GB M* Mac for running models? It is strictly below <64 Gigs or something bigger like Qwen-Flash-Next or GLM5.3-Flash is available at usable speeds too?
I kinda want a Mac machine to test and publish few apps, but I can't justify 128 or 256gb budget right now.
I “upgraded” from an Ubuntu desktop with ryzen 9 and a 3090. I still have the box but it runs like a sauna.
My new set up is a Mac Studio M5 Mac 128gb. Surprisingly it’s still hard to find decent models to fit on 128gb, but Qwen3.8 27b runs FAST so it’s really great as a personal agent that isn’t quantized to heck
Try out qwen.8 flash next via omlx/splash.
The Vardsan97 balanced ple8 version is quite good and fast.
I’ll check it out. I tried the unsloth version but it couldn’t handle anything larger than an 8k context window unless I used a 2 or 3 bit quant
holy crap, thank you. Its actually running at 54 tok/s which is incredible for local inference.
Qwen Next 3.8 @ 3 bit IQ on x2 Quadro 6000 (sm_75) on a Dell T5820 (W-2140B Xeon, 128GiB DDR3). Strata makes the most of the hardware.
heavily customised agentic framework: https://github.com/connollydavid/host gets good work out of the weights
I have an old 48 vcores xeon (5318Y) with 512GB (ddr4) of ram which I bought in 2023, it runs deepseek r4.1-flash; it's very slow; but it works and that's all I need for now.
Local clusters built around old GPUs. I have had cluster of P40 GPUs, MI50s gpu, and now I have a cluster of cmp170hx. I'm always looking for the next cheap GPU dump and then I get as many as I can before it get's popular.
I run a cluster of two Dell Pro Max GB10's 128GB unified mem per node 200G NIC ConnectX
what model can you run with those?
same setup -- I run openai/gpt-oss:120b with decent performance for prototyping
For production and an alternate dev environent I've configured and equipped a Dell PowerEdge XE7745 server that holds 8 NVIDIA RTX Pro 6000 GPUs. Again, running openai/gpt-oss:120b (for now)
I run a M5 Max macbook with 128gb of RAM, 18 cpu cores, 40 gpu cores. It is a beast for local inference, allows offloading of execution. Still use frontier for design / spec work to hand off to the local models to execute.
How did you get so rich? Inheritance? Startup went IPO?
The person you are responding to is not rich by any means in terms of tech standards. If you work at a big tech company (not even a FAANG), you make very decent money in good times and bad.
They could have financed this purchase with their yearly bonus or selling some equity that vested.
7000 dollars is definitely a lot, but not impossible for a tech worker to save up for something worthwhile. some people spend that yearly in baseball alone
Sam Altman is going to put you all on a list. So much precious RAM wasted.
I feel very poor with my M2 MBP with only 16G of RAM
I bought a Framework Desktop mainboard only, 128gb AMD 395+ setup, and put it in an existing case with PSU i had on hand. I had a fan for it. It was $2699 +$15 expedited shipping. The case and PSU/etc was probably $200 more (It's an overkill 850w SFX PSU and cheap but great ITX case from Deepcool).
I started with Ubuntu but pivoted to Fedora over the summer after a recommendation from a security engineer friend. And it is mostly easier.
Generic setup doc here: (not fully updated with GUFO): https://github.com/Selicos/AI_Tools_Public/blob/main/docs/Fe...
Key changes for other distros are how to load drivers/apps and get running. Ubuntu has a recent CVE issue but worked fine. CachyOS/etc set up for more Windows like UI and transplants was fine but not focused on the Stricx Halo setup. Although I do see close to RTX 5070 performance in most games I've tested.
I was running LM Studio and some other runtimes for ease of setup and model switching. Once you settle on a runtime and model setup you like it is easy to set it up, let it run, and manage as needed.
Last week I set up GUFO, as it targets the AMD Strix Halo hardware and is efficient for Qwen Flash Next. I could probably get the work done with Qwen 3.8 27B but I like how QFN thinks and lets me follow along. Starting at about 260k context, I see 88gb of RAM usage when loaded and about 108gb when in use, depending on context overhead. I do have to restart the model every so often as Copilot Chat in VS Code starts failing.
I have also run Lemonade, llama.cpp directly, vllm, and more. Custom or bespoke runtimes for your hardware are key, like Srata or GUFO. I need to look at llama-swap. LM Studio is easy and with their Tailscale backed link feature you can stay connected from any device.
IF it supports OpenAI/etc back end you're mostly there. Get the model running and connect from your preferred harness/etc. GUFO works well with Copilot Chat in VS Code. I need to connect it to Pi and openhands/etc later this week once I get this test project in a good spot.
Qwen 3.8 27B, Gemma 4 26B A4B variants, whisper, and more run well. A Spark or bank of GPUs would do better, of course, but not for under $3k for 128gb of memory. I have tested adding an Nvidia 3060 12gb to the Strix Halo but the iGPU is turned off by Fedora and I didn't have more time to solve.
My secondary Fedora AI system has the 3060 12gb and a 5900XT (16c/32t) plus 64gb of DDR4-3200. I need to get Strata up on it and see how well medium sized models work on the mixed memory. I bet MOE models will work well. I have the hardware so will use it but renting capacity is much cheaper if you just want to mess around and get used to the tools.
Happy to answer questions about how I got here. I'm an infra guy, not a software dev, and what I'm doing with AI at home could definitely be 5x faster with a real dev and dev process.
Upgraded from a 3060 12G to a 4070 12G.
r9700s - running qwen 3.8 flash next with ~1m token kv cache
Running Qwen 3.8 27B on a 5090, 8bit quant and 160k context fills the card but I get about 200 tps. Not as good as claude but its the first local model I have tried where it could easily replace claude. The local setup takes about twice as long as claude to complete a task to the same output quality. Electricity is expensive here so running a 600w gpu 24/7 ends up costing about the same as just buying credits. Free heating for winter and no spying are still worth it.
I went for the 5090 for the software support but I have so many issues running cuda on fedora. Kinda wish I had bought a Nvidia spark or something that was an AI only machine.
Single RTX 4080 (16GB VRAM) on a NixOS Linux workstation running llama.cpp (llama-server on CUDA 12.9).
The biggest unlock on 16GB cards lately has been MoE models plus quantized KV caches. Right now my daily driver is gemma-4-26B-A4B-it (UD-Q4_K_M GGUF, so you get 26B knowledge capacity at 4B active-parameter speed) running alongside a small dedicated llama-server embedding process on the same GPU.
With -fa on (flash attention) and -ctk q4_0 -ctv q4_0 KV-cache quantization, I can keep a 32k context window (-c 32768 -np 2 parallel slots, -ngl 23 when the F16 mmproj vision projector is loaded, -ngl 26 text-only) comfortable without OOMing. For denser single-slot coding runs I swap to Qwen3.8-27B-IQ3_M with the same q4_0 KV cache flags.
Zero desire to build a multi-GPU space heater at home yet - q4_0 KV cache + 26B-A4B MoE stretched what a single 16GB card can usefully do way further than I expected a year ago.