2) The details say that the performance numbers require 2.7x the overall power, so around 13.5W. Power needed for all the memory is not explicit in the numbers.
3) The compiler is also simulated and it's a prototype, so it's not a drop-in replacement for anything like Nvidia ISAAC
4) 5W seems to refer to sub-4B models, so not the 100B param ones.
Statements like these often overlook that a human brain is pretty slow for a lot of tasks. It's hard to make direct comparisons, but for language (related cognitive work) we're at maybe a few tokens per second.
I wonder how even a team of humans (a meat mixture of experts) would perform on the AI benchmarks, measured in energy use per intelligence index/benchmark point.
Back in July, I timed myself manually looking at some data files to create an index out of them, and out of curiosity asked some AI to do the same.
It took me an hour (61 minutes) and I made 4 mistakes in about 80 entries. It took DeepSeek-V4-Flash 3 minutes, it cost $0.034, and only made 2 mistakes. There is no contest. Humans have lost.
I'm getting questions weekly about running AI in-house now, in Europe trust in both the US and China seems to have hit an all-time low and AI is already scary for a lot of people.
I have no doubt that local first is the future that we're heading towards. We're seeing "algorithmic progress" towards more efficient models and _some_ hardware being available (NPUs, this chip, etc).
At the moment it doesn't seem like an appealing offer since it's just a little bit too constrained and since internet is universally available but if we can run a DeepSeek 4.1 Flash class model cheap and local, it becomes a no brainer.
The major problem with making the economics work is utilization. If your users are local to a few time zones, you will naturally have lulls that you need to fill in. Even then, a single firm might have bursty needs compared to a diverse set of firms.
80 TOP/s and 150GB/s memory gives us the fundamentals of where this sits performance wise although its not said if the TOP/s is for INT4 or FP16 or what which changes the prefill performance considerably.
A consumer graphics card (say a 9070 XT) will be something more like 750 TOP/s on INT4 (20TOP/s on FP16) and 650 GB/s. So its probably about a tenth the performance for prefill but maybe its only a third if its FP16 and can't handle more than 15GB per token due to its memory bandwidth (at Q4 that would be ~7B active parameters) for decode to achieve 10 token/s and to get its 30 token/s it would be max 2.5B active parameters at Q4. A full FP16 model on this seems very unlikely unless its a lot bigger than we think it is.
This I think is just about enough for a decent home AI, its a little slow and its only going to be of use for MOE (so no qwen 27B) but depending on price I can see it being of interest to a lot of people as some flash models aiming for 100B and 2-7B active parameters could be really quite a lot of home capability in a relatively low power consumption unit.
If it can work with 100B parameter models then stopping with edge devices makes no sense. This kind of compute-in-memory technology being so much more efficient would make most or all existing data centers obsolete. Or so it seems?
Also, they are saying this efficiency is going to become standard. I think this is the type of technology that will allow AI companies to keep increasing the size of AI models. So they will not be bottlenecked by energy needs to make the next qualitative step in intelligence.
It also points towards the next step for local AI after chips like RTX Spark. So say 18 to 36 months from now (or pick your more optimistic timeline) we expect performant local AI to be everywhere.
This mentions targets for inference of 10 to 30tps and I (albeit just skimming through) didn't find any info about ingestion speed. They also mention that the 1st gen chip is used for vision, though I suspect it's only for specific cases and not anything general.
Overall I like power efficient optimizations like this and let's hope that this is adopted broadly. Although the compute in memory sounds like a big increase in memory prices
My M5U Studio is using about 72w running a long Gemma4 job (and normal day-to-day stuff). I'm in San Diego so that's a tangible addition to electrical bill :'(
Quirks from the article:
1) The performance numbers are simulated
2) The details say that the performance numbers require 2.7x the overall power, so around 13.5W. Power needed for all the memory is not explicit in the numbers.
3) The compiler is also simulated and it's a prototype, so it's not a drop-in replacement for anything like Nvidia ISAAC
4) 5W seems to refer to sub-4B models, so not the 100B param ones.
The chip is designed to run models with 20 billion to 100 billion parameters (approximately 3 billion to 20 billion active parameters)
Doesn't seem like 100B dense models are even on the table, only MoE models. Kinda misleading.
Not really. Models of that size is reasonably often moe. It is also 100b even if it only has 20b active at a time.
13.5W is Jetson nano territory.
I think the human brain uses something on the order of 10-20W so 13.5W seems like there's probably some room to improve on a 4b model!
Statements like these often overlook that a human brain is pretty slow for a lot of tasks. It's hard to make direct comparisons, but for language (related cognitive work) we're at maybe a few tokens per second.
I wonder how even a team of humans (a meat mixture of experts) would perform on the AI benchmarks, measured in energy use per intelligence index/benchmark point.
Most humans probably can't even do many of the AI bench mark tasks.
"Few tokens per seconds"?!
Human brain is an advanced Next Concept Model.
Back in July, I timed myself manually looking at some data files to create an index out of them, and out of curiosity asked some AI to do the same.
It took me an hour (61 minutes) and I made 4 mistakes in about 80 entries. It took DeepSeek-V4-Flash 3 minutes, it cost $0.034, and only made 2 mistakes. There is no contest. Humans have lost.
[flagged]
These days I like to think of myself as a 100T dense model with 20 t/s prefill and 2 t/s decode. I am an RL(HF/VR/CR) policy rollout at this point.
423. Could y'all please start prefixing comments with the number of joules you consumed in generating it, so that I can decide if it's worth reading?
Flashbacks to dial-up internet era, where any website would, parenthetically, indicate how big a linked file was with an estimated download time.
Eleventy billion:watch snl celebrity jeopardy.
^what every advertiser will do if we actually do this lol
I know some people that would beg to differ. They’re at least 7B class.
I'm getting questions weekly about running AI in-house now, in Europe trust in both the US and China seems to have hit an all-time low and AI is already scary for a lot of people.
I have no doubt that local first is the future that we're heading towards. We're seeing "algorithmic progress" towards more efficient models and _some_ hardware being available (NPUs, this chip, etc).
At the moment it doesn't seem like an appealing offer since it's just a little bit too constrained and since internet is universally available but if we can run a DeepSeek 4.1 Flash class model cheap and local, it becomes a no brainer.
The major problem with making the economics work is utilization. If your users are local to a few time zones, you will naturally have lulls that you need to fill in. Even then, a single firm might have bursty needs compared to a diverse set of firms.
It is a hard problem to solve.
It absolutely is but paying a 4x premium for on-premise would absolutely be acceptable in a lot of industries.
The more the labs optimize inference to increase their margins, the more feasible it makes to run models locally
> Europe trust in both the US and China seems to have hit an all-time low and AI is already scary for a lot of people.
And yet in your example you used a Chinese AI model lol.
80 TOP/s and 150GB/s memory gives us the fundamentals of where this sits performance wise although its not said if the TOP/s is for INT4 or FP16 or what which changes the prefill performance considerably.
A consumer graphics card (say a 9070 XT) will be something more like 750 TOP/s on INT4 (20TOP/s on FP16) and 650 GB/s. So its probably about a tenth the performance for prefill but maybe its only a third if its FP16 and can't handle more than 15GB per token due to its memory bandwidth (at Q4 that would be ~7B active parameters) for decode to achieve 10 token/s and to get its 30 token/s it would be max 2.5B active parameters at Q4. A full FP16 model on this seems very unlikely unless its a lot bigger than we think it is.
This I think is just about enough for a decent home AI, its a little slow and its only going to be of use for MOE (so no qwen 27B) but depending on price I can see it being of interest to a lot of people as some flash models aiming for 100B and 2-7B active parameters could be really quite a lot of home capability in a relatively low power consumption unit.
[flagged]
If it can work with 100B parameter models then stopping with edge devices makes no sense. This kind of compute-in-memory technology being so much more efficient would make most or all existing data centers obsolete. Or so it seems?
Also, they are saying this efficiency is going to become standard. I think this is the type of technology that will allow AI companies to keep increasing the size of AI models. So they will not be bottlenecked by energy needs to make the next qualitative step in intelligence.
It also points towards the next step for local AI after chips like RTX Spark. So say 18 to 36 months from now (or pick your more optimistic timeline) we expect performant local AI to be everywhere.
This mentions targets for inference of 10 to 30tps and I (albeit just skimming through) didn't find any info about ingestion speed. They also mention that the 1st gen chip is used for vision, though I suspect it's only for specific cases and not anything general.
Overall I like power efficient optimizations like this and let's hope that this is adopted broadly. Although the compute in memory sounds like a big increase in memory prices
My M5U Studio is using about 72w running a long Gemma4 job (and normal day-to-day stuff). I'm in San Diego so that's a tangible addition to electrical bill :'(
Wonder how it would feel because the details shown seems a bit slow to edge device use
autonomous hunter drones running 100B models
> "large large language models"