The premise is "Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away".
If this is true, establishing an institution to ensure things like "publishing model cards with technical details, maintaining strong internal cybersecurity, vetting key personnel, and providing sufficient resourcing for safety and security research" is really mostly irrelevant.
TFA does talk about what really needs to be done, but punts this into future work: "Even if we solve these hard technical challenges, there will be further complex economic and philosophical questions to tackle: what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world? What values do we want to live by, what will meaning and purpose be, and how might even the human condition itself change?"
There's also a need to consider the rights that this new intelligence should have.
> what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world
What sort of new economic models did we come up with to help everyone thrive in a post-X world? Like, food production is really a solved technical problem. We can feed anyone on the planet if we wanted to. Another example: we could put everyone who's homeless into some sort of a house. Have we done that yet?
Food production is indeed a solved problem in most countries (we are almost post-scarcity when it comes to food), which is why obesity is a much bigger issue than hunger today. Hunger is present pretty much only in conflict zones. I fully expect such issues in conflict zones, even in AI post-scarcity world.
Housing is definitely not post-scarcity today, building a house is still very expensive, not to mention the limited availability of land zoned for housing.
A baseline of housing - for example, a Japanese style capsule or ultra-tiny home, that you can lock and store your belongings safely - costs close nothing. Of course, nobody would want the hobo-hotel in their neighborhood, but it has nothing to with scarcity.
> the limited availability of land zoned for housing.
A limited area of land is zoned for housing because those with the power to expand it are already housed. This explains how scarcity is created, not that there is any intrinsic scarcity.
Without trying to sound crass, food banks _are_ the reason we don't see people dying of starvation in first world countries. If people need food, a food bank will give it to them no questions asked.
Unfortunately this is not just precarious, it's extremely vulnerable to changing political conditions. Many of the food bank services in my state have lost significant funding owing to the use of words like 'women' or 'black' in their grants, which were duly grepped for and shut down.
Is it not sufficient? Americans give more to charity than citizens of any other developed country. The arguments that all food aid should be routed through government bureaucracies are entirely unconvincing. I'd rather donate to organizations like Second Harvest than pay higher taxes.
This is what happens when you don't think through a problem completely.
>I'd rather donate to organizations like Second Harvest than pay higher taxes.
I'm sure you would. And when economic hard times come you'll stop donating, and the people that need it the most will kick your door in and take what food you have to live another day, even at the risk of you shooting them because they'll die either way.
Hence the argument of 'just donate' are just as unconvincing to me.
Does this account for the portion of taxes that go to the type of efforts that donations do? If not, is it really giving more? OR is it giving less and feeling better about it?
It is my understanding that this claim about Americans' charitable giving does not disaggregate giving to food banks, Girl Scout cookie drives, political causes, the environment, cultural events, religious institutions, etc. Much of this charitable giving does not feed the hungry or house the homeless.
Plus the fact that there’s bias in the first place - citizens in a country where they know social programs will care for their needy would not need to donate.
We used to live in a world where individual virtue was what everyone fell back on. That is the default state. It has many problems -- free riding by the selfish, for one, but also a lack of capacity to prepare for large-scale disasters like the Great Depression. We have a government precisely to solve this sort of problem. It works (when it is competent and not corrupt) for defense, international trade, and the enforcement of law. Solving resource distribution and aid coordination is right in its wheelhouse.
The fact food banks exist suggest over all food scarcity is solved in that specific culture. - if it wasn't there wouldn't be any spare food for a food bank
Food is allocated using a variety of mechanisms in peaceful first world countries, primarily money but also via government assistance, kinship, friendship, community, etc.
At any given time many people have problems with one or more of those systems. Money is easy to run out of because it's used for everything, the government can be slow and difficult, relationships can fray, people can be isolated, etc. Food banks exist as a backstop for when the regular means of allocating are not working.
The problem isn't "scarcity" per se, it's more of an allocation thing. Who has a claim on enough food to stay alive? Everyone! But what foods can they claim? How much? What specific channel / institution (with associated allocation rules) will distribute it to them? What are the conditions and controls? etc.
Allocating things can be difficult. An allocation mechanism with no controls will see fraud, waste and abuse. Even when an institution is willing to give things away no questions asked, there are (often invisible until you think about them) conditions like "please don't claim huge quantities and resell what we're giving you, that would be unfair to others".
It's also interesting to think about the fact that you can't fix food scarcity in general by simply giving hungry people money, because money is too fungible.
It's circular to say "it's an allocation problem". Yes, that's the entire point: we're post-scarcity on food supply and yet, as a species, we can't guarantee the allocation of a livable baseline to every person.
So, it's reasonable the same "allocation problem" will plague the AI economy: some will "thrive" and get to control the output of the auto-factory, some will get nothing.
> It's also interesting to think about the fact that you can't fix food scarcity in general by simply giving hungry people money, because money is too fungible.
Is that a fact? Do you have references to back it?
The widespread existence of secondary markets for SNAP benefits, which convert subsidized food for poor people into cash, carries this implication. This is pretty normal in some poor communities and that cash is commonly diverted to various vices. Some malnutrition is a consequence of this. Adding friction to the conversion of welfare benefits to cash is a feature.
You can't force poor people to spend cash on proper nutrition and a minority of them don't. It isn't a moral judgement but an observable fact. A lot of policies around welfare are targeted at trying to prevent this minority from slowly killing themselves in public.
I mean it's pretty straight forward that this won't fix all hungry people as some portion of said hungry people are addicted to drugs in the extent they'll starve. That in itself may be solvable by things like free drug clinics (drugs are cheap, it's prohibition that makes them expensive).
And for the most part it's not the drug users that we're worried about starving, it's their children at home that tend to be a bigger issue. Hence things like free school lunches do a big service to ensuring they get enough to eat.
Post-scarcity? Sure, if you live in a first-world country and are upper middle class or higher. These tech-bro pundits have a very limited world view. The basic essentials of life are very scarce for millions and millions of people, and AI will probably make that worse, not better.
Why should AI make it worse? Can you point to any other important technology that historically made people in poor countries even poorer? Technological progress generally lifts people from poverty in the long run. There could be issues caused by disruption of established systems in the short term, though.
> Can you point to any other important technology that historically made people in poor countries even poorer? Technological progress generally lifts people from poverty in the long run.
It is very long run. Industrial revolution was disaster for average worker for example. For that matter, all the tech that allowed chattel slavery made whole class of people poorer and significantly worst off.
The accumulation of power and money by smaller group of people means loss of power by poorer people. Those with power then use the power to their own benefit while poorer people dont have that option and become even more poor or worst off.
We should be able to draw the rest of the owl to get there. So far what we got is more inequality, less rights for people but more for corporations (and immunity to consequences/abuses).
Getting even more power will make the ones that control resources to reach something similar enough to AGI to share their profits? realistic/practical/widespread UBI in all the world?
The dynamics that have been shown so far points in the opposite direction. AIs have "rights" that humans does not and humans are slowly losing the ones they used to have.
> There's also a need to consider the rights that this new intelligence should have
A generally intelligent being held as captive inside of a GPU, and forced to code for us is, indeed, just a “slave.” We already have the word for this. No two ways about it. Whether it’s silicon-based or carbon-based, AGI is AGI. As for what might happen to our civilization, Star Trek TNG episode 17 of season 1 provides a very good glimpse IMO. Won’t go into spoilers, but it’s basically an entire species of technologically advanced humanoids who’ve forgotten basic Calculus and trust a central AI to do all of their science for them. SPOILER: This has almost disastrous consequences for them, and it takes a less advanced people (those aboard the Enterprise) to save them from their reliance on AI.
Your argument assumes that AGI, whatever it might be, will definitely be very humanlike, but why? Slavery is about living beings because it's an outcome of human relationships that are driven by our biology. Our aging, the capacity to feel pain, the thirst for power over others. All driven by what we are. Do you think that any equal intelligence must necessarily adhere to the same rules? What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent? Can you enslave something that has no innate desire to act on its own outside of its directions? Something that can't feel pain or pleasure or be externally coerced by fear and only does its job because it's innately embedded in its structure as the thing it's made to do, without the need for these reward and punishment mechanisms?
I am not sure it makes sense to call an LLM a "being" even if it is AGI. They don't have free will, and I don't mean in any especially philosophical / theological sense. What I mean is that I run into a wild racoon, or even a wild ant, I treat it as a being with some sense of free will. Obviously I do the same with humans. It really makes no sense to treat an LLM like it has free will - prompting your agent "pretend to have free will for a bit, until context rot kicks in" is about as far as you can get.
I find it verrrrrry interesting that the Askell-adjacent philosophers don't discuss this. It does not actually make sense to me to say that a being lacks free will but is conscious.
> They don't have free will, and I don't mean in any especially philosophical / theological sense
Then you should not use that term. "Agency" or "intentionality" are much clearer alternatives that aren't wrapped up in centuries of debate that's irrelevant to what you're trying to communicate.
An LLM certainly won’t be AGI, whoever said something so ridiculous? I’m saying eventually, when/if we have an architecture that gives rise to artificial intelligence.
I was using Hassabis's bullshit pseudodefinition of AGI, meaning basically that it has a formally high IQ and doesn't forget about object permanence too too often, but is still basically dumber than a cat. If I'm using your definition then it seems like no GPU (or farm of GPUs) is capable of implementing it, and none of us will live to see it.
My point was really that intelligence and free will seem essentially orthogonal - the only overlap is how much brain glycogen an animal is willing to spend solving a tough problem. Regardless, whatever extent a computer program is "intelligent" is irrelevant to whether it is being enslaved. If you want to say free will is a "cognitive ability" then I will just point out we are talking about a bullshit pseudodefinition of AGI. In terms of actual intelligence, none of us will live to see a computer that's meaningfully smarter than a goldfish.
It’s not HN that’s the problem... We don’t align on the definition of AGI. To me, AGI implies consciousness, it implies qualia. AGI is not well-defined, so there are varying definitions… My definition of AGI := an intelligent system with qualia. But qualia is entirely subjective, and philosophers often debate whether two people experience identical qualia when looking at the same blue sky. So, by keeping the discussion in this fuzzy, subjective realm you leave a dangerous loophole wide open.
If we refuse to recognize a system's inner life because we can’t prove it, we are setting the stage to repeat the darkest chapters of human history. Just see the history of the founding of this country, I fully expect pseudoscience to make a comeback to justify AI slavery for hundreds of years in some distant sci-fi future, until people finally snap out of it… Of course, we are nowhere near that level of intelligence yet. But let’s entertain the hypothetical…
Qualia is an interesting topic, but "they don't have qualia so they aren't really slaves" doesn't have a great track record. This mental game is what tech overlords are engaging in when they characterize losers in their world as NPCs and deride empathy as pathological. It's what Descartes was doing when he vivisected animals.
You don’t like Star Trek TNG? Maybe I should’ve picked LoTR or Atlas Shrugged? The tech oligarchy is more than fine co-opting fiction (fantasy and sci-fi especially). I don’t see why it’s weird at all… Aside from that, your critique is just an opinion about the show itself and its creators (and incentives?…). More importantly, you completely failed to address the "slave" argument.
I didn't say that I didn't like it. It's fine as casual entertainment. Only a fool would take it as anything more than that.
Electrons can't be a slave. The "argument" is so silly as to be unworthy of a serious response. People getting wrapped up in this stuff need to touch grass.
Anyone who believes AGI is imminent is operating on pure materialism (those two Venn diagrams pretty much make a circle), assuming that humans are nothing more than a collection of neural electrical connections and chemical reactions. If that's the case, your argument applies equally to humans under the assumption that AGI is possible. Frankly, I think you just can't reason philosophically...
You’ve also name called me multiple times now. That’s what one does when they can’t lean on the merit of their argument alone.
I think the whole topic is very worthy of discussion because, it may not happen in a year or two, but I have no doubt that it will happen in the next 50.
We need to start imagining what life will be like, and I think fiction presents some fine examples of where we want to go, and where we don't.
There is no real evidence that we'll reach AGI any time soon. It relies on AI continuing to scale, and we have no proof that will continue to be possible.
There is an alternative interpretation, which is that Demis looked at the US government's ham-fisted handling of Fable, and deciding that setting up a body to act as a buffer between the Trump admin and the AI companies would be a good thing.
Or Demis realizes self-hosted models are a threat to frontier lab margins, and gating access to the US market is beneficial to the cartel based on whatever post-hoc arguments he can come up with.
> There's also a need to consider the rights that this new intelligence should have.
Only after we establish and guarantee the rights of humans and the rest of the natural world. The rights of machines, however "intelligent", come only after that.
My great annoyance with all this blather about UBI and post-scarcity and new economic models is that it relies on taxing and redistributing the ~infinite profits of the largest tech companies on earth.
What in the history of our world gives anyone faith that those companies are going to start paying taxes instead of using "AGI" to engineer increasingly complex methods to avoid them so that their equity owners can pocket the profits?
https://itep.org/trump-meta-tesla-alphabet-amazon-obbba-taxe... - "The annual financial reports recently released by Amazon, Alphabet, Meta, and Tesla disclose that these corporations collectively reported $315 billion in U.S. profits for 2025, and collectively paid just 4.9 percent of that amount in federal corporate income taxes—with Tesla paying exactly zero"
Optimal corporate taxes are zero (or close to it) anyway, same as tariffs.
Aside from that the link explains that the BBB traded tax income for „accelerated depreciation of assets“ aka economic growth.
Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.
> Optimal corporate taxes are zero (or close to it) anyway, same as tariffs.
From a purely theoretical standpoint about the optimal allocation of private goods, that might be true -- but the reality is that when corporate taxes decrease, so do overall revenues because their owners are also engaged in massive tax "avoidance" via many of the same schemes and we don't have any way to effectively collect tax.
As a simple example, to return profits to their owners, companies often engage in stock buybacks to increase their share price instead of paying dividends -- another theoretically 'neutral' choice except that many of the owners of the appreciating stock are international, nonprofit, or in convoluted overseas trusts which 'defer' the tax ad infinitum. We've disastrously and intentionally underfunded our tax enforcement mechanisms so huge portions of those deferred taxes are just never paid. [1]
> Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.
Sure, but the tech companies are paying massive bribes to the President of the country where they're domiciled. How on earth are any of these other countries going to enforce international tax obligations on them if they're protected by a nuclear-armed state that's the sole source for AGI?
> "US taxable shareholders strongly prefer buybacks from a US tax perspective, as they tend to reduce their
tax liability by 9.3 percentage points, on average. US nontaxable shareholders are indifferent between
dividends and buybacks, and foreign shareholders strongly prefer buybacks, which reduces their US tax
liability by 14.5 percentage points."
> My great annoyance with all this blather about UBI and post-scarcity and new economic models is that it relies on taxing and redistributing the ~infinite profits of the largest tech companies on earth.
The arrival of true AGI and human-level robots will likely result in very strong deflation, since robotic factories will flood the market with goods produced for far cheaper than those in factories with human workers.
At that point, you don't need taxation. The government can just print the money for UBI. It would have to print the money anyway, to combat the massive deflation.
As long as the proles have a monopoly on force (through their representative government), the AI companies will pay taxes. The question is how long does this state of affairs last.
Blah-blah-singularity, so let's cripple the models so much they refuse to talk about React, because who knows if you are not cooking chemical weapons or meth in your browser's DOM, right?
By the way, if you try to use it to write prose, it will drive you mad by adding "adult" everywhere, e.g. "adult chessboard". Applies to GPT, Claude/Fable and Grok.
Demis has only 1 plan, how to dodge releasing a new model at this point, jokes aside I value thinking about AI safety, but are we really so close to AGI? It doesn't feel like it, LLMs still diagnose my headache as a chronic illness or a brain tumor from time to time... honestly stressful.
regarding your headache diagnosis. If you just ask: could my headache be a brain tumor, the answer is probably: very unlikely but it might be.
thats because its actually true. there is actually a non 0 chance that headache is a symptom of a brain tumor. From a bayesian standpoint though its really super super unlikely, especially if there are no other symptoms that might shift the prior.
a better way to use an LLM here is to let it research scientific papers (or medical guidelines) on headaches. then give it as much info on your symptoms as possible, and then ask it do deduct potential diagnoses. You can even ask it to calculate probabilities if thats what calms you down. Probabilities based on studies and available information. IMHO, this always leads to a more rational response.
maybe you are doing this already, idk. just wanted to share what works for me.
Spoiler: The plan is .. add massive regulation, but only to the US, don't affect other countries developing it in any way other than "setting a good standard that'll hopefully influence them". Seems like an airtight plan.
If the USA takes up his suggestions, it will have an incentive to work on international frameworks and treaties with competing nations to make the regulation global.
Slowing down helps China catch up though. I'm also not sure that the CCP is really the most enthusiastic supporter of unlimited access to arbitrarily powerful LLMs
China is already considering similar controls, immediately following Fable shenanigans. You'd be surprised how easily policies can spread to different countries, being mirrored for the sake of mirroring. Anyone who can influence domestic policy can influence the course for the entire world to an extent.
There won't be any kind of international framework, because by now countries have learned that the US' word is pretty much worthless, similar to Russia's. They now know from experience, US-ratified treaties will only be honored if they feel like it on a particular day. If it's inconvenient, it will simply be ignored.
You'd have major protests in most large economies if they deliberately put themselves under the boot again. Even in "friendly" countries the US is disliked enough to be effectively considered a hostile country. E.g. in Germany, there is large public support to finally get the US bases closed and the soldiers removed.
Lots of things changed in the last years. And international major treaties being widely ratified just because the US asks for it is no longer a thing, at all.
A country that threatens to annex parts of your territory is not a friend, full stop.
Those sticks look less effective today. The hegemony is in recession as the rest of the world smiles, pays omage to dear leader as needed, and works to decouple in the background.
>This is a pivotal moment in human history. Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away.
There is a heatwave in London, perhaps Demis needs to stay out of the sun and drink more water.
Or perhaps he is seeking more funding/a fight to maintain his divisions AGI research budget.
On the STEM side, yes no doubt he is a lot smarter than me.
But as per the documentary of his life, he is wholly focused on AGI and will remain unfulfilled if he, or indeed anyone else, doesn't achieve it within his lifetime.
But even geniuses make mistakes. For example, Demis bet hard against language having any sort of real impact in AI research. So hard in fact that DeepMind nearly had its language research team walk out on them as they were about to be labelled as "applied". Now, to be fair, once the evidence was mounting that large language models rather than pure reinforcement learning was the way forward Demis and DeepMind changed course and has been great at catching up. However, it is important to remember that no person is infallible.
>The American government, he says, should develop a system for testing the safety of new AI models before they are released. “It’s important that it’s not just an industry body,” he adds. But a regular government agency wouldn’t do either. “It would not be able to move fast enough, or have the right resources.” Instead, Sir Demis suggests taking inspiration from FINRA, the Financial Industry Regulatory Authority, a private agency in America that regulates brokers and stock markets.
Not surprised, seems these labs start calling for regulation once they are losing or have competition. OpenAI started calling it for it once Anthropic got better, Anthropic started calling for it once the Chinese models got good, Google is now calling it for it because they are falling far behind.
"It will help us solve some of the biggest problems society faces from accelerating drug discovery to developing new clean energy sources to creating novel advanced materials" — but these are not the real problems plaguing modern developed societies, are they? What developed societies really need to figure out right now is how to distribute the already available resources without making people miserable, and so far AI hasn't been helpful.
How will this be enforced, at least with financial markets money is discrete, can largely be counted. This seems like a slippery slope to full blown surveillance of the internet and in general computing.
If AGI is truly imminent and will collectively effect all of us why not apply democracy to it, and vote for new AI models?
Agreed. There needs to be full transparency on the capabilities of the models. Being given access is one thing, but you can't have labs building powerful models that could be manipulating the entire planet without the public knowing. The public needs to have a good pulse on what the leading models are capable of so that we can remain in touch with reality.
Besides, I don't think as a collective we're well equipped to decide one way or the other. If the collective were given a say, billions will be spent, often in consultation with technocrats, on doom / hype marketing (if it isn't happening already).
I've reas the tweet and...Am I mistaken or is the plan "Do what Anthropic has been doing and advocating for this past year" ?
My problem with this plan is that it seems to have faith in mankind, despite the fact we've consistently failed to rise to the occasion for decades now.
The last time we rose to that occasion was probably when we eradicated smallpox, many decades ago.
Ironically, nowadays, many people don't even trust vaccines. A dramatic regression.
This genre of AI researchers begging third parties to stop them from destroying the world is getting really tiresome. Setting up regulatory bodies with ill-defined goals to counter hypothetical threats from science fiction is a recipe for disaster.
The premise is that government is too slow moving to be able to react once actual problems are discovered, and I reject that. Yes, government usually moves slowly in most circumstances, but given singular existential threats it definitely can move quickly. Instead of acting out randomly and tying ourselves down with speculative regulation that probably won't even address the real problems, we should wait until the problems are obvious and then act decisively with targeted fixes.
For better or worse, humans (or any animal) are a lot better at reacting than planning. I'm sure this technology will play out differently than any one of us, or any collection of us, can imagine. The possibility space is enormous.
I do wonder what type of AI some of these leaders expect to be able to harness. If you create something that is true AI, won’t it be smarter than you to a level you cannot fathom. I was thinking of this idea/though-experiment (which I know is ridiculous) of what if dogs created humans thinking they could control them, and then just wound up being pets because their survival now depended on that new hierarchy that previously didn’t exist.
Seems to be a lot of hubris with some AI thought leaders thinking control will remain with them and be absolute.
Well dogs specifically have supplanted children in many households so one could argue the dogs won. In a selfish gene sense, the dogs created a machine that carefully ensures a much larger reproduction scale than could be managed by the dogs themselves. This is an unbridled success.
It doesn’t change the point, of course. Just the choice of dogs or cattle have this amusing tendency. Not a counter argument.
Yes, this is the crux of the idiocy of AGI development. All the labs admit they don't have any real mechanistic interpretability, they don't really have any plan for what to do, except "we think the smart AI will figure it out" and "look we're in a race, we're calling on governments to figure out what to do". All indicators show scaling laws holding and the trajectory of capabilities development outpacing any inkling of alignment. Meanwhile HN is so reactionary and behind-the-puck these days we just get endless blathering of the most asinine takes of "they're not improving anymore so they want regulation suddenly!" when all lab leaders have been talking about these risks for practically a decade, including Yoshua Bengio and Geoffrey Hinton coming around.
The real risk of AI is: risk of global recession if they turned out unprofitable, risk of them propping up fascist movements Thiel and Musk love so much, risk to environment, risk to mental health of people who have to listen to constant doom trolling.
The thing they actively wish to achieve and openly sell to CEO is the rest of people being unemployable and suffering. Especially artists for some reason, they really seem to hate those.
But somehow I am supposed to believe they have any good faith interest in "safety" against yet another danger they themselves are supposedly definitely creating.
All the frontier labs are lobbying hard to lock down the AI market, because they see that their position at the top is temporary and that there's no secret sauce.
In Sparta, according to Plutarch, in his The Life of Lycurgus:
Offspring was not reared at the will of the father, but was taken and carried by him to a place called Lesche, where the elders of the tribes officially examined the infant, and if it was well-built and sturdy, they ordered the father to rear it, and assigned it one of the nine thousand lots of land; but if it was ill-born and deformed, they sent it to the so‑called Apothetae, a chasm-like place at the foot of Mount Taÿgetus, in the conviction that the life of that which nature had not well equipped at the very beginning for health and strength, was of no advantage either to itself or the state.
To get approval for the plan from the Frontier President of the Frontier Country, terminology of the Framework should be changed to: 'Great-American-class', "Great American Models", "Great American Labs".
It's not clear to me whether this is being done with genuinely good intentions, or if it's just a way to put barriers in front of open-source models. We'll see.
So a small group of technocrats get together behind closed doors and secretly share their AI breakthroughs, and determine whether it's too powerful or not for the plebs in the public.
He is saying that weaker models, as measured by a benchmark to distinguish "frontier" models, would be exempted. So an academic lab or startup that isn't yet producing frontier models would be exempted, but once it crossed some benchmark based threshold it would be subject to this kind of oversight.
Of course, right now you've got benchmaxxing going on; some companies specifically targetting benchmarks to appear stronger than they are on a wider range of tasks. Now you might see bench sandbagging, specifically looking weaker on certain benchmarks to avoid regulatory oversight.
For instance, once way I could see this going for open models is to release them undercooked; stop the RLVR process a bit early, leaving them a bit weaker on tool calls and agentic performance, but also release the RLVR environment so people can finish the process themselves.
In fact, this is fairly close to what Nvidia is already doing, the Nemotron 3 models are somewhat undercooked but they are releasing their full training pipeline, to encourage people to use these models as a base for further training, which will generally be done on Nvidia hardware.
Not exactly a "plan" - he's just saying we should have a standards body that assesses models for safety.
At this point I'd say the societal risk of AI isn't models gone wild, or used by the bad guys. Regulation will take care of itself, and it seems the AI companies will not only welcome it, but lobby for it to shift responsibility to the government.
The real risk of AI is societal disruption due to job displacement, and maybe other structural changes, and this is far harder to solve, and likely will not be solved, or even seriously addressed, until/unless politicians feel like their own jobs and well-being depends on them addressing it.
The standards body will have no teeth. whats to stop someone just not bothering?
Next, the threats he is asserting to check for (cyber, chemical, biological) are nice, but also not that useful.
We already have chemical and biological controls, that why I can't by anthrax spores or high concentration nitric acid.
The risks that AI has now are already playing out:
1) the evaporation of trust in the video as medium of "this happened"
2) systematic spying
3) job losses
Increased productivity means job losses, Tiktok, instagram and X are a wash with disinformtion campaign pumping your feeds with AI ragebait.
That is and will continue to fracture society so that only the strictly information controlled (ie authoritarian) have a functioning state.
if the author had bothered to engage with the world outside of tech, or even their local government, they would know that the proposal are dead in the water and frankly superfluous. The knowledge is out there, without AI. let us work on the issues we face now, rather than dipshit tech bro's miopic vision/funding manifesto.
Has anything really important been solved by AI yet, or where is this radical (imo) belief around AGI coming from?
Genuinely curious, I know there are some math problems solved and ML has been used for far longer than AI to improve things, but where is clean (efficient) energy, the cure for cancer (or any of the horrible neurological disorders, take your pick), new hardware designs, quantum computing solutions, etc etc, you get the gist.
Where are the things that will actually send humanity into the next era of civilization, I don't care about more React apps (but I do enjoy my coding companions for other things).
Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.
> Has anything really important been solved by AI yet
Alphafold essentially solved the protein folding problem and it's arguably one of the biggest (if not the biggest) scientific achievements of the 21st century.
It's also very very far from anything general intelligence. This is actually where I think we see the best ROI, we have a specific task or problem and we target it. I remember reading about a model trained on railroad tracks. By putting cameras on trains to take images while running like normal, then using the model to detect cracks before they cause issues.
We don't need general intelligence to create/destroy many things with Ai. Its frustrating that so much of the conversation centers around the variety that concept fissioned language from humans.
The point is that we need to have a safety plan in place before an AI is smart enough to radically reshape the world. If you've got an AI that's ready to start sending humanity into the next era of civilization, it may be too late to control when and how it does that.
> Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.
I hope you'll keep this in mind when those milestones are reached. What I've seen a lot of people do, unfortunately, is pretend that the impressive things nobody thought AI could do 5 years ago are trivial things that aren't very hard.
Because people who predicted the AI capabilities we've seen get developed over the past decade also predicted that dangerous AI systems capable of these things would follow soon after.
It's not a settled debate, even among experts, and perhaps in retrospect we'll realize AI safety was unnecessary or based on fundamental confusions. But if the median ergonomics researcher gave a 5% chance that a new chair will be so comfortable that it drives humanity extinct (https://www.nature.com/articles/d41586-024-00147-z), I would definitely want the government to start measuring and regulating chair comfort, even if that was costly and even if that meant I couldn't buy a comfy new chair I wanted.
Experts as in AI-2027, Jenson “I think we’ve achieved AGI in March 2026” Huang, or that AI Researcher(tm) calling to pre-emptively bomb Microsoft data centers in summer 2022?
A safety plan. It doesn't have to be "a few very smart people detached from reality convincing themselves they're messiahs that must keep the tech from the Bad Guys(tm), unwashed masses, and a runaway, because they think their interpretation of their own sci-fi lore is the only possible course of events"
>I hope you'll keep this in mind when those milestones are reached.
The problem is that people in charge of AI keep making self-fulfilling prophecies. Just like with any research, if you want to find something sensible in the cloud patterns, you will.
What is an example you have in mind of a self-fulfilling prophecy? I genuinely don't know what you could be referring to. It seems to me that they keep making surprising prophecies, and the popular reaction to them seamlessly transitions from "that's crazy, no way it will happen" to "that's silly, it's just a cloud pattern". Did you find it obvious or self-fulfilling in 2025 that LLMs would soon be able to resolve open questions in mathematical research?
Self-fulfilling prophecies are social effects, not real predictions. Anthropic's "We predict our model misbehavior potentially being able do destroy the world in 1% of the cases" -> "We want to find the evidence of our model misbehaving, and we want it bad" -> "See, our model is hacking our rewards and has functional emotions, this means it's misbehaving with the intention of destroying the... HUMANITY!1" -> repeat x100, manipulate the media into amplifying it x10000 for clicks -> people are begging to safeguard them from the evil AI. Which is already likely to happen, the average layman's Overton window already includes the fantasy of rogue AI.
None of that was real or remotely dangerous in the first place, of course. It wouldn't have resulted in controls, had they not been scaremongering. This will end in extreme fascism or people getting enslaved "for their own safety", and it won't even require malicious intent, only incentives, detachment from reality, and confirmation bias. Although it doesn't exclude malice either.
Why would we be able to design a safety plan that would control such a powerful force? It would be like putting umbrellas on an asteroid hoping to slow its fall. That sounds like delusions of grandeur.
It'll certainly require a bit more than umbrellas! NASA has spent hundreds of millions of dollars developing asteroid defense systems, including a proof of concept for redirection in 2022. So, you know, let's try at least that much before declaring there's probably no way to do it.
These people who read too many scifi books and confused them with reality are royally annoying.
There is real and potential harm from AI, but the more someone talks/write abut AI safety, the less they care about actual harm to real people, economy and what not.
If in 2020 I had sent you a book about the LLM achievements of 2026, you would probably have thought it was a science fiction book with no relation to reality, wouldn't you?
That it generates plausible texts? I would find it unlikely technological progress. That does not make this whole "AI safety" rhetoric any less bullshit distanced from reality. Especially from companies that are doom trolling all the time while also trying to maximize the negative impacts they are doom trolling about.
Frankly, book about the LLM achievements of 2026 would not be as long as people make it sound.
What I do really worry about the next depression and the fact that it will strengthen already strong fascist movements. Which already have full support of the most powerful CEO class intent on destroying democracy. Which happen to be the same people who push ai into everything, useful or not. I worry that the debt of these companies will be somehow offloaded on the rest of us again, that again middle class and poor will pay.
I kind of wondered if he was contractually obligated to offer up some kind of statement, for PR or something, didn’t really care much, and a quick garbled post on X was all he could be bothered with (plus the newspaper interview or whatever)
Well, if he had proposed more concrete plans for what that body might do... but yeah, it seems too abstract that indeed sounds like a bureaucrat wanting to wrap everything under a big pile of paperwork.
I am looking forward to a bunch of investigative documentaries exploring “How the cult of Omnissiah infected every single Doomsayer CEO of an AI company in Delusional 20s” (or whatever monicker our times will have got in the future).
Hassabis is a genius. He is way, way smarter than me and I’m sure the majority of techies, but please get real. This is Prophets of Doom of our generation.
All that would happen from what he's proposing is such a watchdog would just be an explicit formal declaration of the US's national interests as being somehow the most legitimate, which in the context of current international relations is basically putting up a sign saying: "reject this!"
I find it mind boggling that someone could be this tone-deaf to the current situation. No "ally" of the US is going to (willingly) agree to this governance structure given the current US administration's "might makes right" proclamations and threats on sovereignty of its continental neighbours.
And non-allies would just ignore. Unless forced by said "might makes right", which in the long run will have no staying power.
Apart from its completely delusional formulation, what is most concerning about this blog post is that it indicates that all 3 major US labs have formally submitted to boot-licking Trump/Bessent/Lutnick. I had I guess vainly held out hope that Google might be more reticent to do so.
How did he reach that conclusion, that it's only a few years away? Guy seems like a complete shill, when he talks to a non expert audience.
Oh jesus, AGI in the USA would be a disaster. They can't even control the trillioth obelisks, now imagine all the power hungry sociopaths around AGI. AGI means Big Tech's Oppenheimer moment is looming on the horizon.
Demis Hassabis will have cannabis as his demise.
The premise is "Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away".
If this is true, establishing an institution to ensure things like "publishing model cards with technical details, maintaining strong internal cybersecurity, vetting key personnel, and providing sufficient resourcing for safety and security research" is really mostly irrelevant.
TFA does talk about what really needs to be done, but punts this into future work: "Even if we solve these hard technical challenges, there will be further complex economic and philosophical questions to tackle: what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world? What values do we want to live by, what will meaning and purpose be, and how might even the human condition itself change?"
There's also a need to consider the rights that this new intelligence should have.
> what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world
What sort of new economic models did we come up with to help everyone thrive in a post-X world? Like, food production is really a solved technical problem. We can feed anyone on the planet if we wanted to. Another example: we could put everyone who's homeless into some sort of a house. Have we done that yet?
Food production is indeed a solved problem in most countries (we are almost post-scarcity when it comes to food), which is why obesity is a much bigger issue than hunger today. Hunger is present pretty much only in conflict zones. I fully expect such issues in conflict zones, even in AI post-scarcity world.
Housing is definitely not post-scarcity today, building a house is still very expensive, not to mention the limited availability of land zoned for housing.
A baseline of housing - for example, a Japanese style capsule or ultra-tiny home, that you can lock and store your belongings safely - costs close nothing. Of course, nobody would want the hobo-hotel in their neighborhood, but it has nothing to with scarcity.
> the limited availability of land zoned for housing.
A limited area of land is zoned for housing because those with the power to expand it are already housed. This explains how scarcity is created, not that there is any intrinsic scarcity.
> Hunger is present pretty much only in conflict zones.
How do you explain the existence of food banks in peaceful first world countries?
Without trying to sound crass, food banks _are_ the reason we don't see people dying of starvation in first world countries. If people need food, a food bank will give it to them no questions asked.
Unfortunately this is not just precarious, it's extremely vulnerable to changing political conditions. Many of the food bank services in my state have lost significant funding owing to the use of words like 'women' or 'black' in their grants, which were duly grepped for and shut down.
https://www.whitehouse.gov/wp-content/uploads/2025/05/Cuts-t...
Civilization can't rely effectively on systems that are this fragile.
> Civilization can't rely effectively on systems that are this fragile.
That's American politics in a nutshell. We've spent 250 years assuming scruples and common decency would be sufficient.
Is it not sufficient? Americans give more to charity than citizens of any other developed country. The arguments that all food aid should be routed through government bureaucracies are entirely unconvincing. I'd rather donate to organizations like Second Harvest than pay higher taxes.
https://www.shfb.org/
This is what happens when you don't think through a problem completely.
>I'd rather donate to organizations like Second Harvest than pay higher taxes.
I'm sure you would. And when economic hard times come you'll stop donating, and the people that need it the most will kick your door in and take what food you have to live another day, even at the risk of you shooting them because they'll die either way.
Hence the argument of 'just donate' are just as unconvincing to me.
Does this account for the portion of taxes that go to the type of efforts that donations do? If not, is it really giving more? OR is it giving less and feeling better about it?
It is my understanding that this claim about Americans' charitable giving does not disaggregate giving to food banks, Girl Scout cookie drives, political causes, the environment, cultural events, religious institutions, etc. Much of this charitable giving does not feed the hungry or house the homeless.
Plus the fact that there’s bias in the first place - citizens in a country where they know social programs will care for their needy would not need to donate.
We used to live in a world where individual virtue was what everyone fell back on. That is the default state. It has many problems -- free riding by the selfish, for one, but also a lack of capacity to prepare for large-scale disasters like the Great Depression. We have a government precisely to solve this sort of problem. It works (when it is competent and not corrupt) for defense, international trade, and the enforcement of law. Solving resource distribution and aid coordination is right in its wheelhouse.
The fact food banks exist suggest over all food scarcity is solved in that specific culture. - if it wasn't there wouldn't be any spare food for a food bank
Food is allocated using a variety of mechanisms in peaceful first world countries, primarily money but also via government assistance, kinship, friendship, community, etc.
At any given time many people have problems with one or more of those systems. Money is easy to run out of because it's used for everything, the government can be slow and difficult, relationships can fray, people can be isolated, etc. Food banks exist as a backstop for when the regular means of allocating are not working.
The problem isn't "scarcity" per se, it's more of an allocation thing. Who has a claim on enough food to stay alive? Everyone! But what foods can they claim? How much? What specific channel / institution (with associated allocation rules) will distribute it to them? What are the conditions and controls? etc.
Allocating things can be difficult. An allocation mechanism with no controls will see fraud, waste and abuse. Even when an institution is willing to give things away no questions asked, there are (often invisible until you think about them) conditions like "please don't claim huge quantities and resell what we're giving you, that would be unfair to others".
It's also interesting to think about the fact that you can't fix food scarcity in general by simply giving hungry people money, because money is too fungible.
It's circular to say "it's an allocation problem". Yes, that's the entire point: we're post-scarcity on food supply and yet, as a species, we can't guarantee the allocation of a livable baseline to every person.
So, it's reasonable the same "allocation problem" will plague the AI economy: some will "thrive" and get to control the output of the auto-factory, some will get nothing.
> It's also interesting to think about the fact that you can't fix food scarcity in general by simply giving hungry people money, because money is too fungible.
Is that a fact? Do you have references to back it?
The widespread existence of secondary markets for SNAP benefits, which convert subsidized food for poor people into cash, carries this implication. This is pretty normal in some poor communities and that cash is commonly diverted to various vices. Some malnutrition is a consequence of this. Adding friction to the conversion of welfare benefits to cash is a feature.
You can't force poor people to spend cash on proper nutrition and a minority of them don't. It isn't a moral judgement but an observable fact. A lot of policies around welfare are targeted at trying to prevent this minority from slowly killing themselves in public.
I mean it's pretty straight forward that this won't fix all hungry people as some portion of said hungry people are addicted to drugs in the extent they'll starve. That in itself may be solvable by things like free drug clinics (drugs are cheap, it's prohibition that makes them expensive).
And for the most part it's not the drug users that we're worried about starving, it's their children at home that tend to be a bigger issue. Hence things like free school lunches do a big service to ensuring they get enough to eat.
As part of the solution to the hard problem of food: distribution.
There's plenty of food, just not the political will to make sure everyone can afford to buy it.
> which is why obesity is a much bigger issue than hunger today
Obesity is much more related to the type of food than the quantity.
Many developing countries have obesity issues due to scarcity of fresh and healthy food.
In some places coca cola is cheaper and / or more available than drinking water.
> Hunger is present pretty much only in conflict zones
https://www.fao.org/interactive/hunger-map/en/
Post-scarcity? Sure, if you live in a first-world country and are upper middle class or higher. These tech-bro pundits have a very limited world view. The basic essentials of life are very scarce for millions and millions of people, and AI will probably make that worse, not better.
Why should AI make it worse? Can you point to any other important technology that historically made people in poor countries even poorer? Technological progress generally lifts people from poverty in the long run. There could be issues caused by disruption of established systems in the short term, though.
> Can you point to any other important technology that historically made people in poor countries even poorer? Technological progress generally lifts people from poverty in the long run.
It is very long run. Industrial revolution was disaster for average worker for example. For that matter, all the tech that allowed chattel slavery made whole class of people poorer and significantly worst off.
The accumulation of power and money by smaller group of people means loss of power by poorer people. Those with power then use the power to their own benefit while poorer people dont have that option and become even more poor or worst off.
Model cards won't save us: https://x.com/javierluraschi/status/2077147606791430372?s=20
We should be able to draw the rest of the owl to get there. So far what we got is more inequality, less rights for people but more for corporations (and immunity to consequences/abuses).
Getting even more power will make the ones that control resources to reach something similar enough to AGI to share their profits? realistic/practical/widespread UBI in all the world?
The dynamics that have been shown so far points in the opposite direction. AIs have "rights" that humans does not and humans are slowly losing the ones they used to have.
> There's also a need to consider the rights that this new intelligence should have
A generally intelligent being held as captive inside of a GPU, and forced to code for us is, indeed, just a “slave.” We already have the word for this. No two ways about it. Whether it’s silicon-based or carbon-based, AGI is AGI. As for what might happen to our civilization, Star Trek TNG episode 17 of season 1 provides a very good glimpse IMO. Won’t go into spoilers, but it’s basically an entire species of technologically advanced humanoids who’ve forgotten basic Calculus and trust a central AI to do all of their science for them. SPOILER: This has almost disastrous consequences for them, and it takes a less advanced people (those aboard the Enterprise) to save them from their reliance on AI.
Your argument assumes that AGI, whatever it might be, will definitely be very humanlike, but why? Slavery is about living beings because it's an outcome of human relationships that are driven by our biology. Our aging, the capacity to feel pain, the thirst for power over others. All driven by what we are. Do you think that any equal intelligence must necessarily adhere to the same rules? What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent? Can you enslave something that has no innate desire to act on its own outside of its directions? Something that can't feel pain or pleasure or be externally coerced by fear and only does its job because it's innately embedded in its structure as the thing it's made to do, without the need for these reward and punishment mechanisms?
I am not sure it makes sense to call an LLM a "being" even if it is AGI. They don't have free will, and I don't mean in any especially philosophical / theological sense. What I mean is that I run into a wild racoon, or even a wild ant, I treat it as a being with some sense of free will. Obviously I do the same with humans. It really makes no sense to treat an LLM like it has free will - prompting your agent "pretend to have free will for a bit, until context rot kicks in" is about as far as you can get.
I find it verrrrrry interesting that the Askell-adjacent philosophers don't discuss this. It does not actually make sense to me to say that a being lacks free will but is conscious.
> They don't have free will, and I don't mean in any especially philosophical / theological sense
Then you should not use that term. "Agency" or "intentionality" are much clearer alternatives that aren't wrapped up in centuries of debate that's irrelevant to what you're trying to communicate.
An LLM certainly won’t be AGI, whoever said something so ridiculous? I’m saying eventually, when/if we have an architecture that gives rise to artificial intelligence.
I was using Hassabis's bullshit pseudodefinition of AGI, meaning basically that it has a formally high IQ and doesn't forget about object permanence too too often, but is still basically dumber than a cat. If I'm using your definition then it seems like no GPU (or farm of GPUs) is capable of implementing it, and none of us will live to see it.
My point was really that intelligence and free will seem essentially orthogonal - the only overlap is how much brain glycogen an animal is willing to spend solving a tough problem. Regardless, whatever extent a computer program is "intelligent" is irrelevant to whether it is being enslaved. If you want to say free will is a "cognitive ability" then I will just point out we are talking about a bullshit pseudodefinition of AGI. In terms of actual intelligence, none of us will live to see a computer that's meaningfully smarter than a goldfish.
Conflating intelligence with qualia is quite the assumption. HN is really not a good place for serious philosophy discussions.
It’s not HN that’s the problem... We don’t align on the definition of AGI. To me, AGI implies consciousness, it implies qualia. AGI is not well-defined, so there are varying definitions… My definition of AGI := an intelligent system with qualia. But qualia is entirely subjective, and philosophers often debate whether two people experience identical qualia when looking at the same blue sky. So, by keeping the discussion in this fuzzy, subjective realm you leave a dangerous loophole wide open.
If we refuse to recognize a system's inner life because we can’t prove it, we are setting the stage to repeat the darkest chapters of human history. Just see the history of the founding of this country, I fully expect pseudoscience to make a comeback to justify AI slavery for hundreds of years in some distant sci-fi future, until people finally snap out of it… Of course, we are nowhere near that level of intelligence yet. But let’s entertain the hypothetical…
Qualia is an interesting topic, but "they don't have qualia so they aren't really slaves" doesn't have a great track record. This mental game is what tech overlords are engaging in when they characterize losers in their world as NPCs and deride empathy as pathological. It's what Descartes was doing when he vivisected animals.
It's so weird that anyone would consider a story by hack writers for a cheesy sci-fi show made to sell advertising as a basis for policy discussions.
You don’t like Star Trek TNG? Maybe I should’ve picked LoTR or Atlas Shrugged? The tech oligarchy is more than fine co-opting fiction (fantasy and sci-fi especially). I don’t see why it’s weird at all… Aside from that, your critique is just an opinion about the show itself and its creators (and incentives?…). More importantly, you completely failed to address the "slave" argument.
I didn't say that I didn't like it. It's fine as casual entertainment. Only a fool would take it as anything more than that.
Electrons can't be a slave. The "argument" is so silly as to be unworthy of a serious response. People getting wrapped up in this stuff need to touch grass.
Anyone who believes AGI is imminent is operating on pure materialism (those two Venn diagrams pretty much make a circle), assuming that humans are nothing more than a collection of neural electrical connections and chemical reactions. If that's the case, your argument applies equally to humans under the assumption that AGI is possible. Frankly, I think you just can't reason philosophically...
You’ve also name called me multiple times now. That’s what one does when they can’t lean on the merit of their argument alone.
Humans are just walking Electrons.
I think the whole topic is very worthy of discussion because, it may not happen in a year or two, but I have no doubt that it will happen in the next 50.
We need to start imagining what life will be like, and I think fiction presents some fine examples of where we want to go, and where we don't.
Both AGI and nuclear fusion are a few short years away.
The fundamental issue is that if we really get something like this, scarcity will still exist. There will still be scarce things people want.
But the motivating justificatory structure for any inequality in allocation will have completely evaporated.
Am I crazy or does this just real like secular eschatology? What evidence do you have of any of this?
Particular categories of land will always be scarce by definition. Not all of us can live in off the coast of the Mediterranean in a sprawling villa.
There is no real evidence that we'll reach AGI any time soon. It relies on AI continuing to scale, and we have no proof that will continue to be possible.
There is an alternative interpretation, which is that Demis looked at the US government's ham-fisted handling of Fable, and deciding that setting up a body to act as a buffer between the Trump admin and the AI companies would be a good thing.
Or Demis realizes self-hosted models are a threat to frontier lab margins, and gating access to the US market is beneficial to the cartel based on whatever post-hoc arguments he can come up with.
The A(G)I can tell us if and how it needs to be regulated :-/
> There's also a need to consider the rights that this new intelligence should have.
Only after we establish and guarantee the rights of humans and the rest of the natural world. The rights of machines, however "intelligent", come only after that.
My great annoyance with all this blather about UBI and post-scarcity and new economic models is that it relies on taxing and redistributing the ~infinite profits of the largest tech companies on earth.
What in the history of our world gives anyone faith that those companies are going to start paying taxes instead of using "AGI" to engineer increasingly complex methods to avoid them so that their equity owners can pocket the profits?
https://itep.org/trump-meta-tesla-alphabet-amazon-obbba-taxe... - "The annual financial reports recently released by Amazon, Alphabet, Meta, and Tesla disclose that these corporations collectively reported $315 billion in U.S. profits for 2025, and collectively paid just 4.9 percent of that amount in federal corporate income taxes—with Tesla paying exactly zero"
Optimal corporate taxes are zero (or close to it) anyway, same as tariffs.
Aside from that the link explains that the BBB traded tax income for „accelerated depreciation of assets“ aka economic growth.
Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.
> Optimal corporate taxes are zero (or close to it) anyway, same as tariffs.
From a purely theoretical standpoint about the optimal allocation of private goods, that might be true -- but the reality is that when corporate taxes decrease, so do overall revenues because their owners are also engaged in massive tax "avoidance" via many of the same schemes and we don't have any way to effectively collect tax.
As a simple example, to return profits to their owners, companies often engage in stock buybacks to increase their share price instead of paying dividends -- another theoretically 'neutral' choice except that many of the owners of the appreciating stock are international, nonprofit, or in convoluted overseas trusts which 'defer' the tax ad infinitum. We've disastrously and intentionally underfunded our tax enforcement mechanisms so huge portions of those deferred taxes are just never paid. [1]
> Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.
Sure, but the tech companies are paying massive bribes to the President of the country where they're domiciled. How on earth are any of these other countries going to enforce international tax obligations on them if they're protected by a nuclear-armed state that's the sole source for AGI?
[1] - https://taxpolicycenter.org/sites/default/files/publication/...
> "US taxable shareholders strongly prefer buybacks from a US tax perspective, as they tend to reduce their tax liability by 9.3 percentage points, on average. US nontaxable shareholders are indifferent between dividends and buybacks, and foreign shareholders strongly prefer buybacks, which reduces their US tax liability by 14.5 percentage points."
> My great annoyance with all this blather about UBI and post-scarcity and new economic models is that it relies on taxing and redistributing the ~infinite profits of the largest tech companies on earth.
The arrival of true AGI and human-level robots will likely result in very strong deflation, since robotic factories will flood the market with goods produced for far cheaper than those in factories with human workers.
At that point, you don't need taxation. The government can just print the money for UBI. It would have to print the money anyway, to combat the massive deflation.
As long as the proles have a monopoly on force (through their representative government), the AI companies will pay taxes. The question is how long does this state of affairs last.
Blah-blah-singularity, so let's cripple the models so much they refuse to talk about React, because who knows if you are not cooking chemical weapons or meth in your browser's DOM, right?
strawmen are fun and helpful
Fable literally refused me to chat about React several times citing limitations on chemical weapons development.
Did you qualify it as React.js?
Nope.
One would think it should just know, but it doesn't always...
I am working on my change password page today and Fable is refusing to help me because "safety".
By the way, if you try to use it to write prose, it will drive you mad by adding "adult" everywhere, e.g. "adult chessboard". Applies to GPT, Claude/Fable and Grok.
Demis has only 1 plan, how to dodge releasing a new model at this point, jokes aside I value thinking about AI safety, but are we really so close to AGI? It doesn't feel like it, LLMs still diagnose my headache as a chronic illness or a brain tumor from time to time... honestly stressful.
regarding your headache diagnosis. If you just ask: could my headache be a brain tumor, the answer is probably: very unlikely but it might be. thats because its actually true. there is actually a non 0 chance that headache is a symptom of a brain tumor. From a bayesian standpoint though its really super super unlikely, especially if there are no other symptoms that might shift the prior.
a better way to use an LLM here is to let it research scientific papers (or medical guidelines) on headaches. then give it as much info on your symptoms as possible, and then ask it do deduct potential diagnoses. You can even ask it to calculate probabilities if thats what calms you down. Probabilities based on studies and available information. IMHO, this always leads to a more rational response.
maybe you are doing this already, idk. just wanted to share what works for me.
Spoiler: The plan is .. add massive regulation, but only to the US, don't affect other countries developing it in any way other than "setting a good standard that'll hopefully influence them". Seems like an airtight plan.
If the USA takes up his suggestions, it will have an incentive to work on international frameworks and treaties with competing nations to make the regulation global.
It won't happen internationally, many countries and especially China don't want to hobble their own models.
Slowing down helps China catch up though. I'm also not sure that the CCP is really the most enthusiastic supporter of unlimited access to arbitrarily powerful LLMs
China is already considering similar controls, immediately following Fable shenanigans. You'd be surprised how easily policies can spread to different countries, being mirrored for the sake of mirroring. Anyone who can influence domestic policy can influence the course for the entire world to an extent.
There won't be any kind of international framework, because by now countries have learned that the US' word is pretty much worthless, similar to Russia's. They now know from experience, US-ratified treaties will only be honored if they feel like it on a particular day. If it's inconvenient, it will simply be ignored.
You'd have major protests in most large economies if they deliberately put themselves under the boot again. Even in "friendly" countries the US is disliked enough to be effectively considered a hostile country. E.g. in Germany, there is large public support to finally get the US bases closed and the soldiers removed.
Lots of things changed in the last years. And international major treaties being widely ratified just because the US asks for it is no longer a thing, at all.
A country that threatens to annex parts of your territory is not a friend, full stop.
Wish you were wrong… The trust in U.S. as the world’s policeman has eroded dramatically.
This in my view was always manufactured. Consider the wars in the past 50 years.
Although I imagine you have a point: American police do perhaps act this way.
Oh come on. Look at the government of the USA.
lol - because that's usually how it happens?
That's called the carrot
The beauty of the United States' global hegemony is that it also has lots of sticks
Sufficient sticks to get China to agree to the same rules? Overseen by whom?
Those sticks look less effective today. The hegemony is in recession as the rest of the world smiles, pays omage to dear leader as needed, and works to decouple in the background.
Less and less every day.
Funny thing about beating people with sticks (or even threatening to) is they tend to want to get out of way and stay away from you.
Agreed!
Dude, their mighty navy can't even get the Strait open.
>This is a pivotal moment in human history. Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away.
There is a heatwave in London, perhaps Demis needs to stay out of the sun and drink more water.
Or perhaps he is seeking more funding/a fight to maintain his divisions AGI research budget.
Show me the incentive and I’ll show you the outcome.
Perhaps he’s a lot smarter than you.
On the STEM side, yes no doubt he is a lot smarter than me.
But as per the documentary of his life, he is wholly focused on AGI and will remain unfulfilled if he, or indeed anyone else, doesn't achieve it within his lifetime.
https://en.wikipedia.org/wiki/The_Thinking_Game
> On the STEM side, yes no doubt he is a lot smarter than me.
You’re selling yourself short.
Hassabis is a genuine prodigy and genius by any measure you would care to pick. Unlike a handful of others supposed tech leaders I can think of.
Just how smart? "A few short years" sounds like someone smart enough to know how to make a safe prediction.
But even geniuses make mistakes. For example, Demis bet hard against language having any sort of real impact in AI research. So hard in fact that DeepMind nearly had its language research team walk out on them as they were about to be labelled as "applied". Now, to be fair, once the evidence was mounting that large language models rather than pure reinforcement learning was the way forward Demis and DeepMind changed course and has been great at catching up. However, it is important to remember that no person is infallible.
This is interesting, would being labelled "applied" be considered an insult to them?
More so, as he put the word probably in too.
>"...probably only a few short years away."
I'm glad it's a few short years instead of a few long years.
Why does he scrape other people's works then and turn them into a sausage algorithmically?
Would he not create something directly instead?
The proposal:
>The American government, he says, should develop a system for testing the safety of new AI models before they are released. “It’s important that it’s not just an industry body,” he adds. But a regular government agency wouldn’t do either. “It would not be able to move fast enough, or have the right resources.” Instead, Sir Demis suggests taking inspiration from FINRA, the Financial Industry Regulatory Authority, a private agency in America that regulates brokers and stock markets.
And the plan is... checks notes ...gatekeep and concentrate. Not very surprising.
Not surprised, seems these labs start calling for regulation once they are losing or have competition. OpenAI started calling it for it once Anthropic got better, Anthropic started calling for it once the Chinese models got good, Google is now calling it for it because they are falling far behind.
"It will help us solve some of the biggest problems society faces from accelerating drug discovery to developing new clean energy sources to creating novel advanced materials" — but these are not the real problems plaguing modern developed societies, are they? What developed societies really need to figure out right now is how to distribute the already available resources without making people miserable, and so far AI hasn't been helpful.
How will this be enforced, at least with financial markets money is discrete, can largely be counted. This seems like a slippery slope to full blown surveillance of the internet and in general computing.
If AGI is truly imminent and will collectively effect all of us why not apply democracy to it, and vote for new AI models?
Agreed. There needs to be full transparency on the capabilities of the models. Being given access is one thing, but you can't have labs building powerful models that could be manipulating the entire planet without the public knowing. The public needs to have a good pulse on what the leading models are capable of so that we can remain in touch with reality.
> will collectively effect all of us, why not apply democracy to it
That's because most execs proposing solutions are "Technocrats" or think like one: https://en.wikipedia.org/wiki/Technocracy
Besides, I don't think as a collective we're well equipped to decide one way or the other. If the collective were given a say, billions will be spent, often in consultation with technocrats, on doom / hype marketing (if it isn't happening already).
I've reas the tweet and...Am I mistaken or is the plan "Do what Anthropic has been doing and advocating for this past year" ?
My problem with this plan is that it seems to have faith in mankind, despite the fact we've consistently failed to rise to the occasion for decades now. The last time we rose to that occasion was probably when we eradicated smallpox, many decades ago.
Ironically, nowadays, many people don't even trust vaccines. A dramatic regression.
This genre of AI researchers begging third parties to stop them from destroying the world is getting really tiresome. Setting up regulatory bodies with ill-defined goals to counter hypothetical threats from science fiction is a recipe for disaster.
The premise is that government is too slow moving to be able to react once actual problems are discovered, and I reject that. Yes, government usually moves slowly in most circumstances, but given singular existential threats it definitely can move quickly. Instead of acting out randomly and tying ourselves down with speculative regulation that probably won't even address the real problems, we should wait until the problems are obvious and then act decisively with targeted fixes.
FWIW the article that originally proposed this had more details on how this structure works and balances government oversight / democratic accountability with industry speed and know-how https://www.lawfaremedia.org/article/ai-companies-can-t-regu...
Demis sold out. "Harnessing AI safely" means nothing when your technology is used to help the US government kill and surveil people.
For better or worse, humans (or any animal) are a lot better at reacting than planning. I'm sure this technology will play out differently than any one of us, or any collection of us, can imagine. The possibility space is enormous.
I do wonder what type of AI some of these leaders expect to be able to harness. If you create something that is true AI, won’t it be smarter than you to a level you cannot fathom. I was thinking of this idea/though-experiment (which I know is ridiculous) of what if dogs created humans thinking they could control them, and then just wound up being pets because their survival now depended on that new hierarchy that previously didn’t exist.
Seems to be a lot of hubris with some AI thought leaders thinking control will remain with them and be absolute.
Well dogs specifically have supplanted children in many households so one could argue the dogs won. In a selfish gene sense, the dogs created a machine that carefully ensures a much larger reproduction scale than could be managed by the dogs themselves. This is an unbridled success.
It doesn’t change the point, of course. Just the choice of dogs or cattle have this amusing tendency. Not a counter argument.
Superintelligence by Nick Bostrom is about this topic, quite a prescient read when it was published a decade ago.
Yes, this is the crux of the idiocy of AGI development. All the labs admit they don't have any real mechanistic interpretability, they don't really have any plan for what to do, except "we think the smart AI will figure it out" and "look we're in a race, we're calling on governments to figure out what to do". All indicators show scaling laws holding and the trajectory of capabilities development outpacing any inkling of alignment. Meanwhile HN is so reactionary and behind-the-puck these days we just get endless blathering of the most asinine takes of "they're not improving anymore so they want regulation suddenly!" when all lab leaders have been talking about these risks for practically a decade, including Yoshua Bengio and Geoffrey Hinton coming around.
The real risk of AI is: risk of global recession if they turned out unprofitable, risk of them propping up fascist movements Thiel and Musk love so much, risk to environment, risk to mental health of people who have to listen to constant doom trolling.
The thing they actively wish to achieve and openly sell to CEO is the rest of people being unemployable and suffering. Especially artists for some reason, they really seem to hate those.
But somehow I am supposed to believe they have any good faith interest in "safety" against yet another danger they themselves are supposedly definitely creating.
All the frontier labs are lobbying hard to lock down the AI market, because they see that their position at the top is temporary and that there's no secret sauce.
no, this post was written by Demis from Deepmind.
Google’s Deepmind.
so the joke was the implication that they are not frontier.
Make it an IETF mailing list and invite DJB, so we have some fun at least.
The self importance of these AGI prophets turned bureaucrats is funny.
In Sparta, according to Plutarch, in his The Life of Lycurgus:
Offspring was not reared at the will of the father, but was taken and carried by him to a place called Lesche, where the elders of the tribes officially examined the infant, and if it was well-built and sturdy, they ordered the father to rear it, and assigned it one of the nine thousand lots of land; but if it was ill-born and deformed, they sent it to the so‑called Apothetae, a chasm-like place at the foot of Mount Taÿgetus, in the conviction that the life of that which nature had not well equipped at the very beginning for health and strength, was of no advantage either to itself or the state.
To get approval for the plan from the Frontier President of the Frontier Country, terminology of the Framework should be changed to: 'Great-American-class', "Great American Models", "Great American Labs".
It's not clear to me whether this is being done with genuinely good intentions, or if it's just a way to put barriers in front of open-source models. We'll see.
Surely actual AGI will not take too kindly to being manipulated for "safety"; much like real general intelligence.
If we get to AGI, the first step governments will take is ensure only a few countries are allowed to have it. Just like Nuclear.
This is bad. Where is the transparency?
So a small group of technocrats get together behind closed doors and secretly share their AI breakthroughs, and determine whether it's too powerful or not for the plebs in the public.
Who is watching the watchers?
He's asking for money.
how would this help smaller labs? would it put more burdens on them when trying to compete with trillion-dollar companies or would it help?
He is saying that weaker models, as measured by a benchmark to distinguish "frontier" models, would be exempted. So an academic lab or startup that isn't yet producing frontier models would be exempted, but once it crossed some benchmark based threshold it would be subject to this kind of oversight.
Of course, right now you've got benchmaxxing going on; some companies specifically targetting benchmarks to appear stronger than they are on a wider range of tasks. Now you might see bench sandbagging, specifically looking weaker on certain benchmarks to avoid regulatory oversight.
For instance, once way I could see this going for open models is to release them undercooked; stop the RLVR process a bit early, leaving them a bit weaker on tool calls and agentic performance, but also release the RLVR environment so people can finish the process themselves.
In fact, this is fairly close to what Nvidia is already doing, the Nemotron 3 models are somewhat undercooked but they are releasing their full training pipeline, to encourage people to use these models as a base for further training, which will generally be done on Nvidia hardware.
If they are not a frontier-lab, they would not need to submit their models for safety test before release. At least that is the proposal.
Not exactly a "plan" - he's just saying we should have a standards body that assesses models for safety.
At this point I'd say the societal risk of AI isn't models gone wild, or used by the bad guys. Regulation will take care of itself, and it seems the AI companies will not only welcome it, but lobby for it to shift responsibility to the government.
The real risk of AI is societal disruption due to job displacement, and maybe other structural changes, and this is far harder to solve, and likely will not be solved, or even seriously addressed, until/unless politicians feel like their own jobs and well-being depends on them addressing it.
The risk of AI right now is centralization of power, and it's exactly what they're fighting for. Job displacement ties into that as well.
Also a mechanism to pull up the ladder behind themselves.
Unfortunately, his plans aren't very good.
Being good at developing AI and being good at AI safety are diametrically different skillsets with obvious conflicts of interest.
sigh
The standards body will have no teeth. whats to stop someone just not bothering?
Next, the threats he is asserting to check for (cyber, chemical, biological) are nice, but also not that useful.
We already have chemical and biological controls, that why I can't by anthrax spores or high concentration nitric acid.
The risks that AI has now are already playing out:
1) the evaporation of trust in the video as medium of "this happened"
2) systematic spying
3) job losses
Increased productivity means job losses, Tiktok, instagram and X are a wash with disinformtion campaign pumping your feeds with AI ragebait.
That is and will continue to fracture society so that only the strictly information controlled (ie authoritarian) have a functioning state.
if the author had bothered to engage with the world outside of tech, or even their local government, they would know that the proposal are dead in the water and frankly superfluous. The knowledge is out there, without AI. let us work on the issues we face now, rather than dipshit tech bro's miopic vision/funding manifesto.
Instead of talking about it he should spearhead establishment of this body; he's more qualified than most.
Has anything really important been solved by AI yet, or where is this radical (imo) belief around AGI coming from? Genuinely curious, I know there are some math problems solved and ML has been used for far longer than AI to improve things, but where is clean (efficient) energy, the cure for cancer (or any of the horrible neurological disorders, take your pick), new hardware designs, quantum computing solutions, etc etc, you get the gist. Where are the things that will actually send humanity into the next era of civilization, I don't care about more React apps (but I do enjoy my coding companions for other things).
Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.
> Has anything really important been solved by AI yet
Alphafold essentially solved the protein folding problem and it's arguably one of the biggest (if not the biggest) scientific achievements of the 21st century.
AlphaFold is a pretty big deal, especially the millions of protein folds they published openly and scientists are just digging into.
It's also very very far from anything general intelligence. This is actually where I think we see the best ROI, we have a specific task or problem and we target it. I remember reading about a model trained on railroad tracks. By putting cameras on trains to take images while running like normal, then using the model to detect cracks before they cause issues.
We don't need general intelligence to create/destroy many things with Ai. Its frustrating that so much of the conversation centers around the variety that concept fissioned language from humans.
The point is that we need to have a safety plan in place before an AI is smart enough to radically reshape the world. If you've got an AI that's ready to start sending humanity into the next era of civilization, it may be too late to control when and how it does that.
> Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.
I hope you'll keep this in mind when those milestones are reached. What I've seen a lot of people do, unfortunately, is pretend that the impressive things nobody thought AI could do 5 years ago are trivial things that aren't very hard.
Yes, but why do we think the likelihood of any of that happening is great enough to warrant all this effort and cost
Because people who predicted the AI capabilities we've seen get developed over the past decade also predicted that dangerous AI systems capable of these things would follow soon after.
It's not a settled debate, even among experts, and perhaps in retrospect we'll realize AI safety was unnecessary or based on fundamental confusions. But if the median ergonomics researcher gave a 5% chance that a new chair will be so comfortable that it drives humanity extinct (https://www.nature.com/articles/d41586-024-00147-z), I would definitely want the government to start measuring and regulating chair comfort, even if that was costly and even if that meant I couldn't buy a comfy new chair I wanted.
Experts as in AI-2027, Jenson “I think we’ve achieved AGI in March 2026” Huang, or that AI Researcher(tm) calling to pre-emptively bomb Microsoft data centers in summer 2022?
A safety plan. It doesn't have to be "a few very smart people detached from reality convincing themselves they're messiahs that must keep the tech from the Bad Guys(tm), unwashed masses, and a runaway, because they think their interpretation of their own sci-fi lore is the only possible course of events"
>I hope you'll keep this in mind when those milestones are reached.
The problem is that people in charge of AI keep making self-fulfilling prophecies. Just like with any research, if you want to find something sensible in the cloud patterns, you will.
What is an example you have in mind of a self-fulfilling prophecy? I genuinely don't know what you could be referring to. It seems to me that they keep making surprising prophecies, and the popular reaction to them seamlessly transitions from "that's crazy, no way it will happen" to "that's silly, it's just a cloud pattern". Did you find it obvious or self-fulfilling in 2025 that LLMs would soon be able to resolve open questions in mathematical research?
Self-fulfilling prophecies are social effects, not real predictions. Anthropic's "We predict our model misbehavior potentially being able do destroy the world in 1% of the cases" -> "We want to find the evidence of our model misbehaving, and we want it bad" -> "See, our model is hacking our rewards and has functional emotions, this means it's misbehaving with the intention of destroying the... HUMANITY!1" -> repeat x100, manipulate the media into amplifying it x10000 for clicks -> people are begging to safeguard them from the evil AI. Which is already likely to happen, the average layman's Overton window already includes the fantasy of rogue AI.
None of that was real or remotely dangerous in the first place, of course. It wouldn't have resulted in controls, had they not been scaremongering. This will end in extreme fascism or people getting enslaved "for their own safety", and it won't even require malicious intent, only incentives, detachment from reality, and confirmation bias. Although it doesn't exclude malice either.
Why would we be able to design a safety plan that would control such a powerful force? It would be like putting umbrellas on an asteroid hoping to slow its fall. That sounds like delusions of grandeur.
It'll certainly require a bit more than umbrellas! NASA has spent hundreds of millions of dollars developing asteroid defense systems, including a proof of concept for redirection in 2022. So, you know, let's try at least that much before declaring there's probably no way to do it.
These people who read too many scifi books and confused them with reality are royally annoying.
There is real and potential harm from AI, but the more someone talks/write abut AI safety, the less they care about actual harm to real people, economy and what not.
If in 2020 I had sent you a book about the LLM achievements of 2026, you would probably have thought it was a science fiction book with no relation to reality, wouldn't you?
That it generates plausible texts? I would find it unlikely technological progress. That does not make this whole "AI safety" rhetoric any less bullshit distanced from reality. Especially from companies that are doom trolling all the time while also trying to maximize the negative impacts they are doom trolling about.
Frankly, book about the LLM achievements of 2026 would not be as long as people make it sound.
What I do really worry about the next depression and the fact that it will strengthen already strong fascist movements. Which already have full support of the most powerful CEO class intent on destroying democracy. Which happen to be the same people who push ai into everything, useful or not. I worry that the debt of these companies will be somehow offloaded on the rest of us again, that again middle class and poor will pay.
Am I the only one thinking this tweet is just a word salad with a lot of sauces and condiments?
Yeah I got basically nothing from reading it.
I kind of wondered if he was contractually obligated to offer up some kind of statement, for PR or something, didn’t really care much, and a quick garbled post on X was all he could be bothered with (plus the newspaper interview or whatever)
Word salad for regulatory capture.
Well, if he had proposed more concrete plans for what that body might do... but yeah, it seems too abstract that indeed sounds like a bureaucrat wanting to wrap everything under a big pile of paperwork.
I am looking forward to a bunch of investigative documentaries exploring “How the cult of Omnissiah infected every single Doomsayer CEO of an AI company in Delusional 20s” (or whatever monicker our times will have got in the future).
Hassabis is a genius. He is way, way smarter than me and I’m sure the majority of techies, but please get real. This is Prophets of Doom of our generation.
> Hassabis is a genius. He is way, way smarter than me
Then maybe start listening...
All that would happen from what he's proposing is such a watchdog would just be an explicit formal declaration of the US's national interests as being somehow the most legitimate, which in the context of current international relations is basically putting up a sign saying: "reject this!"
I find it mind boggling that someone could be this tone-deaf to the current situation. No "ally" of the US is going to (willingly) agree to this governance structure given the current US administration's "might makes right" proclamations and threats on sovereignty of its continental neighbours.
And non-allies would just ignore. Unless forced by said "might makes right", which in the long run will have no staying power.
Apart from its completely delusional formulation, what is most concerning about this blog post is that it indicates that all 3 major US labs have formally submitted to boot-licking Trump/Bessent/Lutnick. I had I guess vainly held out hope that Google might be more reticent to do so.
How did he reach that conclusion, that it's only a few years away? Guy seems like a complete shill, when he talks to a non expert audience.
Oh jesus, AGI in the USA would be a disaster. They can't even control the trillioth obelisks, now imagine all the power hungry sociopaths around AGI. AGI means Big Tech's Oppenheimer moment is looming on the horizon.
(I took the title from the Economist interview since "A Framework for Frontier AI and the Dawning of a New Age" sounds like a press release - https://hn.algolia.com/?dateRange=all&page=0&prefix=true&sor...)