This was interesting right up until "The fund pays every eligible American the same amount each year. "
I'm in Australia. I've contributed my share of dirt to the delta. Why do I not get a share of this?
I get that the frontier companies are (for the moment) US companies. But that's just corporate ownership, it's not what we're talking about. We're talking about compensating the people who wrote the training data for their contribution. That contribution came from all over the world, so the Corpus Fund needs to be paid all over the world.
Set it up in the UN, get the UN to provide the training data sets as a common good, and have the UN collect the money from all AI companies using the training data sets. And the UN should distribute the money in the most equitable manner globally (so most of it going to alleviate poverty, probably).
I'd happily trade my collected years of shitposts to help folks get out of poverty.
Why do you think a UN-hosted process would work? Has the UN done similarly intrusive things before and been effective at it? How do you account for Chinese frontier and open weights models, which are just months behind American models and subsidized by a sovereign state that does not share any of these premises about intellectual property?
A lot of these posts seem subtextually premised on the idea that it's possible to put the genie back into the bottle; that if frontier labs in America didn't sign on to this tolling scheme, our recourse would be to halt the progression of AI completely. But that option does not exist, unless we're going to fight a world war to create it.
Well, I mean, the alternative is for a US-based process to be put into place, followed (or led) by an EU process. The premise of my comment is that no matter who hosts the process, China isn't going to comply, so it really just comes down to the US and Europe.
And the US isn't going to comply either. Just like it never recognized the International Criminal Court in The Hague. In that respect it's at least as bad as China.
The only jurisdictions where training data is going to matter are in the US, Europe, and China. I think we're just talking past each other. My subtextual point is that none of this is ever going to happen.
I have additional essays coming out that will address this exact issue and other issues I know that people will raise.
I’m building the essays series around arguing for practical policy I believe can get implemented and am sequencing it as thoughtfully as I can. I just can’t fit every argument into every essay.
It's a great idea, I hope you get traction with it :)
I am personally coming to the conclusion that having these vast repositories of knowledge that can actually talk to us is actually great. We have some issues to solve, but the end-state of having a global repository of all knowledge that can talk to us and answer questions is actually an amazing outcome.
We just need to solve those problems first; mostly getting past the AI bubble and the massive over-investment, and then solving the hallucination problems. I don't believe either of them are insoluble.
I do worry about how future generations move on from this, though. In the same way that 90's music is still effectively the zeitgeist, and we will never move on from that, because of the way that streaming services work. It's a rare new band that can compete with (e.g.) Nirvana when appealing to that segment of audience, a competition that Nirvana themselves didn't have. So we are effectively locking in Nirvana as The Disaffected Youth Grunge Band for the rest of eternity. So similarly, we are in danger of locking in the current state of the world to the training data, and never being able to move on from that, because any new zeitgeist has to compete with this one on unequal footing.
Imagine that AI lives up to its promise, gets captured by a handful of corporations, and ushers in a world of untouchable über-oligarchs that de facto rule the world. Would it have been worth it then? Can this problem even be solved?
I personally believe that it's more likely that the current batch of massively overleveraged AI companies go broke in the next year or so as the AI bubble bursts. Hopefully the USA manages to get its democratic shit together enough to not bail them out (as happened in 2008), and they just... stop.
The OSS models are almost at the same level, and are progressing fine. The technology continues to be useful, even if all the oligarchs go away.
My spidey sense goes off when we refer to open weight models from Chinese companies trained against frontier models as OSS... as if it was some Torvalds types maintaining a public git repo.
I think those open weight models exist because openai, anthropic, and google rule the roost. And I don't think things would continue as-is if those guys implode.
The technology has certainly moved faster because of all the money thrown at it recently, but it existed before that, and was progressing before that. So no, I think OpenAI, Anthropic only exist because of the open-weight models (that came before them) rather than the other way around. Google is a special case; they invented the transformer and then published it instead of commercialising it, so they actually weren't evil (for once), or they didn't realise what they'd done until too late.
I think things will continue perfectly fine if those guys implode. They didn't invent the technology, they're not the people who are advancing it most (because everything they do is proprietary and undisclosed), and their efforts at commercialising it are just getting in the way of the real research.
Imagine that AI lives up to its promise, gets captured by unaccountable bureaucrats and socialist dictators, ushering in a world of untouchable über-oligarchs that de facto rule the world. Would it have been worth it then? Can this problem even be solved?
I agree UN sounds like a good organization to help distribute the wealth created by AI to the world. But this idea won't be considered by the current U.S. administration. In that case what other countries can do is probably to tax the AI companies at rates higher than regular companies.
The USA helped form the UN as specifically the organisation to do exactly this kind of thing. It's a shame the current administration can't play nicely with others.
The current administration is also playing strange games about export controls (can we run Fable yet? Kinda. Maybe). I think if they keep this up they'll just be shooting the US AI industry in the foot and the Chinese models will take over as the frontier models.
Maybe the UN can levy the USA for this, and leave the USA to collect that levy from its AI companies.
It's about how to benefit the entire human species, or at least to reduce human suffering across the board. If there is famine in North Korea, then the surplus food from the world flowing into there won't be an issue right?
Yes, that would be an issue because it allows the regime to divert more resources to weapons. In general major famines are always caused by corrupt or incompetent political leaders rather than lack of food.
Famine could be caused by drought or flood or other numerous things. When the country can't produce enough food, its people would need to rely on food from outside.
Yes, and when the local officials are corrupt or incompetent then little off that outside food aid reaches the people who need it. That's why giving food to a country like North Korea does more harm than good.
Fortunately we don't have to care much about questions asked by shithole countries today because the UN is virtually powerless. Let's keep it that way.
The UN is a terrible organization for that. It is a corrupt, morally bankrupt institution. Giving it any more power or legitimacy is not a net positive.
100% — publish the hidden research, the value is in the discoveries, not in the dividend. With all due respect to the author, it feels like he missed the entire lesson of history.
Who would pay you? OpenAI has between $600 billion and $1.4 trillion in debt, and no hope for profitability. Anthropic, who have a much more modest $35 billion in debt (by using Sam Altman as a human shield), also has no chance of ever being in the black, pending some freak event.
> This was interesting right up until "The fund pays every eligible American the same amount each year. "
There could be so many other ways to set this up. Enforce a higher tax on any business selling AI models that have capabilities greater than some threshold, and use it to fund development and infrastructure project like roads, hospitals, schools, etc. Or you could even do a negative income tax[1].
Author here. Thanks for reading. I agree there are a lot of levers we can pull to work towards something better. I've structured this essay series as a sequence of nested regulatory solutions so in the next few essays I propose additional structures with instruments like this. They're sequenced in a way I believe that can be pragmatically implemented and start showing progress in the next decade or so by the US. So stay tuned!
I was referring to the potential of actually collecting some of this fund. But yes, essentially in this situation that is correct - I would be doing nothing and also helping people get out of poverty.
As long as places like Taiwan are effectively blocked from being in the UN[0], I don't believe we should be adding more to their power and responsibilities than what they have now, au contraire!
If it was a truly world representation, this might be different. But if things like health are sacrificed (ie. no WHO access either), I don't think they really deserve the benefit of the doubt.
> Frontier science looks different today. It's rooted in model weights and GPUs. It is flooded with token spend and agentic loops. It blooms in data centers.
This seems like handwaving to me. Even the actual frontier science that is using ML (e.g. AlphaFold) isn't based on "token spend" or "agentic loops". I personally cannot think of a single example of frontier science that is rooted in LLMs. I am sure there are a few examples, but the idea that frontier science has somehow completely shifted its trajectory and methods based on LLMs feels like quite a stretch to me.
We cannot always want to capture only the (temporary) winners whenever we see a lucrative business and expect to share a free ride. I'd also assume that most of the revenue these AI labs are making is turned into depreciating fixed capital (hardware) and OPEX at this point.
Why don't we capture Meta and Google as they allegedly take advantage of more publicly available information for profit? Let alone the truly valuable knowledge, like mathematics, has nothing to do with the majority of garbage posts that an average person would "contribute" on social media.
If we really want to tax or nationalize some economic activity, then, in my opinion, the target should be what it takes from society, not what it produces for society. By this logic, we should tax all labs, including those lagging ones, that utilize the public knowledge.
However, if everyone can access the public knowledge without rendering it less useful or reducing its available quantity, there should be no reason to tax it.
I agree we’re in an interesting era where frontier research has shifted from mostly publicly funded to mostly private and it creates challenging incentive structures especially regarding externalized costs of research.
Did you have any thoughts on my argument of how public knowledge does get damaged by the proliferation of AI over time?
I don't understand how knowledge, either public or private, could get damaged.
Though the income of the individuals and businesses that rely on the expertise of the knowledge would be damaged. Is that what you meant?
Edit:
At this stage, the revenue made by the AI labs is almost entirely spent on the formation of fixed capital and opex. The demand is mobilizing physical resources with money. Atoms are relocated and reconfigured into compute racks. But eventually, the created productivity will perhaps make supply-elastic goods extremely cheap and abundant, while the supply-inelastic goods will be worth even more relative to the elastic ones.
After all, money is simply a token for the transmission of physical resources. It doesn't create stuff out of thin air. When new stuff is created, it just makes money cheaper, so that the banks can respond with more money to counteract it. More stuff -> deflation -> more money creation allowed to undo the deflation.
But the "exchange rates" between different goods and services will diverge. That's also why I don't think a direct money transfer like UBI would fix the problem, when it doesn't change the divergence of relative economic values of different goods. Let's say, extremely cheap software and entertainment, but unaffordable healthcare and housing. More money for everyone doesn't make limited resources available. So, what I am leaning into is some sort of Georgist policy. That could hopefully mitigate the price divergence, assuming that AI cannot make every commodity equally abundant.
Well, lots of ways. One is some degree of model collapse, as the slop-enshittified Internet itself is ingested as training data--despite the AI companies' best efforts to prevent this, they won't be altogether successful.
But the more consequential one may be that few are motivated to contribute more training data to make Dario or Sam richer. This is already playing out in open source. People write open-source so humans can use it, in that human way that humans do, not to make Dario richer because his models will emit statistically convoluted copies of that open-source. What is my incentive to open-source something that I could commercialise today, compared to what it was before the LLM age?
(Many will say there's not much point in commercialising it, either, but to the extent that software still has commercial value, the appeal of the alternative path has greatly diminished.)
Fine, but my point was that releasing open source software implies contributing to that, virtually by definition, and then to be met with a déluge of slop PRs and GH issues. Who can blame a developer for saying, "screw that?"
I'm not blaming anyone. A lot of open source developers are paid to do that so presumably most of them will continue doing so if they want to keep their jobs. The major volunteer projects will probably introduce access controls to limit which users are allowed to create PRs and GH issues.
That's pretty much the reaction I had, except that I do find the argument compelling that AI is damaging the ecosystem it feeds upon.
I think Google's AI results are probably the prime example here. Those results are often quite good, and starve off the visits and hence revenue stream of the sites where the results are sourced from. Additionally, there's no way to robustly attribute that information anymore either; those links were already broken pre-AI by freeloading aggregators. So potential information producers can't afford to host their valuable information since they will pay proportionally to their information's value (as translated into bandwidth).
But it's tricky. If you charge AI companies proportional to the damage they do, then you need to assess that damage and you'll be caught in a no-win cat & mouse game where the AI companies outsource the damage and you try to track it back to them. They win if they bump their revenue, but they also win if they conceal the damage. (Just like companies can use shell companies, bankruptcy, and asset-only purchases to avoid Superfund responsibility.) If you charge proportional to revenue, then there aren't conflicting incentives; companies win by increasing revenue full stop. But I do agree that this shouldn't just apply to the big companies; distillation / model extraction (adversarial or not) should not be a way to avoid fees.
Though what really seemed off to me about the proposed solution was the use of the fees (to pay Americans, no less!) Perhaps upcoming essays will justify this more, but to me it seems like those fees should be applied directly to the health of the ecosystem. It should be used to fight pollution, in this case AI slop overtaking the Web. It should support the value creators, eg open source developers being overwhelmed by the AI tsunami. It should go towards serving and moderating online communities that create the very value that the LLMs are trained from. In theory, paying a whole bunch of Americans a trickle of blood money will end up going towards these purposes, but I'm very skeptical: first, the benefit will be very diluted. Second, it's more likely that the excess money will end up in Google or Amazon's pockets. The whole system is already set up to route any advantage to the big players. The payments are at least as likely to damage the ecosystem as they are to support it. They would just feed the capitalistic wolf and further reinforce the setup where monetizers win and value creators lose. If you could magically route the money to value creators, that'd be great. But you can't.
It'd be ok to pour money into a bucket if it had a small hole or two, but pouring it faster into the sieve we have now is not going to help anything.
"Fair use" was always fuzzy. To be honest, I care a lot less about slurping up the public internet and private books to make models than about every profession on the planet being forced by their employers to create skills that automate their knowledge work. The latter is much more directly an expropriation, legitimized only by the shortage of work, i.e., market power.
The troubles over copyright infringement in AI training data remind me a bit of Eli Whitney and the cotton gin.
There he suffered massive patent infringement, that basically stopped being enforced due to the sheer economic importance of the cotton gin.
In a similar manner, I think there is a reasonably strong argument that it was wrong to use copyrighted material for AI training without paying royalties nor even asking for permission. But equally, every country wants to have the most powerful models and enforcing such royalties would make it effectively impossible to train them as the amount of material required would cost an insane amount in royalty fees.
So I expect the law will continue to turn a blind eye (perhaps enforcing some token payments like that $1.5B mentioned in the article) because "if we don't make these models, the Chinese will" etc.
I'd be fine with the nuclear compromise: if AI training is allowed to infringe copyright, then there is no legal protection for the models themselves and their weights. Distillation should be explicitly legal. There will of course be a huge cat and mouse game about it, but let's have competition drive prices down on the stolen IP.
Author here. Appreciate your thoughts and I mostly agree actually.
I'll explain more over the next few essays, but I am designing my proposed regulatory structures to try to accomplish 2 purposes in tension simultaneously like the Fed:
1. Maintain global competitiveness for frontier labs
2. Create a societal hedge against the AI bull case (AKA the economic black hole case)
A % of revenue scales in a way that I think balances the two well while avoiding all the other problems I mentioned in the essay. I’ll get into ratchets, timing, and thresholds in later essays, but I agree the China/competitiveness problem is central.
> ... there is a reasonably strong argument that it was wrong to use copyrighted material for AI training without paying royalties nor even asking for permission. But equally, every country wants to have the most powerful models and enforcing such royalties would make it effectively impossible to train them as the amount of material required would cost an insane amount in royalty fees.
i think you're spot on this is one of the key arguments made beneath the surface. what i find so strikingly frustrating about it is, so many of the ai cultists [0] will imply and sometimes even outright say that writers, artists, musicians are silly useless and overvalued and the work artists do is entirely frivolous. then next breath explain why those artist's work is one of the most important things for a model to be trained on. suddenly art is very important. we absolutely must have access to their work. but also we shouldnt pay them because their work is silly and unimportant.
if an artists (musician, writer, journalist, painter, etc...) work is useless, then obviously you dont need it for training. if their work is imperative and you absolutely must use it, then pay for it.
ive noticed this with ai companies a lot. over and over again they contradict themselves to the core.
1) art is silly and not important enough to pay for but its absolutely foundational and we must be given unfettered access or our models will suck.
2) "our models are the smartest thing in the entire world. also, you're a dipshit if you trust them at all."
ill say it again, if removing art and culture from the training sets would render your model useless, then obviously pay for it.
[0] when i say cultists, im not talking about normal people who use ai. im talking about an entirely different group, we all know the types im talking about.
This is an astute observation. I think it reflects a larger and longer-running strain in the relationship between technocracy and the humanities, though, of which this latest iteration is just even more choleric and rote. The plumbers of capitalism always seem to have had deep contempt for the arts and the humanities, not in the least because they didn't do too well at them in school or didn't understand how philosophy relates to making money, or something.
This has led to some rather fantastical conclusions on both sides, however. On one side, there's an almost sadistic "revenge of the nerds" glee at the notion that these airy-fairy, frou-frou, and "feminised" liberal arts majors will finally crumble before the stochastic parrot machine god, and on the other side, a no less comical notion that after "AI" ushers in utopia, then high-brow artistic and literary pursuits will be all that remains for us to do.
It was a brilliant article, and it succinctly captured the offenses to ethics and humanism posed by LLMs.
I'm not sure it'll get a lot of reception in the technocracy here on HN, whether of the AI booster or AI nihilist sort. However, I think it's a very comprehensive digestion of the questions that will swirl around the idea of LLMs as a public good in the near to medium future.
Author here. Really appreciate you taking the time to read and for the kind comment.
I think the tension between these ethical questions and the practical realities (both the good and the bad) of AI is likely the defining issues for technology and perhaps society in this decade.
It’s important we’re thorough and rigorous with how we think and act here so I really appreciate you engaging with the topic.
Thank you in turn, I have circulated your piece to thoughtful friends.
My immediate, from-the-hip thought is that we are slowly lumbering toward the idea that LLMs ("AI") should be a public utility. It may take us quite a while to get there yet, as an unprecedented concentration of wealth and power is arrayed precisely against this outcome, but I think that will be the eventual effect, in that, "in the long run, we're all dead" kind of way.
I’m working through thoughts on this as well and agree with your read on the incentives.
There is an interesting set of conditions that happens if/when models get so competent that they’re effectively indistinguishable from each other and inference becomes a true commodity. IRL impact will lag this ofc but it’s such a wild time to be alive.
I do think it's easy, in this technology discussion bubble in which we dwell, to overestimate the centrality of LLMs to the arc of developments in our time.
They'll be important, but I don't think they'll be _that_ important, because the rest of society and the economy don't move at the speed of SV. Instead, they'll be overtaken by other, more traditional categories of events, ruptures and dislocations.
Moreover, folks will eventually realise that while they are very impressive derivative databases of knowledge, they're not at all "AI" -- well, not the "I" part, anyway -- as the concept is traditionally understood. There's not any "I". It emits convolutions of its training, and it does so very impressively, and that can even be harnessed by agents to connect them to levers, servos and richer information sources. It's nifty. But it's just not intelligence. It's more of a kind of queryable database than a robot.
Once that realisation diffuses more widely, I think it'll turn out to be a more prosaic and underwhelming development than is presently hypothesised, either here or by the press. It doesn't mean many business and managerial class folks won't try to squeeze everything they can out of so-called AI, but the idea that this can effectuate truly widespread labour displacement will probably quiet down considerably. (The valuations that depend on this assumption may collapse more abruptly and less gracefully.)
Functionally speaking, current models are intelligent by any reasonable definition of the term.
You can test them on their understanding of complex domains such as software systems, and within that domain as an example, you can judge them on their ability to diagnose bugs, fix bugs, explain code to humans, design systems given vague specifications, and implement new systems. On all these tasks, current models are measurably superhuman compared to the average human software developer.
You could make a better case that it’s humans that don’t possess true intelligence.
I don’t agree. I think they clearly lack general intelligence, which is why all the AI companies can think of is sourcing more and more niche domain data to plug more and more holes. When you get enough of that stuff, you can get a convincing illusion of general intelligence, but there is always another car wash test coming that shows it isn’t real.
What you’re doing is the classic No True Scotsman fallacy.
“Isn’t real” is a tell for that. It’s the same as saying it’s not a “true” intelligence.
Define what you mean by “true”, then. But for me, what I’m interested in is functional capability, not some mysterious ineffable quality that only humans can have. And in terms of functional capability, current models are certainly better at software development than you are. You’re just in denial.
Perhaps I was unclear: the part that isn't real is the generalization. The models appear to generalize because they're fitted to so many discrete tasks that it almost doesn't appear to matter. But then it leaks, and the failure modes reveal no coherent model or process that generated the failure. The labs only have one answer for this, which is more duct tape.
> current models are certainly better at software development than you are. You’re just in denial.
I have no ego in this. It could be true. Wikipedia is also "smarter" than I am; it "knows" so many more concepts than I could ever. But regardless, I think the state of slopcoded messes like Claude Code shows that the models are missing something.
Although I agree with you, in fairness, there are some lively controversies in the world of cognitive science and philosophy of mind about whether this meaningfully differs from human thinking at sufficient scale.
The general idea is that the building blocks of "coherent models" and "processes" within the churning of the human intellect are also, in so many words, prior art and existing concepts, and so, while the human mind is not a text model, a sufficiently large and sensorily multimodal neural net would not be too different. Neural nets are, after all, inspired by what we understand of human cognition -- they'd say.
It's quite trivial to show that an LLM doesn't have underlying intent, and that it can only emit direct textual convolutions of its training and not combine tokens in truly novel ways. This is the very thesis of the world-model folks, e.g. LeCun et al, that LLMs are a general intelligence dead-end because they lack any inner concept of the world around them, and do not reason from that.
Furthermore, LLMs clearly do not "reason", despite the marketing around this term; their "chains of thought" are the nothing more than the result of having been trained on explicitly verbalised multi-step processes. There are many cases where the putative result arrived at in the <think>chain of thought</think> does not match the result emitted.
Whether they are "better" at software development than I am greatly depends on whether one is asking them to retrace worn technology paths that are well-represented in their training--in effect, to copy prior art--or to do something in quite obscure technology, or something quite novel altogether. (However, I will happily concede that most everyday business programming involves neither.)
Still, if LLMs were actually intelligent, let alone superhuman in the sense you suggest, then we would expect major scientific breakthroughs to be raining from the sky. If, say, an Einstein, could transform physics with only the knowledge gleaned from a human's feeble capacity to retain the literature of the time, I'd expect LLMs, who retain orders of magnitude more information with far greater fidelity and precision, to have offered at least a small slither of evidence of their superhuman capabilities.
I would also expect the objective progress and capabilities of this galaxy brain to be accelerating, not substantially slowing down as it has. GPT-2 to GPT-3 was truly a quantum leap, GPT-3 to GPT-4 was a substantial jump, GPT-4 to GPT-5 was meh, 5+ is basically unimportant, and so it goes for the other models. There are, of course, holes plugged and benchmarks where these evolutions have been, in various niche ways, consequential, but in the plainspoken meaning of model capability, the low-hanging fruit of pretraining was clearly exhausted quite some time ago. The carnival has been running on "agentic" / MCP / RAG / tool-use fumes since. This is moderately impressive and adds quite a bit of runway, but intelligence it is not.
You're working with mystical definitions of words like "intelligence" and "reasoning", where there's some ineffable quality that you can't define that makes those properties difficult to achieve.
I'm not a mystic, so what I care about is functional behavior. Functionally speaking, models perform the functional equivalent of reasoning. Functionally speaking, they exhibit intelligence. Is it a perfect equivalent of human intelligence? Obviously not, but so what? The ball is in your court if you want to try to nail down the mystical quality you think isn't being achieved.
One hardly needs to soar to the exalted heights of "mysticism" or any sort of "ineffable qualities" to come into contact with the limitations of LLMs.
1) Models do not perform the functional equivalent of reasoning at all. When we reason, we don't simply babble out textual derivations of prior examples of "reasoning" to which we have been exposed, arrive at a conclusion, then occasionally state an altogether different conclusion while pointing at the largely irrelevant reasoning to substantiate it.
2) Models have real-world, not-at-all-mystical functional constraints that are directly relevant to the production of everyday economic work in which one attempts to involve them. Their inability to extrapolate or maintain clear mental models leads to staggering, head-scratching mistakes that even a very feeble and developmentally awry human intelligence would not make.
A basic, if well-worn example that was widely discussed in the last year or two:
However, this is emblematic of a much larger idea: the LLM doesn't have any idea what a letter is or what you're asking it to do. This isn't a question of "ineffable qualia"; when it doesn't know what something is at any essential level, it can't competently solve problems related to it. One bumps up against this in everyday programming and all the time.
Also, what is "mystical" about my demand for the kind of scientific progress--no, forget that, any scientific progress--that a functional superhuman intelligence would yield? I am not a "mystic", either; I want functional results, show me the functionality.
It's so weird how technologists overestimate the importance of LLMs to society at large. It's going to be far less important than real world issues like nuclear weapons proliferation, access to fossil fuels, declining birth rates, and breakdowns in global free trade.
What technocracy lol? people here turn into luddites if there's a positive reception of AI.. I can guarantee that there are at least 40 percent of the top 5 comments of any positive/negative post about AI gonna turn "hackers" to luddites. It was amazing before Covid times and now it's indistinguishable from reddit.
I suppose valid grounds for both perceptions can coexist.
Opposition to AI here is more than understandable; it takes much of the joy out of programming as a craft, while leaving one with so much much more of what is hateable about it as a day-job.
This is a well written essay. I had hoped it might address the role of distillation and open source in diffusing ownership of this technology back to the public that made it possible. And the AI labs’ rank hypocrisy in this area.
There’s a quote about how in some articles a switch is quietly flipped in the middle where the article was talking about what is and suddenly the author has everything to say about what should be.
I googled for the quote but all I got is useless web spam and meme style graphics about quotes from writers. But AI told me it was David Hume and provided the full quote.
The real question is when the day will come that AI become the fertile muck that a new thing grows from and clings to and the legal system needs to adjust to. I hope it’s a good thing.
> In every system of morality which I have hitherto met with [...] the author proceeds for some time in the ordinary way of reasoning [...] when of a sudden I am surprised to find that instead of the usual copulations of propositions is and is not I meet with no proposition that is not connected with an ought or an ought not.
I'm surprised that the Author has missed a very important corollary to the diffusion of "genius theft" that they are bringing up.
And that is the diffusion of the beneficiaries. Maybe they think that OpenAI and Anthropic are actually worth, like a trillion dollars each and can therefore have value extracted from them. I'm not so convinced.
What if there aren't frontier labs spending billions on training a model. What if, instead, open source is at least mostly competitive with the top models. And if the models are open source (or weights, whatever you want to call it) the people benefiting are actually just rando people or startup founders.
What are you going to do if you want to extract this value from this diffuse set of beneficiaries? Put an arbitrary tax on anyone living on San Francisco or something??
The reality is that the author is trying to put the genie back into the bottle. All technological progress has winners and losers. It has people who are even benefiting from the rest of society and making personal gain based on that.
But, at the end of the day, doing accounting math on how much an individual benefited from a specific common good as vague as societal knowledge is impractical. And yet technological progress benefits all of society in a wholistic sense.
Additionally, the author focuses so much on the extraction of a public good, I am surprised that they failed to address that these labs are creating a public good as well. Who's to say that this "theft" is larger than the production of public goods that these labs give to the public in the first place.
I mean, my life has been massively improved by the fact that I have access to these models. And I'm not convinced that I have produce enough myself to outweigh this benefit that they are giving to me, so I consider it to be a fair trade.
Author here - thanks for reading and thoughtfully replying
I’d personally love a world where open weights compete with proprietary ones, but I don’t believe it solves the core concentration issue. In that scenario most value still flows to capital holders, it’s just hardware holders not model weight holders.
I emphatically do not want to put the genie back in the bottle nor do I believe it’s possible. Technology has never been restrained for long (export controls on cryptography textbooks in the 90’s comes to mind here)
However, I believe we may disagree on the definition of a public good. If you’re referring to the free tiers of private models, then I’d argue that unless there is some legal framework passed that forces the frontier labs to offer that to everybody, it’s a customer acquisition cost laundered as a public good. It could disappear at any time and probably will as cutting edge model margins are reduced via competition.
In general, I believe the best AI policy balances allowing for maximum competitive market dynamics while hedging existential economic disruption risk for the general population.
I’ll go into this deeper over the next few essays. Appreciate the feedback
The story about the contribution of Bell Labs' patents to the world was exciting. And the benefit was certainly much greater than the extra amounts people paid to bell labs.
In 20 years, thinking about llm's contribution to new technologies, to improved accessibility of valuable knowledge, to solving problems.
> In economics, a public good (also referred to as a social good or collective good)[1] is a commodity, product or service that is both non-excludable and non-rivalrous and which is typically provided by a government and paid for through taxation.
I can see that we are already excluding people from using models, whether it's China[1] or use in other harnesses[2] so its use can't be considered "non-excludable" or "non-rivalrous". It is also neither provided by the government nor funded through taxation.
I guess one change could be made to force LLM companies to release their (N-1)th model publicly and document architecture and system requirements, which would shift this to make these products actually public goods, however.
> they failed to address that these labs are creating a public good as well. Who's to say that this "theft" is larger than the production of public goods that these labs give to the public in the first place.
Those aren't public goods, they're private goods. The difference is already apparent, and will become more so over time. But yes, they are "goods"; they have a lot of value.
I'm less interested in punishing the past theft that enabled these goods, and more worried about the ongoing damage they are doing to the ecosystem that birthed them. Good artists copy, great artists steal, but if a community is wholly dominated by these "great artists" then it will not survive as a community and will cease to produce anything worth copying let alone stealing.
Sure there are. Ai2's OLMo, EleutherAI's Pythia, and LLM360's Amber are actually open source, top to bottom, training data to checkpoints to code, to name three of them. You also don't have to look far on Huggingface to find smaller models developed with open source processes all the time.
> It's been said a thousand times but the ability to produce copyrighted material is not copyright infringement.
Aside from fair use, yes it is.
> If I draw The Simpsons on a piece of paper - whether or not I used AI to create it - it's not copyright infringement.
Only because of the fair use doctrine, which is limited. If it damages the market for the original, then a court could definitely declare it to be infringement.
> Copyright infringement is if I tried to sell that Simpsons work as my own, putting up for consumption and reaping the monetary benefits.
No. Copyright infringement requires neither sale nor misrepresenting a work as one's one. It also covers derivative works, not just perfect duplicates. Copyright covers reproduction and derivate works and distribution/performance -- you can get in trouble for just one of those. Taken strictly, that would be a horrible world, but fortunately the fair use doctrine weakens those quite a bit. On the flip side, if AI reproduction destroys the original markets -- as it is quite obviously doing right now -- then it's going to have a reckoning at some point, given how its legal status is completely based upon getting a free ride by claiming fair use.
A similar appropriative-use vs. public-trust evaluation is playing out this year as the California State Water Resources Control Board reevaluates Los Angeles’ right to divert water from the Mono Basin in the Eastern Sierra Nevada.
The foundational case for Mono Lake as a public trust resource is National Audubon Society v. Superior Court (1983) [1]. The California Supreme Court evaluated appropriative water rights against the public trust doctrine, took both arguments to their logical extremes, and decided that neither was acceptable in itself.
In a pretty jaw-dropping passage, the Court summarized the Los Angeles Department of Water and Power’s position in relation to appropriative use of water diverted from a unique ecosystem hundreds of thousands of years old:
> Defendant DWP, on the other hand, argues that the public trust doctrine as to stream waters has been "subsumed" into the appropriative water rights system and, absorbed by that body of law, quietly disappeared; according to DWP, the recipient of a board license enjoys a vested right in perpetuity to take water without concern for the consequences to the trust.
The decision in Audubon rejected LADWP’s argument, but it remains a stark example of the beneficiary of a public resource recasting a conditional public license as a permanent private entitlement, apparently free from consequence of accountability for harm inflicted on the public trust.
I think this appropriative-use vs. public-trust/public-benefit discussion is going to define the coming decades. The landscape remains unsettled as it applies to water (especially in a changing climate), much less to data in a period of rapidly evolving technology.
With respect to data, progress could be made by formally establishing a public corpus as an accessible commons, with clear expectations and rights around individual contributions made to third-party platforms. Publicly funded research is still often locked behind paywalls. The contents of the Library of Congress, special collections, municipal libraries, university archives, and museums are publicly owned or publicly supported, yet remain largely inaccessible to the general public.
I expect the “leader” in LLM performance to keep changing, but the accumulated genius of public knowledge to remain far more durable, with periodic and incremental additions. Fighting over small reparations for every scraped post seems less transformative than building a public knowledge commons that anyone can use, converse with, search, train on, and learn from.
reCAPTCHA began as a tool that simultaneously authenticated users while helping verify OCR for the backlog of The New York Times and Project Gutenberg. Maybe it is time for a similar public project to digitize and make accessible the body of public knowledge without surreptitious and ethically dubious appropriation of copyrighted works. Authors, writers, and shitposters could opt in as desired.
I would take a public resource like that well ahead of a few bucks of compensation for my decades of shitposting, just as I'd take a thriving Mono Lake well ahead of compensation for it being relegated into lifeless alkali flat via appropriative water rights.
Also, thank you for including the passage about Elinor Ostrom's work. The conditions outlined as necessary for an enduring public commons go a long way toward explaining why the existing decision governing Mono Lake's public interest resources hasn't delivered the intended outcome.
Glad you noticed. My personal preference is a more aggressive pursuit of a true public data commons, but for better or worse I am trying to write the essay series by balancing what I can be implemented sooner/with less pain + heartache and I think that discussion is a much longer debate with much harder policy to define.
Author here. This is a great point! I explored this case extensively and ultimately didn't include it in this essay because I didn't want to get bogged down in east coast riparian rights vs the west coast judgments. It is certainly a tumultuous time for contested public rights.
This was interesting right up until "The fund pays every eligible American the same amount each year. "
I'm in Australia. I've contributed my share of dirt to the delta. Why do I not get a share of this?
I get that the frontier companies are (for the moment) US companies. But that's just corporate ownership, it's not what we're talking about. We're talking about compensating the people who wrote the training data for their contribution. That contribution came from all over the world, so the Corpus Fund needs to be paid all over the world.
Set it up in the UN, get the UN to provide the training data sets as a common good, and have the UN collect the money from all AI companies using the training data sets. And the UN should distribute the money in the most equitable manner globally (so most of it going to alleviate poverty, probably).
I'd happily trade my collected years of shitposts to help folks get out of poverty.
Why do you think a UN-hosted process would work? Has the UN done similarly intrusive things before and been effective at it? How do you account for Chinese frontier and open weights models, which are just months behind American models and subsidized by a sovereign state that does not share any of these premises about intellectual property?
A lot of these posts seem subtextually premised on the idea that it's possible to put the genie back into the bottle; that if frontier labs in America didn't sign on to this tolling scheme, our recourse would be to halt the progression of AI completely. But that option does not exist, unless we're going to fight a world war to create it.
This comment seems incoherent?
Even if the UN only has a very small amount of legitimacy and credibility… that’s still far far better than literally zero.
Well, I mean, the alternative is for a US-based process to be put into place, followed (or led) by an EU process. The premise of my comment is that no matter who hosts the process, China isn't going to comply, so it really just comes down to the US and Europe.
And the US isn't going to comply either. Just like it never recognized the International Criminal Court in The Hague. In that respect it's at least as bad as China.
What?
There are over 190 nations in the UN. Even if we assume a chunk have no credible claim whatsoever, that’s still well over a hundred.
There are not in fact 100 nations that will seriously contend for frontier models.
You need to re-read the comment chain. Nobody claimed so many nations would contend for frontier models.
I’m talking about training data, and so is the parent.
The only jurisdictions where training data is going to matter are in the US, Europe, and China. I think we're just talking past each other. My subtextual point is that none of this is ever going to happen.
This really makes no sense, as if half the words in the reply were garbled.
e.g. How is whether the training data from this or that jurisdiction “matter”, or not, even relevant to my comment?
I haven’t even thought of any specific “matter” enough threshold to evaluate that.
There are legal matters and ethical matters being discussed. Legally, it does not matter. Ethically, it does.
[dead]
Author here. Thanks for reading!
I have additional essays coming out that will address this exact issue and other issues I know that people will raise.
I’m building the essays series around arguing for practical policy I believe can get implemented and am sequencing it as thoughtfully as I can. I just can’t fit every argument into every essay.
It's a great idea, I hope you get traction with it :)
I am personally coming to the conclusion that having these vast repositories of knowledge that can actually talk to us is actually great. We have some issues to solve, but the end-state of having a global repository of all knowledge that can talk to us and answer questions is actually an amazing outcome.
We just need to solve those problems first; mostly getting past the AI bubble and the massive over-investment, and then solving the hallucination problems. I don't believe either of them are insoluble.
I do worry about how future generations move on from this, though. In the same way that 90's music is still effectively the zeitgeist, and we will never move on from that, because of the way that streaming services work. It's a rare new band that can compete with (e.g.) Nirvana when appealing to that segment of audience, a competition that Nirvana themselves didn't have. So we are effectively locking in Nirvana as The Disaffected Youth Grunge Band for the rest of eternity. So similarly, we are in danger of locking in the current state of the world to the training data, and never being able to move on from that, because any new zeitgeist has to compete with this one on unequal footing.
Imagine that AI lives up to its promise, gets captured by a handful of corporations, and ushers in a world of untouchable über-oligarchs that de facto rule the world. Would it have been worth it then? Can this problem even be solved?
I mean, yes, that's one possible outcome.
I personally believe that it's more likely that the current batch of massively overleveraged AI companies go broke in the next year or so as the AI bubble bursts. Hopefully the USA manages to get its democratic shit together enough to not bail them out (as happened in 2008), and they just... stop.
The OSS models are almost at the same level, and are progressing fine. The technology continues to be useful, even if all the oligarchs go away.
My spidey sense goes off when we refer to open weight models from Chinese companies trained against frontier models as OSS... as if it was some Torvalds types maintaining a public git repo.
I think those open weight models exist because openai, anthropic, and google rule the roost. And I don't think things would continue as-is if those guys implode.
The technology has certainly moved faster because of all the money thrown at it recently, but it existed before that, and was progressing before that. So no, I think OpenAI, Anthropic only exist because of the open-weight models (that came before them) rather than the other way around. Google is a special case; they invented the transformer and then published it instead of commercialising it, so they actually weren't evil (for once), or they didn't realise what they'd done until too late.
I think things will continue perfectly fine if those guys implode. They didn't invent the technology, they're not the people who are advancing it most (because everything they do is proprietary and undisclosed), and their efforts at commercialising it are just getting in the way of the real research.
Imagine that AI lives up to its promise, gets captured by unaccountable bureaucrats and socialist dictators, ushering in a world of untouchable über-oligarchs that de facto rule the world. Would it have been worth it then? Can this problem even be solved?
This reminds me of: Ideas are like hemorrhoids, every asshole sooner or later gets some. ;-)
This is not going to happen: https://www.theguardian.com/environment/earth-insight/2014/m...
[flagged]
I agree UN sounds like a good organization to help distribute the wealth created by AI to the world. But this idea won't be considered by the current U.S. administration. In that case what other countries can do is probably to tax the AI companies at rates higher than regular companies.
The USA helped form the UN as specifically the organisation to do exactly this kind of thing. It's a shame the current administration can't play nicely with others.
The current administration is also playing strange games about export controls (can we run Fable yet? Kinda. Maybe). I think if they keep this up they'll just be shooting the US AI industry in the foot and the Chinese models will take over as the frontier models.
Maybe the UN can levy the USA for this, and leave the USA to collect that levy from its AI companies.
The US formed the UN to distribute funds?
> Maybe the UN can levy the USA for this
The UN has no "levy" powers.
True, good point.
Do you really want countries like Saudi Arabia and North Korea to have a vote in wealth redistribution?
It's about how to benefit the entire human species, or at least to reduce human suffering across the board. If there is famine in North Korea, then the surplus food from the world flowing into there won't be an issue right?
Yes, that would be an issue because it allows the regime to divert more resources to weapons. In general major famines are always caused by corrupt or incompetent political leaders rather than lack of food.
Famine could be caused by drought or flood or other numerous things. When the country can't produce enough food, its people would need to rely on food from outside.
Yes, and when the local officials are corrupt or incompetent then little off that outside food aid reaches the people who need it. That's why giving food to a country like North Korea does more harm than good.
I suppose countries like KSA and DPRK would ask the same question about us.
Fortunately we don't have to care much about questions asked by shithole countries today because the UN is virtually powerless. Let's keep it that way.
at this point I'm just confused whether you are talking about north korea or the us
What's confusing you? Do you want Kim Jong Un to have a say in decisions affecting the US or how money is spent?
The UN is a terrible organization for that. It is a corrupt, morally bankrupt institution. Giving it any more power or legitimacy is not a net positive.
100% — publish the hidden research, the value is in the discoveries, not in the dividend. With all due respect to the author, it feels like he missed the entire lesson of history.
Who would pay you? OpenAI has between $600 billion and $1.4 trillion in debt, and no hope for profitability. Anthropic, who have a much more modest $35 billion in debt (by using Sam Altman as a human shield), also has no chance of ever being in the black, pending some freak event.
> This was interesting right up until "The fund pays every eligible American the same amount each year. "
There could be so many other ways to set this up. Enforce a higher tax on any business selling AI models that have capabilities greater than some threshold, and use it to fund development and infrastructure project like roads, hospitals, schools, etc. Or you could even do a negative income tax[1].
[1] https://en.wikipedia.org/wiki/Negative_income_tax
Author here. Thanks for reading. I agree there are a lot of levers we can pull to work towards something better. I've structured this essay series as a sequence of nested regulatory solutions so in the next few essays I propose additional structures with instruments like this. They're sequenced in a way I believe that can be pragmatically implemented and start showing progress in the next decade or so by the US. So stay tuned!
Agreed. And someone has to manage and enforce that on a global level. Which is what we built the UN for.
The UN is a voluntary association with no enforcement power, like the EU, the WIPO, and the IETF
> I'd happily trade my collected years of shitposts to help folks get out of poverty.
In other words, you'd happily do nothing to help folks get out of poverty?
I was referring to the potential of actually collecting some of this fund. But yes, essentially in this situation that is correct - I would be doing nothing and also helping people get out of poverty.
[flagged]
[flagged]
No one asked you about this. Your government is free to spend its money as it likes, it can send it to the UN.
As long as places like Taiwan are effectively blocked from being in the UN[0], I don't believe we should be adding more to their power and responsibilities than what they have now, au contraire!
If it was a truly world representation, this might be different. But if things like health are sacrificed (ie. no WHO access either), I don't think they really deserve the benefit of the doubt.
[0]: https://en.wikipedia.org/wiki/Taiwan_and_the_United_Nations
> Frontier science looks different today. It's rooted in model weights and GPUs. It is flooded with token spend and agentic loops. It blooms in data centers.
This seems like handwaving to me. Even the actual frontier science that is using ML (e.g. AlphaFold) isn't based on "token spend" or "agentic loops". I personally cannot think of a single example of frontier science that is rooted in LLMs. I am sure there are a few examples, but the idea that frontier science has somehow completely shifted its trajectory and methods based on LLMs feels like quite a stretch to me.
Got any counterexamples to show me I'm wrong?
[flagged]
We cannot always want to capture only the (temporary) winners whenever we see a lucrative business and expect to share a free ride. I'd also assume that most of the revenue these AI labs are making is turned into depreciating fixed capital (hardware) and OPEX at this point.
Why don't we capture Meta and Google as they allegedly take advantage of more publicly available information for profit? Let alone the truly valuable knowledge, like mathematics, has nothing to do with the majority of garbage posts that an average person would "contribute" on social media.
If we really want to tax or nationalize some economic activity, then, in my opinion, the target should be what it takes from society, not what it produces for society. By this logic, we should tax all labs, including those lagging ones, that utilize the public knowledge.
However, if everyone can access the public knowledge without rendering it less useful or reducing its available quantity, there should be no reason to tax it.
Author here. Thanks for taking the time to read.
I agree we’re in an interesting era where frontier research has shifted from mostly publicly funded to mostly private and it creates challenging incentive structures especially regarding externalized costs of research.
Did you have any thoughts on my argument of how public knowledge does get damaged by the proliferation of AI over time?
I don't understand how knowledge, either public or private, could get damaged.
Though the income of the individuals and businesses that rely on the expertise of the knowledge would be damaged. Is that what you meant?
Edit: At this stage, the revenue made by the AI labs is almost entirely spent on the formation of fixed capital and opex. The demand is mobilizing physical resources with money. Atoms are relocated and reconfigured into compute racks. But eventually, the created productivity will perhaps make supply-elastic goods extremely cheap and abundant, while the supply-inelastic goods will be worth even more relative to the elastic ones.
After all, money is simply a token for the transmission of physical resources. It doesn't create stuff out of thin air. When new stuff is created, it just makes money cheaper, so that the banks can respond with more money to counteract it. More stuff -> deflation -> more money creation allowed to undo the deflation.
But the "exchange rates" between different goods and services will diverge. That's also why I don't think a direct money transfer like UBI would fix the problem, when it doesn't change the divergence of relative economic values of different goods. Let's say, extremely cheap software and entertainment, but unaffordable healthcare and housing. More money for everyone doesn't make limited resources available. So, what I am leaning into is some sort of Georgist policy. That could hopefully mitigate the price divergence, assuming that AI cannot make every commodity equally abundant.
Well, lots of ways. One is some degree of model collapse, as the slop-enshittified Internet itself is ingested as training data--despite the AI companies' best efforts to prevent this, they won't be altogether successful.
But the more consequential one may be that few are motivated to contribute more training data to make Dario or Sam richer. This is already playing out in open source. People write open-source so humans can use it, in that human way that humans do, not to make Dario richer because his models will emit statistically convoluted copies of that open-source. What is my incentive to open-source something that I could commercialise today, compared to what it was before the LLM age?
(Many will say there's not much point in commercialising it, either, but to the extent that software still has commercial value, the appeal of the alternative path has greatly diminished.)
Dario and Sam are literally hiring human experts today specifically to create proprietary training data which their competitors can't access.
Fine, but my point was that releasing open source software implies contributing to that, virtually by definition, and then to be met with a déluge of slop PRs and GH issues. Who can blame a developer for saying, "screw that?"
I'm not blaming anyone. A lot of open source developers are paid to do that so presumably most of them will continue doing so if they want to keep their jobs. The major volunteer projects will probably introduce access controls to limit which users are allowed to create PRs and GH issues.
> we see a lucrative business and expect to share a free ride
the lucrative business are the freeloaders
That's pretty much the reaction I had, except that I do find the argument compelling that AI is damaging the ecosystem it feeds upon.
I think Google's AI results are probably the prime example here. Those results are often quite good, and starve off the visits and hence revenue stream of the sites where the results are sourced from. Additionally, there's no way to robustly attribute that information anymore either; those links were already broken pre-AI by freeloading aggregators. So potential information producers can't afford to host their valuable information since they will pay proportionally to their information's value (as translated into bandwidth).
But it's tricky. If you charge AI companies proportional to the damage they do, then you need to assess that damage and you'll be caught in a no-win cat & mouse game where the AI companies outsource the damage and you try to track it back to them. They win if they bump their revenue, but they also win if they conceal the damage. (Just like companies can use shell companies, bankruptcy, and asset-only purchases to avoid Superfund responsibility.) If you charge proportional to revenue, then there aren't conflicting incentives; companies win by increasing revenue full stop. But I do agree that this shouldn't just apply to the big companies; distillation / model extraction (adversarial or not) should not be a way to avoid fees.
Though what really seemed off to me about the proposed solution was the use of the fees (to pay Americans, no less!) Perhaps upcoming essays will justify this more, but to me it seems like those fees should be applied directly to the health of the ecosystem. It should be used to fight pollution, in this case AI slop overtaking the Web. It should support the value creators, eg open source developers being overwhelmed by the AI tsunami. It should go towards serving and moderating online communities that create the very value that the LLMs are trained from. In theory, paying a whole bunch of Americans a trickle of blood money will end up going towards these purposes, but I'm very skeptical: first, the benefit will be very diluted. Second, it's more likely that the excess money will end up in Google or Amazon's pockets. The whole system is already set up to route any advantage to the big players. The payments are at least as likely to damage the ecosystem as they are to support it. They would just feed the capitalistic wolf and further reinforce the setup where monetizers win and value creators lose. If you could magically route the money to value creators, that'd be great. But you can't.
It'd be ok to pour money into a bucket if it had a small hole or two, but pouring it faster into the sieve we have now is not going to help anything.
"Fair use" was always fuzzy. To be honest, I care a lot less about slurping up the public internet and private books to make models than about every profession on the planet being forced by their employers to create skills that automate their knowledge work. The latter is much more directly an expropriation, legitimized only by the shortage of work, i.e., market power.
The troubles over copyright infringement in AI training data remind me a bit of Eli Whitney and the cotton gin.
There he suffered massive patent infringement, that basically stopped being enforced due to the sheer economic importance of the cotton gin.
In a similar manner, I think there is a reasonably strong argument that it was wrong to use copyrighted material for AI training without paying royalties nor even asking for permission. But equally, every country wants to have the most powerful models and enforcing such royalties would make it effectively impossible to train them as the amount of material required would cost an insane amount in royalty fees.
So I expect the law will continue to turn a blind eye (perhaps enforcing some token payments like that $1.5B mentioned in the article) because "if we don't make these models, the Chinese will" etc.
I'd be fine with the nuclear compromise: if AI training is allowed to infringe copyright, then there is no legal protection for the models themselves and their weights. Distillation should be explicitly legal. There will of course be a huge cat and mouse game about it, but let's have competition drive prices down on the stolen IP.
Distillation isn't illegal. But LLM vendors are free to block attempts at distillation or prohibit it in customer contracts.
This is my position, as well, and I have switched exclusively to Chinese models in support of this view (and cost and because they're pretty awesome).
Author here. Appreciate your thoughts and I mostly agree actually.
I'll explain more over the next few essays, but I am designing my proposed regulatory structures to try to accomplish 2 purposes in tension simultaneously like the Fed: 1. Maintain global competitiveness for frontier labs 2. Create a societal hedge against the AI bull case (AKA the economic black hole case)
A % of revenue scales in a way that I think balances the two well while avoiding all the other problems I mentioned in the essay. I’ll get into ratchets, timing, and thresholds in later essays, but I agree the China/competitiveness problem is central.
> ... there is a reasonably strong argument that it was wrong to use copyrighted material for AI training without paying royalties nor even asking for permission. But equally, every country wants to have the most powerful models and enforcing such royalties would make it effectively impossible to train them as the amount of material required would cost an insane amount in royalty fees.
i think you're spot on this is one of the key arguments made beneath the surface. what i find so strikingly frustrating about it is, so many of the ai cultists [0] will imply and sometimes even outright say that writers, artists, musicians are silly useless and overvalued and the work artists do is entirely frivolous. then next breath explain why those artist's work is one of the most important things for a model to be trained on. suddenly art is very important. we absolutely must have access to their work. but also we shouldnt pay them because their work is silly and unimportant.
if an artists (musician, writer, journalist, painter, etc...) work is useless, then obviously you dont need it for training. if their work is imperative and you absolutely must use it, then pay for it.
ive noticed this with ai companies a lot. over and over again they contradict themselves to the core.
1) art is silly and not important enough to pay for but its absolutely foundational and we must be given unfettered access or our models will suck.
2) "our models are the smartest thing in the entire world. also, you're a dipshit if you trust them at all."
ill say it again, if removing art and culture from the training sets would render your model useless, then obviously pay for it.
[0] when i say cultists, im not talking about normal people who use ai. im talking about an entirely different group, we all know the types im talking about.
This is an astute observation. I think it reflects a larger and longer-running strain in the relationship between technocracy and the humanities, though, of which this latest iteration is just even more choleric and rote. The plumbers of capitalism always seem to have had deep contempt for the arts and the humanities, not in the least because they didn't do too well at them in school or didn't understand how philosophy relates to making money, or something.
This has led to some rather fantastical conclusions on both sides, however. On one side, there's an almost sadistic "revenge of the nerds" glee at the notion that these airy-fairy, frou-frou, and "feminised" liberal arts majors will finally crumble before the stochastic parrot machine god, and on the other side, a no less comical notion that after "AI" ushers in utopia, then high-brow artistic and literary pursuits will be all that remains for us to do.
It was a brilliant article, and it succinctly captured the offenses to ethics and humanism posed by LLMs.
I'm not sure it'll get a lot of reception in the technocracy here on HN, whether of the AI booster or AI nihilist sort. However, I think it's a very comprehensive digestion of the questions that will swirl around the idea of LLMs as a public good in the near to medium future.
Author here. Really appreciate you taking the time to read and for the kind comment.
I think the tension between these ethical questions and the practical realities (both the good and the bad) of AI is likely the defining issues for technology and perhaps society in this decade.
It’s important we’re thorough and rigorous with how we think and act here so I really appreciate you engaging with the topic.
Thank you in turn, I have circulated your piece to thoughtful friends.
My immediate, from-the-hip thought is that we are slowly lumbering toward the idea that LLMs ("AI") should be a public utility. It may take us quite a while to get there yet, as an unprecedented concentration of wealth and power is arrayed precisely against this outcome, but I think that will be the eventual effect, in that, "in the long run, we're all dead" kind of way.
Thank you kindly
I’m working through thoughts on this as well and agree with your read on the incentives.
There is an interesting set of conditions that happens if/when models get so competent that they’re effectively indistinguishable from each other and inference becomes a true commodity. IRL impact will lag this ofc but it’s such a wild time to be alive.
I do think it's easy, in this technology discussion bubble in which we dwell, to overestimate the centrality of LLMs to the arc of developments in our time.
They'll be important, but I don't think they'll be _that_ important, because the rest of society and the economy don't move at the speed of SV. Instead, they'll be overtaken by other, more traditional categories of events, ruptures and dislocations.
Moreover, folks will eventually realise that while they are very impressive derivative databases of knowledge, they're not at all "AI" -- well, not the "I" part, anyway -- as the concept is traditionally understood. There's not any "I". It emits convolutions of its training, and it does so very impressively, and that can even be harnessed by agents to connect them to levers, servos and richer information sources. It's nifty. But it's just not intelligence. It's more of a kind of queryable database than a robot.
Once that realisation diffuses more widely, I think it'll turn out to be a more prosaic and underwhelming development than is presently hypothesised, either here or by the press. It doesn't mean many business and managerial class folks won't try to squeeze everything they can out of so-called AI, but the idea that this can effectuate truly widespread labour displacement will probably quiet down considerably. (The valuations that depend on this assumption may collapse more abruptly and less gracefully.)
The challenge is staying solvent until then. :-)
Functionally speaking, current models are intelligent by any reasonable definition of the term.
You can test them on their understanding of complex domains such as software systems, and within that domain as an example, you can judge them on their ability to diagnose bugs, fix bugs, explain code to humans, design systems given vague specifications, and implement new systems. On all these tasks, current models are measurably superhuman compared to the average human software developer.
You could make a better case that it’s humans that don’t possess true intelligence.
I don’t agree. I think they clearly lack general intelligence, which is why all the AI companies can think of is sourcing more and more niche domain data to plug more and more holes. When you get enough of that stuff, you can get a convincing illusion of general intelligence, but there is always another car wash test coming that shows it isn’t real.
What you’re doing is the classic No True Scotsman fallacy.
“Isn’t real” is a tell for that. It’s the same as saying it’s not a “true” intelligence.
Define what you mean by “true”, then. But for me, what I’m interested in is functional capability, not some mysterious ineffable quality that only humans can have. And in terms of functional capability, current models are certainly better at software development than you are. You’re just in denial.
Perhaps I was unclear: the part that isn't real is the generalization. The models appear to generalize because they're fitted to so many discrete tasks that it almost doesn't appear to matter. But then it leaks, and the failure modes reveal no coherent model or process that generated the failure. The labs only have one answer for this, which is more duct tape.
> current models are certainly better at software development than you are. You’re just in denial.
I have no ego in this. It could be true. Wikipedia is also "smarter" than I am; it "knows" so many more concepts than I could ever. But regardless, I think the state of slopcoded messes like Claude Code shows that the models are missing something.
Although I agree with you, in fairness, there are some lively controversies in the world of cognitive science and philosophy of mind about whether this meaningfully differs from human thinking at sufficient scale.
The general idea is that the building blocks of "coherent models" and "processes" within the churning of the human intellect are also, in so many words, prior art and existing concepts, and so, while the human mind is not a text model, a sufficiently large and sensorily multimodal neural net would not be too different. Neural nets are, after all, inspired by what we understand of human cognition -- they'd say.
It's quite trivial to show that an LLM doesn't have underlying intent, and that it can only emit direct textual convolutions of its training and not combine tokens in truly novel ways. This is the very thesis of the world-model folks, e.g. LeCun et al, that LLMs are a general intelligence dead-end because they lack any inner concept of the world around them, and do not reason from that.
Furthermore, LLMs clearly do not "reason", despite the marketing around this term; their "chains of thought" are the nothing more than the result of having been trained on explicitly verbalised multi-step processes. There are many cases where the putative result arrived at in the <think>chain of thought</think> does not match the result emitted.
Whether they are "better" at software development than I am greatly depends on whether one is asking them to retrace worn technology paths that are well-represented in their training--in effect, to copy prior art--or to do something in quite obscure technology, or something quite novel altogether. (However, I will happily concede that most everyday business programming involves neither.)
Still, if LLMs were actually intelligent, let alone superhuman in the sense you suggest, then we would expect major scientific breakthroughs to be raining from the sky. If, say, an Einstein, could transform physics with only the knowledge gleaned from a human's feeble capacity to retain the literature of the time, I'd expect LLMs, who retain orders of magnitude more information with far greater fidelity and precision, to have offered at least a small slither of evidence of their superhuman capabilities.
I would also expect the objective progress and capabilities of this galaxy brain to be accelerating, not substantially slowing down as it has. GPT-2 to GPT-3 was truly a quantum leap, GPT-3 to GPT-4 was a substantial jump, GPT-4 to GPT-5 was meh, 5+ is basically unimportant, and so it goes for the other models. There are, of course, holes plugged and benchmarks where these evolutions have been, in various niche ways, consequential, but in the plainspoken meaning of model capability, the low-hanging fruit of pretraining was clearly exhausted quite some time ago. The carnival has been running on "agentic" / MCP / RAG / tool-use fumes since. This is moderately impressive and adds quite a bit of runway, but intelligence it is not.
You're working with mystical definitions of words like "intelligence" and "reasoning", where there's some ineffable quality that you can't define that makes those properties difficult to achieve.
I'm not a mystic, so what I care about is functional behavior. Functionally speaking, models perform the functional equivalent of reasoning. Functionally speaking, they exhibit intelligence. Is it a perfect equivalent of human intelligence? Obviously not, but so what? The ball is in your court if you want to try to nail down the mystical quality you think isn't being achieved.
One hardly needs to soar to the exalted heights of "mysticism" or any sort of "ineffable qualities" to come into contact with the limitations of LLMs.
1) Models do not perform the functional equivalent of reasoning at all. When we reason, we don't simply babble out textual derivations of prior examples of "reasoning" to which we have been exposed, arrive at a conclusion, then occasionally state an altogether different conclusion while pointing at the largely irrelevant reasoning to substantiate it.
2) Models have real-world, not-at-all-mystical functional constraints that are directly relevant to the production of everyday economic work in which one attempts to involve them. Their inability to extrapolate or maintain clear mental models leads to staggering, head-scratching mistakes that even a very feeble and developmentally awry human intelligence would not make.
A basic, if well-worn example that was widely discussed in the last year or two:
https://medium.com/@JerryCuomo/why-ai-gets-the-strawberry-qu...
However, this is emblematic of a much larger idea: the LLM doesn't have any idea what a letter is or what you're asking it to do. This isn't a question of "ineffable qualia"; when it doesn't know what something is at any essential level, it can't competently solve problems related to it. One bumps up against this in everyday programming and all the time.
Also, what is "mystical" about my demand for the kind of scientific progress--no, forget that, any scientific progress--that a functional superhuman intelligence would yield? I am not a "mystic", either; I want functional results, show me the functionality.
It's so weird how technologists overestimate the importance of LLMs to society at large. It's going to be far less important than real world issues like nuclear weapons proliferation, access to fossil fuels, declining birth rates, and breakdowns in global free trade.
I don't think we're going to agree about many things, but we definitely agree on that!
What technocracy lol? people here turn into luddites if there's a positive reception of AI.. I can guarantee that there are at least 40 percent of the top 5 comments of any positive/negative post about AI gonna turn "hackers" to luddites. It was amazing before Covid times and now it's indistinguishable from reddit.
I suppose valid grounds for both perceptions can coexist.
Opposition to AI here is more than understandable; it takes much of the joy out of programming as a craft, while leaving one with so much much more of what is hateable about it as a day-job.
> returns constrained to a relatively conservative (by today’s standards) ~7% per annum
Damn, we should have something like that market wide. Progressive, with the revenue.
Forbidding vertical integration would be a tremendous blessing too.
This is a well written essay. I had hoped it might address the role of distillation and open source in diffusing ownership of this technology back to the public that made it possible. And the AI labs’ rank hypocrisy in this area.
Author here. Thanks for reading and the kind words. I will talk about distillation and OS in coming essays (the is a multi-part series).
There’s a quote about how in some articles a switch is quietly flipped in the middle where the article was talking about what is and suddenly the author has everything to say about what should be.
I googled for the quote but all I got is useless web spam and meme style graphics about quotes from writers. But AI told me it was David Hume and provided the full quote.
The real question is when the day will come that AI become the fertile muck that a new thing grows from and clings to and the legal system needs to adjust to. I hope it’s a good thing.
Sounds like you're thinking of the https://en.wikipedia.org/wiki/Is%E2%80%93ought_problem . Wikipedia quotes David Hume's "A Treatise of Human Nature" 3.1.1 as follows:
> In every system of morality which I have hitherto met with [...] the author proceeds for some time in the ordinary way of reasoning [...] when of a sudden I am surprised to find that instead of the usual copulations of propositions is and is not I meet with no proposition that is not connected with an ought or an ought not.
Anybody know where that gordon moore quote comes from? A little searching didn't produce sources for me.
Author here! It's from a workshop in 2001 for the National Research Council's Board on Science, Technology, and Economic Policy. He gave a talk.
You can ctrl+f for it at this link
https://www.ncbi.nlm.nih.gov/books/NBK208682/
Thank you!
I'm surprised that the Author has missed a very important corollary to the diffusion of "genius theft" that they are bringing up.
And that is the diffusion of the beneficiaries. Maybe they think that OpenAI and Anthropic are actually worth, like a trillion dollars each and can therefore have value extracted from them. I'm not so convinced.
What if there aren't frontier labs spending billions on training a model. What if, instead, open source is at least mostly competitive with the top models. And if the models are open source (or weights, whatever you want to call it) the people benefiting are actually just rando people or startup founders.
What are you going to do if you want to extract this value from this diffuse set of beneficiaries? Put an arbitrary tax on anyone living on San Francisco or something??
The reality is that the author is trying to put the genie back into the bottle. All technological progress has winners and losers. It has people who are even benefiting from the rest of society and making personal gain based on that.
But, at the end of the day, doing accounting math on how much an individual benefited from a specific common good as vague as societal knowledge is impractical. And yet technological progress benefits all of society in a wholistic sense.
Additionally, the author focuses so much on the extraction of a public good, I am surprised that they failed to address that these labs are creating a public good as well. Who's to say that this "theft" is larger than the production of public goods that these labs give to the public in the first place.
I mean, my life has been massively improved by the fact that I have access to these models. And I'm not convinced that I have produce enough myself to outweigh this benefit that they are giving to me, so I consider it to be a fair trade.
Author here - thanks for reading and thoughtfully replying
I’d personally love a world where open weights compete with proprietary ones, but I don’t believe it solves the core concentration issue. In that scenario most value still flows to capital holders, it’s just hardware holders not model weight holders.
I emphatically do not want to put the genie back in the bottle nor do I believe it’s possible. Technology has never been restrained for long (export controls on cryptography textbooks in the 90’s comes to mind here)
I also have already personally benefited a great deal from LLMs. I actually frame the entire essay series from this perspective in my prelude essay here: https://www.wysr.xyz/p/a-consigliere-on-every-desk-and-in
However, I believe we may disagree on the definition of a public good. If you’re referring to the free tiers of private models, then I’d argue that unless there is some legal framework passed that forces the frontier labs to offer that to everybody, it’s a customer acquisition cost laundered as a public good. It could disappear at any time and probably will as cutting edge model margins are reduced via competition.
In general, I believe the best AI policy balances allowing for maximum competitive market dynamics while hedging existential economic disruption risk for the general population.
I’ll go into this deeper over the next few essays. Appreciate the feedback
The story about the contribution of Bell Labs' patents to the world was exciting. And the benefit was certainly much greater than the extra amounts people paid to bell labs.
In 20 years, thinking about llm's contribution to new technologies, to improved accessibility of valuable knowledge, to solving problems.
How would you tell that story?
> these labs are creating a public good as well.
https://en.wikipedia.org/wiki/Public_good
> In economics, a public good (also referred to as a social good or collective good)[1] is a commodity, product or service that is both non-excludable and non-rivalrous and which is typically provided by a government and paid for through taxation.
I can see that we are already excluding people from using models, whether it's China[1] or use in other harnesses[2] so its use can't be considered "non-excludable" or "non-rivalrous". It is also neither provided by the government nor funded through taxation.
I guess one change could be made to force LLM companies to release their (N-1)th model publicly and document architecture and system requirements, which would shift this to make these products actually public goods, however.
[1] https://x.com/AnthropicAI/status/2025997928242811253
[2] https://www.mindstudio.ai/blog/anthropic-openclaw-ban-oauth-...
> they failed to address that these labs are creating a public good as well. Who's to say that this "theft" is larger than the production of public goods that these labs give to the public in the first place.
Those aren't public goods, they're private goods. The difference is already apparent, and will become more so over time. But yes, they are "goods"; they have a lot of value.
I'm less interested in punishing the past theft that enabled these goods, and more worried about the ongoing damage they are doing to the ecosystem that birthed them. Good artists copy, great artists steal, but if a community is wholly dominated by these "great artists" then it will not survive as a community and will cease to produce anything worth copying let alone stealing.
There aren't any open source LLMs by the way
Sure there are. Ai2's OLMo, EleutherAI's Pythia, and LLM360's Amber are actually open source, top to bottom, training data to checkpoints to code, to name three of them. You also don't have to look far on Huggingface to find smaller models developed with open source processes all the time.
> This is the private capture of public genius.
In other words, the biggest heist in human history. They took all of our collective knowledge and print money by selling it back to us.
It's been said a thousand times but the ability to produce copyrighted material is not copyright infringement.
I'll say it again:
You can produce copyrighted material all day long and it's not copyright infringement.
If I draw The Simpsons on a piece of paper - whether or not I used AI to create it - it's not copyright infringement.
Copyright infringement is if I tried to sell that Simpsons work as my own, putting up for consumption and reaping the monetary benefits.
It has never been illegal to produce copyrighted work. You people are in for a surprise dystopia if you keep pushing to make that a reality.
> It's been said a thousand times but the ability to produce copyrighted material is not copyright infringement.
Aside from fair use, yes it is.
> If I draw The Simpsons on a piece of paper - whether or not I used AI to create it - it's not copyright infringement.
Only because of the fair use doctrine, which is limited. If it damages the market for the original, then a court could definitely declare it to be infringement.
> Copyright infringement is if I tried to sell that Simpsons work as my own, putting up for consumption and reaping the monetary benefits.
No. Copyright infringement requires neither sale nor misrepresenting a work as one's one. It also covers derivative works, not just perfect duplicates. Copyright covers reproduction and derivate works and distribution/performance -- you can get in trouble for just one of those. Taken strictly, that would be a horrible world, but fortunately the fair use doctrine weakens those quite a bit. On the flip side, if AI reproduction destroys the original markets -- as it is quite obviously doing right now -- then it's going to have a reckoning at some point, given how its legal status is completely based upon getting a free ride by claiming fair use.
[dead]
A similar appropriative-use vs. public-trust evaluation is playing out this year as the California State Water Resources Control Board reevaluates Los Angeles’ right to divert water from the Mono Basin in the Eastern Sierra Nevada.
The foundational case for Mono Lake as a public trust resource is National Audubon Society v. Superior Court (1983) [1]. The California Supreme Court evaluated appropriative water rights against the public trust doctrine, took both arguments to their logical extremes, and decided that neither was acceptable in itself.
In a pretty jaw-dropping passage, the Court summarized the Los Angeles Department of Water and Power’s position in relation to appropriative use of water diverted from a unique ecosystem hundreds of thousands of years old:
> Defendant DWP, on the other hand, argues that the public trust doctrine as to stream waters has been "subsumed" into the appropriative water rights system and, absorbed by that body of law, quietly disappeared; according to DWP, the recipient of a board license enjoys a vested right in perpetuity to take water without concern for the consequences to the trust.
The decision in Audubon rejected LADWP’s argument, but it remains a stark example of the beneficiary of a public resource recasting a conditional public license as a permanent private entitlement, apparently free from consequence of accountability for harm inflicted on the public trust.
I think this appropriative-use vs. public-trust/public-benefit discussion is going to define the coming decades. The landscape remains unsettled as it applies to water (especially in a changing climate), much less to data in a period of rapidly evolving technology.
With respect to data, progress could be made by formally establishing a public corpus as an accessible commons, with clear expectations and rights around individual contributions made to third-party platforms. Publicly funded research is still often locked behind paywalls. The contents of the Library of Congress, special collections, municipal libraries, university archives, and museums are publicly owned or publicly supported, yet remain largely inaccessible to the general public.
I expect the “leader” in LLM performance to keep changing, but the accumulated genius of public knowledge to remain far more durable, with periodic and incremental additions. Fighting over small reparations for every scraped post seems less transformative than building a public knowledge commons that anyone can use, converse with, search, train on, and learn from.
reCAPTCHA began as a tool that simultaneously authenticated users while helping verify OCR for the backlog of The New York Times and Project Gutenberg. Maybe it is time for a similar public project to digitize and make accessible the body of public knowledge without surreptitious and ethically dubious appropriation of copyrighted works. Authors, writers, and shitposters could opt in as desired.
I would take a public resource like that well ahead of a few bucks of compensation for my decades of shitposting, just as I'd take a thriving Mono Lake well ahead of compensation for it being relegated into lifeless alkali flat via appropriative water rights.
Also, thank you for including the passage about Elinor Ostrom's work. The conditions outlined as necessary for an enduring public commons go a long way toward explaining why the existing decision governing Mono Lake's public interest resources hasn't delivered the intended outcome.
Glad you noticed. My personal preference is a more aggressive pursuit of a true public data commons, but for better or worse I am trying to write the essay series by balancing what I can be implemented sooner/with less pain + heartache and I think that discussion is a much longer debate with much harder policy to define.
Author here. This is a great point! I explored this case extensively and ultimately didn't include it in this essay because I didn't want to get bogged down in east coast riparian rights vs the west coast judgments. It is certainly a tumultuous time for contested public rights.
tl;dr
Author fears pending/future copyright litigation against AI labs, proposes "Corpus Royalty"