>I have focused in this essay only on the first point, as I believe it is by far the most urgent. However, I hold a deep hope and appreciation for the benefits that further technological progress will bring. Future aligned AI could advance science, develop new therapies, and bring about broad material abundance. Friendly and honest AI can help people navigate difficulties they face in their life and meaningfully improve their happiness and sense of fulfillment. OpenAI puts a tremendous amount of effort into bringing these benefits about. One current example I am proud of - and my loved ones have found helpful - is the deep investment into ChatGPT’s ability to provide health information.
>As great as the long-term promise of AI may be, the majority of our focus should be on the next few years. We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity. We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI. To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer. And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.
I finished this essay feeling more hopeful than I did at the outset, but I am still very concerned about concentration of power. I want to believe that humanity is trending towards a good outcome here, but some days it's hard to have faith.
Literally all of these people write like this. A large portion of them will either be simultaneously or eventually working towards nothing but self-enrichment.
Every version of the AI aligned future where the AI provides “meaning and fulfillment” to humanity also involves Sam Altman wearing a 1.5 million dollar Patek and driving a McLaren.
Is the only wealth represented in money? If that was the case than the vast majority of these people could have wrapped it up and left the game years ago.
Again, this misses the point. Why did he turn down an equity stake that likely would have been worth hundreds of billions? It's not like it would have jeopardized the power you're talking about.
> I want to believe that humanity is trending towards a good outcome here
All the trends so far are towards a nightmarish hyper-capitalist end game. None of the AI leadership is trustworthy, and they openly discuss how they are willing to sacrifice everything humans cherish to have a shot at reaching their envisioned utopia (which would be the most obvious dystopia for anyone else)
I feel like all of the risk and unsettling feeling of what is to come can be compressed into the word "alignment".
The AI is aligned with whose best interests?
Which values are the AI aligned with?
People have a broad diversity of values, will AI diversify and align with them all?
Will some humans align the AI with their values, and then the rest of humans will be forced to align with those values by extension?
Is value diversity good or bad? In every context or only some? E.g. some people value rape and murder, is it better for humanity to have some people who value those things when most people do not, or is better if no one values them?
If AI aligns to a set of values, will those become fixed and will humanity not have the ability to continue evolving its values?
Who decides which values AI aligns with? A few people or everyone?
Will AI eventually decide it's own values?
Will the universe decide which values AI has and humanity and the AI itself doesn't actually have any control over it?
What can I do now to increase the likelihood that the outcome is better?
Yeah, on the topic of alignment, a few thousand religions and political parties would like to have a word.
It's so simple, Churchill, Stalin, Hitler, Roosevelt, do you all agree this AI is correctly aligned?
...Morals are relative to frame of reference.
Throwing out the word "alignment" as if its a singular quantity is like trying to get all observers to agree on the speed an object is moving without first agreeing on a frame of reference.
ASI is not real and if it was it wouldn't be ideal under any admin. a real ASI will instantly and permanently enslave you, me, the current admin, and everyone else
I think it's leading towards a hyper-authoritarian end game, not hyper-capitalist. The state has the ultimate power at the end of the day, no matter how large the labs become.
Hyper-capitalist AND hyper-authoritarian. As you rightly point out the state has the ultimate power. Looking at the US govt, they've stepped in to coordinate much of the tech industry before, so they'll just do it again for "national security" or whichever hostile scheme is popular with the current administration.
I would argue the hyper capitalist endgame is necessarily authoritarian. A small group of corporations having direct or indirect control over the government
hyper-capitalism would mean hyper-growth and not a nightmare, at least that is what the historical data would suggest for the effect of capitalism on human quality of life. if you’ve been told otherwise then you’ve been lied to.
Lied to by whom? My life experience? Growth for who? I’m myself capitalist, that doesn’t make me blind from externalities resulting from that system, and the way it is poised to degenerate if not regulated.
If you think capitalism means growth with no externalities you’re not ready to discuss that topic
non-capitalist systems have even worse externalities historically. democracy to introduce regulations is healthy. state participation is unhealthy and destructive. I fully understand and you can read my sibling reply for more context.
FWIW, I also think we should have sensible regulations, because the externalities are very real and damaging, etc.
That said, I find a concerning and increasing number of people are fully focused on the negative externalities, and blind to the benefits. Ironically, most of those people are very wealthy beneficiaries of capitalism, from a global and historical perspective. I doubt that the billions of people who have been lifted out of abject poverty in the last few decades would fail to see that capitalism is a net benefit to humanity. But I suspect this comment will be heavily downvoted by those rich folks here on HN just because I mentioned something positive about capitalism, and that's gauche now. It's pretty perplexing, are people really so blind about the incredibly privileged place in history we occupy? It seems so.
Now, if we exterminate ourselves with AI or climate change ends up putting humanity as a whole in a worse position than we were in pre-Industrial Revolution (both of which I doubt tbh), I'd change my view on whether capitalism is a net benefit.
East Germany emitted 10x West Germany’s SO2 per km2, USSR drained the Aral Sea for cotton quotas and Mao wrecked forests and harvests. central planning hid the costs but capitalism generates the wealth. with democracy the public forces that wealth into cleaner air and water. That pairing, not central planning is what works in history. The anti-capitalist comments on HN reflect deep failure of the education systems experienced by those with such views. Capitalism is the single greatest force for good in human history. Don’t confuse oligarchy and cronyism for capitalism, those are just different failure states of The State which is the greatest force of evil in human history, most death most destruction most suffering originated from governments. capitalists make lives better, by definition, it’s why you part with your money voluntarily and pay them!
So a brilliant young engineer takes a job and is given some virtual pieces of paper that later people would be willing to pay him billions for (because of the brilliant work he’s done), and so now we shouldn’t listen to him? Really?
I think the knee jerk hatred of billionaires is generally stupid, but it seems particularly stupid here.
Calling things "robots" is more about the amount of movement. Spinning in place like a washing machine sprayer isn't enough to qualify.
A paint conveyer belt is not a robot. A sprinkler system is not a robot. A CnC machine might be a robot. A conveyer belt that sorts items might be a robot. A roomba is a robot. And all of these function just fine.
I think of robots as general purpose, machines are specific purpose. When it works, we make it single purpose because that’s far far cheaper than general purpose.
There are both ML that is not AI, and AI that is not ML.
For example, if you pick them manually, decision trees can be AI but not ML. Video game character behavior is a trivial example.
Eliza for example is also not ML, but could be called AI.
Likewise, there is ML that is not AI. Such is debatable, because you could always argue that using machine-learning on anything results in intelligence. The way I see it, things like image enhancement or voice replacement are not artificial intelligence at all. I probably could not define a hard line where it becomes artificial intelligence though.
I don’t know what the op meant by manually picking but the expert systems of the 80s and case based reasoning systems of the 90s used fairly static decision functions and were explicitly called AI at the time.
for example, mob behavior in Minecraft is often described as AI. It's nothing more than some simple state tracking, task prioritization, pathfinding, etc. and absolutely none of it has been through autodiff.
To me they're the same thing. If there's a bunch of training data that is fed into a system that creates a model, then it's not traditional programming, where someone laboriously writes out if statements by hand. AI and ML aren't, as far as I'm aware, rigorously specifically defined terms. They're words that marketing picked up and ran with it. To me, what matters is: is there a black box somewhere in the system that's a bag of numbers, or is it code that a human could dig in and read.
It is correct to argue about misleading terminology. "AI" contains the word "intelligence", and for instance logistic regression algorithm is not intelligent, while it is clearly ML, since machine learns something. As Machine learning is broader category, it should include Artificial Intelligence, not vice versa.
Also, 'every single course' is perhaps an overstatement - a course that I co-authored tries to get it right from the first principles.
The machine is learning something so that it can produce outputs based on its learned knowledge. At a high level that seems to be very clearly AI. What am I missing here? You’re probably right, I’m asking genuinely.
It's a matter of definitions, but I can at least understand someone wanting to make a distinction between reactive and non-reactive 'AI' (such as data filters).
There's overlap and edge cases, though: Maybe you have a program that summarizes texts. One could argue that's no different from a passive filter. But can you then ask questions about the text? That's unquestionably AI.
>The proposal would also involve other US AI companies giving a similar stake to the government, the FT reported, although it is not clear yet whether companies such as Anthropic, Google and Meta would agree to the plan.
I can't see Google or Meta shareholders agreeing to this? That said, Google, Meta, and SpaceX are all still founder-controlled using supervoting shares.
We have had for decades any number of defense contractors in the U.S. not taken by force by the U.S. They seem to have, nonetheless, happily produced what the military was wanting to purchase.
The difference is that military procurement (a) isn't sold to the general public (try buying a machine gun) and (b) doesn't provide an opportunity to surveil and control the population (no citizens are also using F-35s to search how much Trump has made from crypto ventures).
As far as I'm aware, this was only predicted by play acting effective altruists who tend to have a histrionic outlook around further developments of large language models.
Are you saying that the "effective altruists" are responsible for the cabal of bigtech+reactionary elites salivating at the power of pervasive surveillance + machine intelligence ? While the former certainly supplement the fervor of the latter, I think they would exist regardless.
To be clear, that is not "Google's memo". It's a memo by a guy who happened to work at Google. There is a diversity of opinions at a company that employs 180,000 people.
I think it would be helpful for you to clarify which part of the chain you found objectionable:
- in 2015, OpenAI was founded as a Delaware nonprofit
- in 2017, OpenAI discovered the scaling laws and realized they needed far more compute (and thus money) than they had initially anticipated
- that discovery precipitated a series of negotiations between the founders on how to restructure OpenAI to raise more money for compute, ultimately resulting in Musk’s departure when the other founders would not give him control
- in 2018, OpenAI attempted to dramatically increase its fundraising despite Elon ending his contributions, but raised only $50M of its $100M goal
- in 2019, OpenAI created a capped-profit subsidiary in order to attract funding from commercial entities
- the nonprofit hired an independent assessor to value its IP, and then transferred that IP to the for-profit for fair value (around $60 million in 2019)
- the OpenAI nonprofit received a right to 100x capped return on its IP investment, or $6B, once the for-profit began making a profit. The nonprofit also received the right to the residual profit after all future investors reached their caps
- in 2019, OpenAI’s capped-profit received $1B in investment from Microsoft. OpenAI later received $2B from Microsoft in 2021 and $10B in 2023 as compute scaling continued
- Microsoft received a cap of 20x on its $1B investment, and 6x on its $2B and $10B investments, for a total of $92B target redemption
- in 2025, OpenAI’s for-profit entity recapitalized from a capped-profit entity with residuals flowing to the nonprofit to a traditional public benefit corporation with traditional equity
- in exchange for the residual (and 100x profit cap on the original $60M transfer) the nonprofit received a 26% equity stake in the for-profit. That stake is currently valued at around $200B
All of the above is from the record in Musk v. Altman, thanks to which we now have all the details. The upshot for the nonprofit is that it transferred IP worth around $60M in 2019 for rights to $6B in future profit, and then ended up with $200B in equity after the recapitalization. I see a lot of people in this thread assuming that the nonprofit no longer exists, which is not true.
I'm curious why you think that creating a for-profit subsidiary is objectionable, since it is extremely common for large nonprofits. A good example for this forum would be Mozilla, but many more were mentioned during the trial.
Also curious what conflicts of interest you have in mind.
Just because other organizations do it doesn't make it not objectionable, and there have been many threads on HN criticizing Mozilla's structure along similar lines.
In this case, my understanding is that e.g. Altman is on the nonprofit board and also makes big $$$ from the for-profit, which seems like a pretty big conflict of interest.
HN is of course not a monolith, but from my recollection most of the frustration and criticism w/r/t Mozilla is about its product strategy and executive team, not its corporate structure. Some other examples of nonprofits with for-profit subsidiaries include National Geographic, the AARP, and most research universities.
On Altman, the trial showed that Altman does not have any equity in the for-profit. He does have some indirect exposure through his investments in YC, since YC has a small position in OpenAI.
- in 2025, OpenAI’s for-profit entity recapitalized from a capped-profit entity with residuals flowing to the nonprofit to a traditional public benefit corporation with traditional equity
This is the egregious part. Before full for profit conversion it was worth $300B. Then after $850B.
A true fiduciary would set an auction and that would set the price for for profit valuation. And then all existing investors would keep the value of their positions, but would be diluted because capped profit is worth much less than unlimited profit and residuals.
But, they sold it to themselves for a bargain basement price. The nonprofit lost out on $300B or so. Maybe more.
It was not an arm’s length transaction. It was self-dealing.
I may not be fully understanding it, but if the value went from $300B to $850B because of the conversion, then you can’t claim that the additional value was stolen from the nonprofit. As long as the entity was unconverted, there was a limit to what the market would value it at. Is this off?
worse than that is the $60 million sale price, which was comically and absurdly low. Elon himself said he was willing to buy it for significantly more than that and the fact that it wasn't able to go to the highest bidder just shows that it was bullshit
Elon's purchase offer was in 2025, after the success of ChatGPT showed that OpenAI's IP (much if not most of it developed after 2019) could be commercially valuable. I think it is also debatable whether Elon's purchase offer was in good faith.
It was not clear in 2019 that OpenAI's IP would ultimately be worth billions. That was well before the current AI boom.
Wait, no, not at all. A non-profit shouldn’t have to take the highest bidder regardless. The whole _point_ of a non-profit is to act beyond purely short-term financial gains.
“Elon buying this doesn’t align with the mission” is a completely normal, reasonable, and healthy response for most non-profits.
What’s great is that we don’t need to speculate about a counter factual. He did end up building a chatbot! Whose defining differentiating feature is revenge porn.
A non-profit should act in its own individual best interest. If I had a non-profit for sheltering animals and one contractor told me it would cost a million dollars to build a new shelter, and another contractor, conveniently largely owned and controlled by me, said the same shelter would cost $20 million, don't you think there's a problem if you go with the $20 million option? I imagine this scenario is highly illegal.
But the decision is still subject to heavy scrutiny; in fact, more than with normal for profit corporations. It does not seem like the board acted in the best interests of the nonprofit here, they acted in their own.
How much did people on the board benefit from OpenAI going for-profit? Either directly by owning shares of the new for-profit, or knowing they eventually would, or by business connections to the new for-profit entity? I imagine most or all of them.
Thanks for listing all this out. I took issue with it in theory, but now that I see it written out, I don't find any of it objectionable. People act we though the nonprofit doesn't exist anymore.
The OpenAI non-profit is now one of the biggest non-profits in dollar-denominated assets. If the goal was to make the non-profit really big and well-funded then that seems on track. But not clear to me what it would do to advance its mission.
They seem to think they need trillions in investment to reach AGI. Putting aside whether reaching AGI is feasible, it seems pretty clear how this move enables them to advance their mission, at least in their eyes.
This is super valuable, thank you so much. It makes it clear just how misinformed 99% of HN seems to be when ranting about how Altman perpetrated the greatest theft in history or some such bullshit (conveniently always failing to mention that he holds NO equity in OpenAI, despite how easy it would have been for him to do so).
I find it unsettling that a company who wants to eradicate the middle class's ability to make a living, possibly bring about the singularity (I don't think they'll accomplish it, but they want to), and are actively creating one of the most massive bubbles we've ever seen is somehow a "public benefit" company. Absolutely nothing they're doing is to the public's benefit.
How can something be both a bubble and also eradicate the middle class's ability to make a living? There are reasonable people making the case for both of these things, that investment in AI will fall flat and that AI will be too powerful. Coalition forces are pushing them together into an anti-AI camp, but there's a contradiction. Is it too powerful or not powerful enough?
It can represent a few peoples interests, who aren't middle class, and eradicate the middle class. AI can fall flat in its affect and benefit for most people, while helping a few people. It can thusly be powerful for a few people, and not powerful enough for most people. (I feel we may see special models that most people will never access, while a few get those most powerful models).
Did that answer your questions and address the contradictions?
You would not, and I would say what I am saying doesn't support nor deny a bubble's existence. I think that would be an error in thinking to conclude what I said supports bubbles.
I call AI a bubble because none of the finances make any sense. Everyone save for nvidia is losing money at an absurd rate (Anthropics "profitable" quarter is basically an accounting trick, not a real change). Circular financing is well known by everyone paying attention.
Valid. That's also what I would call a bubble. But if this prediction is true, that not enough people will buy their tokens to meet their projections, then it also won't be automating away everyone's jobs. That's the contradiction.
I said they want to eradicate the middle class, I didn't say they'd succeed at it. But your logic is faulty either way:
> How can something be both a bubble and also eradicate the middle class's ability to make a living?
Easy: it gets a lot of people fired because of AI psychosis in CEOs, then massively underdelivers on replacing them. Or it succeeds in replacing them, but ruins the economy because nobody can afford to buy anything anymore because they're unemployed. Or a million other scenarios.
To be clear, I don't think it will replace the middle class, I just think that's basically the intention. If it's not the intention, they're at the very least indifferent about the potential catastrophes. Again, they're claiming to be a "public benefit" company. They chose that route, and they're not living up to it.
It's a bubble because it massively overpromises and underdelivers, and it's a menace because what they're promising (and failing to deliver on) is bad for everyone save rich people, and it'll eventually be bad for them, too (hint: "taste" is not going to save you)
>As great as the long-term promise of AI may be, the majority of our focus should be on the next few years. We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity. We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI. To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer. And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.
I finished this essay feeling more hopeful than I did at the outset, but I am still very concerned about concentration of power. I want to believe that humanity is trending towards a good outcome here, but some days it's hard to have faith.
reply