A warning to humanity or a political dividend?
Concern about AI safety, even within the AI industry, is nothing new. But recently, what had been a background murmur as reached a crescendo, with the loudest voices coming from the industry itself. Bill Gates, one of the industry’s elder statesmen said on television that AI could kill “one billion people in the wrong hands.” OpenAI announced that safety concerns led it to postpone the release of its newest models due to safety concerns. Other industry leaders have also sounded the alarm.
Maybe they know things the public does not – things that truly frighten them. But they also have large financial and reputational interests in getting ever more powerful AI models to market and frightening their potential customers doesn’t help them do that. So, what is driving their new stance?
To answer that question, start with the ideology that drives the United States’ AI industry. The AI industry is premised on what I call the Replacement Story: AI will inevitably replace humans. Supporters of the Replacement Story have various personal motivations: some believe that AI is a new lifeform that will supplant humans, others look at AI merely as a profit-maximizing tool (increasing margins by substituting for human labor) and some are just excited about working with new technologies. Underlying the shared ideology is an economic and political motivation: power and wealth will inure to those that can control or ration access to AI. The Replacement Story underpins an industry that has become a significant driver of United States GDP.[1]
Not surprisingly, the Replacement Story is not a winning public relations strategy. Negative sentiment towards the AI industry is large and growing larger in the United States. Trust in the AI industry and the government’s approach to it is the lowest of any industrialized country. The most visible manifestation of this distrust is the movement to ban or limit data centers. What had until recently been a local political issue has become a national one, and midterm candidates around the US are putting limits on data centers at the heart of their campaigns.
This leads me to ask: if the AI industry is already alienating people, why make it worse by talking about how it could be very dangerous to humans?
Because making it worse is in the industry’s interest. The sudden public acknowledgement of AI’s dangers is politically motivated and self-serving, as evidenced by their proffered “solutions.” The industry suggests that development of new models should be slowed down to ensure that new models are “safe,” but it does not explain what would make a model “safe.” It puts the burden of defining “safe” on regulators (or the industry itself), a process that even at its best is time-consuming and prone to loopholes and unintended consequences, since regulators are rarely expert in technical matters like this (and the industry is self-interested). And in the ultimate “tell” (in poker jargon), compliance with the regulation would limit liability for harm caused by AI models.
Why now? In part the industry is hedging its bets, in case a new Congress takes a much harder line on the industry than the current one. If regulation is inevitable, it is in the industry’s interest to influence, if not control, the process. Proposing self-regulation, as they did at the White House in signing an “accord” with the president on September 29, is one way to do that. (It also has the additional benefit of allowing them to reduce competition among themselves; whether the “morally binding” agreement they reached that was endorsed by President Trump will save them from liability under antitrust laws remains to be seen.) The leaders of the AI industry read the polls. They know that changes are coming, and attempting to improve their image as they head into a harder regulatory environment is a smart political response.
The bottom line is that while calling attention to AI’s potential to destroy humankind may allow the AI industry to improve its public image by appearing to care about people or to salve the conscience of some of its leaders for producing a dangerous product, the industry is not offering any real change to the status quo.
Not only does the industry’s current position not offer real change, it also leaves the Replacement Story intact. The safety agenda offers to approach that state more carefully and more slowly. But it does not question the Replacement Story’s endpoint at all.
The real issue is autonomy
The real issue for AI regulation is its autonomy: how much can, or should, a frontier model act on its own and make choices? The safety approach being offered by the industry would place guardrails around the behavior of the models to limit bad effects from autonomy. But the inherent design imperative remains: to function as intended, the frontier models must exercise autonomy. The industry’s end state is agents that plan, decide, and act on their own.
The Replacement Story is the only way to make AI work as a matter of economics. The costs are so high that a company wanting to use AI must find a way to offset those costs, and almost universally, labor is the only cost large enough to do that. If AI can do the work of a lawyer, an analyst, a programmer or a customer service agent, the savings on wages flow to whoever owns the model. For the Replacement Story to succeed, AI must become autonomous. A tool that needs a human at every step does not replace the human; it assists one.
The industry also needs to have us become dependent on AI, to make it an essential product that we cannot function without, thus ensuring their market. Autonomy is a key to that. The frontier models are designed to become the place where people think, write, decide and seek counsel. To do that well, they must react, anticipate and lead humans to further interaction. This requires a level of autonomy for the AI to function as needed. Human dependence is valuable to the industry, but it threatens to create liability for human actions and social harms.
Read also: The red string of diplomacy: sense-making, AI, and the perils of infinite connection
And this is where the industry calls for safety and self-regulation reveal themselves as not serious responses. The industry has not backed off from its ultimate goal of AI autonomy at all; at most, it has suggested a slowdown while it regroups. It maintains its vision that the fundamental relationship between human and AI is dependence and replacement.
What the industry proposes is that the fox guard the henhoude. Genuine regulation requires more than self-professed concern about undefined safety issues and voluntary offers to slow development until “safety” catches up. It requires convening all the various stakeholders and starting a process that will be rapid, flexible, responsive, and nuanced. It must have defined risks and goals and be designed to maximize the public interest.
What the AI industry safety agenda leaves out
Consider what a serious AI policy agenda would address.
Autonomy in warfare. The most immediate existential risk is not a rogue chatbot. It is machines authorized to select and engage targets without meaningful human control. International efforts to govern lethal autonomous weapons have stalled for a decade, while AI companies increasingly compete for defense contracts. Pre-deployment testing for commercial models says nothing about what militaries do with them. Mass surveillance and the security of the vast data stores on which AI depends belong on this list as well. Biological and nuclear weapons are regulated. Why should not autonomous weapons be regulated as well?
Replacement as a goal. No regulatory framework currently asks whether a given deployment is designed to augment workers or eliminate them. At best the AI industry is suggesting that we regulate the safety of AI, not the purpose for which it is built. Should policies be put in place that reward companies that augment workers with AI, rather than replace them? If AI results in measurable increases in human creative output, could the margin improvement be used to increase returns on labor, rather than increasing profits for the firm and its AI supplier?
Environmental cost. Data centers’ demand for electricity is growing faster than the electricity generating system can accommodate. New capacity is being added, but in the US, it mostly relies on fossil fuels, since solar- and wind-power have been discouraged. The costs are borne locally; the profits accrue globally. Is there a level of development that balances community concerns and industry needs? Should deciding where data centers can be located be a matter for national regulation rather than local zoning law?
Concentration of power. A handful of firms control the models, the chips, the cloud, and, increasingly, the distribution. Economic concentration on this scale inevitably translates into economic power and political influence. It shapes the very regulation meant to constrain it. Antitrust laws and regulations provide many of the tools necessary to address many of the market and pricing power issues of the AI industry, but enforcement is largely a government activity and thus a reflection of political will – in the US case, at both the state and federal levels. Enforcement patterns and case law will require close consideration to apply these principles to a rapidly growing industry with a small number of dominant players. As they coordinate their actions, merge or acquire technologies, or expand into new markets, antitrust law can play a critical role in ensuring the public interest is protected.
Penetration Pricing. Both the frontier models and the open-weight models have a similar business imperative: drive adoption into value chains in industry and more broadly into society. Both are pricing their utility below the true cost of creation. The frontier models are doing this through large subsidies from the stock and credit markets. The open-weight models take advantage of government subsidies, open-sourced technology and intellectual property theft (levelled at Chinese models primarily) to deliver performance that is close to that of the frontier models. In both instances, the large economic question is what happens when these companies stop using penetration pricing. Will they pass on price efficiencies to consumers or will prices go up? The history of oligopolistic behavior hints strongly at the outcome.
Utility Regulation. Will access to AI become a basic utility for participation in society as a citizen or economic actor? Will it become as essential as water or power? Both are regulated as a matter of investment and pricing as regulated industries. How should the AI industry be evaluated?
Consumer harm and product liability. Recent legal cases in the United States regarding social media suggest that there is a growing appetite for holding the frontier model companies responsible for the effects caused by their products. Growing scientific data tying AI use to human self-harm and cognitive decline creates a growing area of potential legal risk to the frontier model companies. Allocating this risk in a way that protects consumers but balances the growth of the AI industry can be done in the court room in an ad hoc manner, through regulation, or by using both approaches.
Read also: The great unlearning: While AI gets better, human intelligence is collapsing
Intellectual Property. As AI becomes more ubiquitous, it will be more often used in conjunction with human authorship or it will be used without humans at all. In both situations, intellectual property law will need to address ownership when a human is not the only author or inventor. Another issue is how human creators should be rewarded for the use of their intellectual property in frontier models. (Many copyright infringement cases are pending in the US.) In a world where human value may lie in creating differentiation over the baseline of common AI, preserving the human’s ability to be paid for this differentiation will become more important.
The tax base. This may be the most consequential omission. In the United States, federal corporate income tax receipts have fallen from around 4% of GDP in the 1960s to under 2% in recent years. Social insurance, meanwhile, is funded largely by taxes on wages. Here is the arithmetic problem: you cannot sustain a welfare state on payroll taxes in an economy designed to need smaller payrolls. If AI shifts income from labor to capital, the tax base must shift with it or the social contract fails.
Who gets the time. Every productivity revolution poses the same question: when a task that took ten hours now takes one, who keeps the other nine? In the industrial era, the answer was contested and eventually shared — shorter working weeks, higher wages, weekends off. In the AI era, the default answer is that capital keeps the savings. Nothing in the current regulatory conversation challenges that default.
Displacement or human workers. Some people will not benefit from the AI economy, through no fault of their own. The digital divide of the internet era — about access to connectivity — will look modest beside a divide over who owns productive intelligence and who is replaced by it. How societies guarantee a minimum standard of living in that world is not a safety question. It will be the central political question of the coming decades. In other significant technology transitions, workers were ignored as “Luddites,” not given opportunities to retrain, or largely ignored. Things did not just “work out” for them. Generations of workers were displaced before society and the economy adjusted, which required significant social investment, such as community colleges, land grant universities, and the G.I. Bill of 1944.
The outcome the industry should fear
While an advocate for the AI industry could look at the list above and conclude that answering these questions should be avoided, a forward thinking AI leader should proactively engage in these broader issues. The backlash against the industry will not disappear so long as the Replacement Story is the AI industry’s core narrative. Closely aligning itself with the Trump Administration is a risky strategy heading into a mid-term election that results in a shift towards a more pro-regulation stance led by Democrats, as well as Republicans looking to distance themselves from a politically damaged President.
On the other hand, If the AI industry succeeds in reaching the goal of the Replacement Story, the politicization of the industry will make today’s backlash seem tame by comparison. A small number of firms will own the means by which most economic value is produced, while the need for human labor will have steadily declined. History suggests how democratic publics respond when citizens come to believe the gains of a transformation are being captured by the few. They do not simply ask for better safety testing. They ask who should own the machines.
A post-labor economy in which productive capacity sits in a few private hands is exactly the kind of economy in which collectivist solutions become politically plausible — public ownership of compute, nationalization of frontier labs, confiscatory taxation of AI rents. Those who dismiss this as fantasy should notice how quickly the language of anti-collectivism has returned to Western political rhetoric. Political actors rarely campaign hard against threats they consider remote.
The industry therefore faces a choice. It can continue to channel regulation toward narrow procedural safety while the larger distributional questions build pressure unaddressed. Or it can engage now — with tax reform that broadens the corporate base, with mechanisms that share productivity gains with workers, with binding limits on autonomy in lethal systems, and with an honest accounting of environmental cost.
The executives who warn of existential risk are right that the stakes are enormous. They are wrong, or at least incomplete, about where the stakes lie. The question is not only whether AI will be safe. But for whose benefit and costs?
A regulatory framework that answers only the first question is not a safety framework. It is a liability-avoidance strategy and a political strategy likely to fail.
[1] I have a more extensive discussion of this in my book The Originality Dividend.
