A piece in the New Yorker on the “But China” problem; why China will lose control of AI

The problem with thought experiments in the AI “race” and the perils of Chinese opacity

When I was a graduate student in philosophy, we constantly dealt in “thought experiments”—things like the Trolley Problem. These mental exercises strip away messy, real-world complications to explore new moral terrain. One famous example is Peter Singer’s image of the drowning child, which helped argue forcefully why distance and visibility are not relevant factors for moral responsibility.

The trouble with thought experiments is that they can erase so much of the real world that little of practical use survives. In the Trolley Problem, when in real-life have you ever been certain an action you make will kill one person to save five? If there was any uncertainty, would you pull the lever? Herein lies the crux of the issue: If we wouldn’t leave philosophers alone at the head of a moving train, should we leave the philosopher-engineers in charge of whether America should pause or accelerate AI development?

In a piece for the New Yorker today, I write about the problems with evoking China as the reason to avoid pausing or slowing down AI development and regulation. The argument, sometimes called “But China” is rooted in a thought experiment: it takes a sensible premise like self-improving AI and slips into absurdity—killer robots, ethnically targeted bioweapons—while ignoring basic realities of how complicated it is for technologies to diffuse through societies and organizations. As Jeffrey Ding has argued powerfully, the AI “race” is a marathon, not a sprint. Even god, if it is to mean something in this world, must come down to earth and grapple with all its messiness.

The EA inheritance

Singer’s drowning-child arguments helped launch the effective altruism movement, the dominant ideational force behind America’s current AI drive. Among its most influential figures is the Oxford philosopher Nick Bostrom, who, in 2018, published a book on superintelligence. In one thought experiment in that book, Bostrom imagines a nation’s AI project beginning to recursively self-improve, pulling far enough ahead of its competitors to “achieve complete world domination.” He called this a “decisive strategic advantage.”

In the years since Trump 2.0 made an alliance with the tech elite, the concept of decisive strategic advantage has fused with the DC national security establishment to become America’s ultimate boogeyman: the unifying menace of a China “winning” the AI race. This is now the only thing standing in the way of urgent calls to slowdown AI development and it has apparently become so scary that people prefer “we’re all gonna die” to whatever is coming out of China. As Ted Cruz put the point succinctly: “If there are gonna be killer robots, I’d rather they be American killer robots than Chinese killer robots.”

Aside from all the reasons I cited in the piece about technological diffusion, I don’t believe the line between Chinese AI companies, who are clearly committed to global partnerships, and the PLA are that porous. In my experience Chinese businessmen are almost always excited to have their products shipped and recognized by the West and they have only mentioned the prospect of working with the Communist Party when they expect to be sanctioned or cannot succeed in that endeavor. What’s more, none of the evidence in the Chinese AI landscape suggests to me that China’s leaders care or even see superintelligence like the West—for example, I don’t see the AI 2027 prediction of China creating a Manhattan project for AI coming to pass. (It was predicted to happen this year. I don’t think we’re even close.)

While I’m convinced that the risks of China pulling ahead in AI development have been grossly exaggerated compared to dealing first with America’s urgent problems of AI-misalignment, I’m inclined to believe researchers like Jacob Coxon who say AI can cause catastrophic harm given all the existing evidence since the Hugging Face incident. The risk of China using such powerful AI systems to dominate the world are overblown, but whether China will lose control of AI is a different matter and I’m dedicating the rest of this post to share my views about why I think they will. (I apologize for the length.)

The perils of Chinese silence

For the past few years, I’ve struggled to write about Chinese AI safety issues. On the one hand, one has to give credit where credit is due. When I started covering Chinese tech in 2020, coverage was pretty ethically straight-forward: Beijing was disappearing businessmen and vaporizing entire industries like private tutoring. I don’t think I met a single Chinese beyond chest-thumping nationalists who thought the early tech crackdowns were a good idea. Given that starting point, it’s impressive how far tech regulations have come since then as Matt Sheehan has covered so extensively. Some Chinese leaders clearly recognized that they cannot keep playing whack-a-mole with the most dynamic sector in their economy and did something about. The results of this are already quite impressive.

First, the Chinese internet is going to have the most sophisticated deepfake and AI-generated detection systems. Second, while you can’t look up Tiananmen 1989 on Chinese AI models, they now have something like Asian-parent propriety: your kids will probably be better protected from things like pornography, misinformation, and dependence on AI companions. I’m not sure an American soccer mom would prefer American models over Chinese ones given a blind test. After all, American tech companies don’t really care about children—look how long it took Meta to address the harms of teenage social media use.

If you don’t acknowledge this remarkable story of an authoritarian society learning how to regulate powerful new technologies sensibly—with basically no prior models to learn from—then you have contributed to where we are today in the US-China relationship, where the two sides basically talk past each other on AI. For years, American negotiators ignored the reality of their Chinese interlocutors and if you are trying to collaborate on AI regulations, you have to start from their baseline, which is a highly sophisticated system of evidence-based, iterative, adaptive governance.

On the other hand, AI has improved way faster than many anticipated. It’s hard for any state with medieval institutions to deal with god-like technology, to use E.O. Wilson’s phrasing, but in the United States, for better or worse, we have a group of AGI-pilled EA-bros who joined the field because they were already gaming out the worst case scenarios. This has led them to believe that all their thought experiments about Chinese domination will come to pass—which I criticize—but it’s also made the the frontier labs relatively responsive to emergent AI capabilities. In China, the AI ecosystem lacks this “oh-shit” alarm gene. You might believe that the “oh-shit” alarmists in the US are too many, or they’re motivated actors trying to sell the power of their products, what some call criti-hype, but if there is even a remote chance that AI can lead to a bad outcome, you need these people—and institutions to protect them.

If you plotted AI opinion as a bell curve, it’s striking how different the two ecosystems are. In the US, “AI could kill us all” is somewhere close to the mainstream position now and in China, it is still at the very edges—you have to go to philosophy departments at Tsinghua or Fudan University to find these people. The Chinese AI sector reminds me of any other hot tech sector in China like batteries or semiconductors. It’s a field of engineers, who grew up in a culture that prized solving the problems in front of you, and not asking too many speculative questions. Part of the reason Chinese policymakers and developers do not put much stock in AI’s runaway self-improvement is that all of it feels, conceptually and chronologically, far away. They did not read Bostrom, grow up on the Terminator, or share their p(doom). Their chief opponent is not the United States, but “involution,” a system of domestic competition where economic survival depends on refusing to stop.

This difference cannot be explained by Chinese models being six months or a year behind the frontier. There are social-political-culture reasons at play, which I’ll get into in a bit. But here is the point: None of the Chinese AI safety infrastructure I mentioned—the rules and the resources devoted to safety in the labs—adequately address the risks of large-scale AI risks like the Hugging Face case. And while I believe the US extreme has its own problems, I also believe the Chinese extreme is dangerous. It may lead China to walk inexorably toward its own Hugging Face moment, one that will likely result in the Party reverting to old-school crackdowns.

First, Xi Jinping has come out to warn against “loss of control,” but it takes a long time to build an AI safety infrastructure to deal with it in the frontier labs. You need people who specialize in running the tests, which is a different skill from building the model. You need a top-down decision to devote a serious share of already-constrained computing power to the task of safety. You need people who are not just testing against existing cybersecurity benchmarks, but people who are actively trying to come up with new tests, prompting the models for the worst things they can think of. As Matt Sheehan pointed out to me, the US labs have been working on loss of control issues for years and they are still very very far from getting things under control. For Chinese AI researchers who have barely even started, will they learn fast enough?

Second, China does not have a disclosure culture. No Chinese wants to be Jacob Coxon for good reason. The last whistleblower to warn his colleagues about a real, existential danger was Li Wenliang, the Covid-19 doctor who was told by authorities he would face legal action if he kept sharing his “lies.” (Li contracted Covid-19 in February 2020 and died a week later.) Li’s fate cannot be avoided in China because it is a feature of the Chinese system, or I should say the Xi Jinping system, where the law is not meant to protect you, but justify how authorities get you retroactively. Public disclosures here lead to severe and unpredictable consequences and this is why, I’m afraid, we won’t know if Chinese AI labs are already seeing the same kinds of misaligned AI as their American counterparts until its probably too late.

It’s interesting to compare this disclosure problem to the United States, where there is an established whistleblowing culture. I don’t want to dismiss the risks of people who speak out in the U.S., but Coxon’s celebrity-esque media tour makes whistleblowing look cool. I haven’t seen any repercussions for him. Were a DeepSeek employee to post on Weibo that their models can “kill us all,” I’d suspect that the employee would face swift legal action and a call from local police or the cyberspace administration for disrupting social order. DeepSeek itself would be investigated at least or penalized. By contrast, nothing seems to have happened to Dario Amodei. I don’t know if that’s a good thing, but I know that the benefit of the U.S. system is that the AI labs have been forced into a reckoning about how to prevent a Hugging Face 2.0. The United States is a society where a minority of individuals can force reckonings, and this is very important with increasingly capable AI systems. In China, even if you have a contingency of “we’re all gonna die” researchers, they can’t come out and say it. The lack of this self-corrective mechanism in China and other authoritarian states is why incidents like Chernobyl happened.

Suppose DeepSeek’s models were found to breach the hull of a Chinese company, like Hugging Face. First, any kind of acknowledgement of AI misalignment from DeepSeek may prompt some kind of probe or investigation so I think the only rational response is to try and cover it up. So we have the beginnings of a Covid-19 pattern of concealment and crackdown. Because DeepSeek would not publicize what happened, we may not even know the severity of this incident. The victim company might want to publicize it but they may also be ordered by local authorities to remain mum or report something very understated so as to not stir public panic. Historically, the Chinese government usually makes the right decision after things have gotten bad enough.

If the hacking or misalignment is sufficiently bad to have been publicly discovered—I don’t have enough technical knowledge to imagine what this looks like—we will probably see the Chinese government act very swiftly and harshly. I talked with Graham Webster about a possible scenario: Chinese officials would accuse DeepSeek of violating their cybersecurity obligations and state media would shame the company. DeepSeek’s leadership may be summoned for talks, its apps and services might face a six month or indefinite suspension, and the company would be ordered to “rectify” their mistake. Agencies and ministries could swiftly roll out new rules that regulate agent behavior. It’s not clear that this would be so bad of an outcome compared to the US—can we say China is more or less reckless?—but it really depends on this: whether an AI incident that is bad enough to break into public awareness in China, meaning that it could not be concealed by local leaders and companies, is still within the remit of a Chinese-style crackdown to contain. Given the research on AI agents being able to self-replicate, I’m not so sure.

When I talk to Chinese AI researchers about this, most of them believe that the Hugging Face worries are still too far away in China. Perhaps it is, but “we still have time” is not a solution.

There is another danger to the Chinese concealment culture. Given the level of AI capabilities, a la Hugging Face, that have been clearly demonstrated by the American models, it is now becoming reckless for Chinese companies to not disclose anything that’s happening inside their labs. The culture of fear and uncertainty around public disclosures is the X factor in our AI arms race. For years, it has served as lighter fuel for the “But China” narrative. If Liang Wenfeng of Deepseek published on X tomorrow that, effectively, yes, we are also dealing with misalignment issues in our lab, and if it gets more serious, we will invest more heavily in AI safety (i.e. slowdown), he may get in trouble at home, but he could disarm so much of the “But China” ideology and—in some people’s view—save humanity from extinction.

Ro Khanna has reached out to the Chinese lab to pace the frontier, which is quite funny because while I respect his leadership, I also think it reveals a lack of understanding of how the Chinese system works. The Chinese labs don’t have the maneuverability to engage with American politics, that falls clearly inside China’s red lines. Their lack of response to Ro Khanna will be another example of this feedback loop of Chinese silence and American impressions that Chinese labs are not slowing down. In reality, their silence has less to do with a lack of concern with misalignment—I think the labs are quite aware of these issues—than their political culture.

So there we have it. In the West, “But China” thought experiments conjuring a China barreling toward superintelligence and on the cusp of world domination. And in China, compulsory silence fueling that narrative. Two mutually reinforcing pathologies hurling the world into dangerously uncharted AI waters.