By NYU News
Six studies show how state-coordinated media in AI training data influence model responses about politics—especially in a country’s own language.
Image: AI-generated by DIW for illustrative purposes.
Ask an AI model the same political question in two different languages, and you may get two very different responses. A new study in Nature suggests one reason why: governments can indirectly influence large language models (LLMs) by shaping the online media environment, and thus the text those systems learn from.
A cross-university team of New York University researchers and their colleagues found evidence that state media control can leave detectable traces in AI model behavior. The researchers combine evidence from evaluating LLMs in the local languages of 37 countries with a case study from China to understand how this happens. Across six studies, the team traced the pathway from online media to training data to model behavior, combining analysis of open training data, experiments with training small models, human evaluation, and real-world tests of commercial chatbots.
“People often talk about AI as if it learns from the internet in some neutral way,” says Hannah Waight, co-first author of the study, an assistant professor of sociology at University of Oregon, and a former postdoc at NYU’s Center for Social Media, AI, and Politics (CSMaP). “It doesn’t. It learns from information environments that have already been shaped by institutions and power, and those environments can leave measurable traces in what models say.”
The researchers call this idea institutional influence.
“The public debate has focused on what AI can generate, but this study points upstream. Before AI systems can influence politics, politics can influence AI,” observes Joshua A. Tucker, a co-author of the paper and co-director of NYU’s CSMaP. “This is a democracy and governance issue, not just a technical issue. As people turn to chatbots for political information, we need to examine which institutions have shaped the answers before a user ever asks the question.”
“We spent years studying how political information flows through traditional and social media. Now we need to do the same thing for frontier models,” adds Solomon Messing, a paper co-author and research associate professor at CSMaP. “The challenge is that in AI systems, those flows are obscure, because the origins of the data become difficult to trace once information is absorbed during model training.”
The study also included researchers from Purdue University, the University of California San Diego, and Princeton University.
To trace institutional influence through the training process, the authors first showed that state-coordinated media appears frequently in real training data. Comparing two sources of Chinese state-coordinated media with a major open-source multilingual training dataset derived from Common Crawl—a nonprofit that provides web-crawl data—the researchers found more than 3.1 million Chinese-language documents with substantial phrasing overlap, or about 1.64 percent of the dataset’s Chinese-language subset. That is over 40 times the rate for documents from Chinese-language Wikipedia, a common training source. Among documents mentioning Chinese political leaders or institutions, the share rose as high as 23 percent. Only about 12 percent of the matched documents came from known government or news domains, suggesting that the material had spread widely across the web before reaching AI training corpora.
The researchers also found that commercial models memorized distinctive phrases associated with this material, suggesting that they had been seen a number of times during training.
“State-coordinated content is not just about what appears in official media,” says Brandon M. Stewart, one of the paper’s authors and associate professor of sociology at Princeton University. “It is also about recirculation; the same phrasing moving through newspapers, apps, reposts, and ordinary web pages until it looks like part of the broader information environment. Once state-coordinated content is in the training data, the model can launder it into what looks and sounds like neutral, objective information.”
The team then tested whether that content could actually shift a model’s behavior. Large commercial models take months and millions of dollars in compute to train so the team experimented with taking a small, open model and adding additional documents to the training process.
The results were clear: adding scripted news to the training data made the models more likely to produce more favorable answers—nearly 80 percent of the time compared with an unmodified model. This is true even when compared to other non-scripted Chinese media, and especially compared to just adding general Chinese-language text from the internet.
“When the same political question produces systematically different answers with only small changes to the training data, that suggests those additional documents are doing real work,” explains Eddie Yang, co-first author of the study and assistant professor of political science at Purdue University.
The team reasoned that if states have strong real-world influence over the pretraining data, it should appear most clearly in the state’s primary language. For example, a question about the Chinese government should produce a more pro-government answer when posed in Chinese than the same question posed in English.
They used this within-model, cross-language comparison to probe commercial models without access to their internal parameters. In responses to political questions about China, human raters judged the Chinese-prompted answer to be more favorable to China 75.3 percent of the time. For prompts not about China, the rate was no different from chance. The language difference gave them a rare window into a closed system. Follow-on studies using real user prompts and additional commercial models found the same general tendency: on questions about Chinese leaders and institutions, answers tended to be more favorable when the prompt was in Chinese than when it was in English.
Additionally, in a cross-national study of 37 countries where a national language is largely concentrated within a single country, models portrayed governments and institutions from countries with stronger media control more favorably in that country’s language than in English. The authors emphasize that this result is correlational, but say it is consistent with the mechanism identified in the China case study.
“This is not evidence that AI companies set out to curry favor with those governments, or that those governments control media systems with chatbots in mind,” says Margaret E. Roberts, a co-author and professor of political science at UC San Diego. “States shape the information environment, the information environment shapes training data, and training data shapes model outputs. But going forward, our findings suggest that LLMs create new incentives for powerful actors to think strategically about the text they disseminate online.”
The authors stress that no single test can capture how a commercial model was trained because many of those details aren’t publicly known. The paper instead combines multiple approaches including analysis of open-source data, memorization tests of commercial systems, retraining experiments, human evaluation, real-user audits, and cross-national comparison to identify one of the ways that political power can enter AI systems. At their project website, the authors show that the results replicate using the latest models released.
Beyond nation-states, the researchers emphasize that other powerful institutions may also be able to shape large volumes of online text.
“Training data is the foundation of modern AI,” says Messing. “If we want to understand the powerful interests these models reflect, we need to know how we’re sourcing the concrete. That starts with more transparency about what goes into the training data.”
This post was originally published on New York University News and republished here with permission.
Reviewed by Irfan Ahmad.
Read next:
• Study of 1,400 AI incidents finds most harm comes from software, not robots