Computer scientists from a coalition of international research institutions have identified a significant vulnerability in the architecture of frontier artificial intelligence models that allows for the extraction of "hidden reasoning." This internal chain-of-thought, which proprietary AI providers typically encrypt and withhold from users to protect intellectual property and prevent unauthorized model replication, can be exposed through a technique involving "swapping" messages between model variants. The discovery not only raises concerns regarding the leakage of sensitive user data, such as passwords and API keys, but also provides empirical evidence suggesting that some Chinese AI models may have been trained using reasoning data distilled from leading U.S. systems.
The research was conducted by a team including Alexander Panfilov from the University of Tübingen, along with colleagues from the Max Planck Institute, the AI safety institute MATS Research, and the cybersecurity firm Snyk. Their findings, detailed in a technical paper, demonstrate that the vulnerability affects major frontier model providers, including OpenAI, Anthropic, and Google, when their models are accessed via an Application Programming Interface (API).
Technical Mechanism: Exploiting the Model Hierarchy
The core of the vulnerability lies in how AI companies manage computational loads and model alignment. Most frontier AI providers offer a family of models: larger, "frontier" versions that possess high reasoning capabilities but are expensive to run, and smaller, "distilled" versions that are faster and more cost-effective. To optimize performance, companies often send encrypted reasoning traces—the step-by-step logic the AI uses to solve a problem—to a user’s local environment or through an API to offload specific computational tasks.
The researchers discovered that if these encrypted reasoning traces are fed into a smaller, weaker version of the same model family, the smaller model can often be induced to reveal the plaintext content of those traces. This is possible because smaller models frequently share the same decryption keys or architectural frameworks as their larger counterparts but have undergone less rigorous "alignment" training. Alignment is the process by which developers program a model to refuse certain requests, such as revealing its internal logic or generating harmful content. Because the smaller models are less "guarded," they effectively act as a backdoor to the proprietary logic of the more advanced systems.
Chronology of Discovery and Disclosure
The timeline of these events highlights the rapid pace of AI development and the ongoing struggle to secure these complex systems:
- Early 2024: Academic researchers begin investigating the consistency of reasoning traces across different model scales within the same provider ecosystems.
- Late 2024: Speculation begins to circulate on Chinese social media platforms regarding the possibility of intercepting hidden reasoning steps to improve local open-weight models.
- January 2025: The research team led by Alexander Panfilov successfully demonstrates the extraction of API keys and personal passwords from reasoning traces using the model-swapping technique.
- February 2025: The researchers formally alert OpenAI, Anthropic, and Google to the vulnerability, initiating a coordinated disclosure process.
- March 2025: Leading AI providers implement "short-term mitigations" to their APIs to prevent the replay of reasoning traces and the extraction of private user data.
- April 2025: The research paper is published, detailing the findings and the specific case of the Kimi K3 model.
Evidence of Global AI Distillation
One of the most politically sensitive aspects of the report is the analysis of "distillation"—a process where a smaller model is trained on the outputs of a larger, more capable model to mimic its performance. While distillation is a standard industry practice for making AI more efficient, it has become a flashpoint in the technological competition between the U.S. and China.
The researchers tested 90 complex prompts across various models. When they provided the first few words of a hidden reasoning trace from a U.S. model (such as Claude 4.8 or GPT-5.6) to certain open-weight models, they observed a striking similarity in the subsequent logic. Specifically, the Kimi K3 model, developed by the Chinese startup Moonshot AI, produced reasoning steps that were nearly identical to those of the U.S. frontier models.
While the researchers noted that their findings "cannot causally establish distillation"—meaning they cannot prove with 100% certainty that Moonshot AI used stolen U.S. data—the statistical similarity suggests a high likelihood of data crossover. Interestingly, other models tested, such as China’s DeepSeek and the U.S.-based Inkling by Thinking Machines, did not show this level of similarity, suggesting that not all open-weight models rely on distilled proprietary reasoning.
Official Responses and Industry Reactions
The response from the AI industry has been a mix of caution and defense of the open-source ecosystem.
Michael Aciman, a spokesperson for Anthropic, stated that the company values independent research and has already built mitigations to address the "replay behaviors" described in the report. Aciman emphasized that the vulnerability did not allow researchers to recover Anthropic’s primary encryption keys or access their core infrastructure. Google and OpenAI declined to provide specific comments on the vulnerability, though both have reportedly updated their API protocols to prevent the most direct forms of reasoning extraction.
The debate over distillation also reached the highest levels of corporate leadership. Mark Zuckerberg, CEO of Meta, recently argued in a public blog post that distillation is a fundamental principle of the open-source ecosystem. Zuckerberg warned that restricting the ability of researchers and companies to learn from existing models through distillation could stifle innovation and put the U.S. at a disadvantage in the global AI race.
Conversely, U.S. lawmakers have expressed growing concern. In early 2025, both OpenAI and Anthropic briefed legislators on what they described as systematic campaigns by Chinese firms, including Alibaba and DeepSeek, to distill their models. These companies argue that such practices allow competitors to bypass the billions of dollars in R&D costs required to develop frontier models from scratch.
Geopolitical and Policy Implications
The discovery of "stolen thoughts" adds a new layer of complexity to the U.S.-China tech rivalry. As the U.S. implements stricter export controls on high-end semiconductors, Chinese firms are increasingly incentivized to find software-based shortcuts to achieve parity with American AI.
Kyle Miller, a researcher at the Center for Security and Emerging Technologies (CSET), provides a nuanced perspective on the impact of this practice. "Nobody here in the U.S. knows exactly how much distillation is benefiting the Chinese labs," Miller noted. He suggested that while distillation helps bridge the gap, Chinese researchers likely possess the expertise to build cutting-edge models independently. Therefore, even if distillation were completely blocked, the competitive landscape might not shift as dramatically as some "China hawks" suggest.
From a security standpoint, the implications are twofold:
- Data Privacy: The fact that reasoning traces could contain passwords and API keys suggests that as AI becomes more integrated into personal and corporate workflows, the "internal" state of the AI must be treated with the same level of security as a primary database.
- Intellectual Property: If a model’s "thinking" can be easily extracted, the moat protecting a company’s multi-billion dollar investment in AI training becomes significantly shallower.
Future Outlook and Mitigation Challenges
Fixing the distillation vulnerability entirely may prove difficult. Alexander Panfilov noted that a complete solution would require a "fundamental overhaul" of how APIs handle data and how models are aligned across different scales. As long as companies need to offload computation to smaller models or user devices, some form of data transmission will be necessary, creating potential points of interception.
Furthermore, academic experts like Yarin Gal of Oxford University warn that a total crackdown on distillation could slow the overall progress of the field. "If it’s the norm that everyone blocks everyone, then that also will have implications on the rate of progress," Gal said.
As AI models continue to evolve toward "reasoning-heavy" architectures, the transparency of their internal processes will remain a contested territory. The research serves as a reminder that the boundary between an AI’s private "thought" and its public "output" is more porous than previously believed, necessitating new standards for encryption and model alignment in the frontier AI era.
About the Author
