A collaborative investigation by computer scientists from the University of Tübingen, the Max Planck Institute, MATS Research, and the cybersecurity firm Snyk has revealed a significant security flaw in the application programming interfaces (APIs) of leading artificial intelligence providers. The discovery demonstrates a method to extract the "hidden reasoning" or "chain of thought" processes that frontier AI models use to solve complex problems. These internal processes, which companies typically keep proprietary to maintain a competitive advantage and prevent model cloning, were found to be accessible through a technique involving the exploitation of smaller, less-aligned model variants.
The implications of this discovery are twofold. First, it provides empirical evidence—though not definitive proof—that certain open-weight models, particularly those developed in China, may have been trained using "distillation" techniques that leverage these hidden reasoning traces from US-based frontier models. Second, the researchers demonstrated that this vulnerability could be exploited to recover sensitive personal information, such as passwords and API keys, that might be inadvertently embedded within a model’s internal reasoning steps.
The Mechanism of Reasoning Extraction
The core of the vulnerability lies in how AI companies manage computational loads and model alignment. Advanced AI models, such as those developed by OpenAI, Google, and Anthropic, perform "reasoning" by breaking down queries into smaller, logical steps. While the final output is visible to the user, the intermediate steps are often encrypted and sent to the user’s local machine to offload specific computational tasks.
The research team, led by Alexander Panfilov of the University of Tübingen, discovered that because many companies offer a family of models (ranging from "small" or "mini" to "large" or "ultra"), these models often share the same decryption keys for their reasoning traces. By intercepting the encrypted reasoning of a highly aligned, large model and feeding it into a smaller, "weaker" version of the same model family, the researchers were able to bypass security filters. Smaller models typically undergo less rigorous "alignment" training—the process of teaching a model what it should and should not say—making them more likely to reveal the decrypted reasoning traces when prompted.
Florian Tramer, a computer security specialist at ETH Zürich, characterized the technique as a sophisticated "replay attack." By swapping the intended recipient of the encrypted data with a less-restricted variant, the researchers effectively turned the AI’s own architecture against itself to reveal its internal logic.
Chronology of Discovery and Mitigation
The timeline of these findings highlights a rapid cycle of discovery and industry response:
- Early 2024: Speculation begins on Chinese social media platforms regarding the possibility of accessing hidden reasoning traces for the purpose of model distillation.
- Late 2024: Researchers from the University of Tübingen and their partners identify the technical vulnerability across major frontier model APIs, including those of OpenAI, Anthropic, and Google.
- January 2025: The research team officially alerts the affected companies to the vulnerability, specifically highlighting the risk of personal data leakage and large-scale reasoning distillation.
- February 2025: OpenAI and Anthropic begin implementing "short-term mitigations" to prevent the replay of reasoning traces across different model versions.
- Late February 2025: The researchers publish their findings in a comprehensive paper, noting that while data leakage has been mitigated, the underlying architecture that allows for reasoning extraction remains partially vulnerable without a fundamental API overhaul.
Evidence of Global Reasoning Distillation
The practice of distillation—using the output of a large, sophisticated model to train a smaller, more efficient one—is a standard technique in AI development. However, the use of "hidden" reasoning traces represents a more controversial frontier. The researchers tested several open-weight models by providing them with the initial fragments of reasoning traces captured from proprietary US models.
The study found that Moonshot AI’s Kimi K3 model produced outputs that were "strikingly similar" to the hidden reasoning traces of Claude 4.8 Opus and GPT-5.6 Sol. This suggests that the developers of Kimi K3 may have had access to and utilized the internal logic of these US models to enhance their own model’s performance. In contrast, other models tested, such as China’s DeepSeek and the US-based Inkling by Thinking Machines, did not show the same level of similarity, suggesting they were developed using independent datasets or different methodologies.
Geopolitical Context and Official Responses
The discovery arrives amidst escalating tensions between the United States and China over AI supremacy. US policymakers have expressed increasing concern that Chinese firms are "skipping steps" in AI development by distilling the capabilities of US frontier models, thereby narrowing the technological gap at a lower cost.
In February 2025, OpenAI informed US lawmakers that DeepSeek appeared to have distilled its R1 reasoning model from OpenAI’s proprietary systems. Similarly, Anthropic testified in June 2025 that Alibaba’s Qwen models showed signs of systematic distillation from Anthropic’s Claude series. The recent research provides the first independent technical framework to support these claims, though the researchers cautioned that their findings "cannot causally establish distillation" in a legal sense.
Anthropic spokesperson Michael Aciman stated that the company values independent research and has implemented mitigations. He emphasized that the vulnerability did not allow researchers to access Anthropic’s core infrastructure or encryption keys. Google and OpenAI declined to comment on the specific findings, though both have historically advocated for the protection of proprietary "chain of thought" data.
Supporting Data and Technical Analysis
The researchers’ methodology involved a rigorous "replay" test using 90 complex reasoning questions. The data revealed a significant disparity in how different models handled "foreign" reasoning traces:
| Model Tested | Origin | Similarity to US Reasoning Traces | Distillation Likely? |
|---|---|---|---|
| Kimi K3 | China | High (Striking Similarity) | Evidence Suggests Yes |
| DeepSeek R1 | China | Low/Moderate | Inconclusive |
| Inkling | USA | Low | No |
| GPT-4o (Small) | USA | High (Internal Replay) | N/A (Internal Test) |
The data suggests that the "alignment gap" between large and small models is the primary vector for this vulnerability. While a large model might refuse to show its work due to safety protocols, a smaller model with the same "vocabulary" and decryption capability lacks the necessary guardrails to withhold that information.
Broader Impact and Future Implications
The revelation of this vulnerability has significant implications for the future of AI safety and the open-source ecosystem.
1. The "Open vs. Closed" Debate:
The findings provide ammunition for "China hawks" who argue that US AI technology must be more strictly guarded. However, leaders in the open-source community, including Meta CEO Mark Zuckerberg, have warned against over-regulation. Zuckerberg recently stated that distillation is an "important principle" of the open-source ecosystem and that restricting it could inadvertently hamper US innovation by slowing down the rate at which efficient, specialized models can be developed.
2. Data Privacy and Security:
The fact that API keys and passwords were found within reasoning traces highlights a new category of "prompt injection" or "data leakage" risks. As AI models are integrated deeper into enterprise workflows, the "hidden" thoughts of the AI may contain more sensitive data than the final output. While the current vulnerability has been "mitigated," security experts warn that as long as reasoning is offloaded to client-side devices, the risk of interception remains.
3. The Rate of AI Advancement:
Yarin Gal, a computer scientist at Oxford University, noted that distillation has been a primary driver of the rapid advancement in AI capabilities over the last 24 months. If frontier labs begin to "lock down" their models to prevent any form of distillation, the industry may see a plateau in the performance of smaller, more accessible models that are currently used for mobile and edge computing.
4. Strategic Independence:
Kyle Miller of the Center for Security and Emerging Technologies (CSET) offers a different perspective on the geopolitical impact. He argues that even if distillation is occurring, it may only offer a marginal advantage. Miller suggests that Chinese labs likely possess the indigenous expertise to build frontier models regardless of their ability to distill US models, and that the competitive landscape would not be "dramatically changed" even if distillation were completely blocked.
As the AI industry moves toward more complex reasoning architectures, the boundary between a model’s "private" logic and its "public" output is becoming increasingly porous. This research underscores the need for a fundamental re-evaluation of how AI APIs are structured to ensure that "thinking" remains as secure as the "results." For now, the industry remains in a state of adjustment, balancing the need for computational efficiency with the imperative of protecting intellectual property and user data.
About the Author
