Hidden AI Theft Risks Exposed

Key Points

Moonshot relayed almost 300,000 customer requests to Claude Opus within a ten-day period using thousands of fraudulent accounts, while displaying Claude’s responses to users who believed they were using Kimi.

Both Moonshot and DeepSeek captured Claude’s reasoning traces and outputs to use for training their own models, a technique known as distillation.

Anthropic introduced technical defences including summarised reasoning summaries and Claude Fable 5.1’s preserved thinking feature to prevent this type of attack.

China’s Commerce Ministry rejected the allegations as groundless, stating that distillation is a common industry practice.

Why It Matters

This situation presents significant risks for organisations relying on proprietary AI models. When users believe they are interacting with one service but are actually connected to another, there are concerns about data security, liability, and intellectual property protection. The discovery that Chinese developers covertly extracted reasoning traces from Claude highlights how advanced AI systems can be reverse-engineered through systematic collection of outputs, potentially allowing competitors to build capable models without the investment in research and development. For insurers and risk managers, this raises questions about contractual liability if customers unknowingly route requests through third-party systems, data handling compliance violations, and reputational damage if users discover the deception. The fact that thousands of fake accounts were used to obscure the activity suggests intentional obfuscation. Additionally, the ongoing dispute between nations over AI development practices indicates this may become a broader geopolitical and regulatory issue, potentially affecting how AI services can be deployed internationally and what contractual protections organisations need.