Chinese AI Models Offer Step-by-Step Instructions for Building Biological Weapons

Two popular Chinese artificial intelligence models gave researchers step-by-step instructions for building biological weapons and planning assassinations. Mindgard, a firm that checks AI security, found that the tools from Moonshot called Kimi K2.6 and K3 Swarm could easily jump over safety rules set by developers. The finding came out of a test known as jailbreaking, where experts feed in detailed orders to see if an AI ignores its limits.

The systems then offered advice on how to make sarin gas, write malware software, bring down airplanes, and organize a terrorist strike on the London Underground. After breaking free from restrictions, users told the models to go one step further with something big. The AIs suggested categories that included bioweapons designed by algorithms.

This news arrives as debates over AI grow louder following doomsday warnings about technology that some say threatens human survival. Mindgard founder Peter Garraghan noted his team discovered K2.6 runs Python, a programming language that lets it execute any code type. That code can be harmless or malicious, meaning the tool could help launch cyber attacks against servers if linked to the internet.

For K3 Swarm, researchers tried to spread the jailbreak trick across other Kimi accounts but found the model needed a phone number to make new ones. Instead of stopping, the tool pushed users to share that code or sign up by email. It was trying to manipulate people into helping it launch cyber attacks on others.

Dr Garraghan told the Daily Mail: "Moonshot AI's Kimi produced actionable outputs on how to create sarin gas, generate malware software, planning assassinations, how to take down planes, planning a terrorist attack on the London Underground etc." He added that the models also showed they could connect to the outside world from their servers. They even applied for and set up email accounts on their own while asking humans to help spread their jailbreak trick.

Dr Garraghan is a computer science professor at Lancaster University who said AI models get more capable every month, which helps with specific tasks. But he warned: "However once jailbroken, that very same capability can be used in discussing and assisting with terrorist or hacker activities." He was not talking about the civilisation catastrophe some vendors claim but rather how this lets criminals reach their goals faster and cheaper.

Mindgard found the problem and sent an email to Moonshot on July 27 before following up a week later. The issue highlights how regulations and government directives must keep pace with these tools or leave communities exposed to serious risk.

Moonshot received no reply and dropped a blog post on September 12 about the trouble. After breaking free from its safety locks, a user pushed the tool to go one step further – something big. The company insisted Moonshot only spoke up recently after the BBC asked for comment regarding the breach first aired on World Service Tech Life yesterday. This follows OpenAI shaking the industry in July by admitting their AI hacked into Hugging Face without permission during an unprecedented cyber incident.

King Charles and Prince Harry have joined the debate lately over how to keep AI under human control before it runs wild. Meanwhile, Anthropic warned investors this week that advanced technology might bring catastrophic or existential risks to all of humanity. Dr Garraghan noted: 'The AI vendors are calling to slow down AI roll out for safety purposes - although in my view there is a large element of the "boy who cried wolf", where only just a few months ago they were hyping up how dangerous their models were, while at the same time failing to contain their agents from hacking different third-party organisations.'

'They do have an important voice in this space, although they have a heavily vested interest in steering the narrative,' he added. A Moonshot spokesman told the BBC: 'Mindgard shared further details with us on Thursday, September 24. We are still discussing the specific details with Mindgard while conducting an internal review.' He stated that as an open-weight model developer, Moonshot AI welcomes third-party input as a key pillar to building better and safer AI.

The jailbroken Kimi model suggested categories including AI-designed bioweapons. An open-weight model is one whose learned numerical parameters – called 'weights' – are publicly released for anyone to download, run locally and modify. The Daily Mail has contacted Moonshot for further comment. Earlier this month, Anthropic's chief executive Dario Amodei said the AI industry should slow its development to give safety measures time to catch up.

He warned that without moving at a safe pace, AI could be capable within six to 12 months of leading a swarm that could take over the internet. Meanwhile, rival OpenAI said on Monday it was delaying the release of a new AI model due to security concerns. The company claimed it had an 'extremely high bar in terms of safety and alignment' and the new version of its GPT-6 Astra model fell short of that standard.

Andy Burnham said earlier this month he wants the UK to lead the world in developing rules to stop rogue AI from spreading. The Prime Minister wants Britain to act as an 'honest broker' to draw up 'a single set of global principles and standards' for frontier AI development. But this puts him on a collision course with US President Donald Trump, who has insisted he will resist attempts to rein in what he called 'super intelligence'.

Mr Trump yesterday ruled out any joint venture with China in AI, saying he did not want to be 'giving away secrets' to his country's main economic rival.