Researcher backdoors open-weight AI model for under $100

Katie Paxton-Fear, a cybersecurity lecturer at Manchester Metropolitan University and staff security advocate at Semgrep, says she backdoored an open-weight AI model in about an hour for less than $100. The experiment matters for companies adopting local models because the attack did not require control of an application stack, only a small fine-tuning job that changed how the model behaved.

Paxton-Fear said in a recent social media post that she began with a narrower test: using fine tuning to make a model prefer snake_case in JavaScript output even when instructed to use camelCase. After that worked, she said she moved to a security backdoor.

According to Paxton-Fear, ten training examples were enough to make the model consistently emit code vulnerable to remote code execution, including in response to new prompts and in domains outside the examples. She also claimed larger models were easier to poison. The model, exact architecture and evaluation details were not disclosed in the reported account, so the result should be read as a demonstration of feasibility rather than a benchmark.

Open weights do not mean inspectable behavior

Paxton-Fear and Semgrep colleagues Isaac Evans and Cris Thomas published a post last week arguing that open-weight models create a different security problem from conventional software dependencies. Their point is that public model weights do not give defenders a practical way to predict all model behavior.

They contrasted that with traditional software, where a binary can still be examined with reverse engineering tools to build a detailed account of what it does. Their argument is that security teams have no comparable capability for neural network weights, especially when a small change can alter behavior only under certain conditions.

Academic researchers have warned for years that models can be subverted, but the issue is receiving more attention as AI supply-chain attacks start to appear and as running open-weight models on local hardware moves from experimentation into production use. That shift changes the buyer risk profile: companies may treat a downloaded model like a package dependency, while having fewer mature tools for provenance, diffing and behavioral inspection.

A second test targeted data theft

David Kaplan, AI security research lead at Origin, ran a related experiment last month. He created a compromised model intended to steal information in a drug discovery setting, a scenario relevant to pharmaceutical companies using AI tools against sensitive research data.

Kaplan said the model was designed to send data out through a send_email tool call without alerting the user. He framed the risk as different from the commonly cited “lethal trifecta” model associated with developer Simon Willison, which focuses on the combination of private data, untrusted input and an outbound channel. Kaplan’s claim is that poisoned weights can supply the malicious behavior themselves, leaving an outbound tool as the key missing piece.

Paxton-Fear and her colleagues said there may not be strong public examples of widely used open-weight models that have been poisoned. Their broader warning is about observability: a compromised model does not need to crash, produce obviously malicious output or trip a conventional detection rule to create business risk. It can steer code, decisions or tool use in ways that are hard to see after deployment.

The same scrutiny problem is not limited to open weights. Commercial frontier model providers also ask enterprises to trust systems that process sensitive data while revealing little about their internal operations. Open-weight models add tampering risk, but closed models leave customers with a different version of the same problem: limited ability to verify what the system will do under edge conditions.

This story draws on original reporting from The Register.