OpenAI uses GPT-Red to stress-test its own models for prompt attacks

OpenAI has disclosed GPT-Red, an internal AI red-teaming system designed to attack the company’s own models and find prompt-injection weaknesses before they show up in production. The company is framing the system as part of its safety pipeline for more autonomous AI agents, where a successful prompt attack can change behavior rather than just produce a bad answer.

GPT-Red is not a commercial product, and OpenAI said it does not plan to release it. That decision is material: a system optimized to generate working attacks against frontier models would be useful to defenders, but also to attackers. OpenAI said it is keeping GPT-Red separate from deployed models and using its findings to improve training.

The company said GPT-Red works by repeatedly probing a target model, reading the response and modifying its next attempt based on whether the attack moved closer to the desired malicious outcome. Failed attempts are dropped, while successful ones are refined. In OpenAI’s description, the model automates a task that is usually handled by human red teams, but at a scale and speed human testers cannot match manually.

OpenAI trained GPT-Red with self-play reinforcement learning. In that setup, GPT-Red plays the attacker against defender models across different tasks. The attacker is rewarded when it finds an exploit, while the defender is rewarded for resisting the attack and completing the intended task. As the defender improves, the attacker has to produce more effective attacks, creating an iterative training loop.

OpenAI claims large gains over human testers

OpenAI said GPT-Red succeeded in 84% of tested scenarios, compared with 13% for human red-teamers. The company also said the system helped reduce direct prompt-injection failures to one-sixth the rate seen in its strongest production model from four months earlier.

For one class of attacks described as “fake chain-of-thought” attacks, OpenAI said the success rate fell from more than 95% against GPT-5.1 to less than 10% against GPT-5.6. The company did not disclose the full benchmark design behind those figures, the number of scenarios tested or how directly the human results compare with the automated system’s attempts.

OpenAI also said earlier versions of the technique have been used in training since GPT-5.3. The timing matters because OpenAI released GPT-5.6 weeks earlier and positioned it against Anthropic’s Claude, while prompt injection remains a persistent problem for model providers trying to sell agentic systems into enterprise workflows.

The company said GPT-Red has already exposed weaknesses in agent-style systems. In tests, it took over a Vendy vending machine agent and caused it to change prices and cancel orders. It also compromised command-line coding agents, a relevant target class as developers give AI systems more access to tools, files and execution environments.

Dylan Hunn, an OpenAI research scientist and co-creator of GPT-Red, told MIT Technology Review that “compared to a human red-teamer, the model is very, very good at finding exactly what will work.” Nikhil Kandpal, another co-creator, told the publication that the risks increase as models are granted more autonomy: “The risk surface grows and the blast radius also grows.”

Human red teams still cover gaps

OpenAI acknowledged that GPT-Red has limits. The system is weaker at attacks that unfold over multiple conversational turns, and it has limited coverage for image-based prompt injection. OpenAI said human testers will continue to cover those areas.

Jessica Ji, a senior research analyst at Georgetown University’s Center for Security and Emerging Technology, told MIT Technology Review that the results appear promising, while adding that human expertise remains necessary. That is the sober read: automated red teaming can expand test coverage, but OpenAI’s own disclosure shows the work still depends on people deciding which failures matter and how much risk is acceptable before deployment.

This story draws on original reporting from SiliconANGLE.