Dangerous AI coding agent requests slip past reviewers in Wauters’ game
A 40,000-run simulation found players approved one-third of malicious agent requests, underscoring limits of per-command review.
By Renata Fuchs · Policy Reporter
· 3 min read
A browser game examining dangerous AI coding agent requests found that players approved roughly one in three simulated malicious commands across more than 40,000 runs. The result is not a measure of production developer behavior, but it adds evidence that asking people to review every agent action can turn security oversight into a high-volume, low-context task.
Belgian software developer Alex Wauters built the game to mimic permission prompts from a coding agent. Players had 60 seconds to approve or reject as many requests as they could, with penalties for both allowing a risky command and blocking a safe one. The dataset covered 409,000 approval and denial decisions, according to The Register.
The test deliberately contained a far greater share of malicious requests than a developer should encounter in ordinary work. That limitation matters: the one-third figure describes performance in a time-limited, adversarially weighted game, rather than the rate at which engineers miss harmful agent actions in live deployments.
Why do coding-agent permission prompts cause approval fatigue?
The game’s missed requests often looked routine unless players inspected surrounding context. Scope violations, such as attempts to read Kubernetes configuration or lists of AWS credentials, were missed 35% of the time, The Register reported. Requests to unknown APIs and typosquatted packages were also frequently approved.
One prompt, npm run analyze, was approved nearly 65% of the time. The command can execute whatever script is defined in a project’s package.json file. Wauters said the game displayed that script in the agent history, suggesting many participants did not examine the available context before granting permission.
That creates an operational problem for teams using agents to run longer workflows. A reviewer who wants to validate a command may need to inspect files, dependencies and the agent’s intended scope. Repeating that work at every prompt cuts into the productivity benefit that prompted delegation in the first place.
Anthropic has reported a similar acceptance pattern from Claude Code telemetry. In a May engineering post, the company said users approved about 93% of permission prompts and became less diligent as they saw more of them. Anthropic described per-action human approval as fallible, and said its classifier caught roughly 83% of what it calls “overeager behaviors” before execution in its evaluation.
What should teams use instead of approving every command?
Anthropic’s argument is not that a classifier replaces review. It says model-level controls are probabilistic and should not stand alone. The more durable control is containment: limit what an agent can reach with sandboxes, virtual machines, filesystem boundaries, egress controls and narrow tool permissions. A read-only database credential, for example, limits the damage an agent can cause even when it makes a bad decision or processes hostile input.
That is consistent with the broader view that enterprise security is a program, not a product. For coding agents, safeguards need to include the execution environment and access policy, not only a prompt asking a person to click approve.
A position paper by researchers affiliated with Carnegie Mellon, Stanford, Princeton and the University of Illinois Urbana-Champaign makes the related case that coding agents must be judged on how people can supervise, verify and steer them, not only on autonomous task completion. Wauters’ game offers a narrow but concrete warning: human approval is a control with failure modes of its own.
This story draws on original reporting from The Register.
