Study finds AI advice makes users less willing to say they do not know

Researchers from French and Italian universities found that access to AI advice made people far less likely to admit they did not know an answer, even when the AI system was usually wrong. The finding matters for companies rolling AI assistants into search, education and workplace software because the study suggests users may treat machine output as authority while becoming less accurate.

The work was conducted by Valerio Capraro of the University of Milano-Bicocca, Chiara Marcoccia of École Normale Supérieure and Walter Quattrociocchi of Sapienza University of Rome. Their paper is titled AI advice suppresses people’s willingness to say ‘I don’t know’, even when the advice is wrong and accuracy is incentivized.

Capraro said the group wanted to test whether easy AI answers interfere with a basic form of judgment: recognizing the limits of one’s own knowledge. The researchers built a question set around visual details from films, including the color of a team’s uniform in Bend It Like Beckham and the vehicle Monica drives in Like a Cat on a Highway. The rationale was that such details were less likely to appear in model training data.

The experiment used Step 3.5 Flash, which the researchers said typically failed on the questions. They also tested newer frontier models, including GPT-5.5, Claude Sonnet 4.6 and Gemini 3.5 Flash. Those systems missed the vehicle question but were more often correct on other film details. The choice of a weaker model was deliberate, according to the paper, because a drop in human judgment would be harder to explain as sensible reliance on a reliable tool.

Participants were split into two groups. One answered without AI assistance, while the other could ask the AI for advice. Without AI, 44% said they did not know the answer. With AI access, that figure fell to 3%, according to Capraro.

Accuracy also deteriorated. Capraro said 27% of participants in the no-AI group answered correctly, compared with 9% in the AI-assisted group. The result indicates that some participants who might otherwise have answered correctly accepted a wrong suggestion from the model.

Confidence moved in the opposite direction. Capraro said confidence was 30% in the baseline group and rose to 76% when participants had AI assistance. In other words, the AI-assisted group was less accurate and more certain.

The researchers also tested monetary incentives for correct answers. Payment improved performance, but did not restore the baseline. The share of people willing to say they did not know increased from 3% to 8%, and accuracy rose from 9% to 16%. Both remained below the no-AI figures of 44% and 27%.

The experiment used film trivia, a narrow setting. Capraro and his co-authors argue the effect may extend to other areas where people lean on AI systems for answers despite known error rates. That claim will need more evidence across domains, but the mechanism is relevant to product teams: a confident interface can shift users from uncertainty to repetition, even when the answer is bad.

Capraro said model providers could help, but he was skeptical that their incentives fully align with reducing overreliance. He pointed instead to AI literacy and education policy, with particular concern for children who begin using these systems before they have developed independent critical skills.

This story draws on original reporting from The Register.