Experimenting with GenAI: Methodology and Application to Decision Biases of Humans with AI

Despite Generative AI (GenAI) rapidly reshaping how people make decisions, experimental methods available to study human-GenAI interaction lag behind the technology. The authors develop an accessible interactive experimental methodology that embeds a live GenAI assistant directly into survey-based experiments, enabling researchers to capture real-time conversation data, manipulate AI behavior through backend system prompts, and integrate full interaction transcripts alongside standard survey measures. The methodology integrates readily available platforms and requires no specialized programming expertise. The authors demonstrate the methodology’s versatility through two studies grounded in well-established decision biases. Study 1 compares static AI advice via screenshots with interactive GenAI engagement, revealing that the two modalities produce meaningfully different behavioral outcomes and that static presentations can overstate the treatment effect. Study 2 introduces agentic nudging, a scalable form of debiasing delivered through backend system prompts, and demonstrates that these manipulations can systematically shift deeply entrenched decision biases with remarkably large effect sizes. Conversation data from both studies reveal that participants interact with GenAI in highly heterogeneous ways, and that the success or failure of AI-assisted decision-making depends on how users engage with the system.