A study of conversational agent design

chatbots may improve problem-solving times without affecting perceived satisfaction

      As distance programming education grows, exploring ways to provide one-on-one tutoring is important. We¹ conducted an experiment to study how conversation agent-tutors impact programming. Participants solved two problems in an online learn-to-code platform. In the control, they were given no outside assistance. In the treatment, we simulated an assistive chat providing feedback including encouragement and adapted responses from a corpus of relevant conversations. We measured changes in participants’ time-to-completion and surveyed their satisfaction in the experience. The results indicate that conversational agent-tutors may improve the problem-solving speed, but don’t affect perceived satisfaction. This experiment provides insights into how the design of a conversational agent can meaningfully augment learn-to-code interfaces. (link to paper)