More engagement did not mean more learning when AI tutors were steered by prompts
A preregistered ICER 2026 study tested whether system prompts could make AI tutors produce better learning behavior in an authentic introductory programming course. In a three-arm crossover design involving 1,059 students over six weeks, researchers compared a constrained baseline tutor with two tutors prompted to support planning, monitoring, reflection, and deeper cognitive engagement. Across four preregistered confirmatory measures, the study found no statistically significant differences. Exploratory analyses found that students sometimes spent longer, wrote longer messages, and made more constructive contributions with the self-regulated-learning tutors, while the relationship between cognitive load and quiz performance also shifted. Those exploratory patterns should not be presented as confirmed learning gains. The practical signal is narrower and important: changing a tutor's system prompt can change interaction without reliably changing measured learning. Better educational AI may require student choice, adaptive pedagogy, stronger course integration, and evaluation based on durable capability rather than engagement alone.
