More engagement did not mean more learning when AI tutors were steered by prompts

The study tested pedagogy inside real coursework

The six-week crossover study assigned 1,059 introductory programming students to a constrained baseline tutor and two alternatives grounded in self-regulated-learning frameworks. One emphasized planning, monitoring, and reflection; the other encouraged deeper cognitive engagement.

Because students experienced multiple conditions during authentic coursework, the design tested more than a one-off preference survey. It asked whether prompt-level pedagogical steering changed measurable behavior and learning.

Activity moved; confirmed outcomes did not

The four preregistered confirmatory measures showed no statistically significant differences across conditions. Exploratory analysis suggested that some scaffolded interactions lasted longer, included longer messages, and produced more constructive contributions.

That distinction is essential. Exploratory findings can generate hypotheses, but they do not overturn the confirmed result. A dashboard showing more time, messages, or engagement can improve while durable learning remains unchanged.

Educational AI needs outcome accountability

The result does not establish that reflective tutoring is ineffective. It shows that system prompting alone did not reliably produce the intended gains in this setting. Students may need control over when they receive direct help or scaffolding, and tutors may need richer adaptation than one global instruction layer.

Schools and vendors should test retention, transfer, independent problem-solving, and calibration rather than optimizing conversation length. An AI tutor succeeds when the learner becomes more capable without it.

• Predefine learning outcomes before deploying the tutor.
• Separate exploratory engagement signals from confirmed educational gains.
• Test student choice between direct and scaffolded support.
• Measure whether capability transfers to work completed without AI.