Journal of Artificial Intelligence, Virtual Reality, and Human-Centered Computing
The Mediating Effect of an Agentic AI Chatbot in Scaffolding Online Engagement: Evidence from a Three-Cycle Design-Based Study
Abstract
Vincent English
The rapid integration of artificial intelligence (AI) into higher education presents new paradigms for supporting student engagement in online learning environments. This study investigates how an agentic AI chatbot can scaffold dialogue and enhance engagement among postgraduate students enrolled in an online Executive MBA (EMBA) programme. Using a three-cycle design-based research (DBR) methodology, this study analysed the progression of ten design conjectures across a cumulative sample of 145 EMBA students from nine countries. Data were collected from asynchronous forum posts, Moodle learning management system (LMS) metrics, and the Online Student Engagement (OSE) scale. Quantitative analysis employed Kruskal-Wallis H tests, linear regression, and a Naïve Bayes classifier, whilst qualitative analysis utilised the Toolkit for Systematic Educational Dialogue Analysis (Tech-SEDA) to assess dialogue quality and the Structure of Observed Learning Outcome (SOLO) taxonomy to assess cognitive depth. The findings demonstrate a significant and large effect of the interventions on student engagement across all iterations, with Kruskal-Wallis tests yielding large effect sizes (epsilon-squared ranging from 3.78 to 12.08, all p < .001). The integration of the AI chatbot in later iterations was associated with a substantial increase in the quality of dialogue and a progressively stronger correlation between academic performance and participation (R2 = 0.32 in Iteration 1, rising to R2 = 0.49 in Iteration 3; Cohen's f2 = 0.96). The Naïve Bayes classifier achieved 73.4% accuracy in predicting SOLO levels from dialogic features, confirming that dialogue quality is a reliable predictor of cognitive depth. The study introduces the ENGAGE (Engage, Navigate, Guide, Articulate, Gather, Evolve) framework as a model for ethically and effectively integrating AI into online pedagogy. The results confirm that a well-designed, AI-enhanced learning environment can act as a powerful mediating tool, scaffolding deeper cognitive engagement and fostering a more robust dialogic space for adult learners.