This initiative will expand the utilization of an academic chatbot in core courses. The chatbot provides basic academic information, utilizes intelligent agents in the LMS to monitor student engagement and prompts as necessary, evaluates readiness for quizzes and reminds students about assignments and deadlines. GSU received a $7 million Post Secondary Student Success grant from the Department of Education to launch this technology in introductory English and Math courses in Fall 2024. Chatbot technology is already used in American Government, Macroeconomics and Chemistry.
Evaluation Plan and measures: Student academic outcomes in courses supported by the technology
KPIs: DFW rates
Baseline measure (for each KPI): Baseline will be DFW rates in Fall 2023
Current/most recent data (for each KPI): DFW rates in introductory math in Fall 2023 averaged around 30% on the Atlanta campus and between 32 and 50% at PC. DFW rates in introductory English in Fall 2023 averaged 19-25% on the Atlanta campus and between 32-34% at PC.
Goal or targets (for each KPI): Reductions of DFW rates
The student impact of the 2024-205 first academic year of study totaled to 2,148, with 1,086 receiving the intervention. The initial year of study has shown positive indicators for student outcomes, with subsets of students performing similarly to those in previous studies at GSU. Additionally, the DWF rate improvement for focal faculty participating in the study for the Fall 2025 semester improved at a greater rate than the average for the course. With these positive indicators in place, the study has continued into its second academic year, adding an additional 1,143 students to the math study.
This work was expanded to include sections of entry-level English courses for the 2025-2026 academic year. This study currently incorporates 16 different instructors teaching 16 sections of English 1101 and 426 students on the Atlanta campus and 4 Perimeter College Instructors teaching 21 sections and 471 students in total, for a total of 897 students (477 receiving treatment).
To continue in both English and math courses into the spring semester. Additionally, we are conducting randomized control trials in Principles of Chemistry (Chem 1211K) in all available sections to test the scalability of the intervention. The formalized randomized control trails are scheduled to conclude in Chemistry after the spring 2025 semester, Math after summer 2025 semester but will continue in English courses for the 2026-2027 academic year.
In addition to the randomized controlled trials, we plan to continue executing the scaled instance of the chatbot within the American Government course, as well as the Principles of Microeconomics course.
As we move out of formalized study, we anticipate challenges as we scale the intervention across additional faculty and sections. This is a known challenge, and centralized resources for faculty and student training, as well as consistent staff support, will help support and structure the intervention through this time.
