Optimizing an Educational Game Using UpGrade: Challenges and Opportunities
UpGrade is an open-source tool for A/B testing in educational software. We this study, we used UpGrade to run large scale online experiments in an educational game. Our experiments were aimed at increasing the student engagement. We experimented with various features of the game such as question difficulty, game narrative, feedback style, etc. to find out which conditions produced optimal outcomes. One after another, we conducted 3 different experiments. All of these experiments were created and monitored through Up-Grade. We faced several issues during the implementation of these experiments within UpGrade. We discovered that buggy programming logic in the educational software can produce invalid experiment enrollments in UpGrade. We also found out that without tracking version of the educational software, it is possible to get noise in the experimental data. We present several recommendations to avoid these pitfalls. Results of our experiments are not discussed in this paper.