UpGrade is an open-source tool for A/B testing in educational software. In this study, the authors used UpGrade to run large-scale online experiments in an educational game.
The experiments were aimed at increasing student engagement. The authors experimented with various features of the game, including question difficulty, game narrative, and feedback style, to determine which conditions produced optimal outcomes.
The authors conducted three different experiments, one after another. All of these experiments were created and monitored through UpGrade.
During implementation, the authors encountered several issues. They found that buggy programming logic in the educational software can produce invalid experiment enrollments in UpGrade.
They also found that without tracking the version of the educational software, noise can be introduced into the experimental data.
The paper presents several recommendations to help researchers and developers avoid these pitfalls. Results of the experiments are not discussed in this paper.