Large-scale, classroom-based experiments using adaptive instructional software pose unique challenges for experimental design and deployment.
Adaptive software allows students to advance through curriculum at different rates and encounter content at different times, meaning that content targeted for experimentation is often reached asynchronously by students within the same classroom.
Many pedagogical approaches subject to experimentation also require multiple “touch points” with students throughout a course.
Nimble experimental methods require experiments to be able to take place at any time during the school year. As a result, a robust experimentation system may need to account for students’ prior educational experiences, both to exclude students from an experiment when appropriate and to ensure that the pedagogical approach during the experiment is consistent with the student’s experience before and after the experimental period.
The paper discusses several key design considerations for conducting digital experiments within adaptive educational software, including ordering and sequencing, coordination of experimental activities, and exclusion criteria.
The authors then illustrate how these principles were applied in two recently conducted large-scale experiments using the UpGrade A/B testing platform.