Evaluating the Impact of a Mathematics Mastery Learning Platform on Student Achievement: A Large-Scale Longitudinal Analysis
This study evaluates a mathematics mastery learning platform used by over half a million U.S. students in grades 3–8. This platform was examined in prior research via a between-group cross-sectional design on data from a single school year, which found a positive relationship between time spent on the platform and standardized math test scores. Although the authors included statistical controls for prior knowledge, the influence of potential unobserved confounding and self-selection artifacts could not be ruled out. In the present study, we improve on this past work with a robust longitudinal analysis that makes use of multi-year panel data. By examining the changes in individual students’ platform usage and standardized test scores across years, we were able to treat each student as their own control, thereby eliminating confounding from time-invariant student factors (e.g., general ability, motivation). We also examined possible treatment interactions across demographic groups. Using this approach, we replicated the previous findings and found positive treatment effects that did not vary across demographic subgroups - suggesting that all students benefited equally, regardless of racial background. These results are a substantial improvement over earlier research, and further demonstrate the promise of technology enhanced, mastery based learning to enhance outcomes for all students. A limitation of this approach is that we cannot rule out time-varying confounds, such as hiring a tutor or a change to school policy. We discuss additional limitations of these findings and potential avenues for obtaining greater specificity in the measurement of platform efficacy.