This study evaluates a mathematics mastery learning platform used by over half a million U.S. students in grades 3–8.
This platform was examined in prior research via a between-group cross-sectional design on data from a single school year, which found a positive relationship between time spent on the platform and standardized math test scores.
Although the authors included statistical controls for prior knowledge, the influence of potential unobserved confounding and self-selection artifacts could not be ruled out. In the present study, the authors improve on this past work with a robust longitudinal analysis that makes use of multi-year panel data.
By examining the changes in individual students’ platform usage and standardized test scores across years, the authors were able to treat each student as their own control, thereby eliminating confounding from time-invariant student factors, such as general ability and motivation.
The authors also examined possible treatment interactions across demographic groups. Using this approach, they replicated the previous findings and found positive treatment effects that did not vary across demographic subgroups, suggesting that all students benefited equally, regardless of racial background.
These results are a substantial improvement over earlier research and further demonstrate the promise of technology-enhanced, mastery-based learning to enhance outcomes for all students.
A limitation of this approach is that the authors cannot rule out time-varying confounds, such as hiring a tutor or a change to school policy. The article discusses additional limitations of these findings and potential avenues for obtaining greater specificity in the measurement of platform efficacy.