Multimodal AI models offer promising ways to analyze classroom activity using transcripts, audio, and video, but real-world classroom recordings frequently contain missing, noisy, or misaligned information that can reduce model reliability.
This work introduces a Delta-Gated Multi-Forward-Pass (DG-MFP) Longformer for robust multimodal classroom discourse understanding. The approach treats transcript text as the primary modality and models audio and video as additional sources of information.
A shared encoder processes multiple combinations of the available modalities—including text alone, text with audio, text with video, and all three together—to isolate the contribution of each source. Delta gates then estimate the reliability and value of those additional multimodal signals relative to the transcript baseline.
The researchers evaluate the model using the Artificial Intelligence for Advancing Instruction at Scale (AIAIS) dataset, which includes classroom video data used to identify discourse and instructional patterns. They also test performance under deliberately missing and misaligned modalities to approximate challenges found in real classroom recordings.
Results show that the DG-MFP approach improves overall F1 performance over standard multimodal fusion methods and is substantially more robust when modalities are missing or misaligned. The analysis also shows that audio and video provide complementary information for particular classroom discourse tasks, especially when transcript evidence alone is insufficient.
