NBER posted an RCT of Khoaching with Khan Academy (KWiK), a program integrating computer-assisted learning into classroom math teaching. Quick take: Despite study claims of positive results, it found no discernible impact on grade 3-8 math scores on the state test after one year.
Program:
KWiK combines Khan Academy - a computer assisted learning platform in which students watch short videos, complete short exercises, get feedback, etc. - with weekly coaching for teachers on using the platform as part of their curriculum.
Study Design:
The study randomly assigned 224 interested math teachers in grades 3-8 in Arlington, TX schools (grouped together with other teachers in same school and grade) to treatment vs control. The large majority of treatment-group teachers attended the virtual training sessions and met with their coach.
Based on careful review, this was a high-quality RCT (e.g., baseline balance, low attrition, valid analyses).
Findings:
Unfortunately, the study found no discernible impact on the primary prespecified outcome: students' end-of-year scores on the state math test (the non-significant effect size was .025).
The study reports positive math impacts in grades 3-6, as opposed to grades 7-8. But this was a post-hoc (not prespecified) subgroup analysis that isn't reliable under established scientific standards (IES, FDA) as it can easily yield false-positive/chance findings.
The study and a related quasi-experiment found suggestive evidence that positive impacts occurred when teachers gave students sufficient Khan Academy practice time (e.g., 35 minutes a week). This seems like a good hypothesis to test in future RCTs.
Comment:
Unfortunately, as is too common, the study abstract doesn't mention the null finding on the primary outcome and instead portrays the results as unambiguously positive based on the post-hoc subgroup analysis (see below; the Nashville study referenced here was a one-week pilot).
