top of page
NBER posted an RCT of Khoaching with Khan Academy (KWiK), a program integrating computer-assisted learning into classroom math teaching. Quick take: Despite study claims of positive results, it found no discernible impact on grade 3-8 math scores on the state test after one year.

Program:

  • KWiK combines Khan Academy - a computer assisted learning platform in which students watch short videos, complete short exercises, get feedback, etc. - with weekly coaching for teachers on using the platform as part of their curriculum.


Study Design:

  • The study randomly assigned 224 interested math teachers in grades 3-8 in Arlington, TX schools (grouped together with other teachers in same school and grade) to treatment vs control. The large majority of treatment-group teachers attended the virtual training sessions and met with their coach.


  • Based on careful review, this was a high-quality RCT (e.g., baseline balance, low attrition, valid analyses).


Findings:

  • Unfortunately, the study found no discernible impact on the primary prespecified outcome: students' end-of-year scores on the state math test (the non-significant effect size was .025).


  • The study reports positive math impacts in grades 3-6, as opposed to grades 7-8. But this was a post-hoc (not prespecified) subgroup analysis that isn't reliable under established scientific standards (IES, FDA) as it can easily yield false-positive/chance findings.


  • The study and a related quasi-experiment found suggestive evidence that positive impacts occurred when teachers gave students sufficient Khan Academy practice time (e.g., 35 minutes a week). This seems like a good hypothesis to test in future RCTs.

 

Comment:

  • Unfortunately, as is too common, the study abstract doesn't mention the null finding on the primary outcome and instead portrays the results as unambiguously positive based on the post-hoc subgroup analysis (see below; the Nashville study referenced here was a one-week pilot).



bottom of page