American Economic Review published RCT findings for Saga Education's high-dosage math tutoring of low-income 9th and 10th graders in Chicago. Quick take: High-quality RCTs find sizable effect on math achievement that persists 1-2 years after tutoring ends.
Program:
-
Saga provided math tutoring to 9th and 10th graders in low-income high schools - 50 minutes a day, 5 days a week for 1 school year. Tutors were recent college grads, trained and paid a modest stipend. Tutors worked with two students at a time. Cost was approximately $4800 per tutored student.
Study Design:
-
Two RCTs evaluated Saga at scale in 15 high-poverty Chicago high schools, with a pooled sample of 5,343 students randomly assigned to treatment vs control (>95% Black or Hispanic, 87% low-income).
-
Only about 40% of treatment group students received Saga tutoring due to scheduling and related issues, so the studies estimated impacts for both the full treatment group (“intent to treat”) and those treatment group students who actually received at least 1 tutoring session (“treatment on treated”).
Pooled Findings of the 2 RCTs:
-
The RCTs found sizable effects on math achievement at end of the tutoring year, but what's remarkable is that effects persisted. At end of 11th grade — 1-2 yrs after tutoring ended — the effect on district test scores was 0.10 for the full treatment group (intent-to-treat) and 0.23 for treatment group students who received tutoring (treatment on treated).
-
The 0.23 effect size for tutored students represents a gain of approximately 1.6 grade-levels of learning in 11th grade. Saga also increased tutored students' 11th grade math GPA by a quarter point. These effects were statistically significant, p<0.01.
-
The RCTs found no discernible effects on behavior, absenteeism or high school graduation rates. The first RCT also tested a cognitive-behavioral program for males (Becoming a Man) in 2 treatment arms, and found no discernible effect on academic outcomes, suspensions, or arrests.
Comment:
-
Based on careful review, these were well-conducted RCTs (e.g., excellent baseline balance, moderate sample attrition but not differential, valid analyses). Overall, I think the findings are very encouraging.