Mathematica reported long-term RCT results for KIPP charter middle schools. Quick take: RCT finds a suggestive 4% point gain in college enrollment & no impact on college persistence. (Its reported large impact of attending *both* KIPP middle + high school is quasi-experimental & not yet reliable pending confirmation in future RCTs.)

​

Program:

​

• KIPP is a nonprofit network of 280 public charter schools serving mainly low-income minority students preK-12. Prior high-quality RCTs have found that KIPP elementary & middle schools produce sizable, sustained gains (5-10 percentile points) in reading & math achievement.

 

Study Design:

​

• The new RCT analysis estimated long-term impacts for 2,077 students who participated in a randomized lottery for 5th or 6th grade admission at 1 of 21 KIPP middle schools in 8 states + DC. Based on careful review, this was a high-quality RCT (e.g., good baseline balance, negligible attrition).

 

Findings:

​

• On the first prespecified primary outcome: At the 11-14 year follow-up, lottery winners (T) were 4% points more likely to enroll in four-year college than lottery losers (C) - (51.4% of T enrolled vs 47.6% of C). This effect wasn't statistically significant (p=0.12), so is only suggestive (might be a chance finding).

• On the other primary outcome, the study found no significant impact on college persistence (24% T vs 23% C attended a four-year college for six consecutive semesters).

• However, only 70% of lottery winners actually enrolled in KIPP middle school, so the study then estimated impacts for actual enrollees. This "treatment on treated" analysis, which has (almost) RCT-level credibility, found impacts of 6.5% points on college enrollment & 1% point on persistence. But these impacts weren't statistically significant, so are still only suggestive (not strong) evidence.

• Finally, the study analyzed impacts for lottery winners who attended *both* KIPP middle + high school, & found large, statistically significant impacts: 31% points on four-year college enrollment & 20% points on persistence. However, these findings do not have RCT-level credibility for two reasons: (1) This was a quasi-experimental (vs RCT) analysis whose assumptions can't be fully verified. (2) This analysis was post-hoc (vs prespecified), so is vulnerable to researcher bias - i.e., they can try many analyses & consciously or unconsciously select those yielding a desired result.

• That said, these impacts are very large. Even if they can't be taken at face value (for above reasons), their magnitude clearly suggests the value of future RCT(s) directly testing the impact of attending KIPP middle + high school - to see if the results can be confirmed at RCT level.

 

Comment:

​

• Mathematica's full study report appropriately caveats the quasi-experimental results but its evaluation brief & online summary don't - hence recent inflated press reports in the Wall Street Journal and Washington Post. Disclosure: My former employer, Arnold Ventures, funded the study.