top of page
WHY A CONTROL GROUP IS NEEDED TO DETERMINE IF A K-12 EDUCATION PROGRAM IS EFFECTIVE, in 2 simple charts. The example I'll use is Xtreme Reading - a program for struggling 9th graders that provides year-long reading instruction in a small classroom setting (in lieu of an elective).

  • Xtreme Reading was evaluated in a large, high-quality study, commissioned by the U.S. Institute of Education Sciences (sample of 2,833 students in 17 schools nationwide).

  • As shown below, the study found that reading scores of students in the program rose from the 15th to 25th percentile over the course of 9th grade:

  • But the study had a randomly-assigned control group of students who didn't participate in the program, & control-group students saw an almost identical gain:

  • So the study found that students participating in Xtreme Reading didn't do discernibly better in reading than an equivalent group of students who didn't participate (& received usual school services). In other words, the program was not effective.

  • If the program had been evaluated in the usual non-rigorous way - looking at participants' reading gains over time without reference to a control group of nonparticipants - it would've been (erroneously) deemed highly effective.

  • You may wonder why the control group's reading scores - like the treatment group's - increased over the course of 9th grade (& is that atypical?).

  • Two likely (& not atypical) reasons: (1) students naturally make reading gains in 9th grade, based on usual services provided in the 17 study schools; & (2) students were selected for the study based on low baseline test scores & "regressed toward the mean" when later re-tested.

  • CONCLUSION: Simply looking at before & after improvement in student outcomes, without respect to an equivalent group of nonparticipants (controls), often yields the wrong answer about program effectiveness.

bottom of page