Annenberg Institute posted a quasi-experimental study of a California program to train teachers in the science of reading. Quick take: Despite study claims of positive impacts (covered by NY Times & others), it found no discernible effect on 3rd grade reading achievement in its primary analysis.

​

Program:

​

• The study evaluated California’s Early Literacy Support Block Grant, providing low-performing elementary schools with teacher training in the science of reading, new funding (~$1000/student), spending flexibility, expert assistance, & other supports to improve reading.

 

Study Design:

 

• The study's primary, prespecified evaluation method (set out in the study registration) was to compare outcomes over 1y for (i) schools that were just above a threshold percentage of struggling readers (& thus received the program) to (ii) schools just below that threshold (which were ineligible for the program). This type of study is known as “regression discontinuity.”

 

Findings:

​

• Using this method, the study found no statistically significant effect on the percent of 3rd graders scoring near-proficient or higher in reading after 1 school year. In fact, the impact table (shown below) shows a slight decrease of 4-7% points (though not statistically significant).

• The study abstract unfortunately does not mention the disappointing results of the primary analysis, & instead presents the study's findings as unambiguously positive based on post-hoc (vs prespecified) analyses that are far less credible.  

• They're less credible because, in post-hoc analyses, researchers have wide discretion over key study parameters - e.g., which schools comprise the comparison group, & which analysis methods to use - and can consciously or unconsciously select parameters that yield a desired result.

​

Comment:

​

• Unfortunately, the study received uncritical coverage in the NY Times, Education Week, & elsewhere - coverage which largely repeats the study abstract's positive claims.