CAPR published an RCT of Multiple Measures Assessments (MMA), a data analytic tool increasingly used by community colleges to place incoming students into college-level vs developmental (i.e., remedial) courses. Quick take: High-quality RCT finds little impact on student success at 4.5y mark. 

​

Program:

​

• MMA uses predictive algorithms – based on high school GPA, placement test scores, time since HS graduation, etc – to place students into developmental (i.e., remedial) vs college-level courses. Colleges' traditional procedure is to use a standardized test to determine placement.

 

Study Design:

​

• The study randomly assigned 12,796 incoming students at 7 State University of New York community colleges to placement using MMA (treatment) vs standardized test (control). Use of MMA changed the placement of 26% of students in math & 51% of students in English (vs their control group counterparts).

​

Findings:

​

• Despite changing student placements, MMA had little impact on the primary study outcomes at the 4.5y follow-up: (i) % completing college-level math (40.5%T vs 39.3%C) & English (53%T vs 51%C); college credits earned (28T vs 27C); % graduating or transferring to 4y college (23.4%T v 23.7%C).

• Of the above, only the effect on college-level English was statistically significant.

​

Comment:

​

• Based on careful review, this was a well-conducted RCT (e.g., baseline balance, negligible attrition, pre-registered primary outcomes).

• Unfortunately, the study abstract's summary of the results (shown below) doesn't mention the central finding of negligible impacts on the 4 prespecified primary outcomes. It instead paints a rosier picture based on exploratory findings (subgroups etc) that aren't reliable - only suggestive.

• This key omission in the study abstract (which busy readers often rely upon to glean the main take-aways) could easily lead policy officials to consider and/or adopt MMA in the mistaken belief it has been shown effective.