CAPR published an RCT of Multiple Measures Assessments (MMA), a data analytic tool increasingly used by community colleges to place incoming students into college-level vs developmental (i.e., remedial) courses. Quick take: High-quality RCT finds little impact on student success at 4.5 year mark. 

​

Program:

​

• MMA uses predictive algorithms – based on high school GPA, placement test scores, time since HS graduation, etc – to place students into developmental (i.e., remedial) vs college-level courses. Colleges' traditional procedure is to use a standardized test to determine placement.

 

Study Design:

​

• The study randomly assigned 12,796 incoming students at 7 State University of New York community colleges to placement using MMA (treatment) vs standardized test (control). Use of MMA changed the placement of 26% of students in math and 51% of students in English (vs their control group counterparts).

​

Findings:

​

• Despite changing student placements, MMA had little impact on the primary study outcomes at the 4.5 year follow-up: (i) % completing college-level math (40.5% T vs 39.3% C) and English (53% T vs 51% C); college credits earned (28 T vs 27 C); and % graduating or transferring to 4 year college (23.4% T v 23.7% C).

​

• Of the above, only the effect on college-level English was statistically significant.

​

Comment:

​

• Based on careful review, this was a well-conducted RCT (e.g., baseline balance, negligible attrition, pre-registered primary outcomes).

​

• Unfortunately, the study abstract's summary of the results (shown below) doesn't mention the central finding of negligible impacts on the 4 prespecified primary outcomes. It instead paints a rosier picture based on exploratory findings (subgroups etc) that aren't reliable - only suggestive.

​

• This key omission in the study abstract, which busy readers often rely upon to glean the main take-aways, could easily lead policy officials to consider and/or adopt MMA in the mistaken belief it has been shown effective.