21 of the RCTs I've summarized since last summer found disappointing effects (e.g., no discernible positive impacts on the primary outcomes). 7 of these RCTs (i.e., one-third) nevertheless portrayed the results as strongly positive in the study abstract. Here are specifics:
-
4 RCTs found no statistically significant effect on the primary prespecified outcomes, but don't report this fact in the abstract and instead portray results as positive based on other results that aren't reliable (only suggestive) under established scientific standards (FDA, IES).
-
Here are my summaries of the 4 RCTs (including links to the study abstracts): Vision for Baltimore, Working on Womanhood, Core Knowledge charter schools, Procedural Justice police training.
-
2 RCTs found no statistically significant effects after adjusting for their measurement of numerous primary outcomes (which can lead to false-positive findings); yet their abstracts don't mention this fact and instead make strong claims of positive effects.
-
Here are my summaries of these 2 studies (including links to the study abstracts): Padua case management for low-income families, Repairing abandoned housing to prevent crime.
-
Finally, 1 RCT found a very small, short-term effect that quickly faded over time (it was statistically significant due to a huge sample), yet the abstract presents the results as unambiguously positive. NYC Summer Youth Employment Program (SYEP).
-
The good news is that the other 14 RCTs with disappointing results accurately reported their findings in the study abstract. But, bottom line: inaccurate reporting of disappointing results is common, even in top journals (at least in this limited sample of studies).
-
PS: I focus on the study abstracts because readers - who may be too busy to review a full study - often rely on the abstract to provide a balanced, impartial overview of the headline results. That's why accuracy of reporting in the abstract is critically important.