Science Magazine published an RCT at 22 US colleges, reporting that a brief "social belonging" writing exercise increased 1st-year college completion rates. Quick take: These claims are based on post-hoc (vs prespecified) analyses that aren't reliable under established scientific standards.
Program & Study Design:
-
The intervention is a 10-30 minute web-based writing exercise for incoming 1st year students, designed to address their worries about belonging. The study randomized 27k students at 22 colleges to treatment (the social belonging exercise) vs control (an unrelated writing exercise).
Findings:
-
The study abstract (below) says the intervention increased rates of 1st-year full-time college completion, especially among students in groups that had historically progressed at lower rates, and it was effective only at colleges providing opportunities to belong.
-
But the study doesn't actually report effects for the full sample (as implied by the 1st highlighted clause), and the two subgroup effects - for groups with lower historical progression and colleges providing high opportunities to belong - were post-hoc (i.e., not prespecified).
-
The study pre-specification had hypothesized positive intervention effects for other subgroups: societally disadvantaged students (e.g., Blacks), groups self-reporting "higher threat" (e.g., worry that people judge you by race), and colleges providing low opportunities to belong (see relevant excerpt below).
-
The study does not report findings for the first 2 of these pre-specified subgroups and it reports no significant effects in the 3rd (i.e., colleges providing low opportunities to belong). It also reports effects on only 1 of 2 prespecified outcome measures (1st year completion but not GPA).
Comment:
-
Such selective and post-hoc reporting (whether intentional or not) can easily create a false appearance of program effectiveness - yet is too common in social policy, even in top journals like Science.
