Prevention Science published an RCT of High School FLASH, a sex ed program, reporting that it reduces homophobic and transphobic beliefs. Quick take: Not a credible result (only suggestive), as this was a post-hoc analysis that can easily yield chance findings.

Program:

• Per the study: “High School FLASH is a comprehensive sexual health education curriculum … for classroom settings, with the specific behavior change goals of preventing unintended pregnancy, preventing STDs, and preventing the perpetration of sexual violence.”

Study Design & Prior Findings:

• The study randomly assigned 20 schools in the U.S. South and Midwest, with a student sample of 1,597 9th and 10th graders, to (i) a treatment group that received FLASH vs (ii) a control group that received a knowledge-based sexual health curriculum.

• A prior study report found no discernable impact of FLASH vs control on the study’s 2 pre-specified primary outcomes: the full-sample rates of vaginal sex, and vaginal sex without birth control, measured at 3 and 12 months after program completion.

New Findings:

• The newly-published article reports statistically significant positive impacts on students' homophobic and transphobic beliefs at 3 and 12 months post-program.

• Importantly, however, this was not a pre-specified primary or secondary outcome of the study, per the study registration.

  In other words, it’s a post-hoc outcome. Analyses of such outcomes are only suggestive under established scientific standards (FDA, IES), as they can easily yield chance (i.e., false) results.

• The reason is that each statistical significance test has about a 1/20 chance of producing a false result, for a program whose true effect is zero. So a study like this that examines numerous outcomes and subgroups can easily turn up false findings.

Comment:

• Claiming effectiveness based on post-hoc findings is a bit like placing a bet on a horse after watching the race. The findings are not without value but I think only suggestive.