Megan Stevenson authored a thoughtful paper with a central claim that "most reforms and interventions in the criminal legal space are shown to have little lasting effect when evaluated with gold standard methods [i.e., RCTs].” I agree with this claim, but believe her conclusion that RCTs therefore can't drive progress in criminal justice (CJ) misses a key point: Medical RCTs have a similarly high failure rate, yet have led to amazing gains in health.

​

• The difference is that the number of medical RCTs each year is *orders of magnitude* greater than the number of CJ RCTs (think: many billions of $ vs a few million). So a low success rate in medical RCTs has still yielded a large body of proven-effective interventions. To elaborate...

• Only 10-15% of medications that enter the phase 1-2-3 testing process are ultimately found to improve health & thus receive FDA approval. And >50% of positive results in initial clinical studies are overturned in subsequent, more definitive RCTs.

• But the huge volume of medical RCTs has led to amazing successes - e.g., vaccines for measles & COVID, effective treatments for HIV/AIDS & many cancers, and statins & antihypertensive drugs to prevent heart attacks & strokes - all proven/replicated in FDA-required trials.

• In large part, it's a numbers game. To quote Thomas Watson, who built IBM: "If you want to succeed, double your failure rate."

• The same phenomenon is starting to play out in social policy, where (despite *many* null RCT findings) there now exists a modest but growing body of programs shown in large RCTs (with replication) to produce important, lasting improvements in people's lives (e.g., Year Up, Per Scholas, ASAP, Bottom Line, Saga tutoring).

• A few CJ reforms may be approaching this evidence level (e.g., Hot Spots Policing, Big Brothers Big Sisters Mentoring). Better targeting of CJ RCT investments could help increase success. But, per Thomas Watson, we also need to double our failure rate.

​