Evidence-Based Policy Newsletter
We publish an e-newsletter for policymakers, researchers, and others who wish to keep abreast of rigorous evidence on “what works” in social policy. The newsletter, authored by Coalition President Jon Baron, summarizes the latest findings - without spin, just the facts. We focus mainly on randomized controlled trials (RCTs) because, when feasible and well-conducted, they’re considered the strongest method of evaluating program effectiveness.
See all archived newsletter issues below.
Evidence-Based Policy Newsletter #21
August 16, 2024
-
NBER posted an RCT of an unconditional cash transfer program in Texas and Illinois, providing low-income individuals with $1,000 per month ($12,000 per year) for 3 years. Quick take: High-quality RCT finds suggestive (not statistically significant) 4-5% reductions in labor hours and income over 2.5 years. See full summary.
-
JAMA published an RCT of Chelsea Eats - a Chelsea, MA program providing an unconditional cash transfer of up to $400/month for 9 months to low-income individuals. Quick take: High-quality RCT finds a decrease in emergency department visits but (study claims notwithstanding) no credible evidence of improved health over 9 months. See full summary.
-
A recently released RCT of the Denver Basic Income Project - which provided unconditional cash of $12K over 1 year to unhoused people - reported "groundbreaking findings," covered in national and local press. Quick take: The study actually found minimal effects on housing and other outcomes at 10 month mark. See full summary.
-
Four recently published RCTs of US Mayors' programs providing unconditional cash to low-income people - in Los Angeles, Cambridge MA, Stockton CA, and Baltimore - claimed positive results (and got strong press coverage). Quick take: Unfortunately, the results are unreliable due to high sample loss. See full summary.
Evidence-Based Policy Newsletter #20
June 27, 2024
-
Metis Associates & City University of New York (CUNY) reported final RCT results for Accelerate, Complete, Engage (ACE) at John Jay College - a program that provides comprehensive support to help students complete a bachelor's degree. Quick take: High-quality RCT finds remarkable 12 percentage point gain in bachelor's completion at the 5 year mark. See full summary.
-
NBER posted an RCT of Saga Technology, a program that provides 9th graders with high-dosage math tutoring and engagement with educational technology. Quick take: High-quality RCT in NY City and Chicago finds sizable gain (~86% of a grade-level) in math achievement on end-of-year district tests. See full summary.
-
AERJ published a widely-cited systematic review and meta-analysis of RCTs of tutoring in grades Pre K-12. Quick take: Although valuable in certain respects, the study likely overstates (by a lot) the average impact of tutoring. See full summary.
-
New England Journal of Medicine published an RCT of Communities That Heal (CTH), a data-driven program that engaged communities in 4 states to rapidly deploy evidence-based practices to reduce opioid overdose deaths. Quick take: High-quality RCT finds no discernible effect on overdose deaths over 1 year. See full summary.
-
WestEd posted an RCT of ASSISTments, a low-cost online tool that assists 7th graders with math homework and gives teachers real-time reports on student performance. Quick take: High-quality RCT finds sizable gain (~31% of a grade-level) in math scores on the state test at 2 year mark. See full summary.
Evidence-Based Policy Newsletter #19
May 16, 2024
-
PLOS ONE published an RCT of a program providing free reading glasses to adults needing near-vision correction in Bangladesh (where ~1/3 of adults need such correction but only 3% get it). Quick take: High-quality RCT finds sizable 33% gain in workforce earnings at 8-month mark. See full summary.
-
NBER posted an RCT of Khoaching with Khan Academy (KWiK), a program integrating computer-assisted learning into classroom math teaching. Quick take: Despite study claims of positive results, it found no discernible impact on grade 3-8 math scores on the state test after one year. See full summary.
-
Psychiatric Services published an RCT of a "Pay for Performance" approach to delivering care to adults with mental illness, aimed at reducing cost. Quick take: High-quality RCT finds no cost savings over 3 years (and a pre-post study, vs RCT, would've incorrectly found big savings). See full summary.
-
Pediatrics published an RCT of TIPP, a program to help pediatricians prevent injuries in children ages 0-2. Quick take: The main study finding - reduced child injuries - isn't reliable because this was a cluster RCT (4 sites were randomized) and the analysis didn't adjust for clustering. See full summary.
-
Why a control group is needed to determine if a K-12 education program is effective, in 2 simple charts. The example I'll use is Xtreme Reading - a program for struggling 9th graders that provides year-long reading instruction in a small classroom setting (in lieu of an elective). See full summary.
Evidence-Based Policy Newsletter #18
April 2, 2024
-
JAMA Health Forum published an RCT of email feedback to surgeons to reduce over-prescribing of opioids after surgery (thought to be a significant driver of opioid addiction). Quick take: High-quality RCT finds encouraging 5-6% point reduction in over-prescribing over 1 year. See full summary.
-
Social Science & Medicine published an RCT of an Indianapolis program that partners police officers with mental health clinicians in responding to behavioral health emergencies. Quick take: The study findings (no significant impacts) are unreliable due to study limitations. See full summary.
-
Development & Psychopathology published an RCT of home visiting plus doula visits for low-income women starting in pregnancy. Quick take: Despite study claims of positive results, it found no discernible impact on child outcomes and little impact on parent practices, at age 30 months. See full summary.
-
Journal of Adolescent Health published an RCT of Making Proud Choices! (MPC) - "one of the most
popular evidence-based teen pregnancy prevention curricula being implemented nationwide." Quick take: Despite study claims of positive results, it actually found little to no impact on youth sexual behavior and health at 6 mo follow-up. See full summary. -
American Journal of Psychiatry published long-term RCT results of the 1990s Fast Track mental health program for children. Quick take: Not reliable evidence; the study is inaccurately described and analyzed as if children (vs schools) were randomized, likely leading to false positive results. See full summary.
Evidence-Based Policy Newsletter #17
February 20, 2024
-
BMC Public Health published an RCT of iHEAL, a nurse visitation program for women who've experienced intimate partner violence (IPV). Quick take: Mostly well-conducted RCT finds sizable gain in Quality of Life and promising (but not definitive) decrease in PTSD 18 months after study entry. See full summary.
-
National Student Support Accelerator posted RCT results for Chapter One - a program that provides two years of tutoring to struggling readers in grades K-1. Quick take: High-quality RCT finds small impacts (not statistically significant) on reading at the 2 year mark - i.e., end of 1st grade. See full summary.
-
Megan Stevenson authored a thoughtful paper with a central claim that "most reforms and interventions in the criminal legal space are shown to have little lasting effect when evaluated with gold standard methods [i.e., RCTs].” I agree with this claim, but believe her conclusion that RCTs therefore can't drive progress in criminal justice misses a key point: Medical RCTs have a similarly high failure rate, yet have led to amazing gains in health. See full summary.
-
CAPR published an RCT of Multiple Measures Assessments (MMA), a data analytic tool increasingly used by community colleges to place incoming students into college-level vs developmental (i.e., remedial) courses. Quick take: High-quality RCT finds little impact on student success at 4.5 year mark. See full summary.
-
HHS/OPRE published new findings of a major RCT of federally-funded home visiting programs for at-risk families with young children. Quick take: The average impact across the 4 programs on child and mother health and other outcomes is negligible at child age 2.5-3.5 years. However, the study hasn't yet reported impacts of the 4 programs individually, where positive findings are quite possible. See full summary.
-
Annenberg Institute posted a quasi-experimental study of a California program to train teachers in the science of reading. Quick take: Despite study claims of positive impacts (covered by New York Times and others), it found no discernible effect on 3rd grade reading achievement in its primary analysis. See full summary.
Year-End Newsletter (#16): Top Positive RCT Findings
December 20, 2023
This year-end issue of the Newsletter compiles the top U.S. RCT findings of sizable, sustained effects on important life outcomes that I’ve summarized over the past 15 months:
-
Nevada RESEA - a low-cost program aimed at reemploying Unemployment Insurance claimants (earnings gains of $13,000/person over 5 years and net savings to government). See full summary.
-
Accelerate, Complete, Engage (ACE) at John Jay College - providing comprehensive support to help students complete a bachelor’s degree (12%-point gain in bachelor’s completion at 4-year follow-up). See full summary.
-
ASAP Ohio - providing academic, personal, and financial support to low-income community college students (15%-point gain in college graduation and 11% gain in earnings at the 6-year follow-up). See full summary.
-
Saga Education's high-dosage math tutoring of low-income 9th and 10th graders in Chicago (large impact on math achievement sustained 1-2 years after tutoring ends). See full summary.
-
Year Up - a job training program for low-income young adults that focuses on fast-growing economic sectors (long-term earnings gains of 30%, or $8000 per year). See full summary.
-
Per Scholas - an employment/training program low-income adults, preparing them for well-paying jobs in Information Technology (earnings gain of 16%, or $5200 per year, at the 5–7 year follow-up). See full summary.
-
Bottom Line - providing one-on-one advising to help low-income students get into and graduate from college (8%-point gain in bachelor's degree completion at the 6 year follow-up). See full summary.
-
Project QUEST - occupational training of low-income adults for well-paying healthcare jobs (long-term earnings gains of 15–20%, or about $5,000 per year). See full summary.
-
Denver Supportive Housing initiative - providing housing and supportive services to homeless people with over 7 prior arrests (roughly 30–40% reductions in jail and shelter use, emergency department visits, and arrests over 2–3 years). See full summary.
-
Big Brothers Big Sisters Community-Based Mentoring for at-risk youth (1/3 to 1/2 reduction in arrest rate and substance use over 18 months). See full summary.
-
Individual Placement & Support - a job counseling, search, and support program for adults with disabilities (earnings gain of 26%, or ~$2000 annually, over 2 years). See full summary.
Evidence-Based Policy Newsletter #15
November 28, 2023
-
AIR posted RCT results for Literacy First, providing high-dosage tutoring to struggling readers in 2nd grade. Quick take: Well-conducted RCT finds positive impact on end-of-year reading scores, but unfortunately no long-term (grade 5) impact on reading or science achievement. (See full summary.)
-
MDRC published an RCT of NY City P-TECH 9-14 schools, aimed at preparing students for college and careers within six years. Quick take: Small or no impacts on high school graduation or college attainment at 4-7 year follow-up (study claims notwithstanding) but future earnings impacts are very possible. (See full summary.)
-
Impaq just posted long-term RCT results for Nevada's RESEA program, aimed at reemploying Unemployment Insurance (UI) claimants. Quick take: High-quality RCT finds that this low-cost program ($290 per person) yields earnings gains of $13,000 per person over 5 years and net savings to government. (See full summary.)
-
JREE published a long-term evaluation of Reading Recovery (RR) - a tutoring program for 1st graders - that found adverse effects on grade 3-4 reading achievement, and received wide coverage (e.g., recently in the New York Times). Quick take: This is not a reliable finding, due to sample loss (attrition) of over 70% and other problems. (See full summary.)
-
JAMA published 5-year RCT results for Million Hearts - a national HHS pay-for-performance model to prevent heart attacks and strokes in Medicare patients. Quick take: No discernible impact on the study’s primary health or cost outcomes (health findings are misreported as positive in JAMA). (See full summary.)
Evidence-Based Policy Newsletter #14
October 9, 2023
-
An RCT of Fostering Healthy Futures mentoring program for teens in the child welfare system recently reported final results. Quick take: High-quality RCT finds a sizable 14% point decrease in juvenile justice involvement over ~2 years - an encouraging result though not definitive due to lack of statistical significance. (See full summary.)
-
The Baby's First Years RCT, studying the impact of unconditional cash transfers to low-income US mothers with newborns, reported new results on 2 primary outcomes: child overall health and sleep over 3 years. Quick take: No impact on either outcome (but effects could emerge in future). (See full summary.)
-
The US Department of Education's What Works Clearinghouse (WWC) published an evidence review of World of Words preschool program to improve children's vocabulary/language. Quick take: WWC's finding of "strong evidence" is far overstated; the results are preliminary and not yet reliable. (See full summary.)
-
HHS/ACF reported RCT results for HPOG 2.0, a major federal demonstration program that trains low-income adults for skilled healthcare jobs. Quick take: High-quality RCT finds no program impact on participant earnings over 3-4 years, despite increasing completion of training (i.e., % earning any new credential) by 14% points. (See full summary.)
-
Mathematica reported long-term RCT results for KIPP charter middle schools. Quick take: High-quality RCT finds a suggestive 4% point gain in college enrollment and no impact on college persistence. (Its reported large impact of attending both KIPP middle and high school is quasi-experimental and not yet reliable pending confirmation in future RCTs.) (See full summary.)
-
NBER posted an RCT of Sit-D, a police training program teaching the value of developing multiple perspectives on a situation. Quick take: High-quality RCT finds suggestive evidence of a 12% reduction in police use of force over 1 year (findings are moderately encouraging but not as strong as presented in study). (See full summary.)
-
Science Magazine published an RCT at 22 US colleges, reporting that a brief "social belonging" writing exercise increased 1st-year college completion rates. Quick take: These claims are based on post-hoc (vs prespecified) analyses that aren't reliable under established scientific standards. (See full summary.)
Evidence-Based Policy Newsletter #13
September 6, 2023
-
BMJ published an RCT in Switzerland of a smartphone app to reduce drinking among university students with unhealthy alcohol use. Quick take: High-quality RCT finds modest but meaningful 7% decrease in drinks/week at 1-year mark for a low-cost, scalable intervention. (See full summary.)
-
PNAS published an RCT entitled "Unconditional Cash Transfers Reduce Homelessness." Quick take: The RCT found no discernible impact on any primary or secondary study outcome. The claimed effect on homelessness (an exploratory outcome) is unreliable due to sample loss >50%. (See full summary.)
-
Pediatrics published an RCT of a program encouraging school children to drink more water (e.g., in lieu of sweetened drinks), to prevent overweight. Quick take: The study's claims of effectiveness notwithstanding, it found no impact on the prespecified primary outcomes: Body Mass Index (BMI) z-score and BMI percentile. (See full summary.)
-
HUD published a large RCT of Integrated Wellness in Supportive Housing (IWISH), providing a Wellness Director & Nurse for older adults at HUD-assisted multifamily properties. Quick take: High-quality RCT finds no impact on any targeted outcome (e.g., transitions to long-term care, unplanned hospitalization) over 3 years. (See full summary.)
-
Science Magazine published an RCT of the Modeling Practices in Calculus (MPC) approach to teaching college calculus. Quick take: Sizable positive impact on end-of-semester test scores, but findings aren't definitive due to a key study limitation (faculty self-selected to teach MPC). (See full summary.)
Evidence-Based Policy Newsletter #12
August 8, 2023
-
EEPA published long-term RCT findings for Future Forward — a program providing two years of literacy tutoring starting in K-2 in low-income Milwaukee schools. Quick take: High-quality RCT finds sustained, modest gain (1/3 grade level) on district reading test through 7 year follow-up. (See full summary.)
-
Institute for Policy Research posted long-term RCT results for the Graduation program in Ethiopia, providing very poor households a productive asset (e.g. livestock) and training in its use. Quick take: High-quality RCT finds a sizable 18% gain in household consumption at the 3-year mark, fading somewhat to 9% at 7 years. (See full summary.)
-
Mobility published an RCT of English for Advancement (EfA), providing adult English-language learners with language instruction, career coaching, and job placement assistance. Quick take: High-quality RCT finds smallish (7%) earnings gains at 2-year mark, possibly dampened by pandemic. (See full summary.)
-
IZA published an RCT of ReHire Colorado, a state program providing transitional, subsidized jobs to low-income unemployed or underemployed individuals. Quick take: High-quality RCT finds little impact on participant earnings or employment once the transitional jobs end. (See full summary.)
-
US Dept of Labor (DOL) published RCT results for DOL's Ready to Work (RTW) Partnership Grants, providing employment services to long-term unemployed or underemployed workers. Quick take: High-quality RCTs of 4 RTW projects find no discernible positive earnings effects over 3-4 years. (See full summary.)
Evidence-Based Policy Newsletter #11
July 17, 2023
-
Colleagues and I have monitored the RCT literature for many years for policy-important findings, but this study launched in the 80s only recently crossed our radar: A large smoking-cessation RCT found sizable impacts on sustained quit rates & mortality over 14.5 years. (See full summary.)
-
Why a Control Group is Needed to Determine Whether a Social Program is Effective, in 4 simple charts. The example I’ll use is HHS’s Comprehensive Child Development Program (CCDP), which provided intensive case management services to low-income families with young children. (See full summary.)
-
21 of the RCTs I’ve summarized since last summer found disappointing effects (e.g., no discernible positive impacts on the primary outcomes). 7 of these RCTs (i.e., one-third) nevertheless portrayed the results as strongly positive in the study abstract. (See full summary.)
-
US Department of Labor released RCT results for 2 DOL-funded "Pay for Success" projects - Roca (in MA) for justice-involved young men, & Center for Employment Opportunities (in NY) for adult parolees. Quick take: No impacts (yet) on recidivism or employment; the RCTs are ongoing. (See full summary.)
Evidence-Based Policy Newsletter #10
June 12, 2023
-
Metis Associates and CUNY released RCT results for Accelerate, Complete, Engage (ACE) at John Jay College, providing comprehensive support to help students complete a bachelor’s degree. Quick take: High-quality RCT finds large 12%-point gain in bachelor’s completion at 4-year mark. (See full summary.)
-
IES published a large RCT of providing college-intending, disadvantaged adults with personalized texts on the college enrollment & financial aid process. Quick take: High-quality RCT finds no impact on rate of college enrollment or financial aid form completion. (See full summary.)
-
HUD published long-term (20-year) RCT results for Jobs Plus, providing public housing residents with employment services, along with financial incentives & support for work. My quick take: No impact on earnings and employment for full 6-site sample, and the reported positive impacts at 3 sites aren’t reliable. (See full summary.)
-
Science Advances published an RCT of Working on Womanhood — a group counseling program for high school girls from disadvantaged backgrounds. Quick take: High-quality RCT finds no discernible effect on rates of clinical depression, anxiety, or PTSD (primary outcomes) at 1-year mark, but suggestive effects on a broader PTSD measure. (See full summary.)
-
Annenberg Institute posted RCT findings for Core Knowledge Charter Schools. Quick take: Notwithstanding study’s claim of large effects, no discernible effect on English Language Arts (ELA) achievement in 6th grade (primary outcome) — although suggestive effects in grades 4 and 5. (See full summary.)
Evidence-Based Policy Newsletter #9
April 24, 2023
-
The Lancet published an RCT of a low-cost program led by non-physician community healthcare providers to reduce hypertension in rural China. Quick take: Blockbuster results - a 1/3 reduction in cardiovascular events including death over 3 years, with major global health implications. (See full summary.)
-
JPAM published an RCT examining the effect of NY City's Summer Youth Employment Program (SYEP) on youth crime. Quick take: Large, high-quality RCT finds negligible impact on youth crime over a 5-year follow-up (authors' claims notwithstanding). (See full summary.)
-
Prevention Science published an RCT of High School FLASH, a sex ed program, reporting that it reduces homophobic and transphobic beliefs. My quick take: Not a reliable result (only suggestive), as this was a post-hoc analysis that can easily yield chance findings. (See full summary.)
-
MDRC reported RCT results for ASAP Ohio, which provides academic, personal, and financial support to low-income community college students. Quick take: High-quality RCT with blockbuster results - 15% point gain in college graduation and 11% gain in earnings at the 6 year mark. (See full summary.)
-
American Economic Review published RCT findings for Saga Education's high-dosage math tutoring of low-income 9th and 10th graders in Chicago. Quick take: High-quality RCTs find sizable effect on math achievement that persists 1-2 years after tutoring ends. (See full summary.)
Evidence-Based Policy Newsletter #8
March 20, 2023
-
A new Cochrane review of RCTs casts doubt on the effectiveness of masks in community settings for preventing COVID infection. But adherence to proper masking is often low in such settings. We don’t yet know if people who diligently mask are protected. Here’s how to find out. (See full summary.)
-
LEO reported RCT results for Padua, an intensive case management program for low-income adults in Texas, aimed at moving them permanently out of poverty. Quick take: High-quality RCT finds suggestive, but at best modest, effects on earnings, employment, etc over 2–5 year follow-up. (See full summary.)
-
IZA published an RCT evaluating generous unconditional cash transfers ($8K per year) to low-income households in Spain, to reduce poverty. Quick take: High-quality RCT finds sizable (9% point) adverse effect on employment at roughly the 2 year mark. (See full summary.)
-
MDRC published an RCT of Grameen America microfinance program in Union City, NJ, providing low-income women with small loans and training to start or expand a business. Quick take: High-quality RCT finds modestly encouraging gains in economic well-being at 3 year mark. (See full summary.)
Evidence-Based Policy Newsletter #7
February 13, 2023
-
Chris Blattman et. al. report long-term RCT results for a program of cognitive-behavior therapy (CBT) plus cash transfers for criminally-involved men in Liberia. Quick take: High-quality RCT finds sizable reduction in anti-social behavior (e.g., drug-selling, robbery) at 10-year follow-up. (See full summary.)
-
The Vision for Baltimore RCT, evaluating a citywide program of free vision screening and eyeglasses in elementary and middle school, recently published results. Quick take: High-quality RCT finds no significant impact on English language arts (ELA) or math scores on the state test over 2 years. (See full summary.)
-
Mathematica published RCT findings for SSI PROMISE, a $230M federal demonstration program to improve education and workforce outcomes of youth with disabilities. Quick take: High-quality RCTs find none of 6 programs in the demonstration improved education or earnings over 5 years. (See full summary.)
-
NBER posted RCT results for READI Chicago, providing transitional jobs, cognitive behavioral therapy and other services to men at high risk of violence perpetration or victimization. Quick take: High-quality RCT finds no significant impact on overall serious violent crime over 20 months. (See full summary.)
-
Family Relations published an RCT of Love Notes, a pregnancy prevention program for low-income youth ages 14–19. Quick take: RCT finds nearly 50% reduction in pregnancy rate over 12 months that is encouraging but not yet definitive (due to results’ sensitivity to study methods). (See full summary.)
Evidence-Based Policy Newsletter #6
January 10, 2023
-
JAMA Internal Medicine published an RCT examining whether remediation of abandoned housing in Philadelphia reduces surrounding crime. Quick take: Mostly well-conducted RCT finds reductions in gun-related crime that are moderately encouraging but not yet reliable — (might be due to chance). (See full summary.)
-
EEPA published long-term RCT findings for Washington DC’s school voucher program — providing private school scholarships to low-income students. Quick take: High-quality RCT finds no effect on college enrollment rate within 2 years of scheduled high school graduation. (See full summary.)
-
Having recently shared some of the most promising recent RCT findings, I now share one of the most disconcerting: The long-term results of the Tennessee voluntary Pre-K study. Quick take: High-quality RCT finds adverse effects on student achievement and behavior in 6th-grade follow-up. (See full summary.)
-
MDRC recently published RCT findings for Detroit Promise Path (DPP), a program providing coaching and other supports to Detroit-area community college students. Quick take: High-quality RCT finds no discernible impact on college degree or certificate receipt at 4-year mark.(See full summary.)
-
A recent RCT evaluated CenteringPregnancy, a widely-used group prenatal care program that aims to improve racial equity in birth outcomes. Quick take: High-quality RCT finds no impact on rate of preterm birth or low birthweight for whole sample, or Black or Hispanic subgroups. (See full summary.)
Evidence-Based Policy Newsletter #5
November 29, 2022
-
My tweets in coming days [items 2–5 below] will summarize some of the most promising RCT findings — in job training, education, and housing — over the past 18 months. I’ve already shared one: The blockbuster findings for Year Up job training for low-income young adults.
-
In March, MDRC reported results of an RCT of Per Scholas — an employment/training program low-income adults, preparing them for well-paying jobs in Information Technology. Quick take: High-quality RCT finds sizable earnings gain (16%, or $5200 per year) at the 5–7 year follow-up. (See full summary.)
-
Another especially promising RCT finding of the past 18 months: Castleman and Barr’s study of Bottom Line (1-on-1 advising to help low-income students get into and graduate from college). Quick take: High-quality RCT finds large (8% point) increase in 4-year college degree completion. (See full summary.)
-
Another especially promising RCT finding of the past 18 months: Mobility's study of Project QUEST occupational training of low-income adults for well-paying healthcare jobs. Quick take: High-quality RCT finds sizable long-term earnings gains (15–20%, or about $5,000/year). (See full summary.)
-
Another especially promising RCT of the past 18 months: Urban Institute's study of the Denver Supportive Housing SIB initiative, providing housing and supportive services to homeless people with over 7 prior arrests. Quick take: High-quality RCT finds roughly 30–40% reductions in jail and shelter use, emergency department visits, and arrests over 2–3 years. (See full summary.)
-
HHS/OPRE published an excellent summary of results of 10 RCTs of job training programs for low-income adults, funded under HHS’s “PACE” and “HPOG 1.0” initiatives. Quick take: High-quality RCTs find no earnings impacts for 9 programs and large impacts for 1 program. (See full summary.)
Other Items:
-
I’m pleased to share my recent op-ed in The Hill — “The Good and Bad News About Declining U.S. Poverty Rates.” It makes that case that, while we’re alleviating economic hardship, we’re not breaking the poverty cycle. (Link to op-ed).
Evidence-Based Policy Newsletter #4
November 3, 2022
-
Nature Human Behavior just published an RCT in Germany of a program that teaches self-regulation skills in 1st grade (“Mental Contrasting with Implementation Intentions”). Quick take: The RCT had a key flaw, but found large effects on reading achievement — so findings are still somewhat promising. (See full summary.)
-
Earlier this year, HHS/OPRE reported long-term RCT results for the federal Health Profession Opportunity Grants program (HPOG 1.0), which trains low-income adults for healthcare jobs. Quick take: Large, high-quality RCT finds no earnings impacts over 6 years. (See full summary.)
-
BMJ published a large RCT in the UK evaluating the effects of offering pregnant smokers shopping vouchers up to $440 to engage with stop smoking services and stop smoking. Quick take: High-quality RCT finds the offer roughly doubles smoking abstinence in late pregnancy. (See full summary.)
-
The HHS/OPRE and Abt Associates published an RCT of Valley Initiative for Development and Advancement (VIDA)’s college-based occupational training for low-income students. Quick take: High-quality RCT finds sizable effect on degree/credential receipt, but it leads to no earnings gain over 7 years. (See full summary.)
-
Abt Associates published an RCT of Professional Training Corps (PTC) — a lower-cost version of Year Up’s proven-effective job training program for low-income adults. Quick take: High-quality RCT finds PTC has no effect on earnings … yet (the PTC program is still being developed/refined). (See full summary.)
Evidence-Based Policy Newsletter #3
October 14, 2022
-
Researchers David Dubois and Carla Herrera reported interim results of a national RCT of Big Brothers Big Sisters (BBBS) Community-Based Mentoring Program. Quick take: High-quality RCT finds sizable reductions in youth arrests and substance use over 18-month follow-up. (See full summary.)
-
The Lancet published the first RCT of a commercial group weight-loss program — Weight Watchers— with long-term (5-year) follow-up. Quick take: Pretty good RCT found minimal weight-loss impacts (2–4 lbs) at the 5-year mark that weren’t statistically significant. Sizable impacts at the 1-year mark had mostly faded. (See full summary.)
-
A quick detour to a medical RCT, which affects me having newly turned 60. NEJM just published the 1st large RCT of colonoscopy screening to prevent colorectal cancer. Quick take: High-quality RCT shows screening benefit, but much smaller than suggested by prior non-RCTs. (See full summary.)
-
PNAS published an RCT of Procedural Justice training for police, to foster fair/respectful treatment of citizens. Quick take: High-quality RCT found no effect on community perceptions of police legitimacy (primary outcome) but suggestive effects on some other outcomes. (See full summary.)
Evidence-Based Policy Newsletter #2
September 27, 2022
-
EdWorkingPapers posted an RCT of three STEM summer programs for underrepresented high schoolers. Quick take: The findings touted in the study abstract - including large impacts on college STEM degree receipt - are quasi-experimental, not RCT. The RCT impacts are mostly small & not statistically significant. (See full summary.)
-
HHS/OPRE reported an RCT of Individual Placement and Support — a job counseling, search and support program for adults with disabilities. Quick take: High-quality RCT found an impressive 26% gain in two-year earnings, but no significant effect on time employed. (See full summary.)
-
NBER posted an RCT of Bridge private schools in Kenya that’s gotten attention for the big impacts on student achievement. Quick take: Promising findings but not definitive due to key study limitations (especially in methods of testing achievement). (See full summary.)
-
HHS/OPRE published an RCT of Paycheck Plus — Atlanta, evaluating a significant expansion of the Earned Income Tax Credit (EITC) for low-income childless workers. Quick take: High-quality RCT finds no meaningful effects on workers’ economic well-being over three years. (See full summary.)
Other Items:
-
I’m pleased to share my recent Washington Post op-ed on the need for evidence-based policy reform.
Evidence-Based Policy Newsletter #1
September 9, 2022
-
IES recently published results of an RCT evaluating a program that trained schools in “Multi-Tiered Systems of Support for Behavior” (MTSS-B) — an approach used by over 25,000 US schools. Quick take: a mostly well-conducted RCT, with mostly disappointing findings. (See full summary.)
-
NBER recently posted a report on the Baby’s First Years RCT, studying initial effects of providing an unconditional cash transfer to low-income US mothers with newborns. Quick take: High-quality RCT, interesting results but too early to tell if there’s any meaningful child or family benefit. (See full summary.)
-
Another ongoing US RCT of unconditional cash transfers, in Stockton CA reported 12-month findings last year. Quick take: The report doesn’t provide key info needed to gauge study validity, and what info it does provide suggests major problems - e.g. severe differential attrition. (See full summary.)
-
In May, HHS/OPRE reported new findings of a large RCT of Year Up — a job training program for low-income young adults that focuses on fast-growing economic sectors. Quick take: High-quality RCT with blockbuster results: long-term earnings gains of 30%, or $8000 per year. (See full summary.)
-
JAMA recently published initial results of a large South Carolina RCT of Nurse-Family Partnership — a nurse home visiting program for first-time mothers starting in pregnancy. Quick take: High-quality RCT finds no impact on adverse birth outcomes - but study is ongoing with additional results to come. (See full summary.)