A Statistical Case for Qualified Scientific Optimism

PDF Download

Erik van Zwet, Leiden University Medical Center, Andrew Gelman, Political Science and Statistics, Witold Więcek, University of Chicago

We create an open-source database from empirical research in medicine, economics, psychology, psychotherapy, ecology, evolution, political science, education, and more, and make it available at https://github.com/wwiecek/BEAR/. Each result is represented by a z-value: an estimated effect divided by its standard error.

We then provide a modeling framework that allows us to move away from the usual attempts to classify effects as exactly zero or not, and instead focus on identifying the direction of effects and their replicability. 

After adjusting for publication bias, we find two patterns that hold across all datasets and their various subsets: (1) statistical significance alone does not imply a high probability of successful replication, but (2) it does imply a high probability that the direction of the observed effect was correct.

We conclude that replication rate is a poor measure of the quality of a field of research and argue that an attitude of global scientific pessimism based on low replication rates is misguided. Instead, we favor a more optimistic view of scientific research based on the ability to identify the direction of effects. This optimism is not naive as we recognize the many challenges science continues to face such as questionable research practices, paper mills, fake AI papers, and such.

Keywords: empirical science, Benchmarks of Empirical Accuracy in Research

Working Papers

2026

The Eye of the Beholder and Subjective Escalation Theory

Jason Healey

A Statistical Case for Qualified Scientific Optimism

Erik van Zwet
Andrew Gelman
Witold Więcek

Behavioral Signatures of Cognitive Noise

Ryan Oprea
Michael Woodford

Visualizing War under Information Disorder: Generative AI in the Iran–Israel Escalation

Tamar Mitts

Food Labeling Policies: Aggregate Impacts and Heterogeneity Across Categories

Nano Barahona
Cristobal Otero
Sebastian Otero
Joshua Kim

The political economy of differentiated integration in a fragmenting world

Jeffry A. Frieden
Stefanie Walter

Fixed-T Dynamic Spatial Panel Model with Common Shocks

Jushan Bai
Jesse Chieh Chien

The Econometrics of Matching with Transferable Utility: A Progress Report

Pierre-André Chiappori
Dam Linh Nguyen
Bernard Salanié

Finding and Curing Proxy Discrimination

Brendan O'Flaherty

2025

Compensation, Austerity and Populism: Labor Market Spending and Voting in 16 Western European Countries

Chase Foster
Jeffry Frieden

Efficient scenario analysis in real-time Bayesian election forecasting via sequential meta-posterior sampling

Geonhee Han
Andrew Gelman
Aki Vehtari

Unexpected Voices: How Cultural Agents Reshape Political Communication

Eunji Kim
Erin York

Industrial Diversification and the Rise of the Local Chamber

Calvin Thrall

Conformal Prediction and Human Decision Making

Jessica Hullman
Yifan Wu
Dawei Xie
Ziyang Guo
Andrew Gelman

Educating for Control: Evidence of Rebel Intimate Governance from South Asia

Srinjoy Bose
Keshab Giri
Rumela Sen

Do Provinces that Win the Spanish Christmas Lottery Experience a Surge in Incumbent Popularity? An Out-of-Sample Replication of Bagues and Esteve-Volart (2016)

Carolina Bernal
Donald Green
David Vilalta

2024

Meta-analysis with a single study

Erik Van Zwet
Witold Więcek
Andrew Gelman

Russian roulette: Why you can have a deterministic potential-outcome framework, or an asymmetric utility function, but not both

Andrew Gelman
Jonas Mikhaeil

Attitudes toward Globalization in Ranked Ethnic Societies

Nikhar Gaikwad
Pavithra Suryanarayan

Learning the Effect of Persuasion via Difference-In-Difference

Sung Jae Jin
Sokbae Lee

The Perfect Match: Assortative Matching in Mergers and Acquisitions

Maria Guadalupe
Veronica Rappoport
Bernard Salanié
Catherine Thomas

Hedonic and Matching Models

Alfred Galichon
Bernard Salanié

Single-Threshold Food Labeling Policies

Nano Barahona
Cristóbal Otero
Sebastian Otero Ruiz-Tagle
Joshua Kim

Was Freedom Road a Dead End? Political and socio-economic effects of Reconstruction in the American South

Jeffry Frieden
Richard S. Grossman
Daniel Lowery

Refining the Tools to Stymie Government Regulation: Industry’s Misuse of Science in the 1970s

Gerald Markowitz
David Rosner

The decline and the need of the key force of social intermediation

Nadia Urbinati

A Farewell to Equality: Monetary Policy Implications of Heterogeneous Mortgage Refinancing

Anastasia Burya
Martsella Davitaya

ACCESS TO SOCIAL MEDIA AND SUPPORT FOR ELECTED AUTOCRATS: FIELD EXPERIMENTAL AND OBSERVATIONAL EVIDENCE FROM UGANDA

Jeremy Bowles
John Marshall
Pia Raffler

No results

There are no publications with the provided filters.

    Sort Publications By