PDF Download
Erik van Zwet, Leiden University Medical Center, Andrew Gelman, Political Science and Statistics, Witold Więcek, University of Chicago
We create an open-source database from empirical research in medicine, economics, psychology, psychotherapy, ecology, evolution, political science, education, and more, and make it available at https://github.com/wwiecek/BEAR/. Each result is represented by a z-value: an estimated effect divided by its standard error.
We then provide a modeling framework that allows us to move away from the usual attempts to classify effects as exactly zero or not, and instead focus on identifying the direction of effects and their replicability.
After adjusting for publication bias, we find two patterns that hold across all datasets and their various subsets: (1) statistical significance alone does not imply a high probability of successful replication, but (2) it does imply a high probability that the direction of the observed effect was correct.
We conclude that replication rate is a poor measure of the quality of a field of research and argue that an attitude of global scientific pessimism based on low replication rates is misguided. Instead, we favor a more optimistic view of scientific research based on the ability to identify the direction of effects. This optimism is not naive as we recognize the many challenges science continues to face such as questionable research practices, paper mills, fake AI papers, and such.
Keywords: empirical science, Benchmarks of Empirical Accuracy in Research
Working Papers
2026
The Eye of the Beholder and Subjective Escalation Theory
A Statistical Case for Qualified Scientific Optimism
Behavioral Signatures of Cognitive Noise
Visualizing War under Information Disorder: Generative AI in the Iran–Israel Escalation
Food Labeling Policies: Aggregate Impacts and Heterogeneity Across Categories
The political economy of differentiated integration in a fragmenting world
Fixed-T Dynamic Spatial Panel Model with Common Shocks
The Econometrics of Matching with Transferable Utility: A Progress Report
Finding and Curing Proxy Discrimination
2025
The costs of coercion
Compensation, Austerity and Populism: Labor Market Spending and Voting in 16 Western European Countries
Efficient scenario analysis in real-time Bayesian election forecasting via sequential meta-posterior sampling
Unexpected Voices: How Cultural Agents Reshape Political Communication
Industrial Diversification and the Rise of the Local Chamber
Conformal Prediction and Human Decision Making
Educating for Control: Evidence of Rebel Intimate Governance from South Asia
The Test-Optional Puzzle
Do Provinces that Win the Spanish Christmas Lottery Experience a Surge in Incumbent Popularity? An Out-of-Sample Replication of Bagues and Esteve-Volart (2016)
2024
Meta-analysis with a single study
Russian roulette: Why you can have a deterministic potential-outcome framework, or an asymmetric utility function, but not both
Attitudes toward Globalization in Ranked Ethnic Societies
Learning the Effect of Persuasion via Difference-In-Difference
The Perfect Match: Assortative Matching in Mergers and Acquisitions
Hedonic and Matching Models
Single-Threshold Food Labeling Policies
Was Freedom Road a Dead End? Political and socio-economic effects of Reconstruction in the American South
Refining the Tools to Stymie Government Regulation: Industry’s Misuse of Science in the 1970s
The decline and the need of the key force of social intermediation
A Farewell to Equality: Monetary Policy Implications of Heterogeneous Mortgage Refinancing
ACCESS TO SOCIAL MEDIA AND SUPPORT FOR ELECTED AUTOCRATS: FIELD EXPERIMENTAL AND OBSERVATIONAL EVIDENCE FROM UGANDA
No results
There are no publications with the provided filters.