DiSCourse Seminar with Felix Holzmeister
07 October 2022, 12:00 (CET), hybrid
Online in Big Blue Button
DiSCourse - The Digital Science Seminar Series on
The Variability of Scientific Results: Analytical and Design Heterogeneity
In science, evidence is generated to test hypotheses in an evidence-generating process (EGP). This talk summarizes the results of two large-scale, crowd-sourced projects, highlighting that EGP variation across researchers adds uncertainty over and above the sampling variation incorporated into standard statistical testing. In the first study – the Finance Crowd Analysis Project (#fincap) –, we show that the variation in scientific results attributable to different analysis pipelines chosen by independent researchers results in uncertainty on par with the magnitude of standard errors. In the second study – #ManyDesigns –, we provide evidence that various conceivable research designs proposed by independent researchers to test the same hypothesis give rise to systematic variation in effect size estimates. Both studies indicate that the informativeness and generalizability of results based on the implementation of a single path of an EGP multiverse are limited.
Felix Holzmeister, University of Innsbruck, Department of Economics and associated researcher at DiSC
Felix Holzmeister is an assistant professor at the Department of Economics at the University of Innsbruck. Apart from behavioral and experimental economics and finance, his research interests revolve around reproducibility of experimental results in the behavioral sciences as well as open-science and meta-science topics. He is an associated scholar of the Digital Science Center, who also teaches courses for the Minor Digital Science.