Michael Kirchler is one of several researchers at the University of Innsbruck who are working intensively on the topic of metascience. Together with Jürgen Huber and Felix Holzmeister from the University of Innsbruck and, as usual for metascience, numerous other authors, he published the results of a so-called crowd science analysis in the Journal Nature last year: Together with colleagues from Stanford, Tel Aviv and Stockholm, the researchers from Innsbruck conducted economic experiments in an fMRI scanner. However, they did not evaluate the data themselves, but handed them over to 70 research teams, which independently performed the data analysis. Their findings demonstrate, that the scientific analysis of complex research data can lead to very different results and conclusions.
But what does this suggest for science in general? “First of all, we as researchers must be aware that there are different approaches to complex questions. It is therefore all the more important that we openly present our methods and analytical procedures and take a closer look at why we choose certain approaches. Ultimately, it is about how we generate our knowledge,“ says Kirchler. Metascience therefore serves as kind of quality assurance for science. Kirchler and his colleagues from the Faculty of Business and Management and the Faculty of Economics and Statistics primarily investigate the fields of economics and psychology in their metascience studies. “In order to increase the quality of studies in these scientific disciplines, sample sizes of studies must be increased and the entire research process must be made more transparent,“ says Michael Kirchler.
Transparency and Openness
One way to achieve more transparency and openness is through so-called pre-analysis plans, which have been used more and more frequently in recent years. Researchers are required to define their research design and the methods for data selection and analysis before the start of a study. This is intended to prevent a study design - based on data already collected - from being changed at short notice if the desired result is not achieved along the planned path. On different platforms, such as for example Open Science Framework (OSF), pre-analysis plans are uploaded before data collection. “In medicine or chemistry, this process for quality assurance has long been used. But pre-analysis plans are also becoming more common in other scientific disciplines. Some scientific journals now even insist on this procedure,“ says Michael Kirchler.
For the professor of finance, however, this is not the only step toward greater transparency. He also advocates a more open approach to so-called null results. „If the result of our research is that there are none of the expected correlations and no support for our hypotheses in the data, then we have also learned a lot. These null results must be published in the interest of transparency. This is highly relevant, especially for other researchers who are tackling the same or a similar research questions. In the past, however, these findings have hardly been communicated,“ says Kirchler.
Current Research Projects
Kirchler and his colleagues are currently working on several metascience projects. In the Finance Crowd Analysis Project (#fincap), more than 100 research teams worldwide are independently investigating the same set of hypotheses on the same data - 720 million individual transactions in the EuroStoxx 50 Future. Subsequently, all teams produce short scientific papers that are evaluated by renowned scientists. The project coordinators, including Michael Kirchler, Felix Holzmeister, Jürgen Huber and Michael Razen from the University of Innsbruck, finally look at these individual papers to gain new insights into the heterogeneity of the results and, above all, the process of knowledge generation. “At the beginning of January, all teams received the hypotheses and the data for analysis. There is now time for data analysis until the end of March. We want to make the first results of the meta-study public in the fall of 2021,“ says Kirchler.
Therefore, while #fincap focuses on data analysis, another study will examine the process by which the research design develops. „We are currently planning another project in which dozens of international research teams will only receive the research question for the experimental study design. This will be an economic and moral question that has occupied societies for many centuries. I am very excited to see what designs and results we will get from this open and important question. Unfortunately, I can't reveal any more than that at this stage,“ says Kirchler.