repliCATS: Collaborative Assessments for Trustworthy Science
We are an interdisciplinary meta-research team who aim to crowdsource expert predictions of replicability of social science research. This research is critical because of the low rates of replicability of scientific research that has been observed in recent years, calling into question the trustworthiness of published research. The repliCATS project is part of a wider program called SCORE, funded by the US Defense Advanced Research Project Agency (DARPA).
The CATS in repliCATS stands for Collaborative Assessment for Trustworthy Science. Our aim is to crowdsource group predictions of the replicability of 3000 published research claims in the social and behavioural sciences.
We use a structured group discussion method, the Investigate, Discuss, Estimage, Aggregate (IDEA) protocal, developed at the University of Melbourne.
The repliCATS project is a meta-research project, that is, it is a research project about scientific research practices and replication rates in science which is an increasingly important field of research. In the past 10 years, science has faced a 'replication crisis'.
Researchers have been unable to replicate the results of several landmark studies in medicine, psychology, economics and other fields, causing many to question the scientific evidence base we use to make decisions.
We can't afford to test and replicate every piece of research before it's published – it will be too costly and time consuming. This is why developing accurate prediction methods could be so transformative, especially for end-users of research including policymakers.
The repliCATS project is part of a wider research program called Systemizing Confidence in Open Research Evidence (SCORE), funded by DARPA, that eventually aims to build automated tools that can rapidly and reliably assign confidence scores to social science research claims. There are eight fields of research in scope for SCORE: Business research, Criminology, Economics, Education, Political Science, Psychology, Public Administration and Sociology.
The repliCATS project is led by Professor Fiona Fidler. We are a group of interdisciplinary researchers from the School of BioSciences, School of Historical and Philosophical Studies, and the Melbourne School of Engineering at the University of Melbourne, with collaboration from the Centre for Environmental Policy at Imperial College London.
The project has been extended in 2020 to include assessing 100 COVID-19 claims from the behavioural and social sciences.