Projects

Ongoing projects

  • CI Fiona Wilder, Kate Williams, James Wilson (RoRI)

    The ‘replication crisis’ has raised concerns over the credibility of published scientific research. Metascience—or the ‘science of science’—has risen in its wake, aiming to influence and improve the way science is practised, funded, evaluated and disseminated. Metascience has grown rapidly in recent years, and it is now having a significant impact on research policy in the UK and elsewhere. This project aims to document Metascience's origins, connections to prior reform efforts and to other science studies disciplines. Grounded in an understanding of its history and purpose, the expected output is a framework for evaluating Metascience impact, ensuring it delivers relevant, high-quality evidence for research policy in Australia.

  • CI Kate Williams, Jenny Lewis

    This project aims to identify the conceptions of value that underpin national research impact policies and to examine the consequences for research activities, outputs, and outcomes. By studying four countries with different national policy approaches to research impact, it is expected that significant new knowledge about the role of research in society will be produced. Expected outcomes include a framework that links markers of value (i.e. what counts as valuable research) to research policy and assessment principles. Expected benefits include policy learnings to improve how research is evaluated in Australia, thereby enhancing the alignment between what is valued by those who fund research, those who produce it, and those who use it.

  • CI Simine Vazire

    Evaluating the Quality of Scientific Research in Psychology. Buttressing public trust in science has never been more important, yet many sciences are experiencing a crisis of confidence. The current system of relying on journal prestige to calibrate our confidence in individual research findings has created corrupt incentives for scientists, and risks undermining public trust in science. Thousands of scientists and institutions around the world have indicated that research evaluation needs an overhaul by signing the Declaration on Research Assessment. One solution is to create a public, transparent, and valid process for producing and sharing expert evaluations of individual papers. This project aims to launch this reform in psychology, and partner with PREreview to help it spread to other fields.

  • CI Simine Vazire

    Details to be added soon.

  • CIs Fiona Fidler, Fallon Mody

    The repliCATS project will work with the Center for Open Science, extending the work we started on the DARPA SCORE Program. The SCORE program demonstrated the potential of using algorithms to evaluate claims on a large scale efficiently. Under the RWJF’s grant, COS, in partnership the repliCATS project and The Pennsylvania State University, will advance the development of these efforts by expanding beyond the core social and behavioral science disciplines covered by SCORE by adding assessment of health research. Read more here.

  • CI Fallon Mody, Ger Post, Meredith McKague, Alexandru Marcoci

    Our team created a custom evaluation rubric and a cloud-based platform that partners from around the world are using to teach undergraduate and postgraduate students critical evaluation skills! repliCATS in the classroom has been used by academics at Ithaca University, Imperial University, Queensland University of Technology as well as here at the University of Melbourne. At unimelb, repliCATS in the classroom has been incorporated into an innovative first-year psychology module for ~1800 students. Preliminary research shows that students self-report that it has improved their ability to evaluate research, and improve skills in critical thinking and teamwork.
    This work has been supported by two Learning and Teaching Innovation grants from the University of Melbourne.

Archived projects

  • For the DARPA Score program, we crowdsourced evaluations of the credibility of 4000 published research articles in eight social science fields: business research, criminology, economics, education, political science, psychology, public administration, and sociology. Through that process, we were exploring the possibility of peer review as a structured deliberation process.

    In 2020, we completed assessing the replicability of 3000 published claims from eight social & behavioural science fields. Our participants groups achieved 73% classification accuracy for replicated claims (or an AUC>0.7). This was phase 1.

    In 2021, we began phase 2 of the SCORE program. In this phase of research our focus on evaluating a broader set of “credibility signals”— from transparency and replicability, to robustness and generalisability. Data collection for this phase is now complete.

    Learn more about our project and find out what’s next:

  • Deatils to be added soon