Improving the credibility of scientific research.
The ‘replication crisis’ has dominated methodological concerns in the social, biomedical and psychological sciences for almost two decades. In its wake, a host of associated crises have also been declared: the generalizability crisis, the measurement crisis, and the theory crisis. All raise concerns over the capacity of our current peer review and research evaluation systems to maintain a trustworthy and useful evidence base. ‘Metascience’ encompasses the study of interventions and responses to problems with the resourcing, practice and dissemination of research.
We are an interdisciplinary research group, with members from history & philosophy of science, psychology, ecology, biomedical sciences, political science and public policy. We happily embrace the methodological pluralism this brings. Read more about what we do on our projects page.
The beautiful painting of papayas in the cover image was created by Martha Whitaker.
If you would like to get in touch, e-mail us at:
- Fiona Fidler
- fidlerfm@unimelb.edu.au
- Simine Vazire
- simine.vazire@unimelb.edu.au
Research Quality
What are the dimensions we should consider when evaluating research quality? How do researchers evaluate the quality of each other's work, and how do they come to those evaluations? Can we improve the evaluation of research quality by introducing rubrics, group discussions, or other structured processes?
Journal policies and practices
What are common journal policies and informal practices around peer review and publication? How does peer review work and how can we improve it? How is the published literature changing over time?
Statistical inference
What are common errors in statistical inference, and how common are they? How can we improve statistical training and practice? What are the best tools for detecting and correcting errors in statistical inference?
Self-correction
What are the self-correcting mechanisms in science? How common and effective is post-publication critique? Do replications help to correct the scientific record? Who should do the work of verifying and correcting published studies, and how should it be incentivised?
Intellectual humility
Do research claims match the evidence presented? How common are hype and spin? How can we incentivise researchers to make calibrated claims?
Transparency and Reproducibility
Does increased transparency lead to better quality research, or better detection of errors and inaccuracies in research? What kinds of transparency are most important, and how common are they? How can important forms of transparency be incentivised? To what extent and under what conditions does transparency improve reproducibility.
Metascience and Policy
This project is about understanding Metascience: its dramatic rise in influence and its potential to support research policy. To those tasked with making decisions about research systems, Metascience provides evidence about how to improve research policy, funding allocation, or the performance of research organisations towards a greater return on investment. To researchers in the field, Metascience is a response to a crisis of confidence in the credibility of scientific evidence, ushered in by core underlying problems related to replication, measurement, and theory. To them, its value lies in creating trustworthy evidence by addressing the root causes of problems in scientific practice. Metascience also has critics: in their view, its value is diminished by its perceived failure to engage with important scholarship in established fields. Each of these characterisations point to Metascience’s enormous potential for impact, yet its scope and purpose is highly unsettled.
Peer Review and Research Evaluation
The repliCATS project aims to crowdsource predictions about the reliability of 3000 published research claims in: Business research, Criminology, Economics, Education, Political Science, Psychology, Public Administration, and Sociology. The “CATS” in repliCATS stands for Collaborative Assessment for Trustworthy Science.
The repliCATS project is part of a research program called SCORE, funded by DARPA, that eventually aims to build automated tools that can rapidly and reliably assign confidence scores to social science research claims.
Ongoing projects
-
CI Fiona Wilder, Kate Williams, James Wilson (RoRI)
The ‘replication crisis’ has raised concerns over the credibility of published scientific research. Metascience—or the ‘science of science’—has risen in its wake, aiming to influence and improve the way science is practised, funded, evaluated and disseminated. Metascience has grown rapidly in recent years, and it is now having a significant impact on research policy in the UK and elsewhere. This project aims to document Metascience's origins, connections to prior reform efforts and to other science studies disciplines. Grounded in an understanding of its history and purpose, the expected output is a framework for evaluating Metascience impact, ensuring it delivers relevant, high-quality evidence for research policy in Australia.
-
CI Kate Williams, Jenny Lewis
This project aims to identify the conceptions of value that underpin national research impact policies and to examine the consequences for research activities, outputs, and outcomes. By studying four countries with different national policy approaches to research impact, it is expected that significant new knowledge about the role of research in society will be produced. Expected outcomes include a framework that links markers of value (i.e. what counts as valuable research) to research policy and assessment principles. Expected benefits include policy learnings to improve how research is evaluated in Australia, thereby enhancing the alignment between what is valued by those who fund research, those who produce it, and those who use it.
-
CI Simine Vazire
Evaluating the Quality of Scientific Research in Psychology. Buttressing public trust in science has never been more important, yet many sciences are experiencing a crisis of confidence. The current system of relying on journal prestige to calibrate our confidence in individual research findings has created corrupt incentives for scientists, and risks undermining public trust in science. Thousands of scientists and institutions around the world have indicated that research evaluation needs an overhaul by signing the Declaration on Research Assessment. One solution is to create a public, transparent, and valid process for producing and sharing expert evaluations of individual papers. This project aims to launch this reform in psychology, and partner with PREreview to help it spread to other fields.
-
CI Simine Vazire
Details to be added soon.
-
CIs Fiona Fidler, Fallon Mody
The repliCATS project will work with the Center for Open Science, extending the work we started on the DARPA SCORE Program. The SCORE program demonstrated the potential of using algorithms to evaluate claims on a large scale efficiently. Under the RWJF’s grant, COS, in partnership the repliCATS project and The Pennsylvania State University, will advance the development of these efforts by expanding beyond the core social and behavioral science disciplines covered by SCORE by adding assessment of health research. Read more here.
-
CI Fallon Mody, Ger Post, Meredith McKague, Alexandru Marcoci
Our team created a custom evaluation rubric and a cloud-based platform that partners from around the world are using to teach undergraduate and postgraduate students critical evaluation skills! repliCATS in the classroom has been used by academics at Ithaca University, Imperial University, Queensland University of Technology as well as here at the University of Melbourne. At unimelb, repliCATS in the classroom has been incorporated into an innovative first-year psychology module for ~1800 students. Preliminary research shows that students self-report that it has improved their ability to evaluate research, and improve skills in critical thinking and teamwork.This work has been supported by two Learning and Teaching Innovation grants from the University of Melbourne.
Archived projects
-
For the DARPA Score program, we crowdsourced evaluations of the credibility of 4000 published research articles in eight social science fields: business research, criminology, economics, education, political science, psychology, public administration, and sociology. Through that process, we were exploring the possibility of peer review as a structured deliberation process.
In 2020, we completed assessing the replicability of 3000 published claims from eight social & behavioural science fields. Our participants groups achieved 73% classification accuracy for replicated claims (or an AUC>0.7). This was phase 1.
In 2021, we began phase 2 of the SCORE program. In this phase of research our focus on evaluating a broader set of “credibility signals”— from transparency and replicability, to robustness and generalisability. Data collection for this phase is now complete.
Learn more about our project and find out what’s next:
-
Deatils to be added soon
Here are a couple of key papers to introduce you to metascience:
Publications from our people
- A Consensus Statement on self-knowledge conceptualization, measurement, outcomes and changeability, I Thielmann, MD Back, W Bleidorn, H Bukowski, EN Carlson, M Dufner, G Hofer, W Hofmann, CJ Hopwood, LJ Human, CH Jordan, JI Krueger, J Langdon, R Rau, LD Smillie, N Strohminger, J Sun, S Vazire, M Burghart, N Casali, A Seidl (2026), Nat Rev Psychol 5, 338–351.
- Investigating the replicability of the social and behavioural sciences, AH Tyner, AL Abatayo, M Daley, S Field, N Fox, NA Haber, KM Hahn, MK Struhl, B Mawhinney, O Miske, P Silverstein, CK Soderberg, T Stankov, A Abbasi, CL Aberson, B Aczel, M Adamkovič, N Albayrak, PJ Allen, M Andreychik, E Awtrey, E Axxe, F Azevedo, MD Bader, B Bago, J Bailey, M Bakker, G Banik, GC Banks, E Baskin, A Batruch, A Beatteay, SM Behr, N Berente, Z Berry, J Białkowski, B Bodroža, L Boeschoten, M Bognar, C Bokhove, D Bonfiglio, R Bouwman, TF Brady, SR Braithwaite, G Briceño Jiménez, C Brick, T Bricka, R Briker, AN Brown, GDA Brown, RCM van Aert, K Caldwell, S Capitan, T Capitán, J Chandler, T Charles, CR Chartier, R Chawdhary, KJ Cheng, WJ Chopik, B Clark, VE Colvin, CC Comer, G Costantini, T Coupé, J Cummins, A Czernatowicz-Kukuczka, J de Leeuw, D Dobolyi, JN Druckman, J Duan, M Dujmović, DJ Dunleavy, PK Durkee, C Emery, KM Esterling, TR Evans, A Fedor, B Fernández-Castilla, N Fiala, JG Field, N Fong, MA Fonseca, ALJ Freeman, J Freese, SJ Geiger, J Geng, LM Getz, LM Geven, IH Gleibs, DP Gonzales, J Gooty, A Gourdon-Kanhukamwe, C Greculescu, SM Griffin, L Grigoryan, M Grunow, N Gunby, B Hall, PHP Hanel (2026), Nature 652, 143–150.
- Measurement-Reporting Practices in Social-and-Personality-Psychology Articles, KM Lawson, JG Bottesini, LD Khong, S Vazire (2026), Advances in Methods and Practices in Psychological Science 9 (1).
- Evaluation of the Replicability of Systematic Reviews with Meta-Analyses of the Effects of Health Interventions, DG Hamilton, JE Mckenzie, PY Nguyen, ML Rethlefsen, S Mcdonald, SE Brennan, FM Fidler, JPT Higgins, R Kanukula, S Karunananthan, LJ Maxwell, D Moher, S Nakagawa, D Nunan, P Tugwell, VA Welch, MJ Page (2026), Research Synthesis Methods.
- Comparing Expert Assessments of Research Quality between the Global North and East Africa, J Singh, F Mody, M Schmidt, S Shari, J Wambua, D Wilkinson, P Forscher (2025), Philosophical Transactions of the Royal Society of London Series B Biological Sciences 380 (1940).
- Same Data, Different Analysts: Variation in Effect Sizes Due to Analytical Decisions in Ecology and Evolutionary Biology, E Gould, HS Fraser, TH Parker, S Nakagawa, SC Griffith, PA Vesk, F Fidler, DG Hamilton, RN Abbey-Lee, JK Abbott, LA Aguirre, C Alcaraz, I Aloni, D Altschul, K Arekar, JW Atkins, J Atkinson, CM Baker, M Barrett, K Bell, SK Bello, I Beltrán, BJ Berauer, MG Bertram, PD Billman, CK Blake, S Blake, L Bliard, A Bonisoli-Alquati, T Bonnet, CNM Bordes, APH Bose, T Botterill-James, MA Boyd, SA Boyle, T Bradfer-Lawrence, J Bradham, JA Brand, MI Brengdahl, M Bulla, L Bussière, E Camerlenghi, SE Campbell, LLF Campos, A Caravaggi, P Cardoso, CJW Carroll, TA Catanach, X Chen, HYJ Chik, ES Choy, AP Christie, A Chuang, AJ Chunco, BL Clark, A Contina, GA Covernton, MP Cox, KA Cressman, M Crotti, CD Crouch, PB D’Amelio, AA de Sousa, TF Döbert, R Dobler, AJ Dobson, TS Doherty, SM Drobniak, AG Duffy, AB Duncan, RP Dunn, J Dunning, T Dutta, L Eberhart-Hertel, JA Elmore, MM Elsherif, HM English, DC Ensminger, UR Ernst, SM Ferguson, E Fernandez-Juricic, T Ferreira-Arruda, J Fieberg, EA Finch, EA Fiorenza, DN Fisher, A Fontaine, W Forstmeier, Y Fourcade, GS Frank, CA Freund, E Fuentes-Lillo, SL Gandy, DG Gannon, AI García-Cervigón, AC Garretson, X Ge, WL Geary, C Géron, M Gilles, Megan Kate Good, Tanner J Howard (2025), BMC Biology 23 (1).
- The Influence of Public Policy and Administration Expertise on Policy: An Empirical Study, R Haunschild, K Williams, L Bornmann (2025), Evidence and Policy 21 (4).
- Enhancing the Quality and Reproducibility of Research: Preferred Evaluation of Cognitive and Neuropsychological Studies - The PECANS Statement for Human Studies, C Costa, R Pezzetta, E Toffalini, M Grassi, G Cona, C Miniussi, PJ Bauer, S Borgomaneri, M Brysbaert, CD Chambers, N Edelstyn, A Eerland, SJ Gilbert, MA Nitsche, RA Poldrack, A Puce, KR Ridderinkhof, TY Swaab, C Umiltà, M Wiener, C Scarpazza, C Zogmaister, L Zapparoli, JM Zacks, N Yamamoto, SP Woods, ME Wokke, M Wiseheart, CE Wierenga, S Weigelt, EM Wassermann, E Viding, S Vazire, C Vázquez, D Van Ravenzwaaij, AE Van’t Veer, A Vallesi, M Ullsperger, LQ Uddin, EW Twamley, LL Toussaint, M Tops, L Toffoli, CK Thompson, EC Tecwyn, BA Tabak, D Szucs, JS Snyder, AS Souza, JC Snow, J Simner, AJ Shackman, M Scandola, LD Scherer, M Schepisi, W Schaeken, VM Reid, M Pyasik, A Protopapas, G Porciello, G Ponsi, AP Pinheiro, BS Peterson, L Però, MS Panasiti, S Oosterwijk, K Oberauer, M Montefinese, S Migliore, V Miskovic, L Menghini, G Melis, RD McIntosh, RJ McCarthy, JB Mattingley, F Masina, D Mantini, I Mangiulli, CR Madan, T Lanciano, D Lakens, VAF Lamme, C Lamm, S Kiran, P Kanske, L Jobson, RJ Itier, Y Huang, SA Huettel, MW Howe, D Houston, AR Hariri, A Guida, A Giustiniani, GM Gaskell, G Fusco, M Fusaro, GR Fink, T Feraco, V Era (2025), Behavior Research Methods 57 (7).
- Public Opinion and China’s Strategic Communication: Responses to Coercion and Persuasion, GAM Davies, K Edney, K Williams (2025), Foreign Policy Analysis 21 (3).
- Explicit Value Trade-Offs in Conservation: Integrating Animal Welfare, KE Lynch, BL Allen, O Berger-Tal, F Fidler, GE Garrard, JO Hampton, CH Lean, KM Parris, SL Sherwen, TE White, BBM Wong, DT Blumstein (2025), Trends in Ecology and Evolution 40 (6).
- Why Should You Trust Research Published in Psychological Science?, S Vazire (2025), Psychological Science 36 (5).
- Sports Metaresearch: An Emerging Discipline of Sport Science and Medicine, J Warmenhoven, P Menaspà, DN Borg, S Vazire, N White, K Sainani, S Nimphius, AJ Coutts, FM Impellizzeri (2025), Sports Medicine 55 (4).
- Predicting the Replicability of Social and Behavioural Science Claims in COVID-19 Preprints, A Marcoci, DP Wilkinson, A Vercammen, BC Wintle, AL Abatayo, E Baskin, H Berkman, EM Buchanan, S Capitán, T Capitán, G Chan, KJG Cheng, T Coupé, S Dryhurst, J Duan, JE Edlund, TM Errington, A Fedor, F Fidler, JG Field, N Fox, H Fraser, ALJ Freeman, A Hanea, F Holzmeister, S Hong, R Huggins, N Huntington-Klein, M Johannesson, AM Jones, H Kapoor, J Kerr, M Kline Struhl, M Kołczyńska, Y Liu, Z Loomas, B Luis, E Méndez, O Miske, F Mody, C Nast, BA Nosek, E Simon Parsons, T Pfeiffer, WR Reed, J Roozenbeek, AR Schlyfestone, CR Schneider, A Soh, Z Song, A Tagat, M Tutor, AH Tyner, K Urbanska, S van der Linden (2025), Nature Human Behaviour 9 (2).
- Are Women Really (Not) More Talkative Than Men? A Registered Report of Binary Gender Similarities/Differences in Daily Word Use, CA Tidwell, AF Danvers, VA Pfeifer, DB Abel, E Alisic, A Beer, SJ Bierstetel, KL Bollich-Ziegler, M Bruni, WR Calabrese, C Chiarello, B Demiray, S Dimidjian, KL Fingerman, M Haas, DM Kaplan, YK Kim, G Knezevic, LB Lazarevic, M Luo, A Macbeth, JH Manson, JS Mascaro, C Metcalf, KS Minor, S Moseley, AJ Polsinelli, CL Raison, JK Rilling, ML Robbins, D Sbarra, RB Slatcher, J Sun, M Vasileva, S Vazire, MR Mehl (2025), Journal of Personality and Social Psychology 128 (2).
- Repertoires of Research Value: Performing Societal Impact across Countries, K Williams, S Michalska (2025), Research Evaluation 34.
- Snapshots in a Time of Crisis: Trends in Psychology Research Practices Throughout the Replication Crisis, BM Clarke (2025), Edited by S Vazire and T Hardwicke.
- How Do Psychology Journals Handle Postpublication Critique? A Cross-Sectional Study of Policy and Practice, A Whamond, S Vazire, B Clarke, N Moodie, S Schiavone, RT Thibault, TE Hardwicke (2025), Advances in Methods and Practices in Psychological Science 8 (3).
- Context Is Everything: National Variation in Societal Impact Assessment, K Williams, JM Lewis (2025), Higher Education.
- The Benefits of Being between (Many) Fields: Mapping the High-Dimensional Space of AI Research, G Berman, K Williams, E Cohen (2025), Big Data & Society 12 (1).
- Cancer Researchers’ Experiences with and Perceptions of Research Data Sharing: Results of a Cross-Sectional Survey, DG Hamilton, MJ Page, S Everitt, H Fraser, F Fidler (2025), Accountability in Research 32 (4).
- The Cultural Impact of the Impact Agenda in Australia, UK and USA, E Cohen, K Williams, J Grant (2025), Research Evaluation 34.
- What do Australians affected by cancer think about oncology researchers sharing research data? A cross‐sectional survey, DG Hamilton, S Everitt, MJ Page, F Fidler (2024), Asia‐Pacific Journal of Clinical Oncology 20 (4), 522-530.
- Computationally reproducing results from meta-analyses in ecology and evolutionary biology using shared code and data, S Kambouris, DP Wilkinson, ET Smith, F Fidler (2024), Plos one 19 (3), e0300333.
- Attitudes of people living with cancer towards trial non-publication and research participation, DG Hamilton, S Everitt, MJ Page, S Vazire, F Fidler, (2024), BMJ Evidence-Based Medicine 29 (1), 64-66.
- An empirical appraisal of eLife’s assessment vocabulary, TE Hardwicke, SR Schiavone, B Clarke, S Vazire, (2024) Plos Biology 22 (8), e3002645.
- Transparency Is Now the Default at Psychological Science, TE Hardwicke, S Vazire, (2024) Psychological Science 35 (7), 708-711.
- Looking our limitations in the eye: A call for more thorough and honest reporting of study limitations, Clarke, B., Alley, L. J., Ghai, S., Flake, J. K., Rohrer, J. M., Simmons, J. P., Schiavone, S. R., & Vazire, S. (2024), Social and Personality Psychology Compass, e12979.
- The prevalence of direct replication articles in top-ranking psychology journals. Clarke, B., Lee, P. Y. (K.), Schiavone, S. R., Rhemtulla, M., & Vazire, S. (2024), American Psychologist. Advance online publication.
- Construct validity evidence reporting practices for the Reading the Mind in the Eyes Test: A systematic scoping review. Higgins, W. C., Kaplan, D. M., Deschrijver, E., & Ross, R. M. (2024). Clinical Psychology Review, 108, Article 102378.
- Why most research based on the Reading the Mind in the Eyes Test is unsubstantiated and uninterpretable: A response to Murphy and Hall Higgins, W. C., Kaplan, D. M., Deschrijver, E., & Ross, R. M. (2024). Clinical Psychology Review, 115, Article 102530.
- What limitations are reported in short articles in social and personality psychology, Clarke, B., Schiavone, S., & Vazire, S, (2023), Journal of Personality and Social Psychology, 125(4), 874–901.
- Three threats to the validity of the Reading the Mind in the Eyes Test: A commentary on. Neuroscience and Biobehavioral Reviews, Higgins, W. C., Ross, R. M., Polito, V., & Kaplan, D. M. (2023).147, Article 105088.
- Validation of the reading the mind in the eyes test requires an interpretable factor model. Higgins, W. C., Savalei, V., Polito, V., & Ross, R. M. (2023). Proceedings of the National Academy of Sciences - PNAS, 120(52), e2303706120–e2303706120.
RepliCATS-related publications
- Systematizing Confidence in Open Research and Evidence (SCORE), SCORE Collaboration, (2024).
- Predicting reliability through structured expert elicitation with repliCATS (Collaborative Assessments for Trustworthy Science), Fraser et al., (2023).
- AggreCAT: An R Package for Mathematically Aggregating Expert Judgments, Gould et al., (2023).
- Mathematically aggregating experts’ predictions of possible futures, Hanea et al., (2023).
- Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol, Pearson et al., (2021).
- Predicting and reasoning about replicability using structured groups, Wintle et al., (2023).
The best way to keep up to date with our publications is looking up our people on Google Scholar or Find an Expert.
Leadership
Fiona Fidler is a philosopher of science interested in the dynamics of methodological change. She has a long-standing interest in statistical controversies and her current work in metascience focuses on improving research culture and peer review systems to create a more trustworthy scientific evidence base.
E: fidlerfm@unimelb.edu.au
L: https://www.linkedin.com/in/fiona-fidler-b0825128/
B: @fidlerfm.bsky.social
Simine Vazire is a Professor of psychology in the Ethics and Wellbeing Hub in the Melbourne School of Psychological Science and Co-Lead of MetaMelb. She studies the research methods and practices used in psychology, as well as structural systems in science, such as peer review. Her research interests on the meta-science side include assessing the quality and integrity of scientific studies, the peer review process, and the scientific community at large. She is interested in how transparency and criticism are (or aren’t) used to make science more self-correcting. Her training is in social and personality psychology, and her interests in scientific practices and norms stems largely from her experiences in that field, particularly the so-called replication crisis. She has been an editor at several psychology journals, and co-founded the Society for the Improvement of Psychological Science (SIPS) with Brian Nosek in 2016.
E: simine.vazire@unimelb.edu.au
W: http://simine.com/
L: https://www.linkedin.com/in/simine-vazire-55061511a/
B: @simine.com
Fallon Mody is a lecturer in History and Philosophy of Science in the School of Historical and Philosophical Studies and Deputy Director coordinating MetaMelb’s Education and Training theme. This means translating metascience research developments into peer review training workshops for early career researchers, and evaluation training end-users of research. It also includes coordinating the inclusion of repliCATS and error detection modules in the curriculum both at the University of Melbourne — where repliCATS modules have been incorporated into psychology, environment science, and philosophy of science — and beyond, leading our global ‘repliCATS in the classroom’ project.
E: fallon.mody@unimelb.edu.au
L: https://www.linkedin.com/in/fallonmody
B: @fallonmody.bsky.social
Kate Williams is an Associate Professor in Public Policy in the School of Social and Political Sciences and Deputy Director coordinating MetaMelb’s Research Policy theme. This means understanding the impact metascience has on research policy and using metascientific evidence to advise on appropriate evaluation and impact measures. Kate takes a cross-national perspective on how universities and funders assess research impact and value. Her previous policy work includes commissioned reports for the ARC (review of 2018 Engagement and Impact Assessment), ACOLA (advice to the Chief Scientist on reforming research assessment), and Hong Kong’s University Grants Committee (review of national research impact). Kate’s work also underpins the University of Melbourne’s central research impact strategy.
Current Team
Dr David Wilkinson
Dr Cristian Larroulet Philippi
Wendy Higgins
Beth Clarke
Elliot Gould
Steve Kambouris
Dr Hannah Fraser
Carmelina Contarino
Glen Berman
Liam Hourigan
Affiliates
A/Prof Anca Hanea
Mark Burgman
Past Members
Dr Rose O'Dea
Dr Daniel Hamilton
Sarah Schiavone
Dr Aurélien Allard
Julia Bottesini
Andy Head
Aaron Wilcox
