MetaMelb Research Initiative

We are a group of interdisciplinary metascience researchers based at the University of Melbourne.

Meta-researchers investigate scientific practice to help improve confidence in scientific claims as understood by practising scientists. Unlike other approaches to studying science, meta-research is internally focussed – but we think this has a big impact on everyone.

The “meta” in MetaMelb stands for either metascience or meta-research. The terms are commonly used interchangeably, and as an interdisciplinary research group we happily embrace this plurality. What we do doesn’t change.

Leadership

Fiona Fidler is a professor at the University of Melbourne and the Head of the History and Philosophy of Science Program. She is broadly interested in how experts, including scientists, make decisions and change their minds. Her past research has examined how methodological change occurs in different disciplines, including psychology, medicine and ecology, and developed methods for eliciting reliable expert judgements to improve decision making. She originally trained as a psychologist, and maintains a strong interest in psychological methods. She also has an abiding interest is statistical controversies, for example, the ongoing debate over Null Hypothesis Significance Testing. She is a current Australian Research Council Future Fellow, and founded the Association for Interdisciplinary Meta-research & Open Science (AIMOS) in 2019.

E: fidlerdm@unimelb.edu.au
T: @fidlerfm

Simine Vazire is a professor of psychology in the Ethics and Wellbeing Hub in the Melbourne School of Psychological Science. She studies the research methods and practices used in psychology, as well as structural systems in science, such as peer review. Her research interests on the meta-science side include assessing the quality and integrity of scientific studies, the peer review process, and the scientific community at large. She is interested in how transparency and criticism are (or aren’t) used to make science more self-correcting. Her training is in social and personality psychology, and her interests in scientific practices and norms stems largely from her experiences in that field, particularly the so-called replication crisis. She has been an editor at several psychology journals, and co-founded the Society for the Improvement of Psychological Science (SIPS) with Brian Nosek in 2016.

E: simine.vazire@unimelb.edu.au
T: @siminevazire

Current Team

	Dr Tom Hardwicke is a Research Fellow at the Melbourne School of Psychological Sciences at The University of Melbourne. He received his PhD in Experimental Psychology from University College London in 2016 and subsequently completed post-doctoral fellowships at The Meta-Research Innovation Center at Stanford University (METRICS), The QUEST Center for Responsible Research at Charité Berlin, and the Department of Psychology at The University of Amsterdam. Tom works on a variety of meta-research (‘research-on-research’) projects related to transparency, bias, peer review, scientific criticism, and reproducibility.
	Dr Rose O'Dea is a Mary Lugton Postdoctoral Research Fellow at the University of Melbourne. Her research includes behavioural ecology, meta-analysis, and meta-science, and she co-founded the Society for Open, Reliable, and Transparent Ecology and Evolutionary Biology (SORTEE).
	Dr Martin Bush is a Senior Research Fellow in the School of Historical and Philosophical Studies at the University of Melbourne. He has expertise in popular science and public reasoning practices and professional experience in science communication and the museum sector. Martin led the Reasoning Analysis team on the repliCATS project.
	Dr David P. Wilkinson is a Research Fellow School of Agriculture, Food and Ecosystem Sciences. He was the team leader for the Data Management and Analysis team as part of the repliCATS project for the collaborative assessment of trustworthy science.
	Dr Fallon Mody is a Research Fellow in the School of Historical and Philosophical Studies. Fallon’s research interests include science communications, qualitative analysis in mixed methods research, public engagement in science, and history of medicine. Fallon led recruitment and community management on the repliCATS project.
	Dr Bonnie Wintle is a Senior Research Fellow in ecology. Bonnie develops structured methods for eliciting and aggregating quantitative and qualitative judgements from groups of experts, to support better decision and policy making. Bonnie’s also interested in expertise, uncertainty and prediction.
	Beth Clarke a PhD student at the University of Melbourne doing metaresearch primarily in social and personality psychology. She is particularly interested in how science generates and disseminates knowledge. Her PhD research focuses on the claims psychologists make in their published articles, and the statistical evidence they report to support their claims. Beth also studies how research practices have changed in psychology over the course of the replication crisis.
	Elliot Gould is a PhD student in BioSciences. Elliot is studying reproducibility and transparency of decisions in ecology and conservation. Elliot’s research interests include data science, decision analysis, ecological modelling. Elliot has been actively involved in developing new ‘adaptive preregistration’ templates for field work and ecological modelling.
	Steve Kambouris is a PhD student studying reproducibility, open science and meta-analysis. He has a background in physics, psychometrics, education, and philosophy of science, amongst other things.
	Daniel Hamilton is a PhD student exploring data sharing policies, practices and perceptions in oncology. Daniel originally trained as a radiation therapist at Epworth hospital in Melbourne, working both clinically and in a research support role between 2012 and 2017. Following his position at Epworth hospital Daniel worked as a research coordinator at Peter MacCallum Cancer Centre managing a large portfolio of national and international radiation oncology clinical trials. He is the lead author on multiple papers investigating novel radiotherapy treatment techniques for prostate and breast cancer, as well as papers examining ethical issues in scientific publishing.
	Hannah Fraser is an ecologist studying how current ecological research practices relate to the rigor and reproducibility of results. Her research interests include QRPs, replication, preregistration, registered reports and improving scientific practice.
	Sarah Schiavone

Affiliates

	Dr Alexandra de Gendre is the 2022 McKenzie Postdoctoral Research Fellow hosted at the Department of Economics at the Faculty of Business and Economics and an economist with expertise in labor economics and applied microeconomics.
	Dr Nicolas Salamanca Acosta is the Senior Research Fellow at the Melbourne Institute: Applied Economics & Social Research of The University of Melbourne with expertise in applied microeconomics.
	Prof Peter Vesk is a quantitative ecologist who integrates fundamental and applied research, with expertise in plants trait analysis and conservation, management, and landscape restoration.
	A/Prof Anca Hanea is an applied probabilist based at the Centre of Excellence for Biosecurity Risk Analysis (CEBRA) at the University of Melbourne with expertise in mathematics and risk and environmental modelling.
	Prof Mike McGuckin is Deputy Dean and Director Research Strategy in the Faculty of Medicine, Dentistry and Health Sciences at the University of Melbourne. He is the author of over 170 scientific publications focused on mucosal infection, chronic inflammation and cancer in the gastrointestinal tract, and has held four patents.

Past Members

Aurélien Allard is a postdoctoral research fellow at UC Davis. He completed a PhD in political philosophy at Paris 8 University. His research interests include metascience, philosophy of science, moral philosophy and moral psychology.	Julia Bottesini is a PhD student at UC Davis studying metascience, research practices, and quantitative psychology. Her current projects examine how research stakeholders, including study participants and science journalists, view scientists’ research practices.	Andy Head is a research assistant. He recently completed a graduate diploma of psychology at Deakin University and plans to start a PhD in 2020. Andy’s research interests include methodology, improving science practices, and public engagement with science.
Aaron Wilcox recently completed a psychological science degree. Aaron also works as a research assistant at Deakin University under Emma Sciberras. Aaron is an R acolyte and an open science advocate and will be continuing studies into fourth year and beyond.	John Maaga is an undergrad psychology major, with the ultimate goal of becoming a psychiatrist. In his first stats class in community college, he became fascinated by the complexity of research methods. John hopes to influence the use of better research practices, especially as it relates to medical and psychological treatments.	Felix Singleton Thorn recently completed a PhD in psychology. Felix’s research examines how people plan, report and interpret the results of psychological experiments, with the goal of improving the accuracy of inferences and providing practical heuristics to guide the interpretation of results.

Our group is working on a large number of projects organised around the following topic areas.

1. Research quality

What are the dimensions we should consider when evaluating research quality? How do researchers evaluate the quality of each other's work, and how do they come to those evaluations? Can we improve the evaluation of research quality by introducing rubrics, group discussions, or other structured processes?

2. Journal policies and practices

What are common journal policies and informal practices around peer review and publication? How does peer review work and how can we improve it? How is the published literature changing over time?

3. Statistical inference

What are common errors in statistical inference, and how common are they? How can we improve statistical training and practice? What are the best tools for detecting and correcting errors in statistical inference?

4. Self-correction

What are the self-correcting mechanisms in science? How common and effective is post-publication critique? Do replications help to correct the scientific record? Who should do the work of verifying and correcting published studies, and how should it be incentivised?

5. Intellectual humility

Do research claims match the evidence presented? How common are hype and spin? How can we incentivise researchers to make calibrated claims?

6. Transparency and Reproducibility

Does increased transparency lead to better quality research, or better detection of errors and inaccuracies in research? What kinds of transparency are most important, and how common are they? How can important forms of transparency be incentivised? To what extent and under what conditions does transparency improve reproducibility.

repliCATS project

The repliCATS project aims to crowdsource predictions about the reliability of 3000 published research claims in the social and behavioural sciences. The “CATS” in repliCATS stands for Collaborative Assessment for Trustworthy Science.

We aim to estimate the replicability of 3,000 published social scientific research claims in: Business research, Criminology, Economics, Education, Political Science, Psychology, Public Administration, and Sociology.

The repliCATS project is part of a research program called SCORE, funded by DARPA, that eventually aims to build automated tools that can rapidly and reliably assign confidence scores to social science research claims.

More information

The best way to keep up to date with our publications is to look at our individual google scholar pages. Here are a couple of recent papers:

Where are the self-correcting mechanisms in science? S Vazire, AO Holcombe; Review of General Psychology 26 (2), 212-223
Credibility beyond replicability: Improving the four validities in psychological science S Vazire, SR Schiavone, JG Bottesini; Current Directions in Psychological Science 31 (2), 162-168

Publications associated with the repliCATS project:

SCORE collaborative“Systematizing Confidence in Open Research and Evidence (SCORE)”
Fraser et al.,“Predicting reliability through structured expert elicitation with repliCATS (Collaborative Assessments for Trustworthy Science)”
Gould et al.,“aggreCAT: An R Package for Mathematically Aggregating Expert Judgments”
Hanea et al.,“Mathematically aggregating experts’ predictions of possible futures”
Pearson et al (2021),“Eliciting group judgements about replicability: a technical implementation of the IDEA Protocol”
Wintle et al.,“Predicting and reasoning about replicability using structured groups”

If you would like to get in touch, e-mail us at:

Fiona Fidler
fidlerfm@unimelb.edu.au
Simine Vazire
simine.vazire@unimelb.edu.au