Rater cognition and the validity of the BESTEP speaking test
2025-2026
This study aims to illuminate the ‘black box’ of scoring processes and provide actionable recommendations for improving both rater training and the holistic rating scales currently used in the BESTEP speaking test
Project : Rater cognition and the validity of the BESTEP speaking test: A qualitative investigation
While a substantial body of research has examined rater cognition in language assessments, limited attention has been paid to how raters engage with holistic rating scales when scoring integrated speaking tasks. Grounded in an argument-based approach to test validation, this proposed project aims to investigate how raters interpret and apply the holistic rating scales when scoring Parts 2 and 3 in the BESTEP speaking test, both of which involve integrated skills. Two research questions will guide this study:
- How do raters interpret and apply the rating scales and performance level descriptors when rating test takers’ performances on the BESTEP speaking test?
- Do raters’ cognitive processes differ when rating performances on Parts 2 and 3 of the BESTEP speaking test?
To address the research questions, this study will employ two research methods: think-aloud verbal protocols and focus group interviews. The participants will include 15 BESTEP raters. Each rater will be asked to rate 10 spoken performances on Parts 2 and 3 of the BESTEP speaking test and report their thought processes. Two focus group interviews will also be conducted to further explore and clarify how raters have interpreted and applied the rating scale. The collected data will be analysed thematically. By exploring raters’ cognitive processes, this study aims to illuminate the ‘black box’ of scoring processes and provide actionable recommendations for improving both rater training and the holistic rating scales currently used in the BESTEP speaking test.