The blind side

Periodical
Assessment in Education: Principles, Policy & Practice
Volume
29
Year
2022
Issue number
3
Page range
332-360
Relates to study/studies
PISA 2018

The blind side

Exploring item variance in PISA 2018 cognitive domains

Abstract

Communication of International Large-Scale Assessment (ILSA) results is dominated by reporting average country achievement scores that conceal individual differences between pupils, schools, and items. Educational research primarily focuses on examining differences between pupils and schools, while differences between items are overlooked. Using a variance components model on the Programme for International Student Assessment (PISA) 2018 cognitive domains of reading, mathematics, and science literacy, we estimated how much of the response variation can be attributed to differences between pupils, schools, and items. The results show that uniformly across domains and countries, it mattered more for the correctness of an item response which items were responded to by a pupil (27-35%) than which pupil responded to these items (10-12%) or which school the pupil attended (5-7%). Given the findings, we argue that differences between items in ILSAs constitute a source of substantial untapped potential for secondary research.