Automatically analyzing text responses for exploring gender-specific cognitions in PISA reading

Periodical
Large-scale Assessments in Education
Volume
6
Year
2018
Issue number
7
Relates to study/studies
PISA 2012

Automatically analyzing text responses for exploring gender-specific cognitions in PISA reading

Abstract

Background

The gender gap in reading literacy is repeatedly found in large-scale assessments. This study compared girls’ and boys’ text responses in a reading test applying natural language processing. For this, a theoretical framework was compiled that allows mapping of response features to the preceding cognitive components such as micro- and macropropositions from the situation model.

Methods

In total, n=33,604n=33,604 responses from the German sample of the Programme for International Student Assessment (PISA) 2012 reading test have been analyzed for characterizing the genders’ typical cognitive approaches. The analyses mainly explored the gender gap by contrasting groups of responses typical for either gender. These gender-specific responses characterize the typical responding of the genders to PISA reading questions.

Results

Responses typical for girls contained three to five more proposition entities from the situation model, irrespective of the response correctness. They integrated more relevant propositions and constituted better fits to the question focus. That means, in answering questions which ask for explicit information from the stimulus text, the typical girl responses appropriately encompassed more micropropositions, and typical boy responses tended to include more macropropositions—vice versa for questions requesting implicit information.

Conclusion

It appears that typical boy responses to PISA reading questions are characterized by struggling with retrieving and integrating propositions from the situation model. The typical girl liberally juggles these to formulate the responses. The results demonstrate that text responses are a neglected but informative source for educational large-scale assessments made accessible through natural language processing.