PISA 2022 Design
Methodology
Quantitative Study
Method(s)
- Overall approach to data collection:
- Proctored assessment and self-administered questionnaire
- Specification:
- Cross-sectional, every 3 years
Target population
15-year-old students enrolled in education institutions at Grade 7 and higher in their respective countries or economies, including those:
- Enrolled full-time in educational institutions.
- Enrolled in educational institutions but attending only on a part-time basis.
- Enrolled in vocational training programs, or any other related type of educational program.
- Attending foreign schools within the country/economy (as well as students from other countries/economies attending any of the programs in the first three categories).
Sample design
- The international PISA target population in each participating country and economy consisted of 15-year-old students attending an educational institution in Grade 7 and higher.
- In all countries, the sampling design used for the PISA assessment was a two-stage stratified sample design.
The first-stage sampling units
- Consisted of individual schools with 15-year-old students or the possibility of having such students at the time of assessment.
- Schools were sampled systematically from a comprehensive national list of all PISA-eligible schools – the school sampling frame – with probabilities that were proportional to a measure of size.
- The measure of size was a function of the estimated number of PISA-eligible 15-year-old students enrolled in the school. This is referred to as systematic probability-proportional-to-size (PPS) sampling.
- Prior to sampling, schools in the sampling frame were assigned to mutually exclusive groups based on school characteristics (explicit strata) specifically chosen to improve the precision of sample-based estimates.
The second-stage sampling units in countries using the two-stage design were students within sampled schools.
- Once schools were selected to be in the sample, a complete list of each sampled school’s 15-year-old students was prepared.
- For each country a target cluster size (TCS) was set, typically 42 students for computer-based countries and 35 for paper-based countries, although by agreement countries could use alternative values. The sample size within the schools is prescribed, within limits, in the PISA Technical Standards.
- From each list of students that contained more than the TCS, a sample of typically 42 students was selected with equal probability.
- For lists of fewer than the TCS, all students on the list were selected.
PISA in the Pandemic
The test was originally planned to take place in 2021 but was delayed by one year due to the COVID-19 pandemic. The exceptional circumstances throughout this period, including lockdowns and school closures in many places, led to occasional difficulties in collecting some data.
While the vast majority of countries and economies met PISA’s technical standards, a small number did not. In prior PISA rounds, countries and economies that failed to comply with the standards, and which the PISA Adjudication Group judged to be consequential, could face exclusion from the main part of reporting.
However, given the unprecedented situation caused by the pandemic, PISA 2022 results include data from all participating education systems, including those where there were issues such as low response rates (see Annexes A2 and A4, Volume I, PISA 2022 Results). PISA will return to the standard ways of reporting for the 2025 assessment.
* Caution is required when interpreting estimates of the following countries/economies: Australia*, Canada*, Denmark*, Hong Kong (China)*, Ireland*, Jamaica*, Latvia*, the Netherlands*, New Zealand*, Panama*, the United Kingdom* and the United States*.
Sample size
The student sample size is a minimum of 6,300 assessed students per country/economy. The school sample size needs to result in a minimum of 150 participating schools per country/economy.
Some 690 000 students took the assessment in 2022, representing about 29 million 15-year-olds in the schools of the 81 countries and economies.
Data collection techniques and instruments
The assessment
- Computer-based assessment (CBA) was used (by 77 countries/economies), with assessments lasting a total of two hours for each student, in a range of countries and economies.
- In mathematics and reading, a multi‑stage adaptive approach was applied in computer-based tests whereby students were assigned a block of test items based on their performance in preceding blocks.
- Test items were a mixture of multiple-choice questions and questions requiring students to construct their own responses.
- The items were organized in groups based on a passage setting out a real-life situation.
- More than 15 hours of test items were covered, with different students taking different combinations of test items.
- Paper-based assessment (PBA) was used by 4 countries/economies, they used either the existing PBA design implemented in PISA 218 or the new PBA instrument, which was the same one that was used for PISA for Development.
Background questionnaires
- Students also answered a background questionnaire
- Completion time 35 minutes
- Information about the students themselves (attitudes, dispositions and beliefs), their home and school, and learning experiences
- School principals completed a questionnaire that covered the school system and the learning environment.
- Both students and schools responded to items in the Global Crises Module in their respective questionnaires. These items aimed to elicit their perspectives on how learning was organized when schools were closed because of the COVID-19 pandemic.
- In some countries/economies optional questionnaires were distributed to
- Teachers
- Parents
- Countries could choose two other optional questionnaires for students:
- One asked students about their familiarity with computers
- The second sought information about students’ well-being
Techniques
- achievement or student test
- questionnaire
Languages
- The achievement test was administered in 54 different languages.
- The most common languages:
- English (17 countries)
- Spanish (12 countries)
Translation procedures
- Optimization of the English source version for translation through translatability assessment.
- Development of two source versions of the instruments, in English and French (except for the financial literacy and for the operational manuals, provided only in English)
- Double translation design with a final reconciliation
- Preparation of detailed instructions for the localization of the instruments for the Field Trial and for their review for the Main Survey
- Preparation of translation/adaptation guidelines
- Production of item-by-item translation and adaptation notes
- Training of national staff in charge of translation/adaptation of the instruments
- Validation of the translated/adapted national versions, including verification by independent verifiers, review by cApStAn staff and the translation referee or the questionnaires team, countries’ post-verification review and “technical” and linguistic final checks
Quality control of operations
Measures during data collection
- Procedures for sampling, translation, survey administration, and data processing were developed in accordance with the PISA Technical Standards and fully documented.
- The PISA Consortium provided comprehensive manuals explaining the implementation of the survey.
- They included:
- Precise instructions for the work of school coordinators.
- Scripts for test administrators to use during the assessment sessions.
- Proposed adaptations to survey procedures or modifications to the assessment session script were submitted to the PISA Consortium for approval prior to verification.
- They included:
- Participating countries and economies were required to ensure that test administrators worked with the school coordinator to prepare the assessment session, including:
- Reviewing and updating the Student Tracking form
- Completing the Session Report Form, which is designed to summarize session times, any disturbance to the session, etc.
- Ensuring that the number of test booklets and questionnaires collected from students tallied with the number sent to the school (paper‑based assessment countries) or ensuring that the number of USB sticks used for the assessment were accounted for (computer-based assessment countries)
- Sending the school questionnaire, student questionnaires, parent and teacher questionnaires (if applicable), and all test materials (both used and unused) to the national centre after the testing
- The PISA Consortium responsible for overseeing survey operations implemented all phases of the PISA Quality Monitor (PQM) process:
- Phases included:
- Interviewing and hiring PQM candidates in each of the countries
- Organizing their training
- Selecting the schools to visit
- Collecting information from the PQM visits
- PQMs
- Independent contractors located in participating countries who were hired by the international survey operations contractor.
- Visited a sample of schools to observe test administration and to record the implementation of the documented field-operations procedures in the main survey
- Typically, two or three PQMs were hired for each country, and they visited an average of 15 schools in each country. If there were adjudicated regions in a country, it was usually necessary to hire additional PQMs, as a minimum of five schools were observed in adjudicated regions.
- Phases included:
- Quality-assurance data collected throughout the PISA 2022 assessment were entered and collated in a central data adjudication database on the quality of field operations, printing, translation, school and student sampling, and coding. Comprehensive reports were then generated for the PISA Adjudication Group.
Measures during data processing and cleaning
- For Computer-Based Assessment (CBA) participants, the coding designs for the CBA responses for mathematics, reading, science, creative thinking, and financial literacy (when applicable) were greatly simplified through use of the Open-Ended Coding System (OECS), which organizes responses according to the agreed-upon coding designs. It was especially important during the COVID-19 pandemic, as the OECS afforded coders the ability to work remotely.
- Through coder reliability monitoring, coding inconsistencies or problems within and across countries could be detected early in the coding process through OECS and Open-Ended Reporting System (OERS) output reports, allowing action to be taken as soon as possible.
- The OECS/OERS worked in concert with the DME (Data Management Expert) database to generate two types of reliability reports: a score agreement and coding category distributions. Coder reliability studies also made use of the OECS/OERS reports submitted by national centres.
- Data-quality standards in PISA required minimum participation rates for schools as well as for students. These standards were established to minimize the potential for response bias. In the case of countries meeting these standards, it was likely that any bias resulting from non-response would be negligible, i.e., typically smaller than the sampling error.
- All quality assurance data collected throughout the cycle are entered and collated in a central data adjudication database. The technical advisory group (TAG) and the sampling referee review this database to make country-by-country evaluations on the quality of field operations, printing, translation, school and student sampling and coding. The final report by TAG experts is then used for the purpose of data adjudication.
Sources - Technical documentation
Other sources