PIRLS 2021 Design

Methodology
  • International large-scale sample survey of student achievement and educational context
  • Monitors trends by reporting results from successive cycles on a common achievement scale
  • Predominantly quantitative, with qualitative information presented in descriptive country chapters in the PIRLS encyclopedia
TOP
Methodology
Quantitative Study
TOP
Method(s)
  • Overall approach to data collection:
    • Proctored assessment of student achievement
    • Self-administered surveys for students, parents, teachers, and school principals
TOP
Target population

All students enrolled in the grade that represents four years of schooling counting from the first year of ISCED Level 1, provided that the mean age at the time of testing is at least 9.5 years.

Note

At the time of the PIRLS 2021 data collection, there was considerable variation across countries in how primary school operations were affected by the COVID-19

pandemic. This led to the following peculiarities of PIRLS 2021:

  • 43 countries and 5 benchmarking entities assessed students at the end of the target school year, including conducting the assessment a year later than originally scheduled in a few countries (this is annotated in the exhibits of the report). 
  • In 14 countries and 3 benchmarking entities (all in the Northern Hemisphere) the data collection schedule was delayed so that students were assessed at the beginning of the fifth grade. Therefore, the average age of students in these countries/entities was half a year older on average than the average age of students in the other countries and entities. (In the report, color highlighting has been used to identify the results of these participants.)
TOP
Sample design
STRATIFIED TWO-STAGE CLUSTER SAMPLE DESIGN

First sampling stage: sampling school

  • Sampling schools with probabilities proportional to their size (PPS) from the list of all schools in the population that contain eligible students
  • Stratifying schools (optional):
  • According to important (demographic) variables: e.g., region of the country (e.g., states or provinces); school type or source of funding (e.g., public or private); language of instruction; level of urbanization (e.g., urban or rural); socioeconomic indicators; and school performance on national examinations
  • Two forms possible: explicit or implicit stratification
  • Systematic random sampling was used
  • Schools sampled at the same time for field test and main data collection
  • Sampling of two replacement schools for each school sampled (sample selection process only)

 

Second sampling stage: sampling classes within schools

  • Only after sampled schools agreed to participate in the study
  • One or more intact classes from the target grade of each school, selected using systematic random sampling

 

For digitalPIRLS countries: Bridge sample

  • An additional sampling step was required for digitalPIRLS countries to provide “bridge” data between the paper and digital assessments (i.e., between the computer-based data of countries participating in digitalPIRLS in 2021 and their paper-based data in 2016, as well as to the paperPIRLS countries in 2021).
  • The bridge sample consisted of an extra class from a school sampled for digitalPIRLS or from an additional school (equivalent groups design).
  • In schools selected for both the digitalPIRLS and the bridge samples, separate classes were sampled and randomly assigned to either the digitalPIRLS or bridge samples.

 

General notes

  • School sampling conducted by Statistics Canada, class sampling by participating countries
  • All sampling activities monitored and documented by Statistics Canada and IEA Hamburg staffs
TOP
Sample size

PIRLS

  • Per country
  • School sample: Approximately 150–200 schools
  • Student sample: Approximately 4,000 students
  • In total, approximately 400,000 students, 380,000 parents, 20,000 teachers, and 13,000 schools.

digitalPIRLS

  • Approximately 5,000 students for main sample and 1,500 students for the bridge sample
TOP
Data collection techniques and instruments

Student Assessment

Formats provided

  • digitalPIRLS and paperPIRLS; they had the same content.
  • digitalPIRLS
    • Booklets administered on computers through an eAssessment system, including ePIRLS.
    • Administered in 26 countries and seven benchmarking entities
    • Countries participating in digitalPIRLS could choose from the following delivery methods: USB delivery, online administration, local PC server method.
  • paperPIRLS
    • Paper-based booklets  
    • Administered in 31 countries and one benchmarking entity
  • ePIRLS
    • Integrated and administered as part of digitalPIRLS so that no additional testing time was required.
    • Assessed online informational reading (with students reading through information presented in a simulated internet environment and answering a series of comprehension questions about what they had read).

Achievement items

  • Items classified according to process of comprehension being measured by the question
  • Two general item formats:
    • Multiple-choice (a bit less than 40% of the total number of score points represented by all the questions)
    • Constructed response
  • Organization of items
    • digitalPIRLS and paperPIRLS
      • Reading passages with accompanying questions (item blocks)
      • 18 passages arranged into booklets of two passages each, i.e., 
        • one passage assessing reading for literary experience and 
        • one passage assessing reading to acquire and use information
    • ePIRLS: Five reading tasks
  • Items per assessment
    • paperPIRLS: 297 items (18 texts)
    • digitalPIRLS : 349 items (18 texts)
    • ePIRLS: 123 items (5 texts)

Booklets

  • paperPIRLS countries had 18 booklets comprised of 18 passages and item sets.
  • digitalPIRLS countries had three types of booklets
    • 18 regular digitalPIRLS booklets with the 18 passages
    • 20 ePIRLS booklets (the results of all possible pairings of the five ePIRLS tasks)
    • 45 hybrid booklets (so that each of the nine PIRLS informational texts was paired with each of the five ePIRLS tasks)
  • Linking mechanisms between booklets and between cycles (trend)
    • digitalPIRLS and paperPIRLS
      • 12 passages were administered from PIRLS 2016 
      • Six new passages were developed for PIRLS 2021
    • ePIRLS: three trend tasks administered in 2016 and two new tasks
  • Matrix sampling of passages/items (rotated test booklet design)

Group adaptive assessment design

  • Was new in PIRLS 2021
  • Allowed to integrate in the PIRLS main study the previously separately administered less difficult prePIRLS (2011) or PIRLS Literacy (2016) tests.
  • There were three levels of passage/item block difficulty: difficult, medium, and easy.
  • The 18 booklets were divided into two levels of booklet difficulty:
    • Nine more difficult booklets composed of two difficult passages or one medium and one difficult passage.
    • Nine less difficult booklets composed of an easy and a medium passage or two easy passages.
  • Each country administered the entire assessment with the balance of more difficult and less difficult booklets depending on the academic level of the pupils in that country.

Administration of booklets for each test format

  • paperPIRLS countries administered the 18 booklets using the group adaptive design.
  • digitalPIRLS countries
    • Used a 27-booklet rotation that included  
      • all 18 digitalPIRLS booklets following the group adaptive design
      • six of the ePIRLS booklets on a rotating basis, and 
      • three of the hybrid booklets, also on a rotating basis.
    • Administered, in addition, 8 paper bridge booklets 
      • Seven of these booklets were also administered in 2016, and one contained two texts also administered in 2016 but in a different combination.
      • They were administered to an additional sample.

 

Background questionnaires                          

  • Early Learning survey (previously, Learning to Read survey) was the home questionnaire to be completed by students’ parents or primary caregivers, paper-and-pencil and online format 
  • School questionnaire completed by the principal of each school sampled, paper-and-pencil and online format 
  • Teacher questionnaire completed by the reading teachers assessed classes, paper-and-pencil and online format 
  • Student questionnaire
    • PaperPIRLS: paper-and-pencil format 
    • digitalPIRLS: online format

 

Curriculum questionnaire

  • Collected background information about language/reading curricula and education policies to support the encyclopedia chapters 
  • To be completed by NRCs
  • Online format

 

Descriptive encyclopedia chapters

  • One chapter for each participating entity summarizing aspects of the education system and describing the reading curriculum and aspects of reading instruction in the primary grades.
  • Written by experts from ministries of education, research institutes, or institutions of higher education (in countries and benchmarking entities) based on an internationally agreed-upon outline.
TOP
Techniques
  • achievement or student test
  • questionnaire
TOP
Languages
  • Assessment instruments administered in 56 languages for PIRLS in total, i.e., 33 languages for paperPIRLS and 32 languages for digitalPIRLS.
  • Instruments (assessments of achievement and student questionnaires) administered in more than one language in 24 countries and 5 benchmarking entities.
  • The most common languages:
    • English (17 countries and 4 benchmarking entities)
    • Arabic (9 countries and 2 benchmarking entities)
    • Russian (6 countries and 1 benchmarking entity)
TOP
Translation procedures
  • International versions of all assessment instruments (achievement test, context questionnaires, and procedural manuals) were developed in English by TIMSS & PIRLS International Study Center.
  • Participating countries translated and adapted the Instruments into their languages of instruction. 
  • Translations verified by independent linguistic and assessment experts in order to ensure their equivalence with the international version.
TOP
Quality control of operations

Measures during data collection

  • National Research Coordinator (NRC) in each participating country responsible for data collection
  • Standardized survey operations procedures: step-by-step documentation of all operational activities provided with manuals
  • Full-scale field test of all instruments and operational procedures (in each participating country and benchmarking entity)
  • Using random sampling procedures for both field test and data collection concurrently
  • Provision of software tools for supporting activities (e.g., sampling and tracking classes and students, administering school and teacher questionnaires, documenting scoring reliability, creating and checking data files)
  • Intensive training of NRCs, school coordinators, test administrators, etc.
  • International quality control monitors (IQCMs) visited a sample of 15 participating schools in each country during the assessment administration and, in addition, 
    • Five schools in each benchmarking region if the country the region belonged to participated also and
    • Three schools in digitalPIRLS countries to observe the paper bridge booklet administration.
  • National quality control program during data collection 
  • Survey activities questionnaire (SAQ) to be completed by NRCs

 

Measures during data processing and cleaning

  • Material receipt database
  • National adaptation database
  • Testing of all data cleaning programs with simulated data sets
  • Standardized data cleaning rules to verify the accuracy and consistency of the data
  • Four steps of cleaning procedures
  • Checking structure and documentation 
  • Checking identification (ID) variables
  • Checking linkages
  • Context data cleaning
  • Detecting deviations in data patterns and fixing data inaccuracies
  • Three significant interim versions of the data files were checked for database finalization; additional file for digitalPIRLS
  • Repetition of data cleaning and comparison of new data sets with preceding versions
TOP
Sources - Technical documentation
TOP
Other sources
TOP