PIRLS 2016 Design

Methodology
  • International large-scale sample survey of student achievement and educational context
  • Monitors trends by reporting results from successive cycles on a common achievement scale
  • Predominantly quantitative, with qualitative information presented in descriptive country chapters in the PIRLS encyclopedia
TOP
Method(s)
  • Overall data collection approach:
    • Proctored assessment of student achievement
    • Self-administered surveys for students, parents, teachers, and school principals
TOP
Target population

School children in the participating countries in grades representing 4 years of formal schooling

TOP
Sample design
Stratified two-stage cluster sample design

First stage: sampling schools

  • Probability proportional to the size of the school (PPS)
  • Stratifying schools (optional):
    • According to important (demographic) variables (e.g., region of the country, school type or source of funding or languages of instruction)
    • Two forms possible: explicit or implicit stratification
  • Random-start fixed-interval systematic sampling
  • Schools sampled at the same time for field test and main data collection
  • Sampling of two replacement schools for each school sampled (main data collection only)

 

Second stage: sampling classes within schools

  • Only after sampled schools agreed to participate in the study
  • One or more intact classes from the target grade of each school, selected using systematic random sampling

 

General notes

  • School sampling conducted by Statistics Canada, class sampling by participating countries
  • All sampling activities monitored and documented by Statistics Canada and IEA Hamburg staff
TOP
Sample size

Per country

  • School sample: minimum 150–200 schools
  • Student sample: about 4,500 students

 

In total

Approximately 340,000 students

TOP
Data collection techniques and instruments

Student Assessment

  • Written format – PIRLS and PIRLS Literacy
  • Computer-based - ePIRLS
  • Reading passages/online reading tasks and accompanying questions:
    • Multiple-choice (at least half the number of total points)
    • Constructed response
  • Achievement items
    • 12 item blocks (passages) for PIRLS, 12 item blocks for PIRLS Literacy, and 5 ePIRLS tasks
    • 16 test booklets for PIRLS, 16 test booklets for PIRLS Literacy, and 12 task combinations for ePIRLS
      • Items per assessment
        • PIRLS: 175 items
        • PIRLS Literacy: 183 items
        • ePIRLS: 91 items
    • Linking mechanisms between booklets and between cycles
      • PIRLS and PIRLS Literacy have 4 common item blocks/passages
      • New PIRLS passages and items in 2016: 4 blocks
      • 6 PIRLS blocks are trend passages and (unreleased) items from PIRLS 2001, PIRLS 2006, and 2011
      • New PIRLS Literacy passages and items in 2016: 6 blocks
      • 4 PIRLS Literacy blocks are trend passages and (unreleased) items from prePIRLS 2011
  • Matrix sampling of passages/items (rotated test booklet design)

 

Background questionnaires

  • Student questionnaire, in print format; modular design
  • Learning to Read survey (home questionnaire), to be completed by students’ parents or primary caregivers, print and online format (online for some countries)
  • Teacher questionnaire, to be completed by the reading teachers assessed classes, print or online
  • School questionnaire, to be completed by the principal of each school sampled, in print or online format
  • ePIRLS student questionnaire, in computer-based format

 

Curriculum questionnaire

  • To be completed by NRCs
  • Modular design
  • Online format

 

Descriptive encyclopedia chapters

  • One chapter for each participating entity
  • Written by experts from ministries of education, research institutes, or institutions of higher education (in countries and benchmarking entities) based on an internationally agreed-upon outline.
TOP
Techniques
  • achievement or student test
  • questionnaire
TOP
Languages
  • Assessment instruments administered in 40 languages for PIRLS, 10 languages for PIRLS Literacy, and 15 languages for ePIRLS.
  • Instruments (assessments of achievement and student questionnaires) administered in two or more languages in 24 countries and four benchmarking entities.
  • The most common languages: English (17 countries), Arabic (7 countries)
TOP
Translation procedures
  • International version of all assessment instruments developed in English by TIMSS & PIRLS International Study Center.
  • Instruments then translated and adapted by participating countries into their languages of instruction.
  • Translations verified by independent linguistic and assessment experts in order to ensure their equivalence with the international version.
TOP
Quality control of operations

During data collection

  • National Research Coordinator (NRC) in each participating country responsible for data collection
  • Standardized survey operations procedures: step-by-step documentation of all operational activities provided with manuals
  • Full-scale field test of all instruments and operational procedures (in each participating country and benchmarking entity)
  • Provision of software tools for supporting activities (e.g., sampling and tracking classes and students, administering school and teacher questionnaires, documenting scoring reliability, creating and checking data files)
  • Training of NRCs and their staff, school coordinators, test administrators, etc.
  • School visits conducted by international quality control monitors (IQCMs) during test administration (15 schools per country)
  • National quality control program
  • Survey activities questionnaire (SAQ) to be completed by NRCs

 

During data processing and cleaning

  • Testing of all data cleaning programs with simulated data sets
  • Material receipt database
  • National adaptation database
  • Standardized cleaning process
  • Repetition of data cleaning and comparison of new data sets with preceding versions
  • Identification of irregularities in data patterns and correction of data errors
TOP
Sources - Technical documentation
TOP
Other sources
TOP