TIMSS 2019 Design

Methodology
  • International large-scale sample-based assessment of student achievement and survey of the educational context
  • Monitors trends in student achievement since 1995 by reporting results from successive cycles on a common achievement scale
  • Predominantly quantitative, with qualitative information presented in descriptive country chapters in the TIMSS 2019 encyclopedia
TOP
Method(s)
  • Overall approach to data collection
    • Proctored assessment of student achievement
    • Self-administered surveys for students, parents, teachers, and school principals
  • Specification
    • Cross-sectional
    • Some trend reporting possible with previous cycles
TOP
Target population
  • Grade 4: All students enrolled in the grade that represented four years of schooling, counting from the first year of ISCED Level 1, provided that the mean age at the time of testing was at least 9.5 years.
  • Grade 8: All students enrolled in the grade that represented eight years of schooling, counting from the first year of ISCED Level 1, provided that the mean age at the time of testing was at least 13.5 years.
  • In countries where Grade 4 or Grade 8 students were expected to find the TIMSS assessments too difficult, Grade 6 or Grade 9, respectively, were defined as the target populations.
TOP
Sample design
Stratified two-stage cluster sampling design 

First stage: sampling schools

  • Selection probability proportional to the size of the school (PPS)
  • Optional: stratification of schools according to (demographic) variables of interest (e.g., school type or source of funding, level of urbanization, region of the country), either explicit or implicit
  • Random-start fixed-interval systematic sampling
  • Schools sampled at the same time for field trial and main data collection
  • For each sampled school, two replacement schools were assigned where possible

 

Second stage (students): sampling classes

Within schools agreeing to participate:

  • Systematic random sampling was used to select one or more classes from the sampled school in the target grade.
  • Each class had an equal selection probability within the school.
  • If a selected class was smaller than half of the average class size, a pseudo-class was created with other classes.
  • All students within a selected class were asked to participate.

 

For eTIMSS countries: bridge sample

  • Countries participating in eTIMSS were required to also select an equivalent, smaller bridge sample by sampling one additional class from a subset of the sampled schools, by selecting a distinct sample of schools, or by a combination of both strategies.

 

General notes

  • Sampling of schools was conducted by RTI and the sampling team at IEA Hamburg.
  • Sampling procedures within the center/school were carried out by the national study centers, using the Within-school Sampling Software for Windows (WinW3S) provided by IEA.
  • All sampling activities were monitored and documented by Statistics Canada and IEA Hamburg staff.
TOP
Sample size

Intended per country and target grade

  • Approx. 4,000 assessed students
  • A minimum of 150 schools (in countries with fewer than 150 schools, all available schools were included)
  • In each sampled school, selection of one class
  • Required effective sample size: minimum of 400 students
  • Bridging study: countries transitioning to eTIMSS were required to assess an additional sample of at least 1,500 tested students for the bridge data collection.

 

Total achieved

  • Grade 4: Approximately 330,000 students, 310,000 parents, 22,000 teachers, and 11,000 schools from 64 education systems (58 countries and 6 benchmarking entities) participated.
  • Grade 8: Approximately 250,000 students, 30,000 teachers, and 8,000 schools from 46 education systems (39 countries and 7 benchmarking entities) participated.
TOP
Data collection techniques and instruments

Student assessments in mathematics and science

  • Formats
    • Written format (paperTIMSS) and digital format (eTIMSS)
    • Countries participating in eTIMSS could choose from the following delivery methods: USB delivery, tablet delivery, local server method
  • Types of items
    • paperTIMSS and eTIMSS
      • Single and multiple selection of responses
      • Constructed responses (i.e., writing or typing, drawing, dragging and dropping in eTIMSS, matching items in paperTIMSS)
    • eTIMSS (additionally): PSI, i.e., interactive and situational simulation tasks
  • Item organized in item blocks and then into achievement booklets (for paperTIMSS) and item block combinations or digital “booklets” (for eTIMSS)
    • Booklets
      • A total of 14 test booklets per grade
      • Four blocks (2 mathematics, 2 science, in counterbalanced order) per booklet and each item block appearing in two booklets
    • Blocks
      • A total of 28 item blocks per grade (14 with mathematics items, 14 with science items)
      • Items per block
        • Grade 4: 10–14 items in each block
        • Grade 8: 12–18 items in each block
      • eTIMSS contained 4 extra blocks to accommodate PSI tasks
    • Items per assessment in Grade 4
      • paperTIMSS and eTIMSS: 175 items in each of mathematics and science
      • Less difficult mathematics: 179 mathematics items (including less difficult and regular items)
      • PSI: 39 mathematics items (from three PSIs) and 19 science items (from two PSIs)
      • Bridge booklets (trend items only): 92 mathematics items and 111 science items
    • Items per assessment in Grade 8
      • paperTIMSS and eTIMSS: 211 items in mathematics and 220 in science
      • PSI: 25 mathematics items (from three PSIs) and 29 science items (from two PSIs)
      • Bridge booklets (trend items only): 117 mathematics and 122 science items
    • As far as possible, the distribution of items across content and cognitive domains within each block matches the distribution across the item pool overall.
    • Matrix sampling of items (rotated test booklet design)

 

  • Linking mechanisms
    • Between cycles
      • New items in 2019: 12 blocks (6 mathematics and 6 science)
      • The other 16 blocks (8 mathematics and 8 science) are (unreleased) trend items; for each subject, all 8 were administered in 2015 and 3 of them also in 2011.
    • Between regular and less difficult mathematics:  In the less difficult version, one third of the items were from the regular assessment and two-thirds were less difficult items.
    • Between paperTIMSS and eTIMSS: Bridge (in order to control for mode effects):
      • eTIMSS countries administered the complete computer-based eTIMSS 2019 assessment as well as a smaller, paper-based version of the trend items.
      • That is, eTIMSS countries re-administered their eight blocks of trend items from 2015 in paperTIMSS format.
TOP
Languages
  • Assessment instruments administered in 50 languages
  • Instruments administered in two or more languages in 31 countries and four benchmarking entities
  • The most common languages:
    • English (24 countries)
    • Arabic (10 countries)
TOP
Translation procedures
  • International version of all assessment instruments are developed in English by TIMSS & PIRLS International Study Center.
  • Instruments are then translated by participating countries into their languages of instruction.
  • Translations are subsequently verified by independent linguistic and assessment experts in order to ensure equivalence with the international version.
  • Following translation verification, all draft instruments undergo layout verification by the TIMSS & PIRLS International Study Center to ensure they are internationally comparable and of high quality.
  • For the eTIMSS achievement materials, the procedures for translation and verification took place in the eTIMSS Online Translation system, part of IEA Hamburg’s eAssessment system.
TOP
Quality control of operations

Measures during data collection

  • NRC in each participating country responsible for data collection
  • Standardized survey operations procedures: step-by-step documentation of all operational activities provided with manuals, including eTIMSS versions of manuals
  • Full-scale field test of all instruments and operational procedures, including eTIMSS instruments and procedures (in each participating country and benchmarking entity)
  • Provision of software tools for supporting activities (e.g., sampling and tracking classes and students, administering school, teacher and home questionnaires online, documenting scoring reliability, creating and checking data files)
  • Training of NRCs and their staff, school coordinators, test administrators, etc.
  • School visits conducted by international quality control monitors (IQCMs) during test administration (15 schools per grade and country, 5 additional schools in benchmarking entities and 3 additional schools for paper bridge booklet administration in eTIMSS countries)
  • National quality control program
  • Survey activities questionnaire (SAQ) to be completed by NRCs

 

Measures during data processing and cleaning

  • Testing of all data cleaning programs with simulated data sets
  • Data and documents receipt database maintained by IEA Hamburg
  • Standardized cleaning process and documentation of any necessary data recoding
  • National adaptation database
  • Repetition of data cleaning and comparison of new data sets with preceding versions
  • Identification of irregularities in data patterns and correction of data errors

 

TOP
Sources - Technical documentation
TOP
Other sources
TOP