TIMSS 2023 Design

Methodology
  • International large-scale sample survey of students’ mathematics and science achievement, and of their educational contexts – community, home, school, classroom, and student
  • Monitoring of trends since 1995 by reporting results from successive cycles on a common achievement scale
  • Predominantly quantitative, with qualitative information presented in descriptive country chapters in the TIMSS 2023 encyclopedia
TOP
Methodology
Quantitative Study
TOP
Method(s)

Overall data collection approach:

  • Proctored assessment of student achievement and student questionnaire
  • Self-administered surveys for parents, teachers, and school principals
TOP
Target population
  • Fourth grade: All students enrolled in the grade that represents four years of schooling counting from the first year of ISCED Level 1, provided that the mean age at the time of testing is at least 9.5 years.
  • Eighth grade: All students enrolled in the grade that represents eight years of schooling counting from the first year of ISCED Level 1, provided that the mean age at the time of testing is at least 13.5 years.
TOP
Sample design
Stratified two-stage cluster sample design

First stage: Sampling schools

  • Probability proportional to the size of the school (PPS)
  • Stratifying schools (optional):
    • According to important (demographic) variables (e.g., region of the country, school type or source of funding, level of urbanization)
    • Can take two forms: explicit or implicit stratification
  • Random-start fixed-interval systematic sampling
  • Schools sampled at the same time for field test and main data collection
  • Sampling of two replacement schools for each school sampled (main data collection only)

 

Second stage: Sampling classes within schools

  • One or more intact classes from the target grade of each school selected using systematic random sampling
  • Probability inversely proportional to school size
  • Only after sampled schools have agreed to participate in the study
TOP
Sample size
  • For most countries, the TIMSS precision requirements are met with a school sample of 150 schools and a student sample designed to yield at least 4,000 assessed students for each target grade. 
  • Across all countries and both grades, more than 650,000 students (almost 360,000 students in fourth grade and more than 290,000 students in eighth grade)
TOP
Data collection techniques and instruments

Large-scale assessments of student achievement

  • TIMSS uses a matrix sampling approach that involves packaging the entire assessment pool of mathematics and science items at each grade level into a set of student achievement booklets or test forms (a booklet equivalent in the context of computer-based assessments).
  • TIMSS 2023 completed the transition to a fully digital assessment
    • Available in Grade 4 and Grade 8
      • Mathematics and science, Grade 4: 14 test forms 
      • Mathematics and science, Grade 8: 14 test forms
    • The 14 mathematics and 14 science item blocks at each grade were arranged into 14 forms with two mathematics and two science blocks each.
    • Group adaptive design
      • Available in digital format only
      • Three levels of item block difficulty (difficult, medium, and easy) were combined into two levels of test form difficulty (i.e., each student being assigned a block of easy and a block of medium items, or a block of medium and a block of difficult items)
      • Each country administered the entire assessment with the balance of more difficult and less difficult booklets varying with the mathematics and science achievement level of the students in the country.
    • Each student was randomly assigned a test form, consisting of two mathematics item blocks and two science item blocks. 

 

  • paperTIMSS
    • Administered in a small number of countries for whom the fully digital TIMSS may not be logistically feasible
    • Regular paperTIMSS
      • Included only the secured trend items from the TIMSS 2019 paper assessment (and will provide only overall mathematics and science achievement results)
      • Available in the two grades
        • Mathematics and science, Grade 4: 8 booklets
        • Mathematics and science, Grade 8: 8 booklets
    • Less difficult paperTIMSS
      • Included only the secured trend items from the TIMSS 2019 fourth grade less difficult mathematics assessment
      • Mathematics, Grade 4: 8 booklets

 

  • For the first time, TIMSS 2023 offered a longitudinal option in which the same students that participate in TIMSS 2023 could take the same assessment one year later. This optional extension provided a more granular look at education progress over the course of one year.

 

Background questionnaires

  • Student questionnaire
  • Teacher questionnaire
  • School questionnaire
  • Early Learning Survey (also called the “home questionnaire”), grade 4 only
  • Curriculum questionnaire (completed by National Research Coordinators)

 

Descriptive encyclopedia chapters

Each country and benchmarking participant prepares a chapter describing the structure of the country’s education system, the mathematics and science curricula in the primary and secondary grades, the teacher education requirements, the types of examinations and assessments employed, policies associated with their mathematics and science curricula, school organizational approaches, and instructional practices.

TOP
Languages
  • At the fourth grade, the assessment instruments were translated into 92 language versions and the context questionnaires into 96 language versions. 
  • At the eighth grade, the assessment instruments were translated into 66 language versions and the context questionnaires into 65 language versions. 
  • The most common languages:
    • English (21 countries and 5 benchmarking entities)
    • Arabic (11 countries and 3 benchmarking entities)
    • French (5 countries and 2 benchmarking entities)
TOP
Translation procedures
  • All participating countries follow a set of standardized procedures for translating and adapting their national assessment instruments and any related directions or administrative screens.
  • IEA Amsterdam manages the translation verification process, which involves external verifiers comparing each national version to the international text and providing detailed feedback for the National Research Coordinators (NRCs) to improve the accuracy and comparability of their national translations. 
  • Verifiers at the TIMSS & PIRLS International Study Center conduct layout and adaptations verification of national instruments, which includes checking that all national instruments match the international format and that any national adaptations to the 
TOP
Quality control of operations

Measures during data collection

  • In each TIMSS assessment cycle, standardized assessment materials and survey operations procedures are developed and adapted from previous cycles so that the data collection meets the highest standards. 
  • As part of the quality assurance program, International Quality Control Monitors (IQCMs) are selected and appointed in each TIMSS country to implement the program by visiting a sample of participating schools to observe the TIMSS administration. 

 

Measures during data processing and cleaning

  • IEA Hamburg is responsible for checking the data submitted by each country, 
    • applying standardized data cleaning rules to verify the accuracy and consistency of the data, and 
    • documenting any deviations from the international file structure. 
  • This process also includes 
    • importing and cleaning the data collected by the TIMSS Player that delivers the digital assessment to students,
    • transferring student response data that require human scoring and student response data for automated scoring into their corresponding systems, and 
    • importing item scores for further processing.
TOP
Sources - Technical documentation
TOP
Other sources
TOP