ICCS 2022 Design

Methodology
Quantitative Study
TOP
Method(s)
  • Overall approach to data collection
    • Survey
  • Specification
    • Cross-sectional (some trend reporting possible)
TOP
Target population

Students

  • The student target population in ICCS consists of all students enrolled in the grade that represents eight years of schooling, counting from the first year of ISCED Level 1, providing the mean age at the time of testing is at least 13.5 years. 
  • Students older than 17 years are not part of the target population.
  • In countries where the average age of students in Grade 8 is less than 13.5 years, Grade 9 was defined as the target population.

 

Teachers

All teachers teaching regular school subjects to students of the target grade (regardless of the subject or the number of hours taught) during the ICCS testing period who have been employed at school since the beginning of the school year.

TOP
Sample design
STUDENTS: STRATIFIED TWO-STAGE CLUSTER SAMPLE DESIGN

First stage: sampling schools

  • Probability proportional to the size of the school (PPS sampling)
  • Optional: stratification of schools according to demographic variables of interest (e.g., region of the country, school type or source of funding, level of urbanization), either explicit or implicit
  • Random-start fixed-interval systematic sampling
  • Simultaneous sampling of schools for field test and main data collection
  • For each sampled school, two replacement schools also sampled.
  • Bridging study
    • Countries transitioning from the paper-based assessment in the previous cycle to the computer-based assessment for the student survey were required to also select an equivalent, smaller bridge sample. 
    • The samples of schools for the core and the ICCS bridging study were drawn simultaneously. 
      • In a first step, a relatively large sample of schools was drawn. 
      • Then, sampled schools were randomly allocated to participate either in the core survey, the ICCS 2022 Bridge Study, or both (if applicable).

 

Second stage (students): sampling classes within schools

  • Only after the sample school has agreed to participate in the study
  • Selection of one or more intact classes from the target grade of each school using systematic random sampling
  • Bridging study
    • An additional step was required for countries that chose to have an overlap between the core survey and the bridge study school samples. 
    • In such cases, some schools were selected to participate in both studies. Sampled classes were randomly allocated to participate in the core survey or in the bridge study.

 

Second stage (teachers): sampling target grade teachers

  • Only after the sample school has agreed to participate in the study
  • Selection of 15–20 teachers from the target grade of each school using systematic random sampling
  • Optional: oversampling of teachers of civic and citizenship-related subjects
TOP
Sample size
INTERNATIONAL Core STUDY

Per country

  • Expected: approx. 3,000 target grade students per country
  • Intended: a minimum of 150 schools, one intact class per school
  • In countries with less than 150 schools, all available schools were included.
  • 15 target grade teachers (or all if the number of target grade teachers was less or equal to 20)

 

Overall

Approximately 80,000 students and 50,000 teachers from about 3500 schools in 22 countries and two benchmarking participants.

 

REGIONAL PARTS - EUROPEAN

Approximately 65,000 students and 40,000 teachers from 18 European countries and two benchmarking participants. 

 

REGIONAL PARTS - LATIN AMERICAN

Approximately 10,000 students in two Latin American countries.

 

Bridging STUDY

Countries transitioning from the paper-based assessment in the previous cycle to the computer-based assessment for the student survey were required to assess an additional sample of at least 1,500 tested students for the bridge data collection.

TOP
Data collection techniques and instruments

Student Civic Knowledge Test

  • Paper-based format (PBA) or computer-based (online or offline) format (CBA)
  • Conventional items (CBA and PBA)
    • Multiple-choice
    • Constructed-response
  • Computer-enhanced items (CBA only)
    • Drag and drop 
    • Dynamic interactive
  • Test booklets
    • PBA: 11 booklets with 3 clusters each (with items that were common across modes)
      • 11 conventional item clusters
      • 55 trend items
      • 66 newly developed test items
    • CBA: 14 booklets with 3 clusters each
      • 11 conventional item clusters and three computer-enhanced clusters
      • 55 trend items
      • 66 newly developed test items
      • 20 computer-enhanced items
    • Bridging study (paper-based): 8 booklets with 3 clusters each
      • Eight of the 11 conventional item clusters
    • Linking mechanism between booklets and cycles
      • 29 items from ICCS 2016
      • 26 items from ICCS 2009
  • Test administration with matrix-sampling of items, i.e., rotated test booklet design

 

Student questionnaires

  • Student questionnaire, paper-based format or computer-based (online or USB) format
  • Student regional questionnaire (Europe, Latin America), print format or computer-based (online or USB) format

 

School and teacher questionnaires

  • Teacher questionnaire to be completed by a target grade teacher, paper-based or online format
  • School questionnaire to be completed by the principal of each sampled school, paper-based or online format

 

National context questionnaires

  • To be completed by the national research coordinator in each participating entity
  • Online format
TOP
Languages
  • Administration of the Student Civic Knowledge Test in 30 different languages
  • The most common languages 
    • Russian (3 countries)
    • English (2 countries)
    • Polish (2 countries)
    • Spanish (2 countries)
TOP
Translation procedures
  • Development of an international version of all instruments in English by the ICCS 2022 International Study Center at ACER
  • Translation into applicable languages of instruction by participating entities
  • Translation verification by linguistic and assessment experts to ensure equivalence with the international version
TOP
Quality control of operations

Measures during data collection

  • Participants were responsible for data collection within their own respective territories.
  • Standardized survey operation procedures: step-by-step documentation of all operational activities provided with manuals
  • Full-scale field test of all instruments and operational procedures, (in each participating country and benchmarking entity)
  • Provision of software tools for supporting activities (e.g., sampling and tracking classes and students, administering school and teacher questionnaires, documenting scoring reliability, creating and checking data files)
  • Training, e.g., for national research coordinators (NRCs) and their staff, school coordinators, and test administrators
  • School visits conducted by international quality observers (IQOs) during test administration (15 school visits required. In countries who were transitioning from PBA in 2016 to CBA in 2022, IQOs were required to visit three additional schools to observe the paper booklet administration as part of the bridging study.)
  • National quality control program
  • Survey activities questionnaire (SAQ) completed by NRCs

 

Measures during data processing and cleaning

  • Testing of all data cleaning programs with simulated data sets
  • Data and documents receipt database maintained by IEA Hamburg
  • Standardized cleaning process and documentation of any necessary data recoding
  • National adaptation database
  • Repetition of data cleaning and comparison of the new data sets with the preceding versions
  • Finally, identification of irregularities in data patterns and correction of data errors
TOP
Sources - Technical documentation
TOP
Other sources
TOP