ICCS 2016 Design

Methodology
Quantitative Study
TOP
Method(s)
  • Overall approach to data collection
    • Survey
  • Specification
    • Cross-sectional (some trend reporting possible)
TOP
Target population

Students

  • Students in their eighth year of schooling
  • On average, including students who are approximately 14 years of age typically, Grade 8 students, provided that the average age of students in this cohort is 13.5 years or above.
  • In countries where the average age of students in Grade 8 is less than 13.5 years, Grade 9 was defined as the target population.

 

Teachers

All teachers teaching regular school subjects to students of the target grade (regardless of the subject or the number of hours taught) during the ICCS testing period who have been employed at school since the beginning of the school year.

TOP
Sample design
Students: Stratified two-stage cluster sample design

First stage: sampling schools

  • Probability proportional to the size of the school (PPS sampling)
  • Optional: stratification of schools according to demographic variables of interest (e.g., region of the country, school type or source of funding, level of urbanization), either explicit or implicit
  • Random-start fixed-interval systematic sampling
  • Simultaneous sampling of schools for field test and main data collection
  • For each sampled school, two replacement schools also sampled.

 

Second stage (students): sampling classes within schools

  • Only after the sample school has agreed to participate in the study
  • Selection of one or more intact classes from the target grade of each school using systematic random sampling

 

Second stage (teachers): sampling target grade teachers

  • Only after the sample school has agreed to participate in the study
  • Selection of 15–20 teachers from the target grade of each school using systematic random sampling
  • Optional: oversampling of teachers of civic and citizenship-related subjects
TOP
Sample size
International study

Per country

  • Expected: approx. 3,000–4,500 students per country
  • Intended: a minimum of 150 schools, one class in each school
  • In countries with less than 150 schools, all available schools were included.

 

Overall

Approximately 94,000 students and 37,000 teachers from about 3,800 schools in 24 countries participated.

 

Regional parts - European

Approximately 53,000 students in 15 education systems

 
Regional parts - Latin American

Approximately 25,000 students in 5 education systems

TOP
Data collection techniques and instruments

Student tests of civic knowledge

  • Written format
  • Two types of questions
    • Multiple-choice
    • Constructed-response (6 questions)
  • Test booklets
    • 8 booklets with 3 blocks each
      • 88 items
      • 11 items per block
    • Linking mechanism between booklets and cycles
      • 46 new items in 2016
      • 42 items from ICCS 2009
  • Test administration with matrix-sampling of items, i.e., rotated test booklet design

 

Questionnaires

  • Student questionnaire, print format
  • Student regional questionnaire (Europe, Latin America), print format
  • Teacher questionnaire to be completed by a target grade teacher, print or online
  • School questionnaire to be completed by the principal of each sampled school, print or online format

 

National context questionnaires

  • To be completed by the national research coordinator in each participating entity
  • Online format

 

TOP
Techniques
  • achievement or student test
  • questionnaire
TOP
Translation procedures
  • Development of an international version of all assessment instruments in English by the ICCS 2016 International Study Center
  • Translation into applicable languages of instruction by participating entities
  • Translation verification by linguistic and assessment experts in order to ensure equivalence with the international version

 

TOP
Quality control of operations

Measures during data collection

  • Participants were responsible for data collection within their own respective territories.
  • Standardized survey operation procedures: step-by-step documentation of all operational activities provided with manuals
  • Full-scale field test of all instruments and operational procedures (in each participating country and entity, with some countries being allowed a reduced field trial)
  • Provision of software tools for supporting activities (e.g., sampling and tracking classes and students, administering school and teacher questionnaires, documenting scoring reliability, creating and checking data files)
  • Training, e.g., for national research coordinators (NRCs) and their staff, school coordinators, and test administrators
  • School visits conducted by international quality observers (IQOs) during test administration (15 schools per country)
  • National quality control program
  • Survey activities questionnaire (SAQ) completed by NRCs

 

Measures during data processing and cleaning

  • Testing of all data cleaning programs with simulated data sets
  • Material receipt database
  • National adaptation database
  • Standardized cleaning process
  • Repetition of data cleaning and comparison of the new data sets with the preceding version
  • Finally, identification of irregularities in data patterns and correction of the data
TOP
Sources - Technical documentation
TOP
Other sources
TOP