ICILS 2023 Design
Methodology
Quantitative Study
Method(s)
- Overall approach to data collection:
- Proctored assessment and self-administered surveys
- Specification:
- Cross-sectional
- Some trend reporting possible with previous cycles
Target population
- Students: The ICILS target population comprises students in their eighth year of schooling. In most education systems, the eighth year of schooling is Grade 8, provided that the average age of students is 13.5 years or above. In education systems where the average age in Grade 8 is below 13.5, Grade 9 is defined as the ICILS target population.
- Teachers: The population for the ICILS teacher survey was defined as all teachers teaching regular school subjects to the students in the target grade at each school sampled.
- Schools: The population for the ICILS school-level survey comprised schools at which target grade students were enrolled. For this, ICILS administered separate questionnaires to principals and designated ICT-coordinators in each school.
Sample design
STRATIFIED TWO-STAGE CLUSTER SAMPLE DESIGN
First Stage (Schools)
- Random selection of schools with a probability proportional to size as measured by the number of students enrolled in a school.
- Each country was advised for a minimum sample size of 150 schools except some small education systems where all schools were included in the survey.
- Optional: stratification of schools according to demographic variables of interest (e.g., region of the country, school type or source of funding, level of urbanization), either explicit or implicit.
- Random-start fixed-interval systematic sampling.
- Simultaneous sampling of schools for field test and main data collection were possible.
- Optional: overlap control to TIMSS 2023 main survey sample or national assessments, aiming at minimum or maximum overlap.
- For each sampled school, two replacement schools were assigned.
- After a sampled (or its replacement) school had agreed to participate, the second sampling stage was performed.
Second Stage (Students)
- Selection of one or more intact classes from the target grade of each school using systematic random sampling.
- In case the school also participated in TIMSS 2023 either different classes were randomly selected for both studies, or students had a two weeks break between the assessments.
- All students were asked to participate within the selected classes.
Second Stage (Teachers)
- Fifteen teachers were selected randomly from all teachers teaching in the target grade at each sampled school.
- In schools with fewer than 20 teachers, all teachers were invited to participate in the survey.
Sample size
Per Country (intended)
- School sample: a minimum of 150 schools
- In countries with less than 150 schools, all available schools were included.
- For benchmark participants, a school sample size of 150 school was advised.
- Student sample: approx. 3000 students (20 per school)
- Teacher sample: 15 per school (minimum)
Total (achieved)
- Students
- CIL: more than 132,600
- CT: more than 85,200
- Teachers: almost 61,000
- Schools: almost 5,300
Data collection techniques and instruments
Student CIL and CT assessments
- The tests were embedded within modules. In total, there were five 30- minute CIL modules and two 25-minute CT modules.
- Each student completed 2 out of 7 CIL modules. The order and selection of the CIL modules were assigned randomly.
- In countries participating in the CT option, each student completed 2 out of 4 modules following completion of the CIL test and student questionnaire. The order and selection of CT modules were assigned randomly.
Student questionnaire
- Following the CIL assessment, each student completed a 30-minute student questionnaire.
- The student questionnaire was used to investigate student engagement with ICT and included questions related to experience and use of ICT, their attitudes towards the use of computers and ICT and background characteristics.
Questionnaires for teachers, ICT coordinators, and principals
- ICILS 2023 also included a teacher questionnaire, which was completed by 15 randomly selected Grade 8 teachers that teach in the target grade in each sampled school.
- The questions were related to familiarity of teachers with ICT, their use of ICT in educational activities and approaches to teaching, their perceptions of ICT in schools, learning to use ICT in teaching, and their background characteristics.
- The ICT coordinators also had to complete a 15-minute questionnaire, which included questions related to the resources of ICT in school, ICT use in school, ICT technical support, and provisions for professional development in ICT.
- The principal of the respective schools completed a 15-minute questionnaire, answering questions related to school characteristics, policies, procedures, school leadership for ICT and priorities for ICT. An optional questionnaire with questions about generative artificial intelligence was included in ICILS 2023 for principals as well.
National coordinator questionnaires
- National research coordinators (NRCs) collected data from experts in a national context survey (NCS).
- The survey was used for gathering information about the structure of the education system and systematic descriptions of policy and practice in the use of ICT in school education.
Techniques
- achievement or student test
- national contexts survey
- questionnaire
Languages
- Administration of national study instruments in 50 languages. Of the 35 participating entities, 10 administered the instruments in more than one language.
- The most common languages
- English (5 countries)
- Spanish (3 countries)
- Russian (3 countries)
- German (3 countries)
Translation procedures
- Development of an international version of all assessment instruments in English by the ICILS International Study Center
- Translation into applicable languages of instruction by national research coordinators (NRCs)
- Translation verification by linguistic and assessment experts in order to ensure equivalence with the international version
Quality control of operations
Measures during data collection
- Participants were responsible for data collection within their own respective territories.
- Standardized survey operation procedures: step-by-step documentation of all operational activities provided with the operation manuals
- Full-scale field test of all instruments and operational procedures (in each participating country and entity)
- Provision of software tools for supporting activities (e.g., sampling and tracking classes and students; administering school and teacher questionnaires; documenting scoring reliability; creating, and checking data files)
- Training, e.g., for national research coordinators (NRCs) and their staff, for school coordinators, and test administrators
- School visits conducted by international quality observers (IQOs) during test administration (at 15 schools per grade and country)
- Training, e.g., for IQOs for procedures related to arranging visits to schools, interviewing school coordinator and test administrator, and reporting of these activities
- Documentation of data collection activities by IQOs to determine whether the ICILS assessment was administered in compliance with the standardized procedures
- National quality control program by national study centers that involved national quality officers (NQOs) to visit 10 percent of the sampled schools (minimum 15 schools) during testing
- Survey activities questionnaire (SAQ) completed by NRCs
Measures during data processing and cleaning
- Testing of all data cleaning programs with simulated data sets
- Registering all incoming data and documents in a specific database recording the date of arrival
- All systematic data recoding were documented.
- Iterative data cleaning process on each national dataset until all data were consistent and comparable
- Standardized cleaning process
- Repetition of data cleaning and comparison of the new data sets with the preceding version
- Finally, identification of irregularities in data patterns and correction.
Other sources