TALIS Starting Strong 2024 Design

Methodology
Quantitative Study
TOP
Method(s)
  • Overall approach to data collection: self-administered cross-sectional survey, with questionnaires administered online
  • Specification: trend study
TOP
Target population
  • Staff and leaders in ISCED Level 02 ECEC centers that cater to children from three years of age up to the time they enter primary education
  • Staff and leaders in ECEC settings that cater to children under three years of age (U3)
TOP
Sample design
Stratified two-stage probability sample design

Stratification on basis of nationally relevant criteria (e.g., different types of centers, geography, urbanization level, source of funding, language of instruction)


First stage: sampling of centers

  • Systematic random sampling with probability proportional to size (PPS) within explicit strata was used in most cases; systematic equal probability random sampling was used where no measure of size (MOS) was available.
  • Samples for field trial (FT) and main study (MS) were selected at same time to avoid sample overlap.
  • Replacement centers were identified at the time of sample selection (two for each sampled center) if available.


Second stage: sampling of staff

  • Selected randomly from the list of in-scope staff for each of the selected centers.
  • Center leaders were asked to complete the leader questionnaire.
TOP
Sample size

Intended

  • Nominally, 180 centers
  • Nominally, at least eight staff members per center; in the case of fewer staff members, all were selected. A total of 400 effective staff was set to adjust for the clustering effect within selected centers

 

Achieved

  • ISCED level 02: more than 16,000 staff and more than 3,000 leaders in 15 countries/territories
  • U3: approximately 4,900 staff and more than 1,000 leaders in 8 countries/territories
  • Note: Canada (New Brunswick) and Canada (Quebec) were each counted as countries/territories.
TOP
Data collection techniques and instruments

Field trial questionnaire

  • Built-in experiments designed to test various question formats and wording to identify which question version had the better psychometric properties were included.

 

Main survey questionnaire

  • Three questionnaire types: leader questionnaire (LQ), staff questionnaire (SQ), and a combined center questionnaire (CQ). Each was accompanied by a cover letter.
  • Although the instruments were designed to accommodate both center-based and home settings, countries had the option of compiling two separate versions of the questionnaires so as to adjust their wording more closely to the organizational setting of the sampled centers.
TOP
Languages

The four questionnaires were administered in 20 different languages. The most common languages were

  • Spanish (3 countries)
  • Arabic (2 countries)
  • Swedish (2 countries)
TOP
Translation procedures
  • Countries translated all TALIS 2024 Starting Strong survey materials into local languages while preserving the meaning of the international source version.
  • Translations had to follow target-language rules, fit the national education context, and remain conceptually equivalent to the source text.
  • A glossary supported translators by defining key terms, explaining concepts, and preventing inaccurate literal translations.
  • Translators made all translations, implemented required adaptations, and documented all adaptations in the IEA StudyExpert.
  • Independent reviewers checked translations for accuracy, readability, consistency, and suitability for respondents.
  • Translators incorporated reviewer feedback, while the National Project Managers (NPMs) resolved disagreements and approved final versions.
  • TALIS Starting Strong 2018 trend items had to retain previous translations to ensure comparability across survey cycles.
  • Any proposed changes to trend translations required International Study Center (ISC) review and could affect trend comparability.
  • International translation verification was conducted by cApStAn following TALIS 2024 Technical Standards.
  • Verification focused on clarity, completeness, consistency, semantic equivalence, appropriate adaptations, and correct formatting.
  • Verification comments were reviewed and addressed by NPMs; unresolved disagreements could be referred to the ISC and, if necessary, the translation referee.
TOP
Quality control of operations

Measures during data collection

  • Purpose of the quality control program
    • Ensured valid, reliable and internationally comparable data collection across countries and territories
    • Monitored adherence to TALIS technical standards and survey procedures
    • Documented survey implementation quality and identified issues
    • Supported consistent administration across languages, countries, and modes (online/paper)
  • Core components of the program
    • International Quality Observation Programme (IQOs)
      • Independent observers monitored survey administration during the main survey
      • Activities included
        • Visiting national centres and interviewing National Project Managers (NPMs)
        • Reviewing translated manuals against international versions
        • Interviewing ECEC co-ordinators
        • Assessing survey preparation, communication with ECEC settings, adherence to procedures, and implementation challenges.
    • Additional Quality Control Questions
      • Administered to around 20% of the subsample
      • Focused on respondent selection, completion time, confidentiality, and challenges encountered during the survey
    • Survey Activities Questionnaire (SAQ)
      • Completed by NPMs after data collection to document survey implementation
      • Online questionnaire
      • Covered sampling, recruitment, translation, adaptation, administration procedures, technical issues, and suggestions for future improvements
    • National Quality Observation Programmes
      • Conducted by national centres during field trial and main survey
      • ECEC visits and observations
      • Focused on administration procedures and communication with ECEC settings

 

Measures during data processing and cleaning

  • National centers were responsible for ensuring that all collected data were complete, accurate, and correctly documented before submission to the International Study Centre (ISC).
  • Standardized IEA software tools (e.g. WinW3S, StudyExpert and Data Management Expert) were used to prepare survey instruments, support sampling, track participation, capture data and validate entries. 
  • Data from paper questionnaires had to be entered exactly as recorded by respondents (“as is”), without interpretation or correction during the data-entry stage.
  • Automated validation procedures identified problems such as invalid identification numbers, inconsistent records and out-of-range values during data entry.
  • A data-entry error rate above 1% triggered further review and potential re-entry of the data to ensure data quality.
  • Before submission, national centers performed mandatory verification checks to identify duplicate identifiers, invalid codes and missing information, and resolved issues by consulting the original questionnaires.
  • After submission, the ISC conducted further quality control through structured processes including data integration, structural checks, identification and linkage cleaning, and automated consistency checks.
  • Logical inconsistencies, implausible responses and conflicting answers were corrected or recoded according to standardized cleaning rules applied consistently across all participating countries.
  • All data corrections and cleaning actions were documented and reviewed in collaboration with national centers before finalization of the international datasets.
  • Multiple interim data versions were released to national centers and the OECD Secretariat to allow additional validation and plausibility checks before producing the final international database. 
  • Key confidentiality measures
    • Direct personal identifiers (such as names in tracking forms) were removed or redacted before data were submitted for international processing.
    • Respondent identifiers used during data collection were replaced with randomly generated IDs to prevent identification of individuals. 
    • Sensitive or identifying variables and some quality control variables were removed or suppressed in public datasets. 
    • Two versions of the international database were produced: a restricted-use file for authorized users under confidentiality agreements and a public-use file with reduced detail to minimize re-identification risks. 
    • Disclosure avoidance measures such as variable suppression, aggregation, and ID scrambling were applied in accordance with OECD data protection rules and relevant national legislation.
TOP
Sources - Technical documentation
TOP