STEP Design
Methodology
Quantitative Study
Method(s)
- Overall approach to data collection
- Background questionnaire: paper-and-pencil or computer-assisted
- Literacy assessment: always paper-and-pencil
- Specification
- Cross-sectional
Target population
Household survey
- Household level: households in urban and rural areas
- Individual level: population ages 15 to 64 in each country
- Countries were given the option to reduce target population to only urban areas.
Sample design
Household survey
- Household level
- Households were selected on the basis of the national sample designs.
- Individual level
- Participating countries were advised that the target population should be representative of at least 95 percent of the urban working-age population.
- Each country’s formal definition of “urban” was adopted for the household survey.
- A household survey random selection protocol was given to the countries
- Dwelling units as the households
- All household members were listed
- Eligible members were recorded
- The individual respondent was selected from the household using a random number table
- Sample sizes were driven by:
- Scope of the survey research
- Minimum number of reading literacy booklets required to make generalizations
- Specialized stratification for minority populations was utilized if needed
Sample size
Household survey: 2,500–4000 individuals per country
Data collection techniques and instruments
Sequence
- First, a relatively short household questionnaire was administered (by an enumerator using paper-and-pencil or computer-assisted data collection)
- Then, interviewers randomly selected an individual within each household to complete the individual questionnaire.
- This respondent was then asked to participate in a literacy assessment.
- Depending on the outcome of the basic assessment (and on whether or not the country was implementing a full literacy assessment), the respondent might then be asked to participate in an extended literacy assessment.
Booklets
The STEP literacy assessment consisted of two booklets:
- A general booklet
- Section A: Reading components (about foundational reading skills; 10 min.)
- Section B: Core literacy assessment (with basic literacy questions; 7 min.)
- An exercise booklet
- Only for those who successfully complete the core assessment
- Booklet 1, 2, 3, or 4 randomly assigned
- 18 literacy items per respondent
- 28 min.
Techniques
- achievement or student test
- questionnaire
Languages
Administration of background questionnaire and literacy assessment instruments in 15 different languages: Albanian, Armenian, Arabic, Chinese, English, Georgian, Lao, Macedonian, Serbian, Sinhalese, Spanish, Tagalog, Tamil, Ukrainian, and Vietnamese
Translation procedures
- The localization design was developed in close cooperation with the Educational Testing Service (ETS) by cApStAn Linguistic Quality Control, which was also responsible for implementing linguistic quality assurance (LQA) and linguistic quality control (LQC) processes.
- World Bank survey firms were responsible for appointing two professional translators to translate or adapt the materials from the English source files according to the translation and adaptation guidelines provided with the materials.
- The two independent translations were reconciled into a draft national version of the materials and all national adaptations were documented for review by the independent translation verification company cApStAn.
- cApStAn independently verified materials translated by the survey teams and proposed changes before finalization of the assessment booklets.
Quality control of operations
Measures during data collection
- The country survey firms were centrally coordinated and supervised in order to ensure homogenous implementation of technical standards.
- Administration of questionnaires is either done in paper and pencil or computer-assisted personal interviews. The literacy test is always conducted in paper and pencil.
- Survey firms submitted regular fieldwork reports to the STEP team.
- The STEP team monitored data collection progress as well as non-response rates and provided guidance to the teams whenever required.
- All survey firms participated in a workshop organized by ETS to ensure that they would be able to score the booklets adequately.
- The reading literacy assessments were scored by the survey firms in accordance with STEP guidelines.
Measures during data processing and cleaning
- All survey firms were requested to enter the data using a double-data entry process.
- Survey firms could either use the STEP data entry program or develop their own, which were then tested by the STEP team as part of its quality assurance process.
- In the case of the reading literacy assessment data, survey firms were specifically requested to use the data entry program provided by ETS.
- The STEP team checked the household questionnaire data submitted by the survey firms to identify any data entry errors and ensure compliance with fieldwork procedures.
- ETS checked the reading literacy assessment data.
- Quality checks were also carried out to evaluate the handling and pattern of missing values.
Sources - Technical documentation