Mobile education builds resilience during shocks in five countries

Data

In each country, we have two waves of data: baseline and endline. In Nepal, we conducted a baseline survey with a random 50% of the sample, and in Kenya, we relied on administrative data instead of baseline survey data. The endline surveys took approximately 30 min to administer and included approximately 20 questions. These questions included a learning assessment, child wellbeing, parental engagement in educational activities, and parental perceptions of their child’s learning. A portion of the survey was conducted with the parent, and learning outcomes were collected by directly assessing the child over the phone. Endline surveys were conducted a few months after the programme ended. A set of common core questions in the baseline and endline surveys are included in the Supplementary Information.

The learning assessment was adapted from the ASER (Annual Status of Education Report) test, which has been used frequently in the literature to measure learning outcomes^14,15 and is used routinely across 14 countries. The test consists of multiple numeracy items, including two-digit addition, subtraction, multiplication and division problems. In addition, we asked students to solve a word problem and a fraction problem to capture learning outcomes beyond a core set of mathematical operations. Results are presented in several main results figures and tables. Distributions of raw baseline and endline learning levels are shown in Supplementary Fig. A3.

To maximize the reliability of the phone-based assessment, we introduced a series of quality-assurance measures. To minimize family members in the household assisting the child, students had a time cap of 2 minutes per question and we asked each child to explain their work. We only marked a problem correct if the child correctly explained how they solved the problem and enumerators were confident parents were not assisting their child. We also conducted a battery of validity checks to ensure the reliability of the learning outcomes. These results contribute new evidence on the robustness of remote learning assessment data across five countries. Phone assessment has emerged as a common strategy for large-scale household surveys such as the World Bank Living Standards Measurement Survey (LSMS). A growing literature has started to explore the validity of phone-based assessments to measure learning outcomes^20,56,57,58, with emerging evidence suggesting phone assessment can capture meaningful information at high frequency and low cost.

Extended Data Table 5 shows the results of five checks we conducted on the validity of our main learning outcomes of learning assessments via phone. Column 1 shows the first robustness check where we compare in-person to phone-based assessment for a representative sample of the same students in Kenya. An additional test included back-checks, with a random subset of students tested twice on the same competencies. We find a strong relationship as expected, with large positive coefficients and t-statistics ranging from 5 to more than 20. We further randomize students to receive different problems of the same proficiency (for example, four different questions to measure two-digit addition with carryover). Finally, column 5 shows results from a real-effort question to disentangle effects of the intervention on effort on the test versus cognitive skills. Students were asked to answer several effort tasks, for example figuring out the day of the week or counting zeros and ones. Altogether, results suggest measuring learning by phone can be robust, similar to findings from the World Bank LSMS for household measures of consumption.

The survey also included questions on caregiver engagement in their child’s education and beliefs. We measured engagement by asking caregivers how often they spent helping their child with their schoolwork over the previous weeks. We also included a measure of a caregiver’s confidence in their child having made progress in learning over the previous months, and their perception of their child’s numeracy level. Additional questions included information on whether the caregiver has returned to work. We also asked about parents’ demand for remote learning services in the future, and whether they would be willing to pay for such a programme. For students, we asked about child’s mental wellbeing, how much they enjoy school, and the child’s own belief about what mathematics problems they will be able to answer. We also measured non-cognitive skills, such as perseverance and ambition. Results are shown in Supplementary Tables A2 and A3. Finally, we included demographic questions, recording the child’s age, grade and gender.

The overall sample size, pooling all sites, is 16,936 households. The flow of participants from enrolment through randomization to endline follow-up is summarized in a CONSORT diagram (Supplementary Fig. A5). For endline surveys, we randomly sampled households from the full sample to interview for a total sample of 12,331. This was due to time and cost constraints. In Supplementary Table A8, we show that those randomly selected for endline interview are statistically equivalent to the full sample at baseline along a series of indicators such as gender, student grade and baseline learning level.

Supplementary Table A9 presents the response rate to the endline surveys and an analysis of survey attrition for those randomly selected to be part of the endline sample. The follow-up rate was around 76% of respondents at endline. Supplementary Table A9 also presents a test of whether response rates differed by treatment assignment. This provides evidence that our sample has a high and unbiased response rate.

Finally, we also include a survey for teachers to assess their beliefs and instructional practices. These questions include their desire to be a teacher, and teachers’ view that the phone call tutorial programme was helpful for student learning. Questions also include instructional practices, such as involving parents in education further and better targeting feedback to students’ actual learning level. These questions can capture potentially persistent effects on educational systems through teachers changing their beliefs and behaviours beyond the lifecycle of the programme.

Experimental design

We ran randomized controlled trials across five countries. Every trial had a control group and a combined phone call and SMS treatment arm. All studies except the trial in India also included an SMS-only treatment arm. In two countries, Nepal and the Philippines, we further randomized delivery of the phone call by NGO instructors or government teachers to assess scalability within government systems. We exploit random assignment to identify causal effects and estimate the impact of these education in emergencies interventions and various scalable delivery models.

We first estimate intent-to-treat effects per country c as follows:

where Y_i,j,c is a learning outcome for individual i in household j in country c. PhoneCalls_j,c and SMS_j,c take on the value 1 in their respective treatment arms and 0 otherwise. X_j,c denotes a vector of baseline control variables, applied consistently across trials to enhance statistical power and precision: student grade, student gender and baseline learning level. For baseline learning, we have the same assessment as the endline in all countries except Kenya, where we use administrative exam scores, and in Nepal, where we use parent perceptions of baseline learning. Since random subsamples are drawn at baseline in a few cases, we complete the sample with average estimates to enable analysis to consider the full sample at endline when including controls. δ_s denotes the geographic strata fixed effects—the randomization stratum relevant to each country (for example, region, district or constituency). We assess impact on operations gained as well as the highest proficiency achieved in a given country. We estimate the regression using ordinary least squares. Our main specifications are conducted per country. Given the speed and timing of the studies conducted during a historic crisis, rather than register new analysis plans, we rely on specifications published in prior related studies to guide our analysis²⁰. Our specifications are straightforward, with one main learning outcome measure. In the extended data, we conduct corollary analysis where we pool across studies and we include country fixed effects η_s. We run additional regressions using alternative weights by country as well as country-by-country estimations. Our primary analysis focuses on learning outcomes expressed as operation levels gained in raw units as well as the proportion of students who make it to the top proficiency level. We analyse standardized learning outcomes only as a corollary analysis (standardized relative to control group standard deviations at endline and centred at mean zero). All tests are two-sided statistical tests. All analysis was conducted in Stata/MP 18 software.

We also ran a subset of trials in which phone calls were delivered by NGOs or government teachers. We estimate government (Gov) versus NGO delivery effects per country using the following specification:

All standard errors are clustered at the level of the unit of randomization. In most countries this was the household level, which was de facto the same as the student level since one student participated per household; in a subset of treatment comparisons it was the school grade level. In India, Nepal and Uganda all randomization was at the household level. In the Philippines, the NGO arm was randomized at the household level; the government arm was randomized at the school grade level, with a corresponding control group randomized at the same level for each arm. In Kenya, the phone call arm was randomized at the school grade level whereas the SMS arms were cross randomized at the household level. Of note, robustness checks find little to no difference if standard errors are clustered or not across all specifications. This is largely due to the fact that there is substantial variation in learning outcomes within a cluster, such that the intracluster correlation is very low at about 0.03 in the Philippines and Kenya. The above specifications are the core estimation strategies. Additional specifications include other outcomes of interest beyond learning, such as engagement, as well as heterogeneity analysis, and various types of pooling across countries.

Due to randomization, we expect treatment and control groups to be balanced at baseline in expectation. Supplementary Table A10 presents balance tests using baseline data for key demographics and the same learning level variable. We find no statistically significant differences across multiple dimensions, including gender, grade, caregiver type, caregiver education levels and baseline learning. This is robust to multiple specifications, including with and without country fixed effects. Similar results are found for each country, with balanced samples in each individual country. This reveals that each randomization yielded balanced arms in line with expectations.

In all sites, the experimental design had high compliance rates; nearly all randomized treatment households participated in the expected treatment group. Detailed weekly monitoring data shows high take-up and fidelity, as shown in Extended Data Fig. 1.

In a secondary analysis, we embed another randomized trial which randomly allocated a subset of government teachers to deliver the phone call tutorials out of a pool of hundreds of eligible teachers in Nepal. Supplementary Table A6 shows balance tests with no statistically significant differences among treatment groups. We assess whether participating in phone call tutorials changed teachers’ subsequent beliefs and behaviours, which we estimate for teachers i at the school level j in each country c as follows:

where S_i,j,c captures teacher beliefs or instructional practices. These outcomes measure potential spillovers into the education system through teacher beliefs and behaviours which could persist beyond the programme.

Ethics statement

Insitutional review board (IRB) approval was granted by the Health Media Lab IRB (889OXFD21) and Columbia University IRB (21-384). Informed consent was performed in accordance with all relevant guidelines and regulations and obtained from caregivers, child assent and government approvals in each country. An additional country-specific IRB approval was granted by Nepal’s Health Research Council (776/2020) due to local regulations. The trials were registered in the AEA registry: AEARCTR-0009779. After the study period, control group students were later able to be enroled in the intervention to benefit from the programme.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.