metadata: authors: 0: author: Adelman,Melissa Ann
1: author: Haimovich,Francisco
2: author: Ham,Andres
3: author: Vazquez,Emmanuel Jose
count: Guatemala,Honduras
docna: 0: docna: Predicting school dropout with
            administrative data: new evidence from Guatemala and Honduras
docty: Policy Research Working Paper
subtopic: Hazard Risk Management,Secondary Education,Educational Sciences,Educational Institutions & Facilities,Social Risk Management,Effective Schools and Teachers,Inequality,Disaster Management
teratopic: Education
abstracts: cdata!: Across Latin America, school dropout is
            a growing concern, because of its negative social and
            economic consequences. Although a wide range of
            interventions hold potential to reduce dropout rates, policy
            makers in many countries must first address the basic
            question of how to target limited resources effectively for
            such interventions. Identifying who is most likely to drop
            out and, therefore, who should be prioritized for targeting,
            is a prediction problem that has been addressed in a rich
            set of research in countries with strong education system
            data. This paper makes use of newly established
            administrative data systems in Guatemala and Honduras, to
            estimate some of the first dropout prediction models for
            lower-middle-income countries. These models can correctly
            identify 80 percent of sixth grade students who will drop
            out in the transition to lower secondary school, performing
            as well as models used in the United States and providing
            more accurate results than other commonly used targeting approaches.
geo_reg_key: 517191,545133,119228
repnme: repnme: Predicting school dropout with
            administrative data: new evidence from Guatemala and Honduras
display_title: Predicting school dropout with
            administrative data: new evidence from Guatemala and Honduras
listing_relative_url: /research/2017/07/27714679/predicting-school-dropout-administrative-data-new-evidence-guatemala-honduras
topicv3: Teachers Management m1326119 1523,Poverty m1326687 695,Education m1327048 470,Secondary Education m1327058 481,Early Warning Systems m1327419 570
publishtoextweb_dt: 2017-07-10T00:00:00Z
historic_topic: Education
no_of_pages: 42
author_keys: 0: author_key: 1225054
1: author_key: 1381681
model_extractions: WPS8142 Policy Research Working Paper 8142 Predicting School Dropout with Administrative Data New Evidence from Guatemala and Honduras _Melissa Adelman Francisco Haimovich Andres Ham Emmanuel Vazquez_ Education Global Practice Group July 2017
Policy Research Working Paper 8142

# **Abstract**

Across Latin America, school dropout is a growing concern, because of its negative social and economic consequences. Although a wide range of interventions hold potential to reduce dropout rates, policy makers in many countries must first address the basic question of how to target limited resources effectively for such interventions. Identifying who is most likely to drop out and, therefore, who should be prioritized for targeting, is a prediction problem that has been addressed in a rich set of research in countries with strong education system data. This paper makes use of newly established administrative data systems in Guatemala and Honduras, to estimate some of the first dropout prediction models for lower-middle-income countries. These models can correctly identify 80 percent of sixth grade students who will drop out in the transition to lower secondary school, performing as well as models used in the United States and providing more accurate results than other commonly used targeting approaches.

This paper is a product of the Education Global Practice Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at madelman@ worldbank.org and fhaimovich@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Predicting School Dropout with Administrative Data: New Evidence from Guatemala and Honduras**

Melissa Adelman, Francisco Haimovich, Andres Ham, and Emmanuel Vazquez* Keywords: Dropout, Prediction, Early Warning, Guatemala, Honduras JEL Classification: I20, I21, C51, C53

*The authors thank the Ministry of Education of Guatemala ( _Ministerio de Educación_ – MINEDUC) and the Secretary of Education of Honduras ( _Secretaría de Educación_ – SEDUC) for providing access to the anonymized administrative data on which this study is based.
# **1. Introduction**

As a result of sustained public and private investment in recent decades, primary net enrollment rates across all low and middle-income countries now average 91%, while secondary and tertiary enrollment rates are growing quickly. In the Latin America and Caribbean region, primary enrollment is approaching universality, and secondary net enrollment has grown 50% in the last two decades to over 75% (United Nations 2015; World Bank 2016). For many countries, however, attainment continues to fall short of aspirations, as high rates of enrollment in early grades quickly decline due to students dropping out before completing a full course of basic education (Bassi, Busso, and Muñoz 2016). For example, in Guatemala and Honduras, the countries of focus in this paper, education is _de jure_ compulsory through ninth grade, but nearly 40% of sixth graders drop out before getting there. For young people who drop out prematurely, global evidence suggests that, on average, they will earn less and experience more social and economic challenges than their peers with more years of completed education (Patrinos and Psacharopoulos 2004; Oreopoulos and Salvanes 2011; Bentaouet-Kattan and Székely 2014).

Across Latin America, school dropout is a pipeline for expanding the population of underskilled and underengaged youth, contributing to social and economic challenges (Cardenas, De Hoyos, and Székely 2015). Several branches of research from North America, Latin America, and other regions have focused on identifying the causes of dropout, and point to multiple, interacting factors that affect learning, progression through grades, and ultimately dropout. Economics literature on dropout starts from the foundational human capital theory, in which individuals decide whether to persist or drop out of school by weighing the marginal expected costs of continuing investment in education (such as school fees and the opportunity cost of lost earnings) against the marginal expected benefits of acquiring more years of schooling (such as increased future wages from higher skills) (Becker 1967). Several authors have identified a range of individual, community, and broader factors that help shape these costs and benefits for each student, as well as the decisionmaking processes used to compare them, such as household wealth, schooling quality, and labor market conditions (Behrman, de Hoyos, and Szekely 2015; Adelman and Székely 2016). A large education literature focuses on dropout as the ultimate outcome of a process of disengagement from school, and demonstrates that dropouts can be grouped into distinct typologies, based on the factors driving their decision (Fortin et al 2006; Ananga 2011; Bowers and Sprott 2012).

2
Consequently, a range of interventions can be effective at reducing dropout and increasing attainment depending on the context, from conditional cash transfers to providing information on returns to schooling, to training on socio-emotional skills (Barrera-Osorio et al 2011; Nguyen 2008; Jensen 2010; Fryer 2013; Avitabile and de Hoyos 2015; Heller et al 2016).

While understanding why students drop out is critical, the ability of policy makers to respond effectively depends on answering an even more fundamental question – who is most likely to drop out? This question may appear relatively easy to answer, particularly in countries, regions, or localities with high dropout rates, as one might assume dropouts are concentrated in particularly disadvantaged or dysfunctional schools, or among students with particular characteristics. However, dropouts are often spread throughout schools and not readily identifiable by single characteristics, reflecting the complexity of the issue as documented in the dropout typology literature mentioned above. For example, in Guatemala, one of the countries in this study, over half of sixth grade students who drop out in the transition to lower secondary are spread across 70% of the country’s primary schools, where the dropout rate is below 50%, and while 50% of students who score in the lowest quartile of a sixth-grade standardized exam drop out, so do 20% of those who score in the highest quartile.

Accurately identifying students at risk of dropping out in order to target effective interventions to where they are most needed is particularly important in contexts of limited resources and competing priorities, which describes most of the world’s education systems. Largely based on the rich administrative data available in many U.S. school systems, research on dropout prediction is providing an increasingly sound empirical base for accurately predicting who will drop out several months to several years before dropout occurs, and over 30 U.S. states currently have in place some form of “early warning” system (O’Cummings and Therriault 2015). Similarly, the majority of European countries report monitoring “early warning” signs of potential dropout through their management information systems, primarily at the school level, but in some cases nationally (European Commission 2013). For many middle-income countries, which have invested in setting up information management systems in recent years, answering the prediction problem of dropout is now becoming possible through the use of consistent student-level data. In both Guatemala and Honduras, for example, student and school-level data are now digitalized in 3
networked administrative databases, including unique student identifiers that allow tracking students over time, and, in the case of Guatemala, that can be directly linked to standardized test data.

In this paper, we make use of these administrative data to estimate early warning models of dropout in primary and secondary school.<sup>1</sup> Using linear regressions and basic prediction concepts, we are able to accurately predict approximately 80% of the sixth-grade students who dropped out within the next year in Guatemala and Honduras, performing at comparable levels to models used in the U.S. These early warning models, which are based on routinely collected data and relatively simple analytical techniques, are feasible to implement in a wide range of country contexts. By providing an accurate means of targeting, these models could substantially reduce the misallocation of program resources: in a simple simulation of a modest dropout prevention program, targeting students based on these models rather than targeting poor municipalities or high-dropout schools could reduce misallocation of resources by 30 to 80%.

The remainder of the paper is organized as follows. Section 2 briefly reviews the concepts and empirical evidence on early warning systems, which primarily come from the U.S. Section 3 describes the data and the current dropout situation in both Guatemala and Honduras, while Section 4 presents the results of constructing early warning models. Section 5 concludes with a brief summary of the findings and suggestions for future research.

# **2. Predicting dropout: Concepts and evidence**

The quantitative literature on dropout prediction stretches back at least 30 years in the U.S. and Canada, and includes a variety of methods ranging from using single variables (often called “dropout flags”) to applied statistical learning (or “machine learning”) based on large data sets (in terms of both number of variables and number of observations). This research has been primarily concerned with using readily available data to provide accurate predictions to school system managers on who is most likely to drop out prior to completing a full course of primary or secondary education. The variables used for prediction are based on both the conceptual

> 1 The authors thank the Ministry of Education of Guatemala ( _Ministerio de Educación_ –MINEDUC) and the Secretary of Education of Honduras ( _Secretaría de Educación_ – SEDUC) for providing access to the anonymized administrative data on which this study is based.

4
underpinnings discussed in the previous section and on the types of data usually available in school systems’ administrative information systems. This means that in practice most dropout prediction is based on indicators of students’ engagement in school and learning – such as attendance, behavioral infractions, course grades, and exam performance – as well as some socioeconomic characteristics – such as race and qualification for support programs (e.g. free lunch).

While most papers on the subject assert that their preferred method yields accurate predictions, the literature has been limited by use of inconsistent language and metrics for defining accuracy. In a 2013 systematic review, Bowers, Sprott, and Taff lay out a common set of prediction concepts and measure the existing literature against these metrics, finding substantial variation in the accuracy of prediction across methods. Below, we borrow from their review to describe the prediction concepts upon which our results are based.

The accuracy of predicting a binary outcome like dropout is illustrated by the simple event table in Table 1 (Bowers, Sprott, and Taff 2013; Stuit et al 2016). In order to provide useful information to policy makers attempting to target resources, the objective of most dropout prediction is to maximize sensitivity (the share of eventual dropouts correctly identified), while minimizing the rate of false positives (students identified as likely to drop out who in fact will not) (Gleason and Dynarski 2002). The rates of false positives and false negatives (students identified as likely to graduate who in fact will drop out) corresponds to the concepts of Type I and Type II errors, respectively, in statistical hypothesis testing (Sheskin 2004; Rogosa 2005).

These two objectives, of maximizing sensitivity and minimizing false positives, may be at odds with each other, as predictions that identify the majority of dropouts can often also incorrectly classify many students as dropouts. For example, several studies included in Bowers, Sprott, and Taff’s review of single dropout flags achieve sensitivities of over 80% (e.g., correctly predicting over 80% of eventual dropouts), but have false positive rates over 50% (e.g., incorrectly predicting that over 50% of eventual graduates will drop out). The tradeoff between these two objectives can be measured using a Receiver Operating Characteristic (ROC) curve from the signal detection literature, which simply plots sensitivity (or true-positive proportion) against the false-positive proportion in x-y space for a given dropout prediction measure (Bowers, Sprott, and Taff 2013).

5
As Figure 1 illustrates, perfect dropout prediction would correctly classify all eventual dropouts and graduates, while random guessing would on average produce an equal number of false positives and false negatives. For any given dropout prediction measure, a ROC curve can be plotted based on the cutoff value set to define who is classified as a future dropout, and the optimal cutoff value determined through one of several related methods to trade off sensitivity and falsepositives, including by setting the value at the point closest to perfect prediction (Fluss, Faraggi, and Reiser 2005; Liu 2012).

Based on these concepts of prediction accuracy, Bowers, Sprott, and Taff (2013) find that a large share of dropout predictors (across methodologies) perform quite badly, either identifying only a very small share of all eventual dropouts (clustering in the bottom right-hand corner of the ROC graph) or being only marginally better than random guessing (close to the 45-degree line). The most accurate predictors in their review all use growth mixture modeling, a form of multivariate longitudinal analysis that follows the academic trajectory of individual students, for example modeling learning outcomes over time (Muthen 2004; Janosz et al 2008; Bowers and Sprott 2012). These prediction methods achieve 80-90% sensitivity (correctly identifying 80-90% of eventual dropouts as being likely to drop out), and 10-20% false-positive rates (incorrectly identifying 1020% of eventual graduates as being likely to drop out). The relatively strong performance of these approaches accords with intuition, as dropout is generally considered the manifestation of a process of disengagement, rather than a discrete decision (Balfanz, Herzog, and MacIver 2007; Programa Estado de la Nacion 2013; Frazelle and Nagel 2015). However, the high data requirements and relative complexity of these approaches limit their usefulness for most school systems.

The next best performing methods are in fact much more simple and usable. The ‘On-Track Indicator’ used in Chicago is a binary indicator of two underlying measures of students’ performance in the first grade of secondary school: earning the minimum number of course credits required for promotion to the next grade, and not earning more than one semester ‘F’ in any core courses. This indicator achieves 75% sensitivity and a 16% false-positive rate, and is in active use in the Chicago public school system (Allensworth and Easton 2005, 2007). Similarly, Bowers (2010) finds that logistic regressions including indicators of ever having repeated a grade and 6
annual grade point average are able to achieve 81% specificity and a 25% false-positive rate in two school districts in the U.S.

While these results assess dropout prediction based on performance in-sample, the true test for the accuracy of prediction models is performance outside of the sample on which the models are estimated. Using administrative data on attendance, behavioral violations, standardized test scores, and demographic characteristics from the state of Wisconsin, Knowles (2015) estimates a range of models, from standard logistic regression to support vector machines, on a “training sample” and then tests their accuracy out of sample, finding that logistic regression models perform very well relative to more complex algorithms. Similarly, using rich administrative data in Denmark, Sara et al (2015) show that machine learning algorithms “trained” on one data set can achieve sensitivities over 90% (with false-positive rates around 20%) out of sample when predicting high school dropout over three-month intervals.

Overall, then, evidence is substantial that who will drop out can be accurately predicted using highquality administrative data in the U.S. as well as Europe. We are not aware, however, of any studies investigating this question in the context of a developing country, where data requirements have only recently been met and dropout is often a much more widespread issue. This paper therefore provides one of the first applications of dropout prediction methodologies in lower middle-income education systems, made possible by the remarkable efforts of the ministries of education in both Guatemala and Honduras. Furthermore, unlike much of the existing U.S. research, we estimate models at the national level instead that at the state/province level. This is possible because in both countries the administrative data are harmonized, aggregated, and analyzed at the federal level.

7
# **3. Administrative data and dropouts in Guatemala and Honduras**

# **_3.1 Guatemala_**

Over the last several years, the Guatemalan Ministry of Education ( _Ministerio de Educación_ - MINEDUC) has substantially improved and expanded their administrative information system based on student-level records. One of the main innovations was the implementation of a unique student identifier, which started in upper secondary in 2009 and was progressively expanded until all students in primary and secondary schools were assigned unique identifiers in 2011. These identifiers are used by each school at the beginning of the school year to provide data on enrollment to the Ministry. Specifically, schools provide a list of all students who are enrolled in each grade, with their unique identifiers, and the Ministry centralizes and consolidates this information in a database that contains the annual enrollment status of all students (nearly 4 million each year) from 2011 to 2016. This data structure allows the educational trajectories of students to be followed through time, and specifically enables the identification of those who drop out and the year in which they left school. In addition to students’ unique identifiers, the data also include information on student’s gender, school year, and results of their October final examination (promoted, not promoted, withdrew), as well as school’s sector (formal public, private, cooperative, municipal), teaching modality (bilingual, monolingual), municipality, department, and school identifier.<sup>2,3</sup> These basic administrative microdata on enrollment can then be matched with additional data on student and school characteristics from the National Evaluation of Students. These assessments have been conducted in the first, third and sixth grades of primary education, the last grade of lower secondary education (ninth grade), and the last grade of upper secondary, with variable frequency.<sup>4</sup> The evaluations in secondary education are applied to all the students in the

> 2 The formal public sector is the group of schools that are under the direct administration of MINEDUC. The municipal sector is composed of public schools administered by municipalities. The cooperative sector comprises public schools with a tripartite administration that involves the participation of MINEDUC, the municipality, and student’s parents. 3 Bilingual schools are institutions where instruction is offered in both Spanish and an indigenous language. 4 The National Evaluations are conducted every year in the last grade of upper secondary, but the implementation in other grades has been less regular: 2006, 2009, and 2013 for ninth grade; 2006, 2007, 2008, 2009, 2010, 2013, and 2014 for third and sixth grade; and 2006, 2008, 2009 and 2010 for first grade. While this could limit the capacity of implementing an early warning system every year, the Government of Guatemala and the World Bank are working on replicating the analysis but using students‘ grades collected on a yearly basis.

8
corresponding grade, while the assessments in primary school only cover a random sample of the students in the formal public sector, excluding those children attending private, cooperative, and municipal schools.

In this paper, we combine information from the enrollment database and the National Evaluations conducted in sixth and ninth grades in 2013 to follow the educational trajectories of the cohort of students evaluated that year. Specifically, we focus on the random sample of 19,000 students in public schools assessed in sixth grade (6.5% of the sixth graders in the sector, 5.7% of all sixth graders) and the census of 196,000 students in all schools evaluated in ninth grade. As discussed later, these are critical grades for school dropout in both Guatemala and Honduras. National Evaluations provide data not only on students’ reading and math abilities, but also on several selfreported characteristics including their motivation and interest in learning, parents’ characteristics (e.g. education, occupation), and household resources. While these data are quite rich, records are only complete for about 60% of the random sample, as many students do not fully complete the questionnaires – in all the analysis presented, we drop all incomplete records.<sup>5</sup> Because the analysis is done at the national level, we complement these data with information at the department level on the gender-specific high school wage premium from Guatemala’s national household survey, the _Encuesta de Condiciones de Vida_ (ENCOVI) 2014, as an attempt to capture some basic characteristics of the local labor market that may affect the decision to drop out of school. Table A1 in the Appendix presents the definitions of all the variables that are used in the analysis, as well as some descriptive statistics for the cohort of students who were evaluated in sixth and ninth grades in 2013.

Dropout rates calculated using the administrative enrollment data confirm that a large proportion of youth leave school before graduating from secondary school in Guatemala. In particular, we focus on three different periods in which dropout is particularly high: the transition from primary to lower secondary education ( _Ciclo Básico_ ), during lower secondary education, and the transition from lower to upper secondary education ( _Ciclo Diversificado_ ). Figure 2 shows the dropout rates in each of these periods for the cohort of students under analysis. Fully one-quarter of the students

> 5 In results that are not presented, multiple imputation techniques for the missing data are used, producing results that are highly consistent with those presented.

9
evaluated in the last grade of primary education (sixth grade) dropped out in the transition from primary to lower secondary, 18% of the remaining three-quarters (14% overall) left during lower secondary school, and 32% of the students who managed to reach the last year of lower secondary dropped out in the transition to upper secondary. Taken altogether, these figures imply a survival rate from sixth to tenth grade of 42%, meaning that the majority of sixth grade students drop out during this four-year period.<sup>6</sup> This pattern is consistent with the information that emerges from household surveys in Guatemala: for example, Adelman and Székely (2016) find that for the cohort reaching age 18 in 2012-2014, the percentage of youth enrolled in any grade falls from 80% to 47% from age 12 to 16.

While administrative enrollment data have several advantages for the study of school dropout, the records are still in an incipient phase of development and therefore subject to some error. In Guatemala, school principals (and teachers) manually enter information into a web-based information system, creating the possibility of typographical mistakes, incomplete data, and other human errors. Therefore, it is possible that some students who are not listed as enrolled in a particular year are not real dropouts (i.e. our data could overestimate the dropout rate). Some inconsistencies in the educational trajectories of the students have also been found (e.g., students skipping grades from one year to the next). To address this, a new data platform was implemented in 2015, with automatic flags to substantially reduce inconsistencies and improve the quality of data entered. Nevertheless, as we discussed above, when we compare the survival rates estimated with both administrative and household survey data we find similar rates, suggesting that the measurement error of dropout is relatively small in our data set.

# **_3.2 Honduras_**

In Honduras, the _Secretaría de Educación_ (SEDUC) began collecting administrative records in 2013. Student information is reported by teachers in both public and private institutions, who input their rosters into a web-based system. These records gather information for all formal sector students in Honduras and allow the tracking of the same children over time (about 1.5 million).

> 6 As previously mentioned, the dropout rates in the transition from primary to lower secondary education and within lower secondary are based on a sample of students in the official sector, which represents 87% of all the students in sixth grade in 2013.

10
SEDUC estimates a coverage rate of 97%, which has improved in the two subsequent years for which data are available.

Given the recent implementation of the database, it contains limited information. Besides individual enrollment status, the data include some demographics (gender and age), attendance rates, whether the child attends a public or private institution, and current grade. Unlike in Guatemala, more detailed socioeconomic information for students or their household is unavailable. While recent data exist on exam scores, they cannot be matched to our sample because they use different individual identifiers, an issue that has only recently been addressed by SEDUC.

We therefore augment SEDUC’s student data with municipal-level indicators from several sources. First, we add a set of school supply and quality indicators. The number of schools per municipality is included to measure educational supply at the extensive margin. These indicators are weighted by their target population using data from the National Statistics Institute. We first calculate the number of children in each municipality of primary (6-11) and secondary (12-18) age. Then, the number of primary and secondary schools is divided by the target population (in thousands). To measure intensive-margin educational supply, we take the number of available teaching positions in 2015 and weigh them by the target population in each municipality.<sup>7</sup> Quality indicators correspond to municipality-level infrastructure indices from the 2013 school census collected for the School Infrastructure Master Plan ( _Plan Maestro de Infraestructura Educativa_ ). These indices are on a scale from zero to one, and we include a summary measure that is a weighted average of six dimensions: furnishings, basic services, natural disasters, social threats, hydrosanitary issues, and physical infrastructure.<sup>8</sup> Second, we add a set of municipal-level socioeconomic indicators. Honduras is classified as one of the most violent countries in the world (World Bank 2011), but there remains limited evidence

> 7 Available teaching positions do not necessarily correspond to filled vacancies. For instance, a municipality may have 10 available positions, but only 8 employed teachers. The data do not allow us to make this distinction.

> 8 Each index is comprised of multiple indicators (Secretary of Education, 2014). The furnishings index captures the number of desks and blackboards. The basic services index is an average of access to electricity, running water, and sewers. The natural disaster index measures how unlikely it is that a school is affected by floods, landslides strong winds, and earthquakes. The social threat index measures how difficult it is to access and consume alcohol, cigarettes, and illicit drugs. The hydro-sanitary index captures the state of sinks, toilets, and urinals. The physical infrastructure index evaluates how well the building, classrooms, and environments are suited for students to receive lessons.

11
on how crime correlates with schooling outcomes. Students are assigned the annual homicide rate for their municipality (per 100,000 population), taken from the Online Police Statistical System ( _Sistema Estadístico Policial en Línea_ - SEPOL).<sup>9</sup> We also include the following municipal-level indicators from the 2001 Population Census: unsatisfied basic needs (UBNs) poverty, the share of ethnic minority population, average years of education, average pre-school enrollment rates, average rates of child labor, the adolescent birth rate, and the share of households with migrants outside Honduras.

Dropout rates in Honduras vary across the school cycle. While only 8.8% of children drop out in primary school, this fraction increases substantially thereafter. Like Guatemala, dropout is particularly high in the transition from primary to lower secondary education ( _Ciclo Básico_ ), during lower secondary education, and in the transition from lower to upper secondary education ( _Ciclo Diversificado_ ). Figure 3 shows that 37.4% abandon school in the transition from primary to lower secondary, 18.4% drop out within lower secondary, and about 33.2% do not continue into upper secondary. Taken altogether, these figures imply a survival rate from sixth to tenth grade of 34%, meaning that the majority of sixth grade students drop out during this four-year period.

These rates are somewhat higher than household survey estimates but follow the same pattern. For the last available household survey in 2014, which collects data on children’s grade progression, the dropout rate for the transition into lower secondary is 27%, 13% within lower secondary, and 21% during the transition from lower to upper secondary. Adelman and Székely (2016) also report similar trends when using age groups.

As with the Guatemalan data, this method of data collection has only been recently implemented and there remains room for improvement. Teacher reporting of student outcomes is potentially prone to input errors, which may result in measurement bias, although we are unable to determine their extent or direction. As noted in the previous paragraph, one potential outcome is that we may overestimate dropout rates. However, the administrative data present many advantages over household surveys, mainly sample size and nationwide coverage. Data collection is also constantly improving, due to efforts from USINIEH to ensure correct data input and quality control at

> 9 Municipal-level homicide rates are publicly available at the SEPOL website: www.sepol.hn.

12
different stages. Furthermore, as long as the inflated dropout rates are driven by classical (random) measurement error, this would just lower the accuracy of our model. In that scenario, our estimates give us a lower bound of the accuracy levels of the model, and hence we would expect a better performance as the quality of the panel improves over time. Indeed, as we will show in the next section, the models work better in Guatemala, where the measurement error of dropout is much lower.

# **4. Who will drop out: Empirical predictions**

# **_4.1 Results_**

In our analysis, we focus on three periods during which both Guatemala and Honduras lose the majority of students to dropout: the transition between primary and lower secondary school, within lower secondary, and the transition from lower secondary to upper secondary. For ease of exposition, below we present the results of using data available in year t to predict who will drop out in the transition from primary to lower secondary in year t+1. Additional results on predicting dropouts within lower secondary and in the transition from lower to upper secondary, are presented in the Appendix.

We follow the method of Knowles (2015) and Stuit et al (2016) in estimating dropout models in three steps. These prediction models are based on the conceptual framing of dropout as a decision that can be affected by a broad range of underlying factors discussed in Section 1, and include all of the available covariates that could reasonably capture one of these factors. In this way, prediction modeling differs from other exercises that estimate the correlational or causal relationships between specific factors and dropout. First, we estimate linear probability models with dropout as a binary outcome, using the individual/household, school, and municipality/department-level covariates described in the previous section. However, for Guatemala, given the large number of variables available to include in the models we add a ‘zero step’: we combine the information from several highly correlated variables that measure similar characteristics and construct indices using a Principal Component Analysis (Pearson 1901; Hotelling 1933; Jolliffe 2002). These indices were obtained as component scores for the first 13
principal component and enable us to estimate a parsimonious model with almost no loss in the accuracy of the prediction.<sup>10</sup> Table 2 shows the different specifications used in each country.<sup>11</sup> Treating the estimated y-hats as dropout probabilities, we then construct a ROC curve by varying the cutoff point above which a student is identified as a predicted dropout. Each possible cutoff has a particular sensitivity and false-positive proportion associated to it. Therefore, in the third step we select the point that minimizes the distance from perfect prediction (the (0,1) point) and evaluate the models at this optimal cutoff.

Figure 4 shows the ROC curves for each model. The graphs show that even simple models work much better than random guesses (represented by the 45-degree line). In Honduras, model 1— which only includes basic demographic variables and a few municipal-level covariates— is noticeably above of the 45-degree line. For example, if we consider a 25% false-positive proportion, we could gain almost 40 percentage points of sensitivity by including these basic variables. Similar results are observed in Guatemala when we consider individual-level data (including test scores). As we include more information, not surprisingly, the models become more accurate. In particular, adding school fixed effects improves the performance of the models substantially.

Figure 5 tests whether the specification that uses all available variables is statistically different than a random guess. We estimate sensitivity and false positives at 100 different cutoffs by bootstrap to obtain a 95% confidence interval. In both Guatemala and Honduras, we find that these models are significantly better than a random guess.

Table 3 assesses the models in terms of sensitivity, false-positive proportion, and overall accuracy at the optimal cutoff value. In our preferred specification, we are able to correctly identify 80% of sixth grade students who will drop out in the transition to lower secondary, with a false-positive proportion close to 20%. Importantly, these accuracy levels are comparable to those observed in

> 10 See Tables A1(a) and A1(b) in the Appendix and the notes below these tables for a description of the variables involved in their computation.

> 11 Estimation results are presented in the Appendix.

14
developed countries. Furthermore, as will be discussed later, these models allow targeting mechanisms that are more accurate than other commonly used approaches.

# **_4.2 Out-of-sample performance_**

While these models perform very well on the data on which they are constructed, we conduct two validation exercises to assess their performance out of sample. First, following the prediction literature, for each country we conduct a K-fold cross-validation, where the data is randomly divided into K “folds” or sub-samples (Knowles 2015). We present the results here for K=5 folds, but obtain similar results varying K between 3 and 8.<sup>12</sup> The prediction models are then estimated on four of the folds, and predictions made based on the models for each of the five folds. We then calculate sensitivity and false-positive proportions in sample (the four folds on which the model is estimated) and out of sample (the left-out fifth fold). This procedure, starting with the random division of the data into folds, is repeated 100 times to obtain the statistics presented in Table 4. While the models’ out-of-sample performance is worse than in-sample, it remains reasonably good compared to other prediction models used in the U.S. context and significantly better than random guessing (Bowers, Sprott, and Taff 2013).

In Honduras, we can also perform what is arguably the best type of validation exercise for the model’s out of sample prediction. Specifically, we estimate Model 3 in 2014 to predict the likelihood of dropping out in 2015. We use the optimal cutoff for 2013 and classify students at risk and not at risk. Then we check the efficiency of the early warning system. Results are shown in Figure 6. The method performs quite well out of sample, with a sensitivity rate of 76.3% and a false-positive proportion of 15.9%.

Finally, we consider the accuracy of the targeting facilitated by these models in relation to other potential targeting approaches through a simple simulation. In the scenario considered, a dropout prevention program for sixth graders has a fixed budget of either $1M, $2M, or $4M US in total and costs $200 US per student to implement, meaning that 5,000, 10,000, or 20,000 students can

> 12 While no hard-and-fast rules exist around the optimal K, tradeoffs between bias and variance can help guide the choice, and K= 5 and K = 10 are commonly used (Kohavi 1995; Borra and Di Ciaccio 2010).

15
participate. In Guatemala, approximately 90,000 sixth graders drop out before enrolling in seventh grade, while in Honduras, approximately 40,000 do. How should students be targeted for this program in each country? In Table 5, we compare the early warning models to two other possible targeting approaches – targeting students in the poorest municipalities and targeting schools with the highest dropout rates. In both countries, the early warning models perform substantially better than the other options in identifying students who will eventually drop out, particularly for the smaller program sizes, and targeting students based on these models rather than targeting poor municipalities or high-dropout schools could reduce misallocation of resources by 30 to 80%.<sup>13</sup> However, school-level targeting also performs well and could be the most suitable targeting approach for interventions with substantial economies of scale.

# **5. Conclusions**

Many developing countries, including Guatemala and Honduras, are approaching universal primary school enrollment and have also made substantial progress in expanding access to secondary school. However, dropout is a pressing concern for policy makers in both countries, as fewer than 50% of young people succeed in completing a full course of basic education. Policies and programs aimed at reducing dropout may be made more effective when education systems are able to identify with reasonable accuracy the students who are most likely to leave school early.

In both Guatemala and Honduras, substantial advances in the scope and reliability of education administrative data have created an opportunity to do just that, as illustrated in this paper. Using routinely collected administrative data and relatively simple analytical techniques, we show that early warning models can accurately predict which students will drop out, providing potentially actionable information to system and school leaders. The results immediately suggest two areas for further exploration: improving the early warning analysis and moving from early warning analysis to “early warning systems”.

> 13 The calculation on reduced misallocation is based on comparing the dollars wasted in Table 3. In Guatemala, due to the presence of missing data in the covariates used in the early warning system, the sample has been restricted to be exactly the same across the three methods to allocate resources.

16
Our analysis augments the most basic administrative data with additional data to improve the quality of prediction (in the case of Guatemala, with periodic national exam data for primary students; in the case of Honduras, with household survey and census data). This augmentation could affect the consistency of the prediction from year to year, as the availability of household survey and other data varies over time. It could also make the approach more difficult to replicate for practitioners within ministries of education, who may not be familiar with accessing and analyzing these other data sources. However, as mentioned above, in both countries more and more variables are being routinely collected over time, which will reduce the need to pull data from other sources and also could serve to improve the quality of prediction. In addition, as mentioned for Guatemala, a greater focus on obtaining complete data for each student would be critical to improving the quality and usefulness of the predictions.

While this paper has focused on the feasibility of making accurate and replicable predictions of who is most likely to drop out, this is only a first step towards developing an effective early warning system, and a natural question immediately arises – what to do with the data? Maintaining confidentiality, avoiding negative labeling of students, and identifying resources to act appropriately on the predictions are all among the potential important concerns. Moreover, the predictions in and of themselves do not identify the main factors that put any given student at risk of dropping out, information which is critical to providing effective interventions. A complete discussion about setting up an effective early warning system merits its own paper, but many lessons learned have emerged from the experiences of school districts in the U.S. These include clearly communicating the meaning of the predictions; defining roles at all levels in terms of who should receive what information and who is responsible for taking what actions; empowering local school officials to identify and implement relevant, customized prevention measures; and taking an iterative approach to facilitate learning from initial pilots (O’Cummings and Therriault 2015). These and other lessons learned from international experiences, combined with a strong knowledge of local contexts, show a path forward for responding effectively to the dropout crisis in Guatemala, Honduras, and many other countries.

17
# **References**

Adelman, Melissa and Miguel Székely. 2016. “School Dropout in Central America: An Overview of Trends, Causes, Consequences, and Promising Interventions.” World Bank Policy Research Working Paper 7561. Washington, D.C.: World Bank.

Allensworth, E. M., & J.Q. Easton. 2007. _What Matters for Staying On-track and Graduating in Chicago Public High Schools: A Close Look at Course Grades, Failures, and Attendance in the Freshman Year_ . Chicago: Consortium on Chicago School Research.

Avitabile, Ciro and Rafael de Hoyos. 2014. “Heterogeneous Effects of Information about the Returns to Schooling on Student Learning: Evidence from a Randomized Control Trial in Mexico.” World Bank Working Paper. Washington, D.C.: World Bank.

Balfanz, Robert, Liza Herzog, and Douglas MacIver. 2007. “Preventing Student Disengagement and Keeping Students on the Graduation Path in Urban Middle-Grades Schools: Early Identification and Effective Interventions.” _Educational Psychologist_ 42(4): 223-235.

Barrera-Osorio, Felipe, Marianne Bertrand, Leigh Linden, and Francisco Perez-Calle. 2011. “Improving the Design of Conditional Transfer Programs: Evidence from a Randomized Education Experiment in Colombia.” _American Economic Journal: Applied Economics_ 3(2): 167-195.

Bassi, Marina, Matias Busso, and Juan Muñoz. 2016. “Is the Glass Half Empty or Half Full? School Enrollment, Graduation, and Dropout Rates in Latin America.” _Economia_ (forthcoming).

Becker, Gary. _Human Capital_ . New York: Columbia University Press, 1964.

Behrman, Jere, Rafael de Hoyos, and Miguel Székely. 2015. “Out of School and Out of Work: A Conceptual Framework for Investigating “Ninis” in Latin America and the Caribbean.” Washington, DC: World Bank. Background paper for the “Out of School, Out of Work” study.

Bentaouet-Kattan, R., and Székely, M. 2015. “Patterns, Consequences and Possible Causes of Dropout in Upper Secondary Education in Mexico.” _Education Research International_ , Volume 2015, Article ID 676472.

Borra, Simone and Agostino Di Ciaccio. 2010. “Measuring the Prediction Error: A Comparison of Cross-Validation, Bootstrap and Covariance Penalty Methods.” _Computational Statistics & Data Analysis_ 54(12): 2976-2989.

Bowers, Alex. 2010. “Grades and Graduation: A Longitudinal Risk Perspective to Identify Student Dropouts.” _Journal of Educational Research_ 103(3): 191–207.

Bowers, Alex and Ryan Sprott. 2012. “Why Tenth Graders Fail to Finish High School: A Dropout Typology Latent Class Analysis.” _The Journal of Education for Students Placed at Risk (JESPAR)_ 17(3): 129-148.

18
Bowers, Alex, Ryan Sprott, and Sherry Taff. 2013. “Do We Know Who Will Drop Out? A Review of the Predictors of Dropping out of High School: Precision, Sensitivity, and Specificity.” _The High School Journal_ 96(2): 77-100.

Cardenas, Mauricio, Rafael de Hoyos, and Miguel Székely. 2015. “Out of School and Out of Work Youth in Latin America: a Persistent Problem in a Decade of Prosperity.” _Economia_ 16(1).

Estado de la Nación. 2013. _Estado de la Educación Costarricense_ . San José, Costa Rica: Estado de la Nación.

European Commission. 2013. _Early Warning Systems in Europe: Practice, Methods, and Lessons_ . Thematic Working Group on Early School Leaving. Brussels: European Commission.

Frazelle, Sarah and Aisling Nagel. 2015. _A Practitioner’s Guide to Implementing Early Warning Systems_ . Washington, D.C.: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educaitonal Laboratory Northwest.

Hotelling, H. 1933. “Analysis of a Complex of Statistical Variables Into Principal Components.” _Journal of Educational Psychology_ , 24: 417-441 and 498-520.

Jensen, Robert. 2010. “The (Perceived) Returns to Education and the Demand for Schooling.” _The Quarterly Journal of Economics_ 125 (2): 515-548.

Jolliffe, I.T. 2002. _Principal Component Analysis_ . New York: Springer-Verlag New York, Inc.

Fryer, Roland. 2013. “Information and Student Achievement: Evidence from a Cellular Phone Experiment.” NBER Working Paper 19113.

Gleason, P., and M. Dynarski. 2002. “Do We Know Whom to Serve? Issues in Using Risk Factors to Identify Dropouts.” _Journal of Education for Students Placed at Risk_ 7(1): 25–41.

Heller, Sara, Anuj Shah, Jonathan Guryan, Jens Ludwig, Sendhil Mullainathan, and Harold Pollack. “Thinking, Fast and Slow? Some Field Experiments to Reduce Crime and Dropout in Chicago.” NBER Working Paper No.21178. Cambridge, MA: National Bureau of Economic Research (NBER).

Janosz, M., I. Archambault, J. Morizot, and L.S. Pagani. 2008. “School Engagement Trajectories and their Differential Predictive Relations.” _Journal of Social Issues_ 64(1): 21–40.

Knowles, Jared. 2015. “Of Needles and Haystacks: Building an Accurate Statewide Dropout Early Warning System in Wisconsin.” _Journal of Educational Data Mining_ 7(3): 18-67.

Kohavi, Ron. 1995. “A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection.” Presented at the International Joint Conference on Artificial Intelligence.

19
Muthén, B. O. 2004. _Latent Variable Analysis: Growth Mixture Modeling and Related Techniques for Longitudinal Data._ In D. Kaplan (Ed.), The Sage handbook of quantitative methodology for the social sciences (pp. 345–370). Thousand Oaks, CA: Sage Publications.

Nguyen, Trang. 2008. “Information, Role Models, and the Perceived Returns to Education.” Unpublished manuscript.

O’Cummings, Mindee and Susan Therriault. 2015. “From Accountability to Prevention: Early Warning Systems Put Data to Work for Struggling Students.” AIR Early Warning Systems in Education Program. Washington, D.C.: American Institutes for Research (AIR).

Oreopoulos, Philip and Kjell Salvanes. 2011. “Priceless: The Nonpecuniary Benefits of Schooling.” _Journal of Economic Perspectives_ 25(1): 159-184.

Patrinos, Harry and George Psacharopoulos. 2004. “Returns to Investment in Education: a Further Update.” _Education Economics_ 12(2): 111-134.

Pearson, Karl. 1901. “On lines and planes of closest fit to systems of points in space.” _Philosophical Magazine_ , Series 6, 2(11): 559-572.

Rogosa, David. 2005. “Statistical Misunderstandings of the Properties of School Scores and School Accountability.” _Yearbook of the National Society for the Study of Education_ 104(2): 147-174.

Sara, Nicolae-Bogdan, Rasmus Halland, Christian Igel, and Stephen Alstrup. 2015. “HighSchool Dropout Prediction Using Machine Learning: A Danish Large-Scale Study.” _The European Symposium on Artificial Neural Networks (ESANN) Proceedings 2015_ .

Sheskin, David. _Handbook of Parametric and Nonparametric Statistical Procedures_ . Boca Raton: Chapman & Hall / CRC Press, 2004.

Stuit, David, Mindee O’Cummings, Heather Norbury, Jessica Heppen, Sonica Dhillon, Jim Lindsay, and Bo Zhu. 2016. _Identifying Early Warning Indicators in Three Ohio School Districts_ . Washington, D.C.: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educaitonal Laboratory Midwest.

United Nations. 2015. _The Millennium Development Goals Report_ . New York: United Nations.

World Bank. 2016. EdStats. Retrieved from: http://datatopics.worldbank.org/education/ 20
# Table 1. Event table

|||**Actu**|**al event**|
|---|---|---|---|
|||Drop out|Graduate|
|**Prediction**|Drop out|A (True positive)|B (False positive)|
||Graduate|C (False negative)|D (True negative)|

_Sensitivity_ = A / (A+C) _Precision_ = A / (A+B) _Specificity_ = D / (B+D) _False-Positive Proportion_ = _1-Specificity_ = B / (B+D) Figure 1. ROC curve illustration <!-- Start of picture text --> Perfect prediction<br>Random guessing: 45-degree line<br>0 .2 .4 .6 .8 1<br>False positive proportion<br>1<br>.8<br>.6<br>.4<br>.2<br>True positive proportion (Sensitivity)<br>0<br><!-- End of picture text --> 21
Figure 2. Dropout rates in Guatemala <!-- Start of picture text --> 35%<br>31.9%<br>30%<br>24.0%<br>25%<br>18.4%<br>20%<br>15%<br>10%<br>5%<br>0%<br>Primary to lower Within lower Lower to upper<br>sec. transition secondary secondary transition<br>dropped out<br>Percentage of students who<br><!-- End of picture text --> _Source_ : Author calculations based on administrative data on enrollment and the 2013 National Evaluations of students _Notes_ : (1) Dropout rates in primary to lower secondary transition computed as the percentage of sixth grade students evaluated in 2013 who did not enroll in 2014, 2015 and 2016. (2) Dropout rates within lower secondary computed as the percentage of sixth grade students evaluated in 2013 who were enrolled in 2014 but did not enroll in 2015 or 2016, excluding students who dropped out in the transition from primary to lower secondary from the calculation. (3) Dropout rates in lower to upper secondary transition computed as the percentage of ninth grade students evaluated in 2013 who did not enroll in 2014, 2015 and 2016.

Figure 3. Dropout rates in Honduras <!-- Start of picture text --> 40% 37.4%<br>35% 33.2%<br>30%<br>25%<br>18.4%<br>20%<br>15%<br>10%<br>5%<br>0%<br>Primary to lower sec. Within lower secondary Lower to upper<br>transition secondary transition<br>Percentage of students who dropped out<br><!-- End of picture text --> _Source_ : Author calculations based on USINIEH administrative data from 2013-2015 _Notes_ : A dropout is a student who was enrolled in the previous year but did not enroll the following year.

22
Table 2. Covariates included in the models of dropout in the primary to lower secondary transition

|**Specification**|**Guatemala**|**Honduras**|
|---|---|---|
|Model 1|**Individual level:**<br>female, age, indigenous, works,<br>housework, repeater, preschool,<br>all_school_supplies, motivation,<br>test_score|**Individual-level, supply-side, and**<br>**municipal-level:**<br>male, age, attendance rate, public<br>school, rural school, number of<br>secondary schools, number of<br>teachers, summary infrastructure<br>index, homicide rate, poverty<br>(UBN), share of ethnic population,<br>average years of education, average<br>pre-school enrollment, average rate<br>of child labor, adolescent birth rate,<br>share of household with migrants|
|Model 2|**Individual and household level:**<br>covariates in model 1, parents'<br>maximum educational attainment<br>(dummies), help_homework,<br>low_quality_housing, electricity, pc,<br>books, goods_availability_index|<br>**Individual-level and municipality**<br>**fixed effects:**<br>male, age, attendance rate, public<br>school, rural school, municipality<br>fixed effects|
|Model 3|**Individual, household, and**<br>**school/area of residence level:**<br>covariates in model 2, school fixed<br>effects,hs_wage_premium|**Individual-level and school fixed**<br>**effects:**<br>male, age, attendance rate, school<br>fixed effects|

23
<!-- Start of picture text --> So<br>es 2 EE Ee<br>an Pee lo ey<br>oa oeaa © “ ae<br>é a = oo<br>i : [oy :<br>i 4, a qi<br>fog irs, Te)N4 4; f:<br>iy om<br>iga), ,'<br>: o |:<br>lo}<br>fo)<br>0.00 0 . 25 0 . 50 0 . 75 1 . 00<br>1 - Specificity<br>——— Model1 Model2<br>--------- Model3. ———— Reference<br>3+ 8<br>aa | > x<br>4<br>renm 4ge | Te )<br>oa SL foN}<br>ff<br>28=>J y £2 ~ 8<br>° c o<br>a / (a)<br>o ao<br>rs]w Nw<br>oO o | Oofo ° }<br>o 0.00 0.25 0.50 _ 0.75 1.00 lo} 0.00 0.25 0.50 0.75 1 . 00<br>1-Specificity 1 - Specificity<br><!-- End of picture text -->
Table <u>3. Performance of the models of dropout in the primary to lower secondary transition</u>

|**Country and**<br>**specification**|**Sensitivity**<br>**(percent)**|<br>**False-positive**<br>**proportion**<br>**(percent)**|**Overall accuracy**<sup>**1**</sup><br>**(proportion)**|
|---|---|---|---|
|_Guatemala_||||
|Model 1|67.8|32.9|0.67|
|Model 2|71.6|30.7|0.70|
|Model 3|80.0|21.2|0.79|
|_Honduras_||||
|Model 1|68.8|28.9|0.70|
|Model 2|69.1|27.2|0.71|
|Model3|78.2|19.8|0.79|

Source: Author calculations on administrative data from Guatemala and Honduras. Note: This is the area-under-the-curve statistic, which ranges from .50 to 1.00, with higher values associated with higher accuracy (higher sensitivity and lower false-positive proportion).

25
Table 4. Out-of-sample performance in Guatemala and Honduras based on K-fold cross validation

|**Transition from**<br>|Mean<br>**primary to lo**|SD<br>**wer secondary**:**Gua**|Min<br>**temala**|Max|
|---|---|---|---|---|
|_Sensitivity_|||||
|In sample|81.27|0.52|80.10|82.40|
|Out of sample|69.26|0.72|67.95|70.67|
|_False-Positive P_|_roportion_||||
|In sample|20.94|0.47|19.90|21.98|
|Out of sample|24.51|0.52|23.54|25.74|
|**Transition from**<br>|**primary to lo**|**wer secondary**:**Hon**|**duras**||
|_Sensitivity_|||||
|In sample|78.26|0.16|77.94|78.64|
|Out of sample|72.27|0.19|71.87|72.73|
|_False-Positive P_|_roportion_||||
|In sample|19.04|0.18|18.68|19.48|
|Out of sample|21.73|0.19|21.37|22.17|

26
Figure 6. Out-of-sample performance of early warning system in Honduras <!-- Start of picture text --> 90.0<br>78.2<br>80.0 76.3<br>70.0<br>60.0<br>50.0<br>40.0<br>30.0<br>19.8<br>20.0 15.9<br>10.0<br>0.0<br>Sensitivity False alarm rate<br>In sample (2013-2014) Out of sample (2014-2015)<br>Percent<br><!-- End of picture text --> _Source_ : Author calculations based on USINIEH administrative data from 2013-2015 Table 5. Relative performance of the early warning system as a method to allocate limited <u>resources ($200 US per student)</u>

||**Nb f**||||**Me**<br>|**thod of al**<br>|**location**<br>||||
|---|---|---|---|---|---|---|---|---|---|---|
|**Country**|**umer o**<br>**students**|<br>**Munic**|<br>**ipality po**|<br>**verty rates**|<br>**Sc**|<br>**ool dropo**|<br>**ut rates**||<br>**EWS**||
|||_True_<br>_positives_|_False_<br>_positives_|_Dollars_<br>_wasted_|_True_<br>_positives_|_False_<br>_positives_|_Dollars_<br>_wasted_|_True_<br>_positives_|_False_<br>_positives_|_Dollars_<br>_wasted_|
|_Guatemala_|5,000|45%|55%|$ 552,022|93%|7%|$ 72,772|99%|1%|$ 8,319|
||10,000|43%|57%|$ 1,143,022|86%|14%|$ 277,037|89%|11%|$ 210,860|
||20,000|35%|65%|$2,609,953|73%|27%|$1,086,566|78%|22%|$898,115|
|_Honduras_|5,000|58%|42%|$ 418,000|87%|13%|$ 129,000|99%|1%|$ 6,200|
||10,000|60%|40%|$ 798,200|82%|18%|$ 364,000|99%|1%|$ 12,400|
||20,000|56%|44%|$1,748,000|74%|26%|$1,046,800|95%|5%|$188,200|

_Source_ : Author calculations based on administrative data from Guatemala and Honduras.

_Notes_ : The allocation of resources by municipality poverty rates selects the students who live in the municipalities with the highest poverty rates. The allocation of resources by school dropout rates selects the students who are in the schools with highest dropout rates in the sixth to seventh grade transition. The allocation of resources using the early warning system selects the students who have the highest predicted probability of dropout according to the model in Table A6. True positives are the percentage of hypothetical beneficiaries who dropout in the transition from primary to lower secondary, and false positives are the percentage who do not dropout in this transition. Dollars wasted are computed as $200 x Number of students x False positives.

27