metadata: authors: 0: author: Evans, David K.
1: author: Marshak,Anastasia
2: author: Sabarwal,Shwetlena
count: Sierra Leone
docna: 0: docna: The permanent input hypothesis : the case of
            textbooks and (no) student learning in Sierra Leone
docty: Policy Research Working Paper
subtopic: Primary Education,Economics of Education,Educational Sciences,Educational Institutions & Facilities,Educational Populations,Education for Development (superceded),Education Finance,Educational Policy and Planning - Textbook,Effective Schools and Teachers,Education For All
teratopic: Education
abstracts: cdata!: A textbook provision program in Sierra
            Leone demonstrates how volatility in the flow of
            government-provided learning inputs to schools can induce
            storage of these inputs by school administrators to smooth
            future consumption. This process in turn leads to low
            current utilization of inputs for student learning. A
            randomized trial of a public program providing textbooks to
            primary schools had modest positive impacts on teacher
            behavior but no impacts on student performance. In many
            treatment schools, student access to textbooks did not
            actually increase because a large majority of the books were
            stored rather than distributed to students. At the same
            time, the propensity to save books was positively correlated
            with uncertainty on the part of head teachers regarding
            government transfers of books. The evidence suggests that
            schools that have high uncertainty with respect to future
            transfers are more likely to store a high proportion of
            current transfers. These results show that reducing
            uncertainty in school input flows could result in higher
            current input use for student learning. For effective
            program design, public policy programs must take
            forward-looking behavior among intermediate actors into account.
geo_reg_key: 1927320,517191
repnme: repnme: The permanent input hypothesis : the case
            of textbooks and (no) student learning in Sierra Leone
display_title: The permanent input hypothesis : the
            case of textbooks and (no) student learning in Sierra Leone
listing_relative_url: /research/2014/09/20173246/permanent-input-hypothesis-case-textbooks-no-student-learning-sierra-leone
topicv3: Teachers Management m1326119 1523,Primary Education m1326139 480,Textbooks m1326802 2129,Education m1327048 470,User Fees in Education m1327515 1493
publishtoextweb_dt: 2014-09-01T00:00:00Z
historic_topic: Education
no_of_pages: 37
author_keys: 0: author_key: 966560
1: author_key: 950739
model_extractions: WPS7021 Policy Research Working Paper 7021 The Permanent Input Hypothesis The Case of Textbooks and (No) Student Learning in Sierra Leone _Shwetlena Sabarwal David K. Evans Anastasia Marshak_ Education Global Practice Group & Africa Region Office of the Chief Economist September 2014
Policy Research Working Paper 7021

# **Abstract**

A textbook provision program in Sierra Leone demonstrates how volatility in the flow of government-provided learning inputs to schools can induce storage of these inputs by school administrators to smooth future consumption. This process in turn leads to low current utilization of inputs for student learning. A randomized trial of a public program providing textbooks to primary schools had modest positive impacts on teacher behavior but no impacts on student performance. In many treatment schools, student access to textbooks did not actually increase because a large majority of the books were stored rather than distributed to students. At the same time, the propensity to save books was positively correlated with uncertainty on the part of head teachers regarding government transfers of books. The evidence suggests that schools that have high uncertainty with respect to future transfers are more likely to store a high proportion of current transfers. These results show that reducing uncertainty in school input flows could result in higher current input use for student learning. For effective program design, public policy programs must take forwardlooking behavior among intermediate actors into account.

This paper is a product of the Education Global Practice Group and the Office of the Chief Economist, Africa Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank. org. The corresponding authors may be contacted at ssabarwal@worldbank.org and devans2@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **The Permanent Input Hypothesis: The Case of Textbooks and (No) Student Learning in Sierra Leone**

Shwetlena Sabarwal World Bank David K. Evans Anastasia Marshak World Bank Tufts University Keywords: Education, Impact Evaluation, Sierra Leone, Smoothing JEL Codes: O15, I21, I28, J13 Acknowledgements: The team is extremely grateful for the support of the World Bank’s Education Program Development Fund, which provided financing. Keiko Inoue co-designed the evaluation, and Eunice Yaa Brimfah Ackwerh supervised implementation of the intervention. The team benefitted from research assistance from Anna Popova, and comments and guidance from Deon Filmer, James Habyarimana, Peter Holland, Luis Benveniste, Owen Ozier, Christopher Thomas, Ritva Reinikka, Arianna Legovini, Muna Meky, Jee-Peng Tan, and participants in seminars at Brigham Young University, the World Bank, and the Pacific Development Economics Conference. Corresponding authors: Shwetlena Sabarwal (ssabarwal@worldbank.org) and David Evans (devans2@worldbank.org).
# **1. Introduction**

In the macroeconomic literature, volatility in aid flows has been linked to reductions in economic growth (Kodama 2012) and significant decreases in overall welfare (Arellano et al. 2009). This paper explores similar patterns within one sector, education, at a micro level. We provide evidence that unpredictable variability in flows of school inputs interacts with the desire of actors (in this case, school administrators) to smooth input utilization over time, resulting in low current utilization of these inputs for student learning.

In education, a growing body of empirical evidence shows that public provision of school inputs does not always lead to improved learning outcomes. In fact, there is very wide variation in estimated coefficients of school inputs on test scores (Glewwe et al. 2011; Kremer, Brannen, and Glennerster 2013; Krishnaratne, White and Carpenter 2013; McEwan 2014). This may be puzzling, especially with regard to inputs like textbooks which are both demonstrably scarce in developing country schools and at the same time believed to be vital in the education production function.

Clearly, a failure to detect the impact of input provision on student learning could arise due to several reasons. It could be because the inputs are ineffective due to their inherent nature or their quality, resulting in a zero impact on student performance. For instance, a randomized evaluation from Kenya showed that providing textbooks did not raise average test scores, possibly because textbooks were written in English, which is a third language for most students, making it difficult for them to use textbooks effectively (Glewwe, Kremer, and Moulin 2009).

Lack of impacts from input provision could also be because beneficiaries re-optimize their allocation of resources when receiving an input. For example, Das et al. (2011) show in both Zambia and India that when school grants for student learning inputs were anticipated by households, the households compensated by reducing private education spending. Alternatively, a study from Kenya shows that insecticide-treated bed-nets distributed to children under age 5 for use over their beds were actually being used for fishing and for drying fish in Lake Victoria (Minakawa et al. 2008). These and other studies provide evidence that behavioral responses of agents to public programs can mitigate program impacts on ultimate beneficiaries.

In this paper we present evidence of another scenario where the behavioral response of intermediate agents to a public program strongly influenced program outcomes on 2
beneficiaries. In this case the agents in question are school administrators, an often neglected set of actors in empirical work. Analyzing the results from a randomized evaluation of a textbook provision program to schools, we show that the program failed to have any impacts on student outcomes (attendance or performance) because few textbooks reached the students. Instead, publicly provided textbooks were stored by school administrators.

Why should publicly provided school inputs to schools be stored rather than distributed to students? We analyze the correlates of book storage and find that head teachers who exhibit high uncertainty with respect to government transfers of textbooks are significantly more likely to store them. Higher uncertainty is measured by the answers to two queries: if head teachers indicate that they knew how many books the government was to allocate to their school in the previous year, and whether they were able to provide a specific number. This storing behavior in the face of greater uncertainty makes sense from the perspective of consumption theory: Consumption of current government transfers will be dictated in part by expectations regarding future transfers. If current transfers are seen as a one-time (or transitory) shock then their impact on current consumption will be limited. In other words, when there is uncertainty, school administrators have incentives to store current transfers in order to smooth consumption over time, particularly for inputs like textbooks that depreciate quickly when used.

As mentioned earlier, this paper sheds light on the importance of behavioral responses of intermediate agents like school administrators in determining the overall impacts of public input provision programs to schools. In particular, it provides an empirical basis to question the assumption that publicly provided inputs are deployed effectively within a short time period. It also puts at center stage the question of how future uncertainty regarding government transfers and policies impacts the current behavior of service providers.

On the whole, this analysis adds to our understanding of why such input provision programs might be less effective than expected. Broadly speaking, our results are consistent with the premise that merely providing inputs to schools while ignoring the incentive structure within schools and other systemic challenges is unlikely to yield the desired gains in student learning outcomes.

The remainder of the paper is structured as follows: Section 2 provides a broad outline of the education system in Sierra Leone; Section 3 lays out the experimental design.

3
Statistics from baseline, actual implementation of the program, and the empirical strategy are summarized in Sections 4, 5, and 6. Section 7 presents the overall impacts of the program. Results are discussed in sections 8, 9, and 10. Finally, conclusions and policy implications from this empirical work are laid out in Section 11.

# **2. Context**

Over a decade ago, Sierra Leone emerged from a brutal, decade-long civil war (1991– 2002), but the impacts linger. The civil war severely impacted the country’s education system leading to large-scale devastation of school infrastructure, severe shortages of teaching materials, overcrowding in many classrooms in safer areas, displacement of teachers, frequent disruptions of schooling, psychological trauma among children, poor learning outcomes, weakened institutional capacity to manage the system, and a serious lack of information and data to plan service provision (World Bank 2007). The education sector is still in the process of being rehabilitated. The post-war “catch-up” phase and the accompanying construction boom supported by donors, together with the Government’s Education Act of 2004 which abolished primary education fees for all, have all contributed to this rehabilitation, driving a sharp increase in access and gross enrollment. Nonetheless, in 2011 more than half of the working age population in Sierra Leone had no formal education, with an average attainment of just 4 years of schooling (World Bank 2014).

The Education Act of 2004 stipulates universal basic education – 6 years of primary school and 3 years of junior secondary school. Enrollments in primary school more than doubled since 2000, with the gross enrollment rate (GER) reaching 115.7 percent in 2011. Two-thirds of children enrolled in primary education in Sierra Leone attended government schools, 31 percent attended religious schools, and only 6 percent (composed predominantly of children from the two richest quintiles) attended private schools. The primary completion rate was 69 percent in 2011, 11 percentage points below the average for sub-Saharan Africa (World Bank 2014).

Several challenges to the provision of basic education persist, many of them supplyrelated. As of 2008, sixth-grade pupils from 5,000 primary schools competed for places in fewer than 300 junior secondary schools, for example. Although the government abolished school fees, primary education is still not completely free because schools impose a variety of charges on their students, such as informal tuition fees and auxiliary costs such as school uniforms and books. Our baseline data (see Section 3.2 for more details) show that around 28 percent of the schools charged some degree of fees from parents. As many as 70 percent of the schools received subsidy payment for school fees 4
from the government, while 28 percent of the schools also received support from the community, but there is no systematic correlation between the two sources of support – one is not a substitute for the other.

Most schools in Sierra Leone have very poor classroom conditions and still lack sufficient learning materials and adequately qualified teachers (World Bank 2007). The government and development partners have made efforts to provide textbooks, but significant challenges remain. One study reported that, in 2004, a ratio of 1 set of textbooks to 3 pupils in urban areas and 1 set to 6 pupils in rural areas had been reached, whereas the official policy was a ratio of two students for one set of four coresubject textbooks (World Bank 2007).

With the Local Government Act of 2004, the Government of Sierra Leone commenced a national decentralization process: Primary education stood among the first functions scheduled to devolve to the Local Councils. Under the policy of decentralization, local councils would have full control and supervision of all primary schools including such functions as the recruitment and payment of teachers, the provision of textbooks and teaching materials, and the rehabilitation and construction of schools. Our baseline data show that the process of decentralization was still ongoing in the education sector of Sierra Leone as of early 2008. District Education Officers (DEO) in the centralized system and local council officers in the decentralized system were both active, leading to some confusion on the exact chain of command and roles and responsibilities of different agents.

# **3. Experimental Design**

## **3.1 The Intervention**

This paper evaluates a basic textbook distribution program of the Government of Sierra Leone in the year 2008. Under the program, textbooks were provided by the Ministry of Education, Youth, and Sports (MEYS) to primary schools based on student enrollment numbers as captured by the Education Management Information System – or EMIS. The actual transportation of books from central warehouses to schools was undertaken by local service providers who were competitively selected by the Government. The main objective of the textbook distribution program was to provide a set of core textbooks for every child in the treatment schools.<sup>1</sup>

> 1 A set of core textbooks is made up of English, Mathematics, Integrated Science and Social Studies textbooks.

5
While there were just over 5,000 primary schools in Sierra Leone at the start of the program, the impact evaluation (IE) only focuses on schools that were registered with MEYS as having up to grade 6, the year when the National Primary School Examination is taken. The program included government, government assisted, and community schools, but not private schools. The IE focused on grades 4 and 5, as these are the standards by which education is principally in English, the language of the textbooks.<sup>2</sup>

# **3.2 Sampling**

The IE relies on a randomized experimental design using a two-step process. First, 4 out of 19 local councils, stratified at the regional level, were randomly selected. All 19 local councils were interested in participating in the study, thus one council from each of Sierra Leone’s four regions was chosen: Kailahun (Eastern region), Kambia (Northern region), Pujehun (Southern region), and Western Urban and Western Rural (who agreed to enter the randomization jointly). The locations of these councils are illustrated in Figure 1.

Within the randomly selected councils, 360 program schools were randomly selected using the EMIS data. From the universe of schools in these councils, schools that were already being targeted for textbook interventions by other organizations were excluded. Also, to avoid providing textbooks to schools that already had many, this project included only schools which had a 3:1 student-textbook ratio or higher. From the reduced sample, 90 schools were selected in each district (except for Western Rural and Western Urban which together amounted to 90 schools).

Within the local council, sample schools were assigned with equal probability to one of three treatment groups (30 schools in each group in each district): receiving textbooks, receiving textbooks plus teacher training, and a control group. Ultimately, due to multiple implementation challenges, the teacher training component was not undertaken. Therefore, for the IE, we are left with schools that received textbooks (treatment schools) and schools that did not (control schools).

Baseline data were captured on 341 schools (out of a total of 360). The 19 schools that were not included in the sample were mainly due to the inability of enumerators to access the schools due to either poor road conditions or flooded rivers. For a small portion of these schools, the enumerators were able to access the location, but there

> 2 Education at earlier grades is sometimes carried out in the Krio language.

6
was no school at the premises. This could partly be attributed to the large share of “ghost schools” in Sierra Leone.<sup>3</sup> The total number of schools for which data are available at baseline and endline, by district, is summarized in Table 1.

Comparing baseline characteristics for treatment and control schools, we find that the sample is balanced across a wide range of characteristics including infrastructure, student-teacher ratio, school fees, test scores, and many others (Table 2). Of 18 such characteristics, a significant difference (at the 90% level or higher) between treatment and control schools is found only for the student-classroom ratio. Thus, the two groups were well balanced at baseline.

# **3.3 Data**

Baseline data collection took place in May 2008 and final or endline data collection took place in December 2009. The data collected included a head-teacher survey, a classroom teacher survey, a student survey, and student exams.

The data on the school as a whole were obtained through direct observation (e.g., the condition of the buildings, the number of classrooms and other facilities) and a comprehensive interview with the head teacher addressing school finances, record keeping, community participation, management practices, etc. Data were also collected from teachers for grades 4 and 5 through teacher interviews.

One hundred students were selected randomly at each school and were given a written numeracy and literacy test. At baseline (May 2008) the student tests were administered to students in Grades 3 and 4 and at endline the same tests were administered again to the same cohort, which was now in Grades 5 and 6 (Dec 2009). Although we have a large number of tested students, only a small fraction were able to be definitively matched from baseline to endline. Therefore, to measure impacts on student performance, our preferred estimates rely on endline scores only, controlling for schoolcohort-subject mean at baseline and – beyond that – relying on the assumption of balance on average across schools at baseline. We also report results for the change in score of those students who were matched for baseline and endline, but these results may be affected by selective attrition in test participation.

> 3 A 2008 news item highlighted the challenge of “ghost schools,” which the Government was seeking to identify and shut down (Fofana 2008). In 2011, the Government of Sierra Leone announced that they had discovered over 5,000 “ghost teachers,” teachers that do not exist but are used to collect payroll on behalf of corrupt principals and officials (Coker 2011).

7
At endline, we also interviewed students from Grades 4 and 5, in order to characterize actual use of textbooks; the books distributed by the program were targeted to Grades 4 and 5.

We were unable to gather data on 5 percent of the schools originally designated to treatment and control groups. Table A1 shows the correlates of attrition at endline. The dependent variable is a dummy equal to 1 if the school attrited by the time of the endline survey, and we include a number of control variables. We estimate the attrition regressions using a linear probability (ordinary least squares, or OLS) model. We estimate the relationship between attrition and both assignment to treatment (column 1) and actual treatment (column 2); results are consistent across both estimates. In both specifications standard errors are clustered at the school level. Table A1 shows that neither being assigned as a treatment school nor actually receiving additional books predicted attrition. This is robust to controlling for a number of baseline household characteristics: number of classrooms, electricity, distance to the DEO’s office, and number of students. From both models we see that having a greater number of classrooms at baseline decreases the probability of attrition, while having electricity increases it, although none of the other controls are significant predictors (except the dummy variable for having missing data on the distance to the DEO’s office). Overall, these balanced rates of attrition across treatment and control groups suggest that the impact evaluation results are unlikely to be driven by attrition.

# **4. Education Service Delivery in Sierra Leone: Evidence from the Baseline Survey**

One issue that emerged from the baseline survey in the surveyed councils (Kailahun, Kambia, Pujehun, Western Urban and Western Rural) is the stark supply-side disadvantage that remote areas face with respect to education service delivery. Pujehun, which is one of the more remote councils in Sierra Leone, was significantly worse off compared to the other regions in terms of the supply of schooling inputs. In this region, less than 50 percent of all schools were considered to be in good condition, with almost 20 percent falling under the category “no roof, walls are heavily damaged, needs complete rehabilitation.” Further, almost one-third of the schools did not have a working toilet.

During the baseline survey (May 2008), unannounced classroom visits were carried out in Grade 4 and 5 classrooms. Out of 828 such classrooms, teachers were found to be actively teaching in 54 percent of cases. If we include classroom management activities 8
(organizing the class, setting the agenda, or disciplining students) then the corresponding number was 77 percent. Once again, the remote councils performed worse than others: Only half of all teachers were engaged in teaching or classroom management in Pujehun, while the corresponding numbers for some other councils were 90 percent.

As a part of baseline data collection, students in Grades 3 (8,740 students) and 4 (8,815 students) were given multiple-choice mathematics and literacy tests. For mathematics, 49 percent of Grade 3 students and 32 percent of Grade 4 students could not answer the single-digit subtraction question “Calculate 9-4”. Similarly for literacy, when presented with the word “bird”, 45 percent of Grade 3 students and 35 percent of Grade 4 students could not identify which one of four four-letter words provided was identical to it. Samples of results from the math and literacy tests are summarized in Figures 2a and 2b.

A student in grades 4 or 5 requires four textbooks: math, English, social studies, and science. At baseline, students only had 2 textbooks on average, far from the government stipulated level of 4 textbooks per student. Further, only about 40 percent of the teachers claimed that they allowed students to take textbooks home, which implies that student access to textbooks at home is very low. To complement this, at endline, around 36 percent of the sampled schools reported parents complaining about lack of textbooks in the last six months. The likelihood of parents complaining about a lack of learning materials is not systematically related to the existence of a school management committee or to schools charging fees.

# **5. Intervention Implementation**

To determine how well the intervention was implemented, two sources of data are used: (i) information from the head teacher at endline and (ii) data on textbook distribution collected by the IE team from district education officers and service providers.<sup>4</sup> Using these data we determine how many schools actually received books from the project.

As a second step, head teacher data are used to determine if the school received textbooks from other sources (such as from local government, non-government

> 4 Head teacher data are given precedence; administrative data are used when head teacher data are missing. In case head teacher and administrative data contradict each other, head teacher data are used, except in 26 cases where the head teacher claims the school received books from the project but is unable to provide information on the number of books received or date books were received.

9
organizations, or from MEYS but outside the project). This allows us to create a variable for actual treatment, i.e., schools in the program that actually received textbooks in 2008-09.

Contrasting the measure of actual treatment with the original, randomized allocation, we observe some non-compliance. Fifteen percent of schools assigned as treatment did not receive any textbooks, and 46 percent of schools assigned as control received textbooks from one source or another. Still, assignment increased the probability of receiving books from 46% to 85%. This information is summarized in Table 3.

# **6. Empirical Strategy**

Given the imperfect compliance to random assignment, the evaluation first reports Intent-to-Treat (ITT) analysis for identification of program impacts. In ITT analysis, participants are analyzed as if they received the treatment to which they were assigned (Begg 2000). This analysis yields an unbiased estimate only of being assigned to a treatment and not of actually receiving the treatment. Hence, using this methodology we estimate the impact of being assigned to receive textbooks and not of actually having received textbooks. These are likely to be underestimates of the true impact of receiving textbooks.

Outcomes are tested using OLS regression models that use assignment to the treatment group as the explanatory variable and thereby calculates how much an outcome of interest – e.g., test scores – differs between schools that are assigned as treatment schools and those that are assigned as control schools.

where Outcomei is the outcome in school _i_ , and _Assignment_ = 1 if the school was randomly assigned as a treatment school and 0 otherwise.

Outcomes are tested at the classroom and student levels when appropriate. At the classroom and student level the specification is as follows: where Outcomeis is the outcome in classroom _i_ and school _s_ for the classroom level regression, and it is the outcome for student _i_ and school _s_ for the student level regression. The variable _Assignment_ = 1 if the school was randomly assigned as a 10
treatment school and 0 otherwise. The error term εis is clustered at the school level in these specifications to account for intra-school correlation of outcomes. The parameters of interest are β1 and γ1 , which are the average effect of being assigned as a treatment school in the textbook provision program.

To supplement this analysis, treatment-on-the-treated (TOT) estimates are also presented. Because actual provision of textbooks to schools outside the randomized assignment could be a function of observed and unobserved school characteristics which may also be correlated with the outcomes of interest (i.e., endogenous), we cannot simply compute the difference between outcomes of schools that received textbooks with those that did not.

To correct the potential endogeneity, we employ instrumental variables estimation, using the randomly assigned treatment or control status of a school as an instrument for actual school treatment. The treatment status of a school has a high correlation with actual provision of textbooks to schools (satisfying the first stage requirement for instrumental variables), and it is orthogonal to school characteristics because it is randomly assigned (satisfying the exclusion restriction).

For TOT estimation we rely on a two stage least squares model. In the first stage we estimate: where Pi is an indicator of whether school _i_ actually received textbooks during the intervention period. In the second stage, the predicted values of Pi – labelled P<sup>�</sup> ı – are included as an independent variable: The key parameter of interest is again β1 , which represents the average impact of the program for those schools that received books due to the random assignment.

We calculate both ITT and TOT estimates for all outcomes and present both: Panel A contains ITT estimates and Panel B contains TOT estimates throughout the results tables. In general, the pattern of results is highly consistent across ITT and TOT estimates, and we discuss the estimates separately only in cases where the direction or significance varies by estimation technique. However, in the discussion and conclusion 11
sections we rely more heavily on the ITT analysis, as the cleanest estimate of the impact of being assigned as a treatment school for receiving textbooks.

# **7. Overall Impacts**

At the school level, we find no impact of the program on student enrollment. We do find, however, that the likelihood of parents complaining to head teachers about the lack of textbooks is substantially lower in treatment schools (Table 4).

Information on teaching practices was collected through interviews with teachers for Grades 4 and 5 at endline. These impacts are summarized in Table 5, wherein each teacher interview is a distinct observation and standard errors are clustered at the school level. We find that the program had no impact on the likelihood of a teacher assigning homework from textbooks (Table 5, Columns 1 and 2). This is most likely because the reported propensity to assign homework from textbooks is fairly high even in control schools (93 percent). One may question how teachers are able to assign homework from textbooks if very few students have access to the textbooks. Anecdotal evidence suggests that this is very common: Teachers copy textbook questions onto the blackboard and ask students to copy them down and answer them at home as a part of their homework. Likewise, the program did not influence the likelihood of physical punishment for students found mistreating textbooks (Columns 3 and 4).

We also find that the program did not impact the likelihood of a teacher producing pamphlets (informal notes or study guides) for students (Columns 7 and 8). However, we do find significant positive impacts of the program on the likelihood of a teacher having a lesson plan (Columns 5 and 6).

Student attendance data were collected through unannounced visits to classrooms (Grades 4 and 5) at endline. We see no pattern of improved attendance, although we do observe a positive significant effect on girls’ attendance in grade 5. No other discernible program impacts on student attendance by grade or by student gender are found (Table 6).

These unannounced classroom visits also yielded data on teacher activity (Grades 4 and 5) in class. The program did not affect the likelihood of finding a teacher in class but did lead to an increased likelihood of finding a teacher in class _teaching_ (Table 7). Note that in the control group about 18 percent of the teachers were absent from the classroom at the time of the visit; however, of the teachers in class, only 57 percent were found to 12
be actually teaching. TOT estimates suggest that the program dramatically increased inclass teaching, to about 90 percent (Table 7 Column 2).

However, we find no program impacts on learning outcomes as measured by the literacy and mathematics exams (Table 8). As noted above, impacts on student performance are measured by comparing student test scores in literacy and mathematics at endline between control and treatment schools, controlling for baseline school-cohort-subject average scores. Standard errors are clustered at the school level.<sup>5</sup> Therefore, on the whole we find that the textbook provision program had some positive impacts on teacher behavior (increased presence of lesson plan and increased likelihood of being in class _and_ teaching) but no discernible impacts at the student level (either on attendance or performance).

# **8. What explains the lack of program impacts on students?**

In order to fully understand the reasons for absence of program impacts at the student level, we exploit student survey data to examine how the program impacted textbook use by students. These data come from interviews conducted with students in Grades 4 and 5 at endline.

We find that the government textbook provision program did not significantly impact students’ use of textbooks. Table 9 summarizes program impacts on indicators of student textbook use and related behaviors. We see that the program did not increase the likelihood that students in treatment schools were using a textbook in class for a core subject, have a new textbook, or use a textbook given to them by a teacher. Likewise, the program did not impact the number of students with whom textbooks were shared, or the likelihood of receiving pamphlets (informal notes or an exam guide) from teachers.

Information was also collected on materials used by students for exam preparation, including books, pamphlets, and notes (Table 9 Columns 6-8). On these indicators as well, the program had no discernible impacts.

> 5 The test estimates do not use matched scores for individual students between baseline and endline because of extensive difficulties in matching, mainly because certain first and last names occurred with high frequency and students used inconsistent placement of first, middle, and last names. As a result, less than one-third of students could be matched, and we cannot rule out selective attrition in the matched sample. Nevertheless, we estimated program impacts on test scores even in that sub-sample and found no evidence of impact.

13
Student use of textbooks is often heavily contingent on teacher behavior. We examine program impacts on teacher behaviors with relation to textbooks in Table 10. This information also comes from student interviews. We see that while the program had modest impacts on the likelihood of teachers encouraging the use of textbooks by students, it did not impact any of the more direct predictors of textbook use, such as students reading from textbook in class, students being allowed to take textbooks home, and homework assignment in general.

Given the lack of program impacts on student textbook use, it is not surprising that we observe no impacts of the textbook provision program on student performance. We are, however, faced with a puzzle – when schools, by their own admission, received textbooks from the government, why then did the students’ actual use of these textbooks did not increase? What happened to the government-provided textbooks?

To answer this question, we again turn to directly observed data collected at endline. During unannounced classroom visits, data were collected on the number of textbooks per student in class. We find that, at endline, textbooks per capita in classrooms were not significantly higher in treatment schools than in control schools (Table 11 Columns 1 and 2).

During data collection, enumerators were instructed to request access from the head teacher to the place where textbooks were stored and to count the textbooks (of the 4 core subjects for grades 4 and 5) in storage. We find that in treatment schools the number of core textbooks stored per students present in school is significantly higher than in control schools (Table 11 Columns 4-6). Comparing books stored per capita at endline and baseline, we find that the program led to a strong increase in the number of books stored in treatment schools (Table 11 Columns 7-10). Therefore, a major reason for lack of student access to textbooks either in the classroom or at home could be that many books are stored by schools and not distributed to students.

Presumably, this lack of access did not extend to teachers, given that the program positively impacted teacher behavior on a number of dimensions.<sup>6</sup> This suggests that teachers may have used a few books to facilitate teaching (hence the significantly higher rates of teaching in treatment schools) but without providing broad access to students.

> 6 These results suggest that it would be of interest to derive impacts of textbooks on student performance after controlling for storage. However, book storage is endogenous to other factors (e.g., school needs, teacher capacity) and the authors were unable to find a credible instrument for storage.

14
# **9. Why are textbooks stored and not used?**

If school administrators exhibit a high propensity to store inputs instead of providing them to students, input provision programs to schools are likely to have only limited effectiveness. Therefore, it becomes imperative to understand what prompts school administrators to save government provided inputs instead of using them.

To investigate underlying factors that predict the propensity to store, we estimate a cross-sectional model in which book storage (per student present) is regressed on a set of school level controls.

books stored per capitai = α+ β1Ai + β2Xi + εi (5) where books stored per capitai is the number of books stored per student present in school _i_ at endline, Ai is a dummy for whether school _i_ expected to receive books in the academic year 2008-09, and Xi includes the number of books stored at baseline as well as other school level controls that could influence propensity to store. These include the presence of the head teacher at the time of interview, remoteness of the school (distance from nearest paved road and from DEO’s office), parental pressure (whether parents complain to head teacher about lack of textbooks), whether the school charges fees, and whether the school received books from the government in the previous academic year.<sup>7</sup> As noted above, among independent variables we include a proxy for whether the school administrators “expected” to receive books or not ( Ai) . The proxy for expectations of receiving inputs from government is created using two head teacher questions at endline:

- Expectations proxy 1: If the head-teacher answers yes to the question: “Do you know how many books the government was to allocate to your school for Grades 4 and 5?”

- Expectations proxy 2: If the head-teacher is able to answer the question: “How many books was the government to allocate to your school for Grades 4 and 5?”

> 7 In our sample of 341 schools, 71 schools reported receiving textbooks from the Government in 2008 (the previous academic year).

15
The underlying assumption in treating these variables as proxies for expectations is that head teachers who claim to know whether books were allocated to their schools or, more stringently, who claim to know the number of books that were allocated – must have had some expectations of receiving textbooks from the government in the current academic year. This proxy is not perfect due to the possibility of rationalization: Schools that actually received books may be more likely to claim that they expected to receive books.

Results are shown in Table 12. Columns 1 and 3 show results with expectations proxy 1 for ITT and TOT, and Columns 2 and 4 show results with expectations proxy 2. Book storage is significantly higher in treatment schools that did not expect to receive books. This behavior makes sense if we assume that current expectations are positively correlated with future expectations. Head teachers who have low current expectations (and hence low future expectations) of government transfers are more likely to store a significant portion of what they see as a transitory infusion of inputs, so that they can smooth “consumption” in future periods. In this context, consumption is defined as “use of textbooks by students.” The level effect of expecting books then represents the impact for comparison schools. The positive, significant coefficients are consistent with updating of expectations: Schools that expected books but did not receive them reacted by increasing their book storage. Although the direction of these – updating of expectation effects for schools that expected books but did not receive them – is consistent for the TOT estimates and their magnitude is greater, only the ITT estimates are significant.

This interpretation calls for the following assumptions, all of which are plausible in this context: (a) the head teacher’s (or class teacher’s) performance is measured by the average performance of students in her school (or class); (b) the head teacher’s and class teacher’s objective function is to maximize their performance over their tenure period; (c) the tenure period extends for more than one year; and (d) head teachers (or class teachers) believe increased student access to textbooks leads to better student performance.

In this scenario, low expectations and high uncertainty mean that current transfers are seen as a transitory positive shock by head teachers or teachers with a multi-year planning horizon or objective function.<sup>8</sup> To maximize their performance over a multi-

> 8 This further assumes that current transfers are not sufficient to lead to an updating of head-teacher or class teacher expectations from “no transfers” to “likely transfers every year.”

16
year period it is therefore optimal for school administrators to store part of the current transfers of textbooks to smooth future use.

There are other interpretations of this result. For instance, β1 could simply be capturing unobserved components of head-teacher quality that are correlated with the propensity to store books. It is possible, for example, that head teachers who store books are also those head teachers with less access to information on future transfers from government, potentially because of a lack of information channels. To this point, Table 12 highlights the role of road accessibility in influencing the propensity of input storage in schools. More remote schools are indeed significantly more likely to store books. But even after controlling for road accessibility, the impacts of head teacher expectations remain significant.

Similarly, it is possible that head teachers who store books are also those who have worse information regarding the benefits of distributing textbooks to students, potentially as a result not of limited information but because of poor motivation. However, it is not clear that less competent or motivated head teachers would be more likely to store books. Distributing books to students is not a time or labor intensive exercise, and storing books requires space and at least some level of organization, especially when at least a few of the books are being distributed. Note that we do not observe complete storage: “books in class” rises positively but insignificantly with treatment (Table 11). Note also that our results are robust to controlling for at least one – admittedly imperfect – proxy for head teacher motivation, the likelihood of finding the head teacher in school at the time of the unannounced data collection visit. Furthermore, we observe that for the comparison group, greater expectations actually led to higher storage (i.e., unfulfilled expectations led to storage), which is inconsistent with expectations being a simple indicator of head teacher quality.

There are of course other omitted variables that could be correlated with head-teacher expectations. In the absence of reliable time-series data on past transfers of textbooks from government to schools, it is difficult to make strong claims regarding expectations and storage behavior on the part of school administrators. That said, the interpretation of rationing or “consumption smoothing” is consistent with the available data.

The “consumption smoothing” hypothesis, with few books distributed to classrooms, is particularly striking in conjunction with the observed positive impacts of treatment on teacher behavior. As seen above, the treatment – provision of textbooks to schools – leads to a significant increase in the likelihood of teachers having lesson plans (Table 5) 17
and the likelihood of teachers being in class teaching (Table 7). This behavior is consistent with teachers in treated schools receiving textbooks. Because the number of teachers in a school is much lower than the number of students, distributing books to teachers will still leave enough books in storage for future “smoothing.” Distributing a few books to teachers but not to students may well be rational behavior on the part of head teachers. Teachers have more organizing and bargaining power within schools, which increases their likelihood of gaining access to “rationed” resources. Also, head-teachers might believe that within a smoothing framework, distributing books to teachers is the most effective way to maximize current learning outcomes of students because a teacher is likely to use information from the textbooks to impact student learning.

# **10. Expectations of schools with regard to government transfers**

Why should expectations of school administrators regarding current and future transfers of textbooks from government be low? One of the challenges that societies face as they move from conflict and fragility towards development is that trust in the state has to be re-built. Lant Pritchett (2009) argues: “In seeking to build legitimacy the state will be under pressure to create confidence in the state, in part by delivering on identifiable ‘quick wins’ but also at the same time to build and transform institutions that are capable of delivering on development.” While a government transfer program for textbooks can be seen as a “quick win,” the system that is expected to deliver these transfers to schools is still far from perfect. There are inefficiencies and uncertainties in the functioning of government systems in Sierra Leone as evidenced by the fact that in 2009, 17 percent of the head teachers and 36 percent of classroom teachers reported not receiving their full pay in the last year.

In addition, relating specifically to the textbook provision program, process data from the impact evaluation reveals that none of the DEOs or Local Councils had a completely clear picture of who was responsible for book pick-up and distribution, official signatories, or monitoring of textbook delivery. Likewise with regard to the actual disbursement of books, knowledge was spotty and written records were rarely found.

Furthermore, only in 20 percent of the cases (out of 325) were textbooks ever delivered directly to the school (in other cases, presumably, textbooks are delivered to a central location in the district, like the DEO office) and in as many as 25 percent of the cases the 18
head teacher claimed that she had to pay for textbook retrieval from her personal resources with no expectation of reimbursement by the government.

These factors indicate that school administrators might view transfers from government as a one-time positive shock, with little expectation of further replenishment in the short-run. This would explain some of the observed storing behavior.

# **11. Conclusions and Policy Implications**

There is considerable preoccupation in the development community with the disappointing impacts of input provision programs on beneficiary outcomes. This paper provides one possible explanation for this phenomenon. We show that public provision of textbooks to schools failed to have any impacts on student outcomes because a large share of the publicly provided textbooks were stored by school administrators instead of being distributed to students.

At the same time, improved teacher access to textbooks had significant positive impacts on teacher behavior. One pathway for this might be through enhanced teacher motivation. It has been argued that having a textbook makes teaching easier. It can allow for better planning and measurement of student progress, and it can provide ready-made content for copying to the blackboard and reading aloud. The link between teachers’ access to textbooks and their classroom behavior has important policy implications: Ensuring that teachers have materials is important for their being willing and able to teach. However, the lack of test score results suggests that this is insufficient, by itself, to translate into student learning.

We find that school administrators who did not expect to receive books were more likely to store them. Based on this result, we hypothesize that administrators are forward looking and that in times of high uncertainty they store part of current transfers to smooth future textbook consumption. Therefore, in line with the Permanent Income Hypothesis, when current transfers are seen as transitory they will have only limited impact on current consumption of textbooks in schools.

Clearly, the behavior of intermediate agents, like school administrators, is likely to exercise a strong mediating influence on the impacts of public programs providing inputs on ultimate beneficiaries. However, the types of intermediate agent behavior that are usually linked to program failure include issues like organizational constraints, problems of human error, and program capture for private gain. In this paper we provide empirical evidence of intermediate agent behavior that is rational and not 19
necessarily driven by private gain, but that nonetheless subverts intended program objectives.

These results have clear policy implications. Public (or non-government organization) provision of inputs that depreciate quickly when used, like textbooks, will be fully consumed only if agents have expectations of replenishment. For public programs that are envisaged as recurring transfers, there is a need to establish a reputation of consistent delivery and to reliably communicate the timing of the next transfer to intermediate agents, with the aim of optimizing the use of current transfers. As in many other empirical studies, we see here that mere input provision does not guarantee results without adequate attention to actual input use. For the latter, it is the incentives and expectations of intermediate agents and ultimate beneficiaries that can make all the difference.

20