metadata: authors: 0: author: Buehren,Niklas
1: author: Goldstein,Markus P.
2: author: Gulesci,Selim
3: author: Sulaiman,Munshi
4: author: Yam,Venus
count: Tanzania
docna: 0: docna: Evaluation of an adolescent development
            program for girls in Tanzania
docty: Policy Research Working Paper
subtopic: Labor Markets,Labor Policies,Vocational & Technical Education,Rural Microfinance and SMEs,Microfinance,Educational Sciences,Gender and Development,Rural Labor Markets
teratopic: Poverty Reduction
abstracts: cdata!: This paper evaluates a program targeted
            to adolescent girls in Tanzania that aims to empower them
            economically as well as socially. The program was found to
            be highly successful in Uganda in terms of economic, health,
            and social outcomes. In contrast, this evaluation finds that
            the program did not have any notable effect on most of these
            outcomes in the Tanzanian setting. The evaluation also
            measures the impact of the program with and without
            microcredit services. The findings show that the addition of
            microcredit improves the take-up of the program and savings
            of the participants. The paper explores programmatic
            implementation information that helps explain the marked
            difference in outcomes between Uganda and Tanzania. This
            research shows that layering additional microfinance
            services onto an adolescent development program can be an
            effective tool to attain greater inclusion of youth in
            financial services, and brings out important issues of the
            generalizability of the research findings.
geo_reg_key: 517191
repnme: repnme: Evaluation of an adolescent development
            program for girls in Tanzania
display_title: Evaluation of an adolescent
            development program for girls in Tanzania
listing_relative_url: /research/2017/02/27168294/evaluation-adolescent-development-program-girls-tanzania
topicv3: Life Skills m1326038 857,Vocational Education and Technical
            Training m1326570 1573,Education m1327048 470,Microfinance m1327227 367,Gender m1327621 615
publishtoextweb_dt: 2017-02-07T00:00:00Z
historic_topic: Poverty Reduction
no_of_pages: 25
author_keys: 0: author_key: 1176301
1: author_key: 769522
2: author_key: 1272457
3: author_key: 1114462
fullavailablein: node_id: 27168294
node_id: 27168294
model_extractions: WPS7961 Policy Research Working Paper 7961 Evaluation of an Adolescent Development Program for Girls in Tanzania _Niklas Buehren Markus Goldstein Selim Gulesci Munshi Sulaiman Venus Yam_ Africa Region Office of the Chief Economist & Gender Cross Cutting Solution Area February 2017
Policy Research Working Paper 7961

# **Abstract**

This paper evaluates a program targeted to adolescent girls in Tanzania that aims to empower them economically as well as socially. The program was found to be highly successful in Uganda in terms of economic, health, and social outcomes. In contrast, this evaluation finds that the program did not have any notable effect on most of these outcomes in the Tanzanian setting. The evaluation also measures the impact of the program with and without microcredit services. The findings show that the addition of microcredit improves the take-up of the program and savings of the participants. The paper explores programmatic implementation information that helps explain the marked difference in outcomes between Uganda and Tanzania. This research shows that layering additional microfinance services onto an adolescent development program can be an effective tool to attain greater inclusion of youth in financial services, and brings out important issues of the generalizability of the research findings.

This paper is a product of the Office of the Chief Economist, Africa Region and the Gender Cross Cutting Solution Area. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank. org. The authors may be contacted at nbuehren@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Evaluation of an Adolescent Development Program for Girls in Tanzania***

**Niklas Buehren     Markus Goldstein     Selim Gulesci Munshi Sulaiman     Venus Yamᵻ**

_JEL Classifications: I25, J13, J24, O12. Keywords: ELA program, adolescent girls’ empowerment, microfinance, Tanzania._

* We thank colleagues at BRAC Tanzania for active support in conducting the evaluation. We have benefited from discussions with conference participants at Improving Institutions for Pro‐Poor Growth (iiG) and Centre for Studies of African Economies (CSAE). We are grateful to the Africa Gender Innovation Lab, the Umbrella Fund for Gender Equality, Nike Foundation, Youth Employment Network and iiG for support. All views in this paper are those of the authors and not any particular institution.

ᵻ Buehren: World Bank [nbuehren@worldbank.org]; Goldstein: World Bank

[mgoldstein@worldbank.org]; Gulesci: [selim.gulesci@unibocconi.it]; Sulaiman: Save the Children [sulaiman.munshi@savethechildren.org]; Yam: Independent Researcher [venus.yam@gmail.com]
# **1. Introduction**

Many African countries are going through a phase of demographic transition marked by a very high share of adolescent and youth population. This has led to a focus on programs that are aimed to facilitate the transition of adolescents into the labor market through vocational training and transfers. A number of more recent initiatives have also been trying to extend financial services to the youth as an entry point for their economic empowerment.<sup>1</sup> Such initiatives are often motivated by the limited participation of youth in mainstream financial services.<sup>2</sup> However, there is lack of evidence on whether savings and credit services for youth have any impact on their overall financial market participation and subsequently on their transition into the labor market.<sup>3</sup> In this paper, we evaluate the impacts of an intervention designed to improve the human capital of young women in Tanzania by providing them with vocational training and information on sex, reproduction and marriage, so‐called life skills. We also test for the impact of providing microcredit services to program participants in addition to the other components of the program.

This paper uses data collected in Tanzania for this evaluation of the Empowerment and Livelihood for Adolescents (ELA) program by BRAC.<sup>4</sup> The intervention package of this program is intended to combine the twin goals of social and economic empowerment. The core model has been found to be very effective for both goals in Uganda (Bandiera et al, 2015). Examining effects after two years, the evaluation in Uganda found an increase in income generating activities, increased monthly consumption expenditures, improved reproductive health knowledge and practices, and large decreases in teen pregnancies and early entry into marriage/cohabitation. Positive changes are also observed in several other outcome indicators related to gender roles and norms. In a medium‐term follow‐up, it appears that many of these effects are sustained four years after the program was first implemented.

Our evaluation in Tanzania is a replication of the Ugandan study with one key difference: In Uganda, we only evaluated the standard ELA model, consisting of vocational and life skills training. In the Tanzanian setting, we were able to introduce a second treatment arm which combined the standard ELA model with microcredit. After launching the core interventions of ELA (i.e. setting up adolescent girls clubs, and conducting life skills, livelihood and vocational training) in all treatment communities,

> 1 Among several examples, the YouthStart and Youth‐Inclusive Financial Services Linkage Program initiatives by UNCDF, MasterCard Foundation and Making Cents have been aiming to play the catalytic role of making financial service providers more pro‐youth.

> 2 According to the Global Findex Database, the ‘youth gaps’ in access to financial services (i.e. the difference in likelihoods of having an account by 15‐24 year‐olds compared to adults aged over 24) are larger in developing countries than OECD countries, and have not reduced between 2011 and 2014 (Demirguc‐Kunt, 2015). Using information of over 66,000 young account holders at formal financial institutions in four countries, Johnson et al (2015) argue that there is huge latent demand for financial services by the youth and they will save if affordable services are on offer.

3 Although there are a growing number of experimental evaluations of the impacts of credit interventions, the evidence is somewhat mixed and rarely focused on adolescents. Reviewing evidence from six randomized evaluations of credit programs, Banerjee et al (2015) conclude that credit does not have any transformative effects on the livelihood outcomes. However, the evaluations do not focus on young micro‐entrepreneurs. 4 BRAC is one of the largest southern NGOs originating in Bangladesh and expanded their multidimensional development programs in several countries in Asia and Africa. Youth development through various initiatives for adolescent girls is one of their priority areas in Africa. For more details on the NGO visit www.brac.net, and <u>http://brac.net/ela</u> for the program.

2
half of the clubs were provided with microcredit services. This way, we are able to evaluate the effects of the standard ELA model in Tanzania (as in Uganda) and to estimate the marginal effect of layering microfinance onto the program. Our evaluation yields three sets of key findings: First, we do not find any robust effects of the standard ELA model on young women’s social and economic outcomes. We look at the same set of outcomes as in the Uganda evaluation – likelihood of being involved in any earning activity, income, plans for starting new activity or financial skills, knowledge of safe sexual practices and productive health, fertility preferences, perceptions of gender roles and control over life – and we find that the program did not lead to significant improvements in any of these outcomes. This is in sharp contrast to the findings in Uganda where the core model is found to be extremely successful in empowering girls both socially and economically. For example, in Uganda the interventions raised the likelihood of girls being engaged in earning activities by 72% (mainly driven by small‐scale self‐employment), raised their private consumption by 38%, reduced teen pregnancy by 26%, and reduced early entry into marriage/cohabitation by 58% (Bandiera et al, 2015). These contradicting findings raise questions about the replicability of successful intervention models across settings.

Second, we find that layering of microfinance services onto the standard ELA model led to greater interest among the target girls about the ELA program, which was reflected by a higher take‐up rate of ELA club activities in communities with microfinance (19%) compared to treatment villages without (13%). The take‐up rate in control villages was 7%.<sup>5</sup> Third, the program led to an increase in savings among adolescent girls from communities that received the ELA program with microfinance. Interestingly, offering this formal microfinance service also increased participation in informal savings groups by both the ELA participants and non‐ participants in these communities. We provide suggestive evidence that this spillover effect on informal savings by non‐participants may have been driven by their interactions with club members. We do not find any significant effect on savings in communities with the standard ELA interventions.

Finally, we use qualitative evidence to discuss the reasons behind the stark differences in the program’s effectiveness between the Ugandan and Tanzanian settings. Differences in the quality of implementation due to resource constraints and several contextual factors appear to be important drivers of these differences. Comparing these two evaluation cases brings forward important issues of scalability across contexts, particularly in terms of resource requirements in pilots and adaptations. There is a popular narrative that the quality of implementation, and hence the magnitudes of impacts, go down as a particular intervention is scaled up, due to weaker implementation, weak fidelity to design and loss of novelty (e.g. White, 2015; Brown, 2016). For example, in a systematic review of farmer field schools, Waddington and While (2014) find that the positive results are limited to small‐ scale pilots and there is no evidence of positive effects from such initiatives at national scale. In another example, Berge et al (2012) evaluate a business training for micro‐entrepreneurs and conclude that the “... effect of research‐led interventions should be interpreted as an upper bound of what can be achieved when scaling up ...”. We argue that the relationship between scale and quality

> 5 The two‐year take‐up rates of ELA in Uganda (without microfinance) were 21% in treatment villages and 5% in control villages (Bandiera et al, 2015).

3
is not necessarily inverse or linear, given that the Uganda program operated at a larger scale than its counterpart in Tanzania. In particular, the minimum scale required for quality assurance in such interventions is larger than ‘typical’ pilots if the cost‐effectiveness analyses from pilots are to be used in scaling up decisions. Assessing cost‐effectiveness of such a pilot can be problematic if such projects are made resource intensive for quality assurance.

Our findings are also related to the policy discussion on potential approaches of making financial services more pro‐youth. These discussions mostly revolve around product adaptations (CYFI, 2012), ‘youth‐friendly’ approaches and the use of ICT in service delivery (Zimmerman and Arnold, 2013), and combining financial services with potentially complementary skills training (Jamison et al, 2014). There are also some other initiatives, although less frequently focused on the adolescent population, for strengthening various informal institutions to make savings and credit services accessible (Karlan et al, 2012). This evaluation looks at a somewhat different model of combining financial services with primarily non‐financial skills training. Since there is a large number of programs for adolescents offering life skills and health education, these programs can potentially be leveraged as vehicles of offering financial services. This evaluation measures the marginal effects of layering credit services on such a non‐financial intervention.

The remainder of the paper is organized as follows: Section 2 describes the intervention and our evaluation design; Section 3 describes sample attrition, program take‐up and estimation strategy; Section 4 describes the main findings, Section 5 provides robustness checks; Section 6 discusses plausible explanations of the contrasting results between Uganda and Tanzania; and Section 7 concludes.

# **2. Program and Evaluation Description**

The ELA program draws on BRAC’s experiences of working with adolescent girls for over two decades in Bangladesh. With the goal of facilitating the transition of young girls into adulthood as responsible and empowered citizens, this program is being implemented at various levels of scale by BRAC in a number of countries.<sup>6</sup> To this goal, the program assists adolescent girls<sup>7</sup> in achieving greater economic and social empowerment through life‐skill and livelihood training, and a safe space for sharing their experiences. Since there are project specific variations in ELA programs across different countries, we explain the core approach followed by a description of the key differences in the Tanzanian pilot.

# **2.1 The ELA Program**

The key components of ELA are adolescent development centers, life‐skills training, livelihoods training, financial education, and sensitization meetings with the parents and village elders. The _adolescent development centers_ , typically referred to as clubs, are usually set up in a one‐room house at a convenient and easy‐to‐reach location for adolescent girls in the community. One of the objectives of this club is to create a safe space for the club members. Each club is provided with books, and

> 6 In Bangladesh, BRAC has reached over a million girls between 1993 and 2013 with 40,000 adolescent clubs. The second largest program in Uganda has reached over 50,000 girls with 1,200 clubs established since 2008. Several pilots ranging between 60 and 250 clubs have been initiated in Tanzania, South Sudan, Sierra Leone and Pakistan.

> 7 Although 13‐19 years is typically used as a functional definition of adolescence, participation is voluntary and not strictly restricted to this age bracket.

4
equipment for indoor and outdoor games. The girls congregate in the club five days a week in the afternoons to engage in various learning and recreational activities ‐ reading and exchanging books, playing games, staging dramas, singing and dancing, and socializing among themselves. The club and its associated recreational activities are meant to attract the girls to voluntarily participate in the program, and leverage this interest to conduct skill trainings at the clubs.

The second component of ELA is _life skills training_ . A range of topics are covered in the life skills trainings including sexual and reproductive health, menstrual disorders, dangers of early pregnancy, sexually transmitted infections, HIV/AIDS awareness and family planning. An adolescent leader, recruited from the community, is assigned to each club to conduct these trainings and to facilitate club activities more generally. The adolescent leaders, also known as mentors, are usually a few years older than typical club participants. Each mentor is provided with a one‐week residential training on club maintenance and training‐of‐trainers for the life skills component. Each mentor receives a monthly stipend of about $15.

A third component of ELA is _livelihood training_ . The club members are provided with training on an income generating activity (IGA) suitable to their local market context. Typical IGAs include small‐scale agriculture or vegetable cultivation, poultry and livestock rearing, hair‐dressing, tailoring, computer operating, and other small‐scale trades. These trainings are provided through entrepreneurs who are engaged in these activities in the community or by hired professionals. BRAC’s agriculture and livestock program also assists in conducting specific IGA training. Sometimes financial education is provided in addition to the IGA trainings.

The fourth component of ELA is to promote community participation by conducting periodic _meetings with the parents and village elders_ . Meetings are conducted with parents and village leaders to sensitize them about the issues of adolescent girls and to create an enabling environment for the girls, especially during the early stages after a club is set up. This community consultation is considered instrumental to generate community ownership of the program and to derive community support.

_Microfinance_ , the fifth component, is only provided to older adolescents. The objective of adding credit to the ELA program is to assist low‐skilled girls to become self‐reliant by providing them with the financial means to engage in self‐employment activities. Financial literacy training and individualized support for business planning are implemented as part of the loan process to improve the likelihood of business success. In most of these communities, BRAC also operates its mainstream microfinance program. While many of these girls are eligible to participate in the mainstream microfinance program, only ELA microfinance conducts the hands‐on coaching on business planning and management, and offers smaller loan sizes to cater for the needs of adolescent girls. In other dimensions, the product structure of ELA microfinance is similar to the mainstream microfinance particularly in terms of interest rates and repayment frequency.<sup>8</sup>

# **2.2 Implementation of the ELA Pilot in Tanzania**

> 8 The loan charges a 25% flat interest rate and the repayment is done in 40 equal instalments. Borrowers also make a 10% deposit as loan security at the time of disbursement and this deposit is not interest bearing.

5
There were important variations between the general program design as described above and its implementation in Tanzania. To begin with, there were challenges in selecting implementation sites. Initially 10 BRAC branch offices in and around the urban setting of Dar es Salaam were selected as implementation sites. Field activities within these branches were then launched with the listing of adolescent girls in about 200 communities. Following this listing intended to identify suitable communities for the intervention, the program implementation team observed that these were not ideal communities for the program. Lack of interest in the program in urban settings and limited cooperation from the communities were highlighted as the reasons for this assessment.<sup>9</sup> Consequently, the pilot sites were relocated to 10 relatively less urban branches in the Dodoma and Iringa districts. This early setback put a constraint on project resources – both financially (having to incur expenses for redeployment of staff and restart surveys for listing eligible girls) and time (to meet deliverable requirements according to the initial project timeline).

Among the intervention components described in the above section, there was an important variation in terms of the club space. As already discussed, these clubs are expected to be set up in a one‐room house for monthly rents paid by the program. In Tanzania, the clubs were not usually rented (in contrast to Uganda). Consequently, field officers had very limited control over the type of space they could ensure for housing the clubs. In many cases, they failed to find a donated space and worked out an arrangement with a local school or church to use their space. Using these public spaces reduced BRAC’s ability to decide on the timing of club activities. In addition, the donated club houses were often insecure and had to adjust to the preference of the owner. Overall, ensuring a ‘secure club space’ was an ongoing challenge throughout the pilot.

Another important variation to highlight concerns the livelihood training package. The participating girls are most frequently trained on agriculture and small‐scale poultry or livestock farming under this component. In some of the programs in other countries, including in Uganda, they were provided with in‐kind support (worth about $30) in the form of seeds, tools or chicks in addition to the training. In Tanzania, the livelihood trainings were not complemented by such input support.

We discuss the reasons and consequences of these challenges and variations further in Section 6. Overall, these variations have important implications for the interpretation of the impact results – especially for understanding the contrasting results of the otherwise identical programmatic model in Uganda.

# **2.3 Research Design**

Implementation of the field activities of ELA in Tanzania started in 2009 with 100 adolescent centers in 10 branches located in Irigna and Dodoma, with a mix of urban and rural locations. We over‐selected the number of communities to construct a control group. Prior to launching the interventions, the program implementation team identified 15 communities in each branch as potential sites for setting up the clubs. Stratified at the branch level, these 150 communities were randomly divided into two groups‐ 100 treatment communities and 50 control communities. All the treatment communities received the ELA intervention (i.e. a club, life‐skills training, livelihood training and community meetings). For simplicity, we refer to this set of interventions as ‘Club’. The treatment communities

> 9 Few of these branches have started ‘ELA‐like’ programs with new project support in later years.

6
were then further randomized into two groups –‘Club only’ and ‘Club + microfinance’. Therefore, there are three types of communities:

1. Group A: ‘Club only’ communities offering adolescent development clubs and associated interventions

2. Group B: ‘Club + MF’ offering microcredit in addition to Group A interventions 3. Group C: Control communities with no interventions In every community, a census of all eligible girls was done by the program team and used as our sampling frame. A baseline survey of 30‐40 girls from each village was conducted during January‐July, 2009. Table A1 shows the balance checks of the three groups in terms of their baseline characteristics. The normalized difference between the two treatment groups and the control group is less than 0.1 for all variables, which is lower than the rule of thumb value suggested by Imbens and Wooldridge (2009), and significant in only 1 of 40 cases. The clubs were established in 2009 after the baseline survey. Microfinance rollout started in early 2010 in the second treatment group after all the clubs were set up. In order to establish a panel data set, the follow‐up survey was conducted during June‐ November of 2011 on the same adolescent girls surveyed at baseline.

# **3. Sample, Take‐up, and Estimation Method**

Our baseline covered 5,454 adolescent girls from these 150 villages. In the follow‐up survey, we were able to track 3,179 of these girls. Therefore, we face a very high attrition rate (42%) between baseline and follow‐up. Although such high attrition for similar target populations is not uncommon,<sup>10</sup> this poses challenges in both interpreting the representativeness of the panel sample and measuring impacts. Table A2 presents the correlates of attrition in our sample. On average, girls who were enrolled in school at baseline are more likely to be surveyed in the follow‐up. Similarly, girls who had a child at baseline are also less likely to attrit. The treatment dummies are not significant in any of the specifications showing similar tracking rates across the three groups. However, the main concern is whether there is a selection bias introduced by losing certain types of girls from our initial sample between control and treatment groups. However, the estimated coefficients on interactions of baseline characteristics with the treatment dummies are jointly not significant. The adolescent girls, who have been successfully tracked, are not statistically different in these baseline characteristics among the three groups. We also try to account for attrition in our estimates in the section on robustness checks.

# **3.1 Take‐up and Selection into the Program**

Table 1 shows the take‐up rates of different ELA activities across the three types of communities. Over half of the girls in each of the three groups know about the ELA intervention. Although ELA clubs were not set up in any control village, 54% of the girls from this group report being aware of ELA activities, demonstrating the high visibility of the program. Girls in ‘Club + MF’ communities were relatively more likely (64%) to be informed about ELA than the control communities. There is no significant difference in knowing about ELA activities between ‘Club only’ and control communities.

> 10 For example, in their evaluation of vocational training for youth in Malawi, Cho et al (2013) observed 33% attrition rate in their follow‐up after 1 year. Duflo et al (2014) have observed 51% attrition rate in their experiment in Kenya after 7 years. In the evaluation of ELA in Uganda (Bandiera et al, 2015), attrition rates are 18% and 41% after 2 and 4 years, respectively.

7
Conditional on having heard about the ELA program, girls in both types of treatment communities are more likely to participate than in the control group. Overall the take‐up rate, defined as having ever participated in any club activity, is the highest in treatment communities with microfinance (19%), and significantly higher than both treatment communities without microfinance (13%) and control (7%) communities. It is important to note here that the participation rate of the program in Uganda after 2 years was 21% in treatment communities without microfinance and 5% in control communities. In Tanzania, the introduction of microfinance increased overall take‐up of the all program activities by 6 percentage points. However, this additional take‐up is not necessarily driven entirely by girls who are joining ELA clubs to be able to borrow from ELA microfinance, although being an ELA club member is a pre‐requisite. Only 4% of the girls in ‘Club + MF’ communities ever participated in the microfinance component of the program. In fact, there seems to be more persistence in girls’ interest in participating in ELA activities in these communities. The likelihoods of joining ELA activities recently (in the last 6 months) or currently participating in the program are also the highest in ‘Club + MF’ communities. Therefore, it seems that microfinance activities strengthened the overall ELA approach by generating more interest in the program.

Given the differences in take‐up rates, it is important to assess whether there are differences in the profile of girls joining ELA clubs between the two treatment groups and as compared to girls from control communities. In Table 2, we investigate differential selection in becoming a club member across the three groups in terms of various baseline characteristics of the participants. Looking at these characteristics individually, we find little differences between club members and non‐members. Among the characteristics with statistically significant differences, girls who do not yet have a child and expressed greater interest in the clubs at baseline are more likely to be a club member. Among the household characteristics, having a member participating in a regular BRAC program also significantly increases the likelihood of a girl becoming an ELA club member. There is no major significant difference in characteristics between club members from control communities and club members in the ‘Club only’ communities. In the ‘Club + MF’ communities, however, the girls who reported less interest in ELA at baseline are more likely to participate. It seems that offering microfinance services encouraged relatively less interested girls to participate. It is also important to note that, across the three groups, the girls reported extremely high interest with an average intention to participate of over 9 on a 10 point scale.

In Table 3, we report the joint significance of all these determinants of club participation. For this, we use the following regression.

�� ��� �������� ������ ����� ����� ∗����� ����� ∗��� ��� (1) Yi is a dummy variable indicating whether individual _i_ ever participated or not; _Xi_ are baseline characteristics; and V _Club_ and V _MF_ are the dummies for the community being assigned to ‘Club only’ and ‘Club + MF’ treatments respectively. In the regression of predictors of club participation including branch fixed effects, the F‐statistics for individual and household characteristics is 4.16 (p‐value 0.00). However, the joint significance of the interaction terms ( ∑��� is 1.72 (p‐value 0.07) for ‘Club only’ villages, and 1.04 (p‐value 0.259) for ‘Club + MF’ villages (Column 4 in Table 3). Therefore, we conclude that participants from ‘Club only’ communities were significantly different relative to participants 8
from control communities. While we do not detect a significant difference between ‘Club + MF’ and control communities in terms of the set of observable characteristics we test for here, there could still be differences in unobservable characteristics of the two groups. As such, comparing participants across different treatment groups is likely to yield estimates that are in part driven by such differential selection.

# **3.2 Estimation Method**

The intention to treat (ITT) effect is generally the most reliable estimate of impacts in randomized experiments. Partial compliance and/or contamination make this a more conservative estimate. Therefore, we start our examination with ITT effects with individual fixed effects using the following specification

- ���� ���� ����������������������∗����� �����������∗��� ����� (2)

In this equation, ���� are the outcome indicators for girl _i_ in village _v_ at time _t_ . ci is an individual fixed effect, �� measures the average change over time in control villages, and �� and �� are the estimated impacts for ‘Club only’ and ‘Club + MF’ interventions respectively.

Along with this ITT specification, we also look at the effects on participants (the local average treatment effect – LATE) by using our randomized assignment to treatments as instruments for predicting club participation.

In this two‐stage estimation, the dependent variables in the second stage are the changes in the variables from baseline to follow‐up with �� measuring the impacts on participants (equation 3). ������ is the predicted participation estimated from the first stage. In the first stage, participation in club activities is regressed on the randomized assignment (the instrument) with baseline characteristics and branch dummies as controls. The strength of the instruments in predicting participation is one of the major concerns for reliable estimates in this specification. An F‐statistic of 10 for the instruments is often used as a rule of thumb for a minimum value required. The F‐statistics of our instruments are 10.01 and 32.23 for ‘Club only’ and ‘Club + MF’ respectively. The instruments pass the weak identification test marginally for measuring the effects of ‘Club only’ interventions, and comfortably for ‘Club + MF’. This essentially reflects the larger difference in take‐up between ‘Club + MF’ and control compared to the difference between ‘Club only’ and control.

# **4. Main Impact Results**

We start by analyzing the effects of the ELA program on economic outcomes. Table 4 presents both the ITT and the IV estimates from specifications (2) and (3) respectively. The outcomes we look at include the likelihood of being involved in an income‐generating activity and to have a daily income, logarithm of labor income, whether the respondent indicated that she was planning to start a new income‐generating activity and a measure of financial skills. None of the estimates shows a significant 9
impact of the program. This is in sharp contrast with the findings for the evaluation of ELA interventions in Uganda, which found a significant positive increase in the probability that girls were in an income generating activity. Aside from income, the point estimates of the ITT effects in the Tanzania evaluation are also very close to zero. Point estimates for the LATE estimates are higher but also not statistically significant.

Table 5 shows ITT and IV estimates for the impact of the program on participants’ financial market participation. We find that the combined intervention of club and microfinance service has increased the likelihood of adolescent girls having savings, and particularly at informal institutions, such as rotating savings and credit schemes (ROSCAs), locally known as _upatu_ . The ITT estimates show a 2.8 percentage points increase in the likelihood of having savings at ROSCAs, which is fairly large relative to the low base of only 2% at baseline. The point estimates for having savings at home and savings at an NGO are also large, but not statistically significant. There is a statistically significant positive impact on total amount saved by the girls, but only in the Tobit, not the OLS specification. The IV estimates in Panel B show positive impacts on savings at home as well as overall savings for the ‘Club + MF’ intervention only. There are, however, no impacts observed on borrowing. Both the ITT and the IV estimates in column (7) show that the participants did not experience any increase in their likelihood of having a loan. This is surprising, given that the microfinance services offered are primarily offering credit and not saving products. This implies that mechanisms other than the direct availability of microfinance services may be driving the effects on saving. In section 6, we will provide suggestive evidence that these effects were mainly driven by spillover effects through the social networks of club participants.

Finally, in Table 6, we provide results on reproductive health, education, and empowerment outcomes. We fail to reject the null of zero impact for almost all indicators. The ITT results for these indicators are statistically not significant and in most cases very close to zero – e.g. the point estimates for HIV knowledge, on a scale of 0 to 5, are 0.15 for ‘club only’ and 0.07 for ‘Club + MF’ as opposed to a large effect of 0.47 for the program in Uganda. Similarly, the ITT effects on perception of ideal number of children are 0.003 and 0.024 and not significant compared to a statistically significant decline of 0.28 in Uganda. The only meaningful changes observed are on perceived gender roles and business network. While the positive effect on perception of gender roles in ‘Club + MF’ villages is interesting, we conclude that the clubs – both with and without microfinance – seem to have had no discernible impact on reproductive outcomes or participants’ empowerment, a marked contrast from the results in Uganda. The positive effect on girls’ likelihood of talking with friends about business, on the other hand, is more likely to be linked to the microfinance component and we reflect on this in the next section.

# **5. Discussion on the Impacts on Savings**

In this section, we discuss social learning as a plausible mechanism of the effects on informal savings. In the follow‐up survey, we collected network information by asking girls about their interactions with other girls in their respective communities. More specifically, we asked about their connection with five girls from their village (randomly selected from the baseline sample). Conditional on knowing the girl, the respondents were asked whether they talked to them about business or social issues. Since this information is collected only at the follow‐up survey, we are only able to look at the correlations between social network and having informal savings. Therefore, this analysis is not sufficient to claim 10
social learning or a demonstration effect. While the ELA program can influence the girls to talk about business and finance with their friends from their informal savings group, the reverse causality is equally possible whereby girls are more likely to talk about businesses after forming their savings groups. However, positive correlations between the likelihood of talking about business with fellow girls in their network and having informal savings could suggest social learning. ROSCA being a collective activity,<sup>11</sup> it is almost a necessary condition for girls in such groups to discuss business plans and other topics related to finance.

Figure 1 shows that the girls who reported discussing business issues with their friends are more likely to have savings with a ROSCA compared to the rest. While we see the same pattern in the likelihood of ROSCA participation in all three groups of communities, the difference is much sharper for ‘Club + MF’ villages. We then use discussion of social issues as a type of placebo comparison. Although there is a significant positive correlation between these two measures of networks, we find a weaker correlation between having discussed social issues with their social network and ROSCA participation. As pointed out earlier, this measure does not provide conclusive evidence on social learning, but it provides suggestive evidence of a mechanism by which the provision of formal financial services (microfinance) may have crowded‐in demand for informal financial services.

# **6. Why are the impacts of ELA different between Tanzania and Uganda?**

As we have discussed so far, we find starkly different impacts of the ELA program in Tanzania as compared to a similar intervention by the same NGO in Uganda. It is important to highlight here that it is not unique to find that similar programs show different effects depending on the implementation team.<sup>12</sup> While it is extremely important to understand why these are so different from a generalizability perspective, an attempt to answer this question _ex post_ can be dangerous. After all, these are only two cases of implementing a ‘similar intervention model’ by an NGO that differ in many aspects, including (and certainly not limited to) two different country contexts, different implementation teams, implemented at different levels of scale, with different amounts of resources available. Any combination of these differences could potentially be the underlying factor behind this difference. With this important caveat, we discuss several key factors that seem to be driving these contrasting results. For this, we heavily draw on qualitative research comparing ELA in Uganda and Tanzania as case studies by BRAC’s research and evaluation division (Yam, 2013).

This qualitative process evaluation identifies several weaknesses in the implementation of the program in Tanzania. Discussions of these weaknesses with implementation teams reveal that resource constraints are at the core of the quality of implementation. Unlike in Uganda, where the program was scaled up while the evaluation pilot was underway, there were important resource constraints in Tanzania. The initial delay in project rollout and having to redeploy teams to new sites depleted the program’s ability to implement the interventions according to the original design.

> 11 In a typical ROSCA, group members put in their savings at regular intervals and loans are given out predominantly to group members.

> 12 For example, in an evaluation of scaling up a previously successful contract teachers intervention in Kenyan schools, Bold et al (2013) find different results on impacts of students’ learning by types of implementing agencies (Govt. vs. NGO). This study highlights differences in monitoring and implementation quality being major drivers of different outcomes.

11
Firstly, as indicated above, the provision for renting club houses was removed due to funding constraints. The initial approach had been to ask the community to donate space, but the response to this provided no or inadequate housing for the clubs. In Uganda, funding was provided to rent suitable spaces, but in Tanzania such funding was not available. Consequently, the club meetings and trainings were held either in public spaces (e.g. in local schools or churches) or in donated houses. Conducting club activities in public spaces created uncertainty over timing, and donated houses were often not ideal, as clubs typically had less space than was needed for 20‐30 girls to congregate and lacked lighting or toilet facilities. The second difference in quality was in club materials. Not replenishing club materials (such as instruments for dance and drama, books, game boards, etc.) that became worn, broken, or unusable was also found to have reduced club members’ interest in regular participation. There were also claims that some materials were lost as they could not lock the doors. Thirdly, there was inadequate provision for the training of new mentors. Since the pilot was operating at a very small scale, it was not feasible to organize trainers’ training for only a few mentors. For example, the mentors are trained in batches of 20‐30 individuals. With less than five mentors dropping out in a given month, the replacement mentors often had to wait for months before they could be trained. Finally, the qualitative report also identifies less frequent monitoring and supervision visits as a reason for low implementation quality. In particular, supervision of the implementation process was considerably stretched in Tanzania compared to Uganda: the area manager in Tanzania supported 10 branches, while area managers in Uganda supported five branches.

Because of relatively quick scale‐up in Uganda during the evaluation period with funding from the MasterCard Foundation,<sup>13</sup> the program there was able to reduce overhead costs per club, utilizing economies of scale in procurement of materials, in mentors’ training and in supervision. There were also additional initiatives that could leverage the scale of operations. There have been several initiatives of inter‐club competitions (on cultural events, sports, and handicrafts) in Uganda to rejuvenate club activities, the like of which was not done in Tanzania. These examples of advantages in scale are important to note in the context of the external validity of small pilots in different program environments. These suggest that there may not necessarily be a negative relationship between the scale of an intervention and its quality, which is often assumed to be the case.

Besides these issues with the quality of implementation, the report also highlights a difference in the priorities expressed by the girls in Uganda versus Tanzania. The club members in Uganda were more interested in economic and livelihood components, but their counterparts in Tanzania expressed greater interest in receiving supplemental tutoring to support their education. This is in line with a higher rate of school enrollment among the members of Tanzanian ELA clubs than in Uganda (80% vs. 71%). There have been a few adjustments in the overall ELA program in Tanzania more recently (since 2013) by focusing on training mentors on tutoring club members who are enrolled in school. An evaluation is currently underway to assess the ELA program with this adjustment and initiatives for quality improvements.<sup>14</sup>

> 13 The foundation provided financial assistance to open 1,200 additional clubs in that many villages across the country within the first year of initiating the pilot evaluation.

> 14 This project is part of the Girl’s Education Challenge Fund of DFID. See the link for more information on the ‐ project, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/415489/GEC <u>Project‐Profile‐booklet‐March2015.pdf, accessed on June 7, 2016.</u>

12
Despite many differences, this is an intervention implemented by the same NGO in two neighboring countries of East Africa with markedly different results. When scaling up this model or taking it to different contexts, there are risks of getting it wrong despite the potential strength of the core model. We argue that building the external validity of this program would require further work that tests it as a complete, well‐implemented and resourced package instead of cutting corners to make it less costly. Alternatively, a more structured approach to examining various aspects of the program (e.g. the livelihoods training vs. the life skills training) with a multi‐arm experiment could provide insights on whether it is possible to run this type of program effectively with fewer resources. Overall, the key factor for long‐term success at the institutional level is continuous learning ‐ both from evaluations and field level implementations for contextual adaptations. This is an endeavor the policy community engaged in adolescent programming should pursue, with adequate provisions for sharing lessons of both successes and failures.

# **7. Conclusion**

We measured the impacts of an Empowerment and Livelihood for Adolescents (ELA) program in Tanzania, based on the same implementation model as a program in Uganda that was successful in both socially and economically empowering young women. The program consists of a safe space, access to recreational activities, life skills training, livelihood training, and community sensitization meetings. We also tested the effects of adding microcredit services to this composite ELA program. Overall, the results show that the same positive effects of ELA could not be replicated in Tanzania, although we find that layering microfinance influenced more girls to participate in the ELA program and to have higher savings. More importantly, the availability of formal financial services seems to have crowded in informal financial mechanisms in the form of ROSCA participation.

Extending financial services is increasingly becoming a priority in the policies around labor market transition of the young population. This is a more prominent challenge in African countries that are going through demographic transitions. This research indicates that layering financial services on other (non‐financial) training programs is a plausible avenue of achieving youth‐inclusive financial services. Such a strategy can be effective in enhancing overall participation in such programs and have an impact on their economic empowerment.

The contrasting results on the effects of the ELA program, on the other hand, bring out the important issue of the generalizability of individual studies for the purpose of scaling up both within the same context and in new settings. A qualitative comparison of these two case studies indicates that implementation fidelity is a key driver of the contrasting results. The case studies also suggest that the typical narrative of a negative association between scale and quality may not be accurate.

13