metadata: authors: 0: author: Lara Ibarra,Gabriel
1: author: Mckenzie,David J.
2: author: Ruiz Ortega,Claudia
count: World
docna: 0: docna: Learning the impact of financial education
            when take-up is low
docty: Policy Research Working Paper
subtopic: Health Care Services Industry,Legal Institutions of the Market Economy,Educational Sciences,Financial Structures,Financial Literacy
teratopic: Education
abstracts: cdata!: Financial education programs are
            increasingly offered by governments, nonprofits, and
            financial institutions. However, voluntary participation
            rates in such programs are often very low, posing a severe
            challenge for randomized experiments attempting to measure
            their impact. This study uses a large experiment on more
            than 100,000 credit card clients in Mexico. The study shows
            how the richness of financial data allows combining
            nonexperimental methods with the experiment to yield
            credible measures of impact, even with take-up rates below 1
            percent. The findings show that a financial education
            workshop and personalized coaching result in a higher
            likelihood of paying credit cards on time, and of making
            more than the minimum payment, but do not reduce spending,
            resulting in higher profitability for the bank.
repnme: repnme: Learning the impact of financial education
            when take-up is low
display_title: Learning the impact of financial
            education when take-up is low
listing_relative_url: /research/2017/11/28139796/learning-impact-financial-education-take-up-low
topicv3: Education m1327048 470,Credit Reporting m1327250 357,Health m1504661 3480,Financial Literacy m1504979 3391,Financial Institutions m1505015 3797
publishtoextweb_dt: 2017-11-07T00:00:00Z
historic_topic: Education
no_of_pages: 38
author_keys: 0: author_key: 1050590
1: author_key: 437864
2: author_key: 1174517
model_extractions: WPS8238 Policy Research Working Paper 8238 Learning the Impact of Financial Education When Take-Up Is Low _Gabriel Lara Ibarra David McKenzie Claudia Ruiz Ortega_ Development Research Group Finance and Private Sector Development Team & Poverty and Equity Global Practice Group November 2017
Policy Research Working Paper 8238

# **Abstract**

Financial education programs are increasingly offered by governments, nonprofits, and financial institutions. However, voluntary participation rates in such programs are often very low, posing a severe challenge for randomized experiments attempting to measure their impact. This study uses a large experiment on more than 100,000 credit card clients in Mexico. The study shows how the richness of financial data allows combining nonexperimental methods with the experiment to yield credible measures of impact, even with take-up rates below 1 percent. The findings show that a financial education workshop and personalized coaching result in a higher likelihood of paying credit cards on time, and of making more than the minimum payment, but do not reduce spending, resulting in higher profitability for the bank.

This paper is a product of the Finance and Private Sector Development Team, Development Research Group and the Poverty and Equity Global Practice Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at glaraibarra@worldbank.org, dmckenzie@worldbank.org and cruizortega@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Learning the Impact of Financial Education When Take-Up Is Low**<sup>**#**</sup>

Gabriel Lara Ibarra _World Bank_ David McKenzie _World Bank_ Claudia Ruiz Ortega _World Bank_ _JEL Classification Codes:_ D14, G21, G28, O12.

_Keywords:_ financial literacy; credit-card behavior; low take-up

> # We thank Adolfo Albo, Juan Luis Ordaz, David Cervantes and the BBVA Bancomer Financial Education Team for their collaboration on the experiment and for providing us with the anonymized customer data used in this paper. We would also like to thank the participants at the EduFin Summit 2017 in Mexico City for comments and suggestions. The authors of the paper received no funding for this work from BBVA Bancomer, and maintained independence in their analysis and reporting of the results. The findings, interpretations, and conclusions expressed in this paper are those of the authors and do not necessarily represent the views of the World Bank, its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.
# 1. **Introduction**

Low levels of financial literacy are pervasive in both developed and developing countries. Higher financial literacy is associated with a wide range of better financial decisions, including more retirement planning, greater stock market participation, and higher savings (Lusardi and Mitchell, 2014). Credit card users with low levels of financial literacy are more likely to carry balances, pay only the minimum payment, or incur late fees (Mottola, 2013; Lusardi and Tufano, 2015). As a result, a large number of governments, international organizations, non-profits, and financial institutions have launched efforts to provide financial education in many countries around the world (Fernandes et al, 2014; Miller et al, 2015).

However, voluntary participation in many financial education efforts is often very low. Willis (2011, p. 230) notes that “voluntary financial education is widely available today, yet seldom used”. For example, Brown and Gartner (2007) report on experimental efforts by different credit card providers in the United States that aimed to provide online financial literacy training to delinquent and at-risk credit card holders: Target Financial Services made calls to 80,982 cardholders, reached only 6,417 of them, offered half of them the program, and had only 28 log in, and only 2 people completed the course; U.S. Bank had only 384 cardholders out of the 42,000 it attempted to reach complete its online program. In Mexico, Bruhn et al. (2014) sent 40,000 letters to bank clients to get them to enroll in a financial education course, of which only 42 responded with interest in the course; they also displayed 16 million Facebook ads, to receive only 119 responses. In Peru, Chong et al. (2010) abandoned a randomized experiment after only 7 percent of their treatment group listened regularly to a radio program with financial education messages, despite being given financial incentives to do so. Low take-up rates are not unique to financial education, but are common among many offers of financial products and services by financial institutions. For example, in the United States the response rate to direct mail credit card solicitations fell from 2.2 percent to 0.6 percent between 1991 and 2012 (Grodzicki, 2015).<sup>1</sup> Such low take-up rates present a severe challenge to randomized experiments attempting to measure the impact of financial education on those who do participate. Due to the inverse-square rule, an experiment with 1 percent take-up requires 10,000 times the sample as an experiment with 100 percent take-up in order to have the same statistical power. This can be one reason why many experimental evaluations of financial literacy training struggle to find a significant effect (Fernandes et al, 2014; Miller et al, 2015). Yet there is often still interest on the part of researchers and financial institutions in learning whether the program had impacts for those individuals who do choose to participate. Moreover, while take-up rates may be low in response to specific offers, widespread availability can still mean the total number of program participants can be large – just as many people have credit cards despite few people responding to any particular credit card offer.

> 1 Take-up rates can be substantially higher in underdeveloped economies in which consumers have few alternatives for the product or service. For example, Bursztyn et al. (2017) report 21 percent take-up for a platinum credit card offer in Indonesia. Nevertheless, Karlan et al. (2010) note that low take-up is an issue for many evaluations attempting to measure impacts of savings, loan, and insurance products offered by microfinance institutions. Crépon et al. (2015) use a pilot phase to then over-sample households with a higher ex-ante propensity to borrow, but find that take-up is not that much higher in this group than in the average population, highlighting the difficulty of predicting in advance who will take-up borrowing.

2
The key question this paper attempts to address is whether we can still obtain reliable measures of the impact of financial education when take-up rates are too low to enable estimation by experimental methods alone. Our context is an experiment in Mexico, whereby the bank BBVA Bancomer worked with over 100,000 of its credit card clients, inviting the treatment group to attend its financial education program _Adelante con tu futuro_ (Go ahead with your future). The program has had over 1.2 million participants between 2008 and 2016, yet only 0.8 percent of the clients in the treatment group attended the workshop. A second experiment which tested personalized financial coaching also had low take-up, with 6.8 percent of the treatment group actually receiving coaching. Standard experimental estimation then finds no significant impact of either treatment, but with very wide confidence intervals for the estimated treatment effect on the treated (the impact of actually receiving training or coaching).

Nevertheless, we argue that the richness of financial data allows combining non-experimental methods with the random assignment from the experiment in a way that yields credible estimates of the treatment impact for those who do take up financial education. We show that those who participate in financial education tend to pay more than the minimum payment on their credit cards, which is in line with the claim of Willis (2011) that those who participate voluntarily tend to already have more financial knowledge and better financial practices than those who do not. This means that simple comparisons of those who participate to those who do not will be biased. Instead, we use the rich time-series data we have on credit card clients to match those who take up the workshops or coaching in the treatment group to clients in the control group who display similar levels and trends in key outcomes month-after-month for 16 months pre-treatment. Matched difference-in-differences are then used to estimate the treatment impacts. This approach helps overcome several common concerns about the use of matching and difference-in-differences: the assumption of common trends becomes more plausible when we can show the two groups we select displayed similar behavior for 16 time periods beforehand; while the question of why matched individuals in the control group did not take up treatment if they are so similar to the treated has a ready answer in that they were randomly assigned to not be invited.

We find that both attending the financial education workshops, and receiving personalized coaching, do lead to changes in financial outcomes for participants. Attending the workshop or receiving coaching results in a 6 to 11 percentage point increase in the likelihood of paying more than the minimum payment, a 3 percentage point lower likelihood of not paying by the payment due date, and a 3 percentage point increase in the likelihood of also having a deposit account at the bank. While paying more responsibly, clients do not reduce the amount of credit card spending, and actually spend more. The result is that financial education of either type leads to an increase in the likelihood that the client is considered profitable by the bank. This demonstrates an additional motive for financial institutions to offer financial education to their clients, beyond the social responsibility motives typically given.

The remainder of the paper is structured as follows: Section 2 discusses the context of the experiment, details of the two interventions, the experimental assignment, and the low take-up; Section 3 provides the treatment impacts using pure experimental methods, showing these to be 3
uninformative; Section 4 then discusses our methodology for combining experimental with nonexperimental methods and the resulting estimates of treatment impact; and Section 5 concludes.

# **2. The Intervention, Samples, and Take-up**

# **2.1 Context**

Mexico has experienced a remarkable increase in the penetration of financial services in the population in recent years. For instance, the number of credit cards owned was estimated at 27.5 million in December of 2016 – a number 12.4 percent higher than that registered at the end of 2015.<sup>2</sup> In a country with 76 million adults, this implies that there is one credit card for every three adults.<sup>3</sup> As of June 2016, the balance in credit card debts represented 39.4 percent of the total credit to personal consumption issued in the country.<sup>4</sup> This rapid growth in credit card usage in Mexico may clash with the amount of time that people have had to learn what an appropriate use of the cards may be. In Mexico, only 32 percent of adults were found to be financially literate, positioning it around the 85<sup>th</sup> place in a worldwide survey of 142 countries (Klapper et al. 2015). Equally worrying, Ponce et al. (2017) illustrate how Mexican cardholders with more than one card typically leave money on the table by using the higher rate card, and experience 31 percent higher cost than the minimum to finance their existing debt.

The experiment of our study was rolled out in several cities across Mexico by BBVA Bancomer, our partner bank. BBVA Bancomer has a dominant position in the credit card market in Mexico. In 2016, BBVA Bancomer and Citigroup Banamex held more than half the credit card market in Mexico and around the same share of the total balance in such credits.<sup>5</sup> Clients of our partner bank appear to be good candidates for improvement in their financial literacy skills. More than half are classified as either medium or high risk, and practically one in five is considered by BBVA Bancomer to be high-risk. BBVA Bancomer credit card clients also appear to use their cards regularly and heavily: they spend on their credit cards about 6,600 Mexican pesos (MXP) per month, which is not far from the 7,365 MXP monthly salary in 2014.<sup>6</sup>

# **2.2 The Interventions**

In this paper, we tackle the question of how effective is financial literacy in improving financial behavior through two distinct treatments. One treatment arm is based on providing the typical approach to financial training: a classroom setting. Since this is the most common tool used by private and public institutions alike to promote financial education, understanding the potential effects and benefits remains an important question. The second treatment arm is based on personalized coaching sessions. This arm responds to the concern in Willis (2011) that due to the heterogeneity of households’ circumstances and needs, effective financial education needs to be

> 2 <u>https://www.publimetro.com.mx/mx/economia/2017/05/03/numero-tarjetas-credito-se-dispara-12-4-condusef.html</u>

> 3 Based on INEGI’s population Count 2015 and including those 20 years and older.

> 4 <u>http://www.banxico.org.mx/sistema-financiero/publicaciones/reporte-de-tasas-de-interes-efectivas-de-tarjetas/%7B3A787547-BAAA-A0F2-4D1A-151E96D32321%7D.pdf</u>

> 5 Based on comparable market of credit cards (Banxico, 2016).

> 6 Informe Anual del Observatorio de Salarios 2016 “Los salarios y la desigualdad en México” IBERO/EQUIDE

4
structured “in a one-on-one setting, with content personalized for each consumer”. Due to the more personal approach, individuals may internalize better the information provided and receive more actionable recommendations based on their own situation and thus be more likely to change their financial behavior than in a situation where only generic advice was provided. However, such an approach can be costlier to provide and scale. We now describe each of the treatment arms in detail.

# _Financial Literacy course_

The first treatment arm is a financial literacy training course that is made freely available by our partner bank at a large-scale throughout Mexico and online. BBVA Bancomer launched the financial literacy courses in 2008, winning an award for innovativeness in fostering financial education in 2011. The courses offered include savings, retirement savings, credit card use, mortgages, life insurance, as well as a series of workshops for small and medium enterprises. As of November 2016, over 4.8 million workshop sessions have provided training to about 1.2 million participants.

The courses are offered in person and online. All courses follow generally the same structure: twohour-long interactive sessions, with material being provided in multimedia: a facilitator presents the material, some videos are shown, and each participant receives a notebook to conduct personal evaluations of their financial knowledge and behaviors, as well as a personal computer to work on. Participants get to take home the notebook that also contains all the information reviewed, and a CD with exercises. Participants are evaluated at the end of the workshop and receive a certificate of completion.

Under this treatment arm, we focus on the provision of the course covering modules on _Credit Card Use and Financial Health_ . The first module of this two-hour course delves into the use of credit cards, associated fees, and how to decipher a credit card statement. Hands-on exercises make participants go through the explanation of what the different parts of a credit card are (digits, expiration date, the security code in the back of the card), understand and differentiate between the payment period and the closing-date of purchases, and read all the elements of a bank credit card statement. The second module focuses on good credit card debt management practices. In this module, individuals learn about the credit score and credit history and their determinants, including failure to pay old debts or keeping high balances frequently. Participants are asked to go through a self-evaluation of their financial health, and based on this assessment a discussion is held on the steps individuals can take to preserve and improve their credit management.<sup>7</sup> Throughout the modules, participants are reminded of rules of thumb labeled “golden rules” for good credit card behavior as a way to make the messages and advice concrete and easy to remember. These rules are presented in Appendix 3b, and emphasis the importance of paying on time, and paying more than the minimum payment.

Invitations and attendance to the workshops were conducted from July through December 2016. Invitations were done through email and BBVA Bancomer call center. Given the scope of the study, participants were contacted and invited to participate in the face-to-face training in their city

> 7 The detailed topics covered by the training are included in Appendix 3a.

5
of residency. Clients from Mexico City, Puebla, Guadalajara, Morelia, Cuernavaca, Mérida and Tijuana were included in the workshop treatment arm. They were offered BBVA Bancomer rewards points for completing the course, which could be redeemed towards small rewards like a meal or merchandise.<sup>8</sup>

# _Personalized coaching_

A second treatment arm was based on a series of personalized coaching sessions. The content was developed by BBVA Bancomer with the objective of “bringing information and tools to participants so that they have the capacity to make an adequate use of their credit card and to keep an excellent credit health”. This coaching entailed calling and scheduling a series of conversations with the participant to discuss her credit history, health, and behavior and help solve any issues or doubts she may have. The coaching was provided by highly trained _asesores_ (financial advisers). Each of these asesores was assigned a group of individuals who they would call and invite to engage in these coaching sessions. If an individual agreed to participate, the asesores would ask them about the doubts and questions they may have about their credit card and credit card use. Asesores were equipped to provide suggestions on how to improve the individual’s credit and help them pursue healthy financial behavior. The recommendations followed closely the contents and advice that the workshops had, such as: payment more than the minimum to lessen the total interest payment overall, remembering the payment date, and budgeting as to avoid unnecessary use of credit: “the credit is not an extension of your salary”.

After the initial call and introduction, the asesores would discuss and agree on a follow-up call with the participant at a time that fitted her schedule. The calls were aimed to be roughly two weeks apart, with a total of four calls with each participant. The calls were intended to be thematic and follow a specific progression: diagnostic, budget, credit, and credit health. Each call was planned to last about 10 minutes. Overall, participants were expected to be engaged in this treatment in a span of two calendar months. The list of topics and guide for the discussion are presented in appendix 3c.

As with the workshops, coaching sessions took place between June and December 2016. While the call center was situated in Mexico City, participants of the coaching sessions could be residing in any of the nine cities that were part of the study. As with the workshops, participants were offered rewards points for completing the program.

It is worth noting that the delivery methods studied in our treatment arms are easily scalable and of relatively low cost from the point of view of a financial institution. Each participant in the workshop cost BBVA Bancomer 86 MXP or around 5 USD, while the cost per person coached was estimated at 131 MXP (7 USD). To put this cost in perspective, the annual fee for a set of BBVA Bancomer’s credit cards is between 631 and 5,275 MXP. The average yearly profitability of BBVA Bancomer credit card clients is 1,056 MXP. Thus, expanding the provision of such interventions should not be necessarily seen as a high toll to pay. While credit card holders are already profitable to the bank, the expected effects of financial education through better (and

> 8 They were given up to 4,000 points, which are valued at approximately 300 pesos (US$16).

6
increased) use of their credit cards may further strengthen the business case to continue BBVA Bancomer financial literacy program.

# **2.3 Outcomes and Measurement**

We obtained administrative data from BBVA Bancomer on 136,104 credit card clients from December 2014 to February 2017. The data set contains a rich set of information summarizing the monthly evolution of each client’s credit card balance, payments, purchases, delays, profitability for the bank and ownership of basic deposit accounts with the bank. The data set also includes the seniority of clients with our partner bank and their background characteristics such as age and gender.

We center our analysis on a set of outcomes that we believe are more likely to be affected by the interventions, based on the material covered both in the workshops and coaching. More concretely, there are three rules that the workshops and coaching emphasize to help participants achieve a more responsible use of their credit cards and avoid extra fees and future over-indebtedness. The first rule is to cover at least the minimum payment required by the bank. The second one is to identify from the credit card statements the payment due date and make sure to pay before that date. A third advice is to limit the use of credit to an amount that the client can comfortably pay later. Along these three rules, the importance of saving and better managing expenses is frequently discussed.

We first test whether participants of the interventions are more likely to follow these rules by analyzing key outcomes directly related to them. These outcomes are: i) the likelihood of paying more than the minimum payment; ii) the likelihood of paying past the payment due date; and iii) total credit card purchases. As the interventions highlight the importance of saving, we also analyze whether participants are more likely to own a basic deposit account after the interventions. Finally, we investigate how profitable these financial education interventions are for our partner bank. We do this by analyzing a variable that measures the profitability of each client for the bank at every month.<sup>9,10</sup>

# **2.4 Samples and Random Assignment**

Given that we expect financial education interventions to have a greater impact among participants with riskier credit card management practices, we stratified the sample into six groups based on two risk measures of the clients. The first measure corresponds to the risk classification of each client. Every client is classified by our partner bank as low risk, medium risk and high risk. Since clients who struggle to cover the minimum payment are at a higher risk of facing credit card management issues in the future, we produce a complementary risk measure that classifies clients

> 9 Our profitability variable corresponds to the difference between the revenue obtained from a client and the expenditures he or she generated for the bank. The revenue is measured as the interest income plus paid commissions and fees. The expenditures include operational costs, cost of capital, loan losses and reserves.

> 10 On one hand, profits to the bank may increase if clients spend more on their credit cards and thus the revenues from commissions and interest income rise. In addition, if clients are more likely to pay on time, the costs of monitoring and recovering loans drop. On the other hand, profits may decline if better credit card management translates in lower interest payments and fees.

7
according to how often their payment exceeds the minimum required by the bank. We define clients as “with frequent low payments” if more than one-third of the time they pay the required minimum or less.<sup>11</sup> While our partner bank had no capacity constraints to deliver the financial education workshops, only 300 coaching interventions could be given.<sup>12</sup> Therefore, we decided to restrict the coaching group to clients belonging to the stratum with the highest-risk clients (i.e., clients with frequent low payments and classified by the bank as high risk). The highest-risk clients were randomly assigned into three groups: workshops, coaching and the control group. For all other strata, clients were randomly assigned into either the workshop or the control group.

Clients in the coaching group were also randomly divided into three lists: the main list and two wait lists. Clients in the first wait list would only be contacted if there were still sessions available after coaching was offered to clients in the first list.<sup>13</sup> After contacting clients from the first two lists, all coaching sessions were exhausted. Therefore, 1,354 clients that were assigned to the third list were never contacted to participate in the intervention and were dropped from the sample.

To have at least one year of pre-intervention data to get accurate counterfactuals, we also dropped from the sample 20,524 clients. These clients were new to the bank and only had six months of data before the interventions. Therefore, our final sample consists of 114,226 clients.

Table 1 presents the summary statistics of our sample, divided by the group of clients assigned to the workshops (Panel A) and the set of clients assigned to coaching (Panel B). We divide each panel in four columns. The first column presents the characteristics of clients in the control group. The second and third columns of each panel show the characteristics of clients assigned to and that effectively attended each intervention. The fourth column presents the mean differential of the characteristics of clients assigned to the control group with clients taking up each intervention.

Clients in the sample are on average 46 years old and about half of them are women. Most clients live in Mexico City and have been clients with our partner bank for about 12 years. In terms of their risk profile, 19 percent of clients are classified by the bank as being high-risk, 37 percent as medium-risk and 44 percent as low-risk. Per our definition, 26 percent of clients struggle to pay more than the minimum required. That is, their payments do not exceed the minimum payment required in at least 4 of the 12 months that we observe them before the intervention. Each month, clients tend to spend about 7,000 Mexican pesos on their credit cards. On average, 86 percent of clients pay more than the minimum payment required by the bank and only 1 percent make their payments past the due date. 70 percent of clients own a deposit account with our partner bank. In terms of profitability, each month the bank obtains approximately 1,000 Mexican pesos for each client in the sample.

> 11 On average, 27 percent of individuals in the sample paid the minimum required payment or less in the preintervention period.

> 12 From the institution’s point of view, the intervention was thought as a pilot that could be scaled up based on the results and lessons learned.

> 13 Likewise, clients from the second list would only be contacted if there were still coaching sessions available after having invited all clients from the first wait list.

8
Currently, BBVA Bancomer offers eleven different credit cards in Mexico (Appendix 4). After the closing date (the last day of the monthly billing cycle), all clients have 20 days to pay at least the minimum payment (typically 20% of the balance) before incurring late fees. Even though the terms vary across credit cards, APRs on BBVA Bancomer cards range from 18.6 to 115.6 percent, with those for the most common cards ranging from 68.2% to 91.6%. The fixed penalty for not paying on time is approximately 377 MXP (or 24 dollars) without counting the added interest. The annual fee of BBVA Bancomer credit cards is about 631 MXP (33 dollars), though it can be much higher for certain types of cards.

# **2.5 Take-up**

As in many other settings analyzing financial behavior, the implementation of the study faced a major challenge regarding take-up rates. From a total of 114,226 clients, 36,946 were assigned to the control group, 73,654 were assigned to the workshop treatment arm and 3,636 were assigned to the coaching treatment arm. At the end, only 0.8 percent of the workshop treatment arm clients actually received the treatment, and 6.8 percent of the clients in the coaching treatment arm participated in the sessions (table 2). There are several reasons explaining this low take-up rate.

In the case of the workshop treatment arm, the resources needed to reach out to such a large number of clients were underestimated. Thus, during the implementation phase that lasted about six months, of the original group assigned to the workshop treatment, contact was attempted only to about 47.3 percent (34,818 clients). Next, despite repeated efforts to contact the individuals in this group, only 8,900 clients were effectively contacted. This means that over 25,000 clients did not pick up the phone during the outreach or that they answered and asked to be called later (without success). Thus, only a little over 12% of the group assigned to the workshops could be contacted and actually invited to participate in the treatment. From this group, 2,672 clients agreed to participate in the workshop and a mere 583 attended and completed the workshop.

Similar challenges were faced in the roll out of the coaching intervention. Due to the relatively low number of clients assigned to the treatment arm, the vast majority had at least one attempted contact (88.5%). From these, only about a third picked up the phone and less than a sixth agreed to participate in the coaching sessions (14% of the original treatment group). Finally, 246 clients completed at least one session with the coach, translating to a take-up rate of 6.8%. While these take-up rates seem dire, as discussed in the introduction, they are unfortunately not unusual in the RCT universe, nor in the marketing reach out campaigns of financial institutions. Anecdotal evidence from BBVA Bancomer deposits department puts the typical response rate of the bank’s marketing campaigns at 2%.

The challenge of low take-up rates can pose an even bigger problem if it is selective. For instance, it is easy to argue that bank clients will be less likely to answer a call from their bank if they are having trouble keeping their finances in order, are often late in paying their cards, or have typically large balances on their cards. Thus, when such clients get a call from the bank to be invited to take a training, a coaching session or any other reason, they are less likely to answer the call in the first place. If good (i.e. more financially literate) clients self-select into participating in the treatment, while bad clients tend to self-select out of the treatment, a direct comparison of their outcomes 9
with the control group will yield biased results. Financially literate clients are expected to have less to learn from more financial education, thus hinting that the workshops or coaching sessions may not affect individuals’ financial behavior significantly.

We find evidence of selective participation in our study. Clients that end up taking the workshop or the coaching sessions appear to be in lower need of financial education than the average client. The more often an individual paid above the minimum payment in her card, the higher the likelihood she signed up for the workshop (figure 1 panel A) or the coaching (Figure 1 panel B). For instance, an individual assigned to the coaching group who paid more than the minimum for six months is more than twice as likely to complete the coaching session than an individual who paid more than the minimum in three months only. Other characteristics also hint at positive selection among treatment takers. Clients who were contacted, accepted participating and actually took the workshop (or coaching) are more likely to make payments above the minimum required, less likely to pay late, and more likely to also own a deposit account than those who were assigned to the same treatment group but did not sign up and received the treatment (table 1). Within the workshop treatment group, the takers are also more likely to avoid making low payments on a regular basis.

We now turn to the description of the methodological approaches used to estimate the effects of the treatment arms on financial behavior. We first describe the pure experimental approach that is applied in most RCTs. Next, to overcome the low take-up rate problem, we apply a combination of non-experimental methods to get at a cleaner estimate of the impact of the workshops and coaching sessions. We describe these approaches in detail below.

# **3. Pure Experimental Results**

# **3.1 The Challenge of Low Take-up for Statistical Power**

Consider a simple comparison of treatment and control means in a randomized experiment which allocates a proportion _P_ of subjects to the treatment, and _1-P_ to the control. This intent-to-treat effect can then be estimated in a regression of the form: Where _Ti_ is a dummy variable denoting assignment to treatment, and the error ε is i.i.d. with variance �<sup>�</sup> . Let _c_ be the take-up rate in the treatment group, _s_ the take-up rate in the control group ( _s=_ 0 if no one in the control group gets the treatment), and _E_ the impact of treatment for those who actually receive the treatment (the treatment effect on the treated). The sample size _N_ needed to detect effect size _E_ at significance level α and power _β_ is then (e.g. Duflo et al, 2008): With more rounds of data and the use of difference-in-differences or Ancova estimation, the variance term becomes more complicated, but the influence of the take-up rates remains the same (McKenzie, 2012).

10
We see that the sample size required is proportional to the inverse of the difference in take-up rates squared _1/(c-s)_<sup>_2_</sup> . The consequence is that low take-up rates dramatically increase the sample size required to detect the impact of training: if take-up is 10 percent, 100 times the sample is needed than with full take-up; if it is 5 percent, 400 times the sample is needed; and if it is the 0.5 percent that is common in responses to bank direct mail promotions, 40,000 times the sample is needed. This makes it extremely challenging for experimental methods to detect the impact of interventions when take-up rates are very low.

# **3.2 Experimental Treatment Impacts**

The offer of a financial education workshop or of coaching was randomly assigned, and so comparing post-treatment outcomes for the treatment group to the control group gives an unbiased estimate of the intention-to-treat (ITT) effect, which is the effect of being offered the program. Consider outcome _Yi,t_ measured for client _i_ in period _t._ McKenzie (2012) shows that with multiple rounds of follow-up data, maximum power comes from estimating an average effect _γ_ over the entire nine-month post-intervention period _t=1,2,...,9_ via the following Ancova specification: Where �<sup>�</sup> �,��� is the mean of the outcome over the pre-treatment periods, �� are strata fixed effects, �� are time period fixed effects, and the standard errors ��,� are clustered at the client-level.

Under the assumption that the invitation to financial education or coaching has no impact on outcomes for those who do not take-up the treatment, we can also estimate the local-average treatment effect (LATE) by replacing _OfferedTreatment_ with _ReceivedTreatment_ in equation (3), and then instrumenting the receipt of treatment with its randomly assigned offer. This identifies the local-average treatment effect (LATE), which is the effect of receiving training or coaching when offered it, and not otherwise. If no one in the control group takes up the treatment, then this also gives the treatment-effect-on-the-treated (TOT). This is the case for the coaching intervention, but it is possible that a few individuals in the control group for the training intervention may have attended a workshop without being invited.

Figure 2 plots the trajectory of two key outcomes – paying more than the minimum payment, and having a delay in payment – over time by treatment status. The top two figures show this for the sample assigned to workshops, and the bottom two figures for the sample assigned to coaching. In both cases we see that the treatment and control groups track each other very closely over time before the intervention (as would be expected by randomization with a large sample), and continue to track each other closely after the intervention. With such low take-up, the average for the treatment group as a whole is dominated by the behavior of those who do not receive the treatment.

Table 3 then reports the results of estimating the ITT effect in equation (3) of being offered the financial education workshop (panel A), and of being offered the coaching (panel B). These estimates are all small in magnitude, and very close to zero. That is, the offer of treatment has a very small, and insignificant impact on financial behavior. Underneath each ITT, we then report the LATE/TOT and a 95 percent confidence interval around it. We see that the confidence intervals are very wide for the impact of actually taking up either treatment, as a result, the experiment is 11
not very informative about the impact of these interventions. For example, the 95 percent confidence interval for the impact of coaching on whether or not the client pays more than the minimum payment ranges from -25 percentage points to +18 percentage points. The control mean is 54 percent, so this is equivalent to almost halving the percent paying on time, or increasing it by one-third.

This is where standard analysis using experimental methods would stop. We would conclude that there is no significant impact of either intervention, but that we have insufficient power to rule out a wide range of positive and negative impacts. We therefore turn to combining non-experimental methods with the experiment to obtain more informative results.

# **4. Combining Experimental and Non-Experimental Methods to Measure Impact for Those Who Actually Take Up Treatment**

# **4.1 Empirical Approach**

The basic challenge for identifying the impact of training and coaching for those who actually took part in these interventions is that take-up is not random. Section 2 showed that those who attended the workshops or received coaching differ in current and past financial behaviors from those who did not. As a result, simply comparing those who took-up the interventions to the full set of individuals in the control group would yield biased results.

Our solution is to use the richness of the financial data available on credit card clients to combine experimental and non-experimental methods. We use propensity score matching to match individuals in the treatment group who took up the treatment to similar individuals in the control group, and then difference-in-differences on this matched sample to estimate the impact of attending the workshop or receiving coaching. That is, we estimate the following equation for the matched sample: Where _ReceivedTreatmenti,t_ takes on value one for individuals in the treatment group in postintervention periods, and zero otherwise; _EverTreatedi_ is an indicator of whether individual _i_ is in the treatment group and ever took-up the treatment; and the time fixed effects are now included for up to 18 months pre-treatment, as well as 9 months post-treatment. The standard errors are again clustered at the client level.

There are several concerns that typically apply when applying propensity score matching. The first is a concern of omitted variables: individuals who look similar in terms of baseline observable variables might differ in terms of unobserved characteristics that also matter for client outcomes. A particular concern here is that of dynamic selection, similar to the problem of an Ashenfelter dip in labor economics experiments. For example, people might be more willing to engage in financial education if they suddenly find themselves struggling with their credit card, whereas those who have been experiencing problems for a while may be less likely to participate. Matching on current behavior only would not be able to distinguish between these two types. Secondly, a critique 12
underlying all matching studies is to explain why, if these two groups are so similar, only one group ended up taking the intervention. Our rich data and experiment help in addressing both concerns. We have up to 18 months of pre-intervention financial data for these clients, so can match not only on current financial behavior, but on the monthly trajectory of this behavior over many months. This helps alleviate concerns about dynamic selection. Moreover, by only matching to individuals in the control group (and not those in the treatment group who did not take up treatment), we have a plausible reason why some individuals do not take up treatment – they were not invited to under the random invitations.

Difference-in-differences further enables us to difference out any time-invariant unobservable differences between the two groups. Thus, if those who participate in training or coaching always tend to be better re-payers than those who do not, we can difference this out. The underlying assumption for difference-in-difference analysis is that of a common trend, so that the two groups would follow the same time paths as each other in the absence of an intervention. This assumption is more credible if the individuals are more similar to begin with (which is where matching helps), and if we see the two groups have the same dynamics prior to the intervention. Researchers relying on survey data typically do not have multiple rounds of pre-intervention data with which to test this assumption. In contrast, the monthly administrative data enables us to not only test whether the two groups follow similar linear trends prior to the intervention, but also to test whether they follow the same non-linear trend. To test this, we estimate over the pre-intervention period: And test that all the β’s are jointly zero.

As we move away from the pure experiment, there is no one universally agreed control group. We examine several different plausible ways of choosing this control group. We then view the resulting estimates as more credible if these different methods give similar results, even though they end up choosing different individuals from the control group to match to those who actually take up treatment in the treatment group.

We begin by estimating the difference-in-differences estimator using the full control group. If there is self-selection into treatment, those receiving training or coaching will differ in levels, and potentially trends, from this full control group. A first step towards refining the control group to more comparable individuals is to restrict the analysis to individuals in the common support of the propensity score. For this approach, we estimate the propensity score as a function of gender, and pre-treatment monthly levels of all five outcomes. This involves matching on 73 variables in total, and eliminates 38% of the control group and 22% of the treatment group for coaching. We then go further by choosing the nearest neighbor within this common support for each client who received treatment. Using all the outcomes simultaneously to form these matches has the advantage of making clients similar on average in terms of existing financial behavior, but, because it is attempting to match on so many variables, may not match especially well on any particular single outcome. We therefore also consider two alternatives to forming the propensity score and then matching on the nearest neighbor. The first is to use Lasso to choose a parsimonious set of variables to match on. This chooses 8 of the 73 variables to use in forming the match. The second, and our 13
most preferred approach, is to match just on the month-by-month pre-intervention data for an outcome at a time. This last method ensures that the control individuals look as similar as possible on levels and dynamics as those receiving treatment, but does mean, in contrast to the other approaches, that different controls are used for each outcome.

Figure 3 illustrates how the five different approaches define counterfactuals for the coaching treatment and outcome of paying more than the minimum payment owed. The top left panel compares the full control group to those receiving treatment. We see that the group which received coaching starts from a much higher mean level than the control group, reflecting positive selection into training in terms of pre-existing credit behavior. The trends seem broadly similar preintervention, suggesting difference-in-differences may be able to control for this selection. This difference in baseline means becomes smaller, but is still there, when we condition on being in the common support. In contrast, all three nearest neighbor approaches look much more similar on baseline levels, and appear to match reasonably well on baseline trends. These different nearest neighbor approaches do select different individuals from the control group: only 2 clients are selected by all three methods, so we are forming multiple plausible counterfactuals.

Appendix Table 1 shows how these three nearest neighbor matches achieve samples from the control group which are much more comparable on baseline observables to those who took up treatment than is the case for the full control sample. The first column shows baseline means for those who took-up treatment. We then follow Imbens and Rubin (2015) in considering the normalized difference ��<sup>�</sup> � −�<sup>�</sup> � �� + �̅���/2 as a measure of balance, where �<sup>�</sup> � and �̅�� are �/���̅ the sample mean and variance of the variable for those receiving treatment ( _j=T_ ) and the comparison subsample from the control group ( _j=C_ ) respectively. These normalized differences provide a scale-invariant measure of the difference in locations, with differences less than 0.2 standard deviations typically considered to indicate balance. We see that normalized differences exceed this level for pre-intervention averages in our key outcomes when using the full control sample, or the sample within the common support, but are all less than this when using any of our three nearest neighbor methods. As a result, we cannot reject equality of means of our financial outcomes averaged over all pre-intervention periods when using nearest neighbor.

# **4.2 Impacts of Workshops**

Table 4 presents the difference-in-difference estimates of the impact of workshops on our five outcomes of interest (Panels A through E). Each column presents the results of one of the five approaches we use to form a control group. For each outcome and approach, we test whether the treated and control groups followed common linear and non-linear trends in the pre-intervention period. The p-values of the tests are included in the table. In Figure 3, we show a graphic representation of the different approaches. The figure shows the trajectory of the outcome of paying more than the minimum payment for the clients that participated in the workshops and the clients assigned to the different control groups.

The last column of the table presents our preferred specification, the nearest neighbor approach that matches on the monthly pre-intervention data for a specific outcome of interest. As Figure 4 14
shows, this approach allows us to generate a control group that tracks very closely the evolution of each outcome for the treated group in the pre-intervention period. After the intervention, the mean outcomes of the clients that participated in the workshops begin to separate from the mean outcomes of the control group. While the outcomes of clients in the control group deteriorate over time (i.e., lower fraction of clients paying more than the minimum required and increased likelihood of delayed payment), the outcomes of clients who took the workshops remained stable.

The results from Table 4 show that these differences are statistically significant for all outcomes, except for bank profitability. The p-values of the common trends tests suggest that we cannot reject that the outcomes of clients who took the workshop and the control group formed by our preferred approach followed common linear and non-linear trends before the interventions. The economic impact of the estimates is also robust across the different matching approaches. The results of our preferred specification suggest that participating in the workshop results in an 11 percentage point increase in the likelihood of paying more than the minimum payment, a 3.4 percentage point reduction in the likelihood of delaying payment, 63.7 percent higher monthly spending on the credit card, and a 2.7 percentage point increase in the likelihood of owning a deposit account with our partner bank.

# **4.3 Impacts of Coaching**

Table 5 provides the difference-in-difference results for our five different approaches to forming a control group, along with tests of whether the two groups follow a common linear trend, and a common non-linear trend, before the intervention. Figure 5 shows the trajectory of mean outcomes for the group that received coaching, compared to our different matched control groups.

Figure 6 shows that after the time of the intervention, the control group is becoming progressively more likely to not pay more than the minimum payment, to delay in their credit card payments, to no longer have a deposit account with the bank, to cut back on spending, and are becoming less profitable clients for the bank. The coaching treatment is halting these trends from occurring, so that those who receive coaching appear more similar to their pre-intervention levels.

Table 5 shows that these impacts are statistically significant after matching for all but having a deposit account, are reasonably robust in magnitude to different plausible ways of defining this matched control group, and that we cannot reject that the matched control groups display parallel linear or non-linear trends pre-intervention. Using our preferred specification in the last column (which matches Figure 5), we find that receiving coaching results in a 5.9 percentage point increase in the likelihood of paying more than the minimum payment, a 2.6 percentage point reduction in the likelihood of delaying payment, 51.9 percent higher monthly spending on the credit card, and a 7.8 percentage point increase in the likelihood that the client is profitable for the bank.

# **4.4 Discussion**

This combination of nearest neighbor matching, difference-in-differences, and the random assignment enables us to find a subset of clients within the full experimental control group who look similar on baseline observables to those who take up the interventions, and who also follow similar pre-intervention trends. Using this strategy, we detect treatment effects of the interventions 15
that were not able to be detected using experimental methods alone. It is worth re-iterating how little power experimental methods will have to detect treatment effects of the size we find, given the low take-up levels. For example, our power to detect the 5.9 percentage point improvement in the likelihood of paying more than the minimum that we see for the coaching treatment, given that take-up is 6.8 percent, is only 17.9 percent.<sup>14</sup> Note that our estimated treatment effects are all within the (very wide) confidence intervals seen for the LATE in Table 3.

It is important to note that these treatment effects are for the set of clients who will take up the interventions when invited. We have seen there is positive selection into participation, so that individuals who have the worst initial financial behavior in terms of late payments, not paying more than the minimum required, etc., are less likely to participate. The treatment effect may be larger for these individuals if they could be induced to participate, since they have more room for improvement, or potentially smaller if such individuals are less likely to implement the changes suggested in the workshops.

Finally, the cost per client of providing these programs was 131 MXP (7 USD) per person coached, and 86 MXP (5 USD) per person participating in the workshop. Using the impact on paying on time, this equates to a cost of 7 MXP/0.026 = 269 MXP per additional client induced by coaching to pay on time. If this were the only impact, this would appear an expensive way for banks to get clients to pay in a timelier manner. However, we see that the training and coaching get clients to pay their accounts on time and pay more of their bills, but do not get them to cut back on spending. In fact, perhaps because they are not experiencing as many payment problems, they spend more on their cards. The result is that this training does increase the likelihood these clients remain profitable for the bank.

# **5. Conclusions**

The reliable estimation of treatment effects in impact evaluations relies heavily on the implementation efforts of teams to get individuals assigned to a treatment to be effectively treated (and to the extent possible ensure those assigned to the control group do not receive treatment). In settings where individuals have little incentive to participate and those that do tend to be selfselected, the identification of the program effects through experimental methods is challenging. Unfortunately, and despite their recent popularity, financial education programs constitute a perfect example of these issues.

In this study, we take advantage of the richness of administrative data from a financial institution to implement a novel approach that overcomes the low-take-up problem in RCTs. The availability of monthly administrative data over a two-year period for a large pool of individuals assigned to the control group allows for a clearer estimation of the impacts of the workshops and coaching sessions. By selecting a group of individuals within the control group that is statistically comparable in their financial behavior (previous to the treatment) to that of the effectively treated group, our approach improves the empirical evaluation of the treatment in several ways. The approach improves upon the simple application of experimental methods. The experimental

> 14 This uses the autocorrelation of approximately 0.4 in our data, and the following command in Stata: sampsi 0.69 0.694012, n1(2504) n2(3626) sd1(0.46) pre(14) post(9) r01(0.4) r1(0.4) r0(0.4).

16
method estimate gets “diluted” as it compares a large pool of individuals assigned to the treatment group where only a handful were effectively treated, to a large pool of individuals assigned to the control group. By using several rounds of administrative data, our approach also allows the verification of the parallel trends assumption required for the application of non-experimental approaches.

The combination of experimental and non-experimental methods along with rich administrative data present a new avenue for empirical applications of impact evaluations when take-up is low. Examples of widespread reach out efforts with small response rates include credit card offers in the U.S. where response rates were estimated to reach 0.2% in 2006,<sup>15</sup> and the banking/finance industry in the U.K., where the overall click rate (i.e. the percent of clicks that are made among all the emails sent) of email marketing campaigns was estimated at 0.48% in 2016.<sup>16</sup>

> 15 Federal Deposit Insurance Corporation “Credit Card Activities Manual”

> <u>https://www.fdic.gov/regulations/examinations/credit_card/ch5.html [accessed October 15h 2017].</u> 16 Sign-up.to 2017 report https://www.signupto.com/email-marketing-benchmarks/email-benchmark-2017/  [accessed October 15<sup>th</sup> 2017].

17