metadata: authors: 0: author: Khandker,Shahidur R.
1: author: Pitt,Mark M.
count: World
docna: 0: docna: Replicating replication :  due diligence in
            Roodman and Morduch's replication of Pitt and Khandker (1998)
docty: Policy Research Working Paper
theme: Rural non-farm income generation,Rural policies and institutions
majtheme: Rural development
subtopic: Rural Settlements,Rural Microfinance and SMEs,Microfinance,Gender and Development,Food Security,Agricultural Growth and Rural Development,Inequality
teratopic: Macroeconomics and Economic Growth,Science and Technology Development,Education
abstracts: cdata!: "The Impact of Microcredit on the
            Poor in Bangladesh: Revisiting the Evidence," by David
            Roodman and Jonathan Morduch (2011) is the most recent of a
            sequence of papers and postings that seeks to refute the
            findings of the Pitt and Khandker (1998) article "The
            Impact of Group-Based Credit on Poor Households in
            Bangladesh: Does the Gender of Participants Matter?"
            that microcredit for women had significant, favorable
            effects on poverty reduction. In this paper the authors show
            that these latest Roodman and Morduch claims are based on
            seriously flawed econometric methods and theory and a lack
            of due diligence in formulating models and interpreting
            output from packaged software. On the basis of Roodman and
            Morduch's preferred two-stage least squares regression,
            an alternative calculation of the standard errors would lead
            one to conclude that the problem with Pitt and Khandker is
            that they underestimate the positive and statistically
            significant effect of women's credit on household
            consumption. As in their previous efforts, the methods of
            Roodman and Morduch are shown to bias the findings in the
            direction of rejecting the results of Pitt and Khandker. We
            also further examine two aspects of our instrumental
            variable approach that have been attacked by Roodman and
            Morduch. The first is the validity of the exclusion
            restrictions underlying the use of interactions between
            program choice and the set of exogenous variables (including
            the village fixed effects) as instruments. The second is the
            application of the "one-half acre" program
            eligibility rule. The authors show that identification does
            not require both of these, and present new results dropping
            each assumption in turn. The results originally reported in
            the Pitt and Khandker paper hold up extremely well in this
            new analysis.
repnme: repnme: Replicating replication :  due diligence in
            Roodman and Morduch's replication of Pitt and Khandker (1998)
display_title: Replicating replication : due
            diligence in Roodman and Morduch's replication of Pitt
            and Khandker (1998)
listing_relative_url: /research/2012/11/16985792/replicating-replication-due-diligence-roodman-morduchs-replication-pitt-khandker-1998
topicv3: Agriculture and Food Security m1326205 435,Poverty m1326687 695,Microfinance m1327227 367,Gender m1327621 615,Rural Development m1504505 2667
publishtoextweb_dt: 2012-11-01T00:00:00Z
historic_topic: Macroeconomics and Economic Growth,Science and Technology Development,Education
no_of_pages: 55
author_keys: 0: author_key: 92956
1: author_key: 123538
model_extractions: WPS6273 Policy Research Working Paper 6273

# Replicating Replication

Due Diligence in Roodman and Morduch’s Replication of Pitt and Khandker (1998) _Mark M. Pitt Shahidur R. Khandker_ The World Bank Development Research Group Agriculture and Rural Development Team November 2012
Policy Research Working Paper 6273

# **Abstract**

“The Impact of Microcredit on the Poor in Bangladesh: Revisiting the Evidence,” by David Roodman and Jonathan Morduch (2011) is the most recent of a sequence of papers and postings that seeks to refute the findings of the Pitt and Khandker (1998) article “The Impact of Group-Based Credit on Poor Households in Bangladesh: Does the Gender of Participants Matter?” that microcredit for women had significant, favorable effects on poverty reduction. In this paper the authors show that these latest Roodman and Morduch claims are based on seriously flawed econometric methods and theory and a lack of due diligence in formulating models and interpreting output from packaged software. On the basis of Roodman and Morduch’s preferred twostage least squares regression, an alternative calculation of the standard errors would lead one to conclude that the problem with Pitt and Khandker is that they underestimate the positive and statistically significant effect of women’s credit on household consumption. As in their previous efforts, the methods of Roodman and Morduch are shown to bias the findings in the direction of rejecting the results of Pitt and Khandker. We also further examine two aspects of our instrumental variable approach that have been attacked by Roodman and Morduch. The first is the validity of the exclusion restrictions underlying the use of interactions between program choice and the set of exogenous variables (including the village fixed effects) as instruments. The second is the application of the “one-half acre” program eligibility rule. The authors show that identification does not require both of these, and present new results dropping each assumption in turn. The results originally reported in the Pitt and Khandker paper hold up extremely well in this new analysis.

This paper is a product of the Agriculture and Rural Development Team, Development Research Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The author may be contacted at skhandker@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Replicating Replication:  Due Diligence in Roodman and Morduch’s Replication of Pitt and Khandker (1998)**

# Mark M. Pitt and Shahidur R. Khandker<sup>1</sup>

JEL code: C01; C21; C26; ARD.

> 1 Brown University and World Bank.  We thank Martin Ravallion, Will Martin, and Tiemen Woutersen for helpful comments and Kelley Smith for editorial suggestions. This paper and the code and data are available from <u>http://www.brown.edu/research/projects/pitt. Views expressed in this paper are entirely those of authors and do not</u> reflect in any way views of the World Bank or its affiliated organizations.
# **Replicating Replication:  Due Diligence in Roodman and Morduch’s Replication of Pitt and Khandker (1998)**

Mark M. Pitt and Shahidur R. Khandker

## **1.  Introduction**

“The Impact of Microcredit on the Poor in Bangladesh: Revisiting the Evidence,” by David Roodman and Jonathan Morduch (2011), hereafter referred to as RM, is the most recent of a sequence of papers and postings that seeks to refute the findings of Pitt and Khandker (PK) 1998, which found that microcredit for women had significant poverty-reducing impacts.

In the most recent iteration of their papers and web postings, RM claim that they have found three substantive and related weaknesses in PK: weak instruments, non-normality, and bimodality of the log-likelihood function. RM state that bimodality is “the most striking” finding, and implies that the PK estimates are inconsistent. They claim that this can be explained ‐ by “instrument weakness caused by disaggregating credit by gender” and that “non normality appears to interact with the instrument weakness to generate the bimodality.” They present test statistics to back up their claim of instrument weakness and non-normality, and suggest that instrument weakness and non-normality cause bimodality. They further suggest that the parameters estimated by PK lack validity based upon these three claims; additionally, using their preferred two-stage least squares (2SLS) linear LIML estimator, they fail to find the positive and statistically significant female credit effects found by PK, which they attribute to instrument weakness and non-normality. Like the earlier critiques by RM, we believe that these claims are based on flawed evidence.

This paper demonstrates that the tests of instrument weakness and non-normality presented by RM are invalid. As we will show, they have chosen testing procedures that necessarily tend to invalidly accept the hypothesis of weak instruments and reject the hypothesis of normal errors. Hidden from view in the paper are methods and results that reflect a serious lack of diligence in responding to and reporting errors that are plainly and repeatedly reported in the Stata output generated by the commands contained in the authors’ Stata “do” files.

In the weak instrument case, RM “pad” the matrix of instruments they use for the test so that it is over 58 times larger than that actually used by PK. We show that this augmented matrix of instruments biases standard 2SLS coefficients in the direction of the least squares bias; that is, they underestimate women’s credit effects. We provide simulation results that demonstrate that the PK model is very robust to variation in the weakness of the instruments, with instrument weakness measured using the methods of RM. The PK maximum likelihood approach recovers 2
the true parameters even when the instruments are quite “weak” and when the RM 2SLS approach demonstrates substantial bias. In addition, the linear LIML method that RM use generates estimates that are even less credible than standard two-stage least squares.

As for the test of non-normality, the skewness-kurtosis test for normality is the <u>only</u> parameter or test statistic that RM present in their paper that has not been adjusted for choicebased sampling. This failure to adjust for choice-based sampling results in over-rejection of normality when it is in fact true. No mention of this omission in the testing procedure is made in the paper.

RM do not investigate the econometric properties of the PK maximum likelihood when making their claims about bimodality and its causes and implications. We show below that, contrary to their implied claims, bimodality is a “standard” feature of this estimator even with strong instruments and normal errors, and provide evidence from a simulation exercise that the parameters identified at the global maximum mode are those of the true data generating process. As all agree that the PK results are those of the global maximum, there is no reason to suspect that PK have not recovered the true parameters.

RM also make incorrect claims concerning econometric theory. First, they wrongly believe that the bimodality of the PK log-likelihood implies that even when the model is properly specified (when errors are truly normal) the estimator is inconsistent, and that any claim otherwise is a violation of the theorem of the consistency of maximum likelihood. Second, RM maintain that the consistency of the maximum likelihood IV model requires normality, although it does not. Third, RM apparently believe that the recent literature on “weak instruments” applies to the PK model. There is no reason to think that it does, and indeed we present evidence that it does not.

There are other serious issues in the RM paper that we detail below. The lesson to be learned from this and the previous effort of RM (2009) is that their econometric methods are faulty and that their mistakes bias the findings in the direction of rejecting the results of PK. In the next section of this paper we briefly review the recent history of the RM replication attempts and our response to them. Section 3 addresses the supposed weak instrument problem and demonstrates that the RM two-stage setup used to generate tests of instrument weakness is wrong. Section 4 examines the faulty test of normality carried out by RM and shows that the PK model is robust to deviations of normality, just as econometric theory suggests. Section 5 examines the issue of bimodality, establishes that it is a “standard” feature of the PK model even with strong instruments, normal errors, and fairly large samples, reviews and cites the theorem on the consistency of maximum likelihood with respect to multi-modal likelihood functions, and demonstrates that the PK log-likelihood is not “flat” as RM suggest when appropriately graphed in the metric of probability. Section 6 addresses the issue of RM’s lack of care in estimating their two-stage linear LIML models for which most t-values are exactly 0.01 in absolute value. Section 7 further examines two aspects of our instrumental variable approach that have been 3
attacked by RM. The first is the validity of the exclusion restrictions underlying the use of interactions between program choice and the set of exogenous variables (including the village fixed effects) as instruments. The second is the application of the “one-half acre” program eligibility rule. We will show that identification does not require both of these, and present new results dropping each assumption in turn. The results originally reported in the PK paper hold up extremely well in this new analysis. Section 8 summarizes our results.

# **2.  Replication repeatedly redone**

Although Roodman and Morduch’s views on the validity of PK have never been published in a peer-reviewed journal, they have received broad exposure in media such as Roodman’s “Microfinance Open Book Blog,” tweets, Roodman’s (2011) book _Due Diligence: An Impertinent Inquiry into Microfinance_ , an extensive worldwide book tour, at conferences of development practitioners, and in testimony to the US Congress (posted on YouTube). The book and Roodman’s subsequent book tour specifically target Pitt and Khandker, and conclude that “on current evidence, the best estimate of the average impact of microcredit on poverty is zero.” His book recommends that microfinance institutions and donors “eschew any drive to extend credit to the poorest.” This is a very strong statement that appears to arise in large part from Roodman’s experience in replicating and seeking to refute PK.

Roodman and Morduch used their original replication paper (2009) to support the claim that the PK evidence is faulty, that the development community did not perform “due diligence” in evaluating the effectiveness of microfinance, and that microfinance has no discernible impact on the lives of the poor. Their latest revised version (2011) of that paper is just the most recent iteration in a continuing saga that dates back to Morduch’s 1998 attempt at replication prior to the official publication of PK.

David Roodman’s critical reviews on scholarly research cannot be taken lightly. His claim of a very high ranking among the world’s economists is a prominent feature of his biographies. His book (Roodman 2011) bio says that “in 2011, Roodman ranked in the top 10 in the Research Papers in Economics (RePEc) list of young economists in the world.” This is indeed remarkable given that his book bio also notes that “he has never taken a course in economics or statistics.”<sup>2</sup> Furthermore, he is a member of the four-person Advisory Board of the _Replication Programme Advisory Group_ of the International Initiative for Impact Evaluation (3ie).

> 2 We note that the RePEc ranking is based in large part on file downloads, which include journal articles posted on RePEc, working papers, and software contributions.  The vast majority of Roodman’s RePEc downloads are software add-ons for _Stata_ (xtabond2, in particular) and papers explaining how to use those add-ons.  To illustrate how writing some computer code can affect the ranking, note that in the most recent ranking “Top 5% of Authors” of all ages based on downloads (May 2012), Roodman ranks 10<sup>th</sup> in the world, above most living Nobel Laureates including Sims, Prescott,  Engle, Becker, Akerlof and Diamond.

> (http://ideas.repec.org/top/top.person.adownloads.html#pro120).  As another example, we note that number 9 on this list is a Swiss professor of sociology whose major download is a piece of Stata code for formatting regression output for printing.

4
Although this paper addresses the critiques of Roodman and Morduch (2011), it is useful to review the recent history of Roodman and Morduch’s replications. In short:

1. **RM (2009) found statistically significant and negative female credit effects in their 2009 version of the paper, the opposite of the PK findings** . This finding came about because of what can only be called a typographical error on their part. They included the variable named “ _crcensored_ ” when they should have used “ _nontar_ ” in a single line of their Stata code.   That is it.  This error was not the result of being confused by some complicated bit of mathematics, econometric modeling and programming, or by us not sharing computer code.  The error that turned the signs of coefficients around was simply the lack of the word “ _nontar_ ” and inclusion of the word “ _crcensored_ ” in one line of their code.<sup>3</sup>

2. **RM claim a lack of sharing made their attempts at replication difficult and resulted in their errors.** Although there were certain details of the PK model that are not found in the PK paper, these details were all inconsequential in the RM replication. In reference to “sharing” it should be noted that RM had the PK data for at least 11 1⁄2 years prior to their posting of the RM (2009), with Morduch being sent the data by us at least 9 months prior to the PK publication date of October 1998.<sup>4</sup> Morduch had apparently constructed all of the variables required for estimation for at least that long. Morduch, in the final version of his paper dated June 1998, lists all of the PK variables in the Appendix (most relevantly Appendix Table 1<sup>5</sup> ) to his paper along with means and standard deviations. RM (2009) confirm that these variables are appropriately constructed, noting that “Our replication data set matches Morduch’s original quite well, not surprisingly. “ So both the raw data and the actual variables used in estimation have been in Morduch’s possession since early 1998.<sup>6</sup> 3. **RM claim in their original replication that the signs of the credit effects do not matter.**<sup>**7**</sup> **They claim that the real issue is a lack of causality based upon test**

> 3 The correct variable _nontar_ is in the list of included exogenous variables in the 2009 RM paper, in PK (1998), in Morduch (1998), and in a 2008 email from Roodman asking Pitt for assistance.  The incorrect variable _crcensored_ is in none of these.  Consequently, the results that RM presented were inconsistent with what their paper and emails to us said that they did.

> 4 Roodman wrote the complete code for estimating the PK model by maximum likelihood prior to any contact between us, and has never suggested that there were any errors in his _cmp_ code that affected its ability to estimate PK.

> 5 This table is missing from the version available from Morduch’s site at NYU, although it is included in identically dated versions available elsewhere.

> 6 Roodman and Morduch did not make public the code that they used to convert the raw data into the variables used in estimation until December 2011 even though issues about the data and variables began almost three years previously.  Their code, which was publically distributed two and one-half years after their paper was distributed, is in the form of a binary file written in Microsoft SQL format.

> 7 Although they claim that “this is not about our inability to match the sign,” the difference in sign is front and center in their only academic publication (of which we are aware) that discusses their replication, the chapter titled “Access

5
**statistics that suggest that the PK instruments are invalid.** This is the same claim made in the current version of their paper, and refuted below.  However, in the 2009 version and in blog updates to that version, RM used different methods to argue this point.  As in the current version of the paper, the methods used in the earlier version are faulty.  The particular errors of those previous attempts to disprove causality in PK include invalid Sargan test statistics and wrong Hansen J-tests critical values, both as a consequence of their neglect of the implications of sample weights. Furthermore, the incorrect two-stage setup used by RM means that their Sargan and Hansen J-tests would be wrong even if there were no need to use sample weights. This error is discussed at length in this paper in the context of Kleibergen-Paap test statistics (Kleibergen and Paap, 2006), the test statistics adopted in this most recent iteration of RM, but it applies as well to Sargan and Hansen J tests.

# **3.  Weak Instruments**

The essential nature of the model as formulated in PK is quite simple, although we are unaware of any previous models of this sort in the literature. Some individuals are randomly offered treatment.<sup>8</sup> Those offered treatment choose a level of treatment c (credit), which can be zero. Other individuals are randomly not offered treatment, and their level of treatment is deterministically zero (c = ε = 0, where ε is an error term). The upper portion of the vector contains the level of treatment chosen by those offered treatment, and the bottom portion contains the non-stochastic zeros of those not offered treatment.<sup>9</sup> Analogously, the upper portion of the matrix contains a set of K exogenous variables (which may include dummy variables for place) that affect the level of treatment c among those with choice. The bottom portion of this matrix is zero, as is the bottom portion of the vector of errors , consistent with the deterministic nature of c=ε=0 of those without choice.

> to Finance” authored by Morduch and Dean Karlan in the widely respected _Handbook of Development Economics_ (Karlan and Morduch, 2009):

> “Roodman and Morduch (2009) attempt to find closure to the issue by returning to the data and rebuilding the analysis from scratch. They are unable to replicate results from Pitt and Khandker (1998) or Khandker (2005). In fact, their estimates carry the opposite sign. Rather than concluding that microcredit harms borrowers, however, they unearth a raft of identification issues which are not solved with panel data. Their revised analysis casts doubt on all of the findings from the related set of papers, including Morduch (1998)’s finding on consumption smoothing. “

> 8 More generally, we require only that the offer of treatment be random conditional on a set of exogenous covariates that might include location fixed-effects, the assumption actually made in PK.

> 9 Thus, elements of c can be stochastically or deterministically zero depending on if individuals are offered treatment.

6
An outcome of interest y, such as total household expenditure, is observed for both those offered treatment choice and those not offered treatment choice.

where is a vector of errors. To be clear, all of the elements of X in the first term of the left hand side of equation (1) are identical to the elements of X in the upper sub-matrix of the first term on the left hand side of (2). To keep the notation simple, and because we need only to distinguish between observations that are deterministically zero and observations that are not, the lower sub-matrix of the first term on the left hand side of (2) is also called X because it contains the same variables, but the values of the variables (elements) in the lower sub-matrix correspond to a different set of individuals -- those individuals without credit program choice.

We are interested in estimating the treatment effect parameter δ. Least squares estimation of (2) yields biased estimates of the parameter of interest δ if the errors ε and ν covary. In PK, it is assumed (fairly innocuously) that the joint distribution of the errors is The setup presented above is sufficient to provide a consistent instrumental variable estimate of . This model is identified even though there are no exclusion restrictions of the usual sort; that is, there is no set of exogenous variables Z that affects c but does not affect y conditional on c.<sup>10</sup> The model is identified by the deterministic lack of credit program choice for a subsample of the data. PK estimate the model by maximum likelihood. The lower submatrices in (1) are all zero and thus provide no information to help identify the first-stage parameters πm, and hence are not in the portion of the likelihood identifying those parameters. The likelihood is presented in detail in an appendix to PK.

Roodman and Morduch Stata forgo ML in favor of the packaged 2SLS test statistics obtainable from Stata. The question is whether the model, given in its simplest form in (1) and (2), can actually be estimated by 2SLS packages without doing violence to the model’s setup.<sup>11</sup> The requirements imposed by the packaged software (Stata) used by RM of particular concern are that (1) there must be a vector of identifying instruments Z with at least as many columns as there are endogenous variables, and (2) all the included exogenous variables in the second-stage equation must also appear in the first-stage.

> 10 Note that adding exogenous variables with associated parameter vector βp to equation (2) does not aid identification.

> 11 RM use the _ivreg2_ package of _Stata_ with the “liml” option that estimates linear 2SLS by limited information maximum likelihood.  We will say more about this choice below; however, the “linear LIML” model estimated by _ivreg2_ is essentially two-stage least squares and not the “LIML” estimated by iterating over a nonlinear loglikelihood until convergence as carried out in PK.

7
To see whether these requirements of the packaged software are consistent with the PK model, we re-write equation (1) to be consistent with the 2SLS format and where c0 = , Z = , **X** = , and e = : or The first-stage regression equation (3) will yield x<sup>= 0.0, which is sufficient to retain the</sup> property that C is deterministically zero (ε=0), the property that underlies identification, and πm=πz will be consistently estimated. Two-stage least squares estimation carried out in this way is entirely consistent with PK, except that it includes the bottom sub-matrices containing all zero elements (recall that x<sup>= 0.0). That is, it simply adds observations containing only zeros to the</sup> vector of dependent variable observations and to the matrix of independent variable observation. Adding all of these deterministically zero observations still recovers a consistent m,<sup>but not the</sup> correct parameter covariance matrix.

The model defined by equations (1) and (2) illustrates the source of identification in PK, but in practice the PK model extends this model by allowing for the two endogenous variables Cm and Cf (male and female credit), the effects of which vary for three different program providers for each gender. To start, consider only the gender disaggregation. As before, there is random assignment to credit choice. In the maximum likelihood approach of PK, the “firststage” uses only the (informative) stochastic observations: Male credit: And, female credit: Where cmo refers to observations on the credit treatment choice of males where only males have choice, cmf refers to observations on the credit treatment choice of males where both males and females have choice, cfm refers to observations on the credit treatment choice of females where both females and males have choice, and cf0 refers to observations on the credit treatment choice of females where only females have choice. The second-stage with gender-specific credit is 8
In the packaged 2SLS estimation of the authors, the set-up for the first-stage equations is: Male credit: And, female credit: Note that πmx= πmf=0 is required for deterministic zero C in equation (7), and πfx= πfm=0 is required for deterministic zero C in equation (8). However, in general, least squares estimation will result in mx<sup>≠ 0,</sup> mf<sup>≠0</sup> fx<sup>≠ 0, and</sup> fm<sup>≠0, as the three matrices of exogenous variables</sup> in each first-stage are not perfectly collinear. The consequence of this is that rather than having K parameters in the first-stage, there are 3K parameters. The additional instruments that have been added are necessarily “weak;” that is, they have individual t-statistics and group F-tests that are close to zero, if the data generating process set out in equations (1) and (2) is true. In particular, πmx and πfx , the gender-specific versions of parameter πx of equation (3) – a parameter that would be estimated as identically zero in equation (3) – are now identified (that is, are nonzero) as a consequence of the artificial addition of the zero sub-matrices to the dependent and independent variable data matrices.

In the informative stochastic part of the sample, there is perfect collinearity between the first-two matrices on the right-hand side of equations (7) and (8). In equation (7), the matrix of 9
0 x x 0 <!-- Start of picture text --> Cm<br>0<br><!-- End of picture text --> <!-- Start of picture text --> 0<br>o.<br><!-- End of picture text --> ¥ x Cm 0 y) \x 0 o. v <!-- Start of picture text --> y)<br><!-- End of picture text -->
<!-- Start of picture text --> it<br><!-- End of picture text --> it <!-- Start of picture text --> fm0.1<br>0<br>0<br>Cm.<br>0<br>0<br>i]<br>0<br>o<br>0<br><!-- End of picture text --> <!-- Start of picture text --> i<br><!-- End of picture text --> i <!-- Start of picture text --> xX<br>xX<br>xX<br>xX<br>x<br>x<br>0O<br>0<br>i]<br>0<br><!-- End of picture text --> <!-- Start of picture text --> x<br>X<br>X<br>x<br>Xx<br>x<br>x<br>Xx<br>x<br>x<br><!-- End of picture text --> <!-- Start of picture text --> i]<br>0<br>0<br>x<br>x<br>x<br>xX<br>Xx<br>x<br>0<br><!-- End of picture text --> <!-- Start of picture text --> Emo<br>)<br>i)<br>emf<br>0<br>0<br>0<br>0<br>0<br>0<br><!-- End of picture text -->
PK estimate the male credit first-stage equation (4) using a matrix of independent variables that contains 54,534 elements; that is, 61 variables times 894 households with choice to join male credit programs. RM estimates the same first-stage equations using a matrix of independent variables that contains 58.6 times more elements than PK.<sup>14</sup> This wrongly expanded matrix consists of 204 independent variables times 5218 observations times 3 program types for a total of 3,193,416 elements. (The number of independent variables is only 204 in RM because Stata drops out 102 more independent variables as a result of perfect collinearity).

In the case of male participation in the Grameen Bank, the first-stage equation used by RM contains a matrix of independent variables for which fully 87.5 percent (931,872 out of 1,064,472) of the elements correspond to households in which males have (deterministically) no choice of participation in the Grameen Bank. In effect, RM’s effort to examine the weakness (and other qualities) of the instruments used in PK uses a method that multiplies the size of the matrix of instrumental variables by a factor of 58 and, as a result, yields test statistics that have hardly any relevance to PK at all. It is no wonder that such methods allow RM to conclude that “The linear LIML estimates in Table 6, with their large standard errors, seem closer to the truth.” (p. 32) Issues arising from this artificial matrix remain even when RM aggregate credit effects across groups, resulting in only two first-stage equations, one for each gender. In this case the first-stage equation of the determinants of male credit program participation has 1,069,690 elements (205 independent variables times 5218 observations). This is 19.6 times the number of elements in the matrix of instrumental variables in PK.<sup>15</sup> RM’s claims fall short in another important way. There is no reason to believe that _any_ of the literature on detecting weak instruments, and on the effect of weak instruments on parameter estimates, has any direct bearing on a PK-type model. The PK model is not a usual sort of instrumental variables model in which there is a set of identifying instruments associated with exclusion restrictions that meet the rank and order conditions for identification. Identification in PK clearly arises from having observations on the dependent variable y and the exogenous variables X for individuals that exogenously have no choice to obtain (credit)

> 14 A relatively small part of this 58 times inflation is the result of RM replicating the data matrix for the first-stage into three rounds so that it fits into the single-command framework of the Stata _ivreg2_ command. 15 In an email David Roodman recently sent to Mark Pitt in response to an early undistributed version of this paper, Roodman maintains that the setup for two-stage estimation of PK is that given by equations (9) and (10) because it is the classical two-stage least squares setup.  He argues that if test statistics show that the instruments in (10) are weak, then that means just what it says.  In his email elucidation of this argument he fails to recognize the fact that credit is deterministically zero for a sizable part of the sample and that these deterministically zero “dependent variables” are the salient difference between the PK model and the classical two-stage least squares problem he is fixated on.  Once again, if one tries to explain deterministic zero with a regression containing a set of independent variables, one gets nonsense.

12
treatment. Clearly, the number of observations without choice of treatment critically affects identification – at the limit, if there is only one such observation in the data, the model with both exogenous variables X and treatment c is not identified – and yet the authors proceed as if this were just another two-stage least squares problem. They seem to recognize there might be a problem, but then fail to properly address it: _Because of the unusual design of the PK estimator, no theory is available to measure instrument weakness in terms of its potential for distorting estimates. We find some evidence, however, that credit eligibility is a weak instrument for credit uptake. Differentiating credit by gender exacerbates the weakness_ . (p. 27) If there is no statistical theory, then on what basis is there evidence?

Finally, one can easily simulate PK-like data sets and (i) estimate the model as PK do, by maximum likelihood, and (ii) estimate the model as RM do, using the wrongly augmented set of instruments and 2SLS.<sup>16</sup> The results of generating 200 such datasets are provided in Table 1.

Using the RM approach, the simulated data generating process has a positive bias for males and a negative bias for females<sup>17</sup> . Using PK’s maximum likelihood method there is absolutely no evidence of bias arising from “weak instruments.” The mean value for the F- statistic for the first-stage regressions given by equations (4) and (5) are 3.47 and 3.41, respectively. Using RM’s 2SLS method, however, there is strong evidence of bias in the direction of the least squares bias. The male parameters are almost twice as large as the true values, and the female parameters are one-third too small, so that RM’s method finds that male and female credit effects are about the same while the true female effect is three times as large as the true male credit effect. The Kleibergen-Papp rk rank test F statistic for weak instruments in the 2SLS has a mean value of 0.736, and the null hypothesis of “weak instruments” is never rejected in the 200 simulations. It is no surprise that the instruments are weak in RM’s approach,

> 16 RM actually estimate their model with linear LIML _not standard_ two-stage least squares.  This turns out to be another crucial error on their part that is discussed at length below.  We use standard 2SLS here so as to highlight the effect of the weak instruments bias that arises as a consequence of RM’s padded matrix of instrumental variables.  The further degree of bias arising from the use of linear LIML is estimated with these same simulated datasets below and discussed at length.

> 17 For the simulation, 40 variables Xi were generated from independent normal (0,1) random variables.  The treatment chosen by females who are offered treatment is cf = 0.5∑x + 12μ + 8εf, and the treatment chosen by males who are offered treatment is cm = 0.5∑x - 12μ + 8εm, where μ , εf, and εm are independent normal (0,1) random variables.  The outcome is y =   0.75*cf + 0.25*cm + 0.5∑x  - 12μ + 8ν, where ν is an independent normal (0,1) random variable.  Female treatment was made available to 40 percent of the sample, male treatment to 40 percent, and no treatment to 40 percent.  Half of the observations with female treatment also had male treatment, so when taking into account the overlap the percentages sum to 100 percent.  Those treated were randomly assigned to one of three groups.  Two hundred data sets with 5000 observations each were independently generated and the models estimated with maximum likelihood using Roodman’s _cmp_ estimation package, and the _ivreg2_ (2SLS) package of _Stata_ .

13
since they predict both behavior and deterministic zeros. All the 2SLS results of RM are the product of a flawed model that ultimately looks very little like the model of PK.<sup>18</sup>

# **4. Test of normality**

Roodman and Morduch also claim that “non-normality appears to interact with the instrument weakness to generate bimodality” and that PK have an “incorrect likelihood function, as εo [the second-stage residual] is not normally distributed” and claim that “the PK structural error is not” normal based upon the skewness and kurtosis test for normality of D’Agostino, J. Belanger, and D’Agostino Jr. (1990) and implemented as part as the Stata command _sktest_ . Just like their tests of instrument weakness, the normality tests that they carry out have errors that will necessarily lead to the over-rejection of the null hypothesis when it is true. That is, the authors have altered the PK model in such a way to make the instruments appear weak, and they have also calculated the test of normality in a way that makes the errors seem non-normal.

Their mistake is that they do not account for the sample weights in constructing the test statistic. This is odd, since they use sample weights in computing all other econometric estimates that they present in the paper, including in the table of means and standard deviations. So why is this not the case with the test of normality? The _sktest_ in Stata, perhaps unique among the many Stata estimation commands that they use in the paper, does not allow for sample weights. Rather than go to the trouble of computing a sample weighted version of _sktest_ , they proceed with computing unweighted test statistics, making no reference whatsoever in the text or table that sample weights were not used in only this peculiar case.

If the data sample arises from stratified random sampling (choice-based sampling in the PK case), then not correcting for the sample design will lead to over-rejection of tests of normality based upon skewness and kurtosis. Consider a sample r obtained by drawing N times from a normal distribution with zero mean and unit variance. Now sample (without replacement) a proportion p<1 of the values of r that are less than, say, 2.0, and “sample” a proportion p=1 all of the values of r having values greater than or equal to 2.0. Let q denote this stratified sample of r. An unweighted skewness/kurtosis test on q will necessarily over-reject the

> 18 None of this means that the PK model cannot be estimated with two-stage least squares.  Moreover, it should not be very difficult to accomplish.   As shown above, however, one cannot expect to get it right by arbitrarily inserting a few million new data points into the procedure just so a two-stage model can be estimated with a single command from pre-packaged software.  Estimating the first-stage equations (4) is just OLS.  From those first-stage equations it is straightforward to predict credit for those with credit program choice, and make this prediction exactly zero for those without choice.  The second-stage is just OLS substituting the predicted credit for the actual credit.  All econometrics textbooks provide the formula for computing the standard errors for the case of predicted regressors, and these formulas can be appropriately adjusted for this case where some of the observations are exogenous in that they do not have predicted regressors.  In every simulation do file that was appended to Pitt (1997), this two-step approach (using predicted credit for those with choice and exactly zero for those without choice) was the method used to illustrate PK.  This was then followed in every instance with a check for the equivalence of 2SLS.  The 2SLS was equivalent to this two-step method in every case where there was one gender and not equivalent in the case in which there were two genders, exactly as the simple algebra above predicts.

14
null hypothesis of normality even though the true process that generated the data is normal.<sup>19</sup> In a large sample such as PK’s, in which sample proportions vary greatly across strata, it is to be expected that an inappropriate skewness/kurtosis test that did not apply sampling weights would reject the null hypothesis of normality.

To further demonstrate the inaccuracy of their approach, we have constructed a simulation of the PK model of the following sort. Thirty variables x1 through x30 were generated from independent normal (0,1) random variables. The treatment chosen by those who are offered treatment is c = 0.40∑x + 12μ + 8ε where μ and ε are independent normal (0,1) random variables. The outcome y = 1*c + 0.40∑x + 12μ + 8ν, where ν is an independent normal (0,1) random variable. Half of the sample of 30,000 was randomly assigned to treatment. There is only one gender and one program (credit) source. Eighty percent of the observations for which c < 10 were randomly dropped. One hundred data sets were generated and the model estimated with ML using Roodman’s cmp estimation package with each data set, using sample weights. The mean value of the estimated parameters on c was 1.00 (true value = 1) with a standard deviation of 0.028. The predicted errors of the second stage were calculated and tested with the Stata _sktest_ command, exactly as in the authors’ paper. The median p-value of the skewness tests was 1.43e-06 and the median p-value for the combined skewness/kurtosis test was 8.49e-06. As the errors in this simulation exercise were normally distributed, these very low p- values rejecting the null hypothesis of normality are solely the result of not using sample weights in calculating the test statistics.

It is useful to consider whether non-normality of the errors should be a concern at all. In general, consistency of the maximum likelihood estimation of the multivariate regression model – also known as Seemingly Unrelated Regression (SUR) -- does not require that the errors actually be normal when that is the assumption made in constructing the log-likelihood (Davidson and MacKinnon 1993, p.315). In a simultaneous equations model estimated by LIML or FIML, such as PK, the same holds. The assumption that the errors terms are multivariate normal need not be true to insure that the parameter estimates are consistent and asymptotically normal (Davidson and MacKinnon, 1993, p. 641). Although it is well known that non-normality or heteroskedasticity yield biased estimates of Tobit models, the Tobit part of the PK model is the first-stage equations, and as is well known, consistent estimation of the first-stage equations is not necessary for consistency of the second-stage equations where the credit effects are estimated. Moreover, the RM tests of normality are only for the second-stage, which is absolutely continuous for the outcome of interest (household expenditure), and not for the Tobit

> 19 The following Stata code demonstates this: set obs 50000 gen x = rnormal() sktest x drop if uniform() < 0.9 & x < 2.0 sktest x

15
first-stages. Consequently, we can see no reason for the suggestion by RM that the actual distribution of the second-stage errors in PK has any important empirical ramification at all.

Even though RM’s concern with bias arising from non-normality draws no support from econometric theory, we pursue the matter using simulation. This goes much further than RM’s faulty skewness and kurtosis tests of the second-stage errors reported above. We allow for nonnormality in both the first-stage and second-stage errors, and furthermore we specify two separate deviations from normality, one that generates skewness and one that generate excess kurtosis. The first-stage equations in every case are Tobits with a high proportion of censored values. Specifying a Tobit reveals how well the second-stage credit effects are estimated when the first-stage Tobit, which requires normality, is inconsistent.

The skew distribution was generated by summing a (non-negative) half-normal distribution and a normal distribution. Table 2 presents the results of generating 200 independent datasets with skew error distributions for both first-stage and second-stage equations. The median p-value for the tests for normality based on skewness is 3.49E-08, so skewness is quite significant. The RM 2SLS estimates perform poorly (although certainly as a consequence of the faulty construction of the RM instruments rather than because of skewness). The means of the maximum likelihood estimates of the credit effects using the PK log-likelihood are within 0.003 of the true value of 1.

The excess kurtosis was generated by adding a t(20) distribution to a normal. Table 3 presents the results of generating 200 independent datasets with error distributions characterized by excess kurtosis for both first-stage and second-stage equations. The median p-value for the tests for normality based on excess kurtosis is 7.64E-11, so excess kurtosis is quite significant. The RM 2SLS estimates perform poorly, again a likely result of the RM method. The means of the maximum likelihood estimates of the credit effects using the PK log-likelihood are within 0.005 of the true value of 1.

In short, econometric theory suggests that the PK maximum likelihood results are consistent when errors are non-normal, and the simulation exercises are in agreement.

# **5.  Bimodal log-likelihood**

RM claim to have discovered one symptom in the PK regressions “known to be associated in 2SLS and linear LIML with weak instrumentation: bimodality (p. 32).” Narrowing down the source of the problem, they conclude that it is explained by instrument weakness caused by disaggregating credit by gender (p. 35). The reasoning for this causal assertion is that, as described above, when they estimate their model by linear LIML they obtain Kleibergen-Papp rk rank test statistics suggesting weak instruments, although they fail to note that in calculating these test statistics they add scores of extraneous “instrumental variables” and thousands of deterministically zero observations to the first-stage matrix. In addition, as we show below, their use of linear LIML also results in an ill-conditioned parameter covariance matrix and outlandish 16
‐ parameter estimates. Furthermore, they claim that “The non normality appears to interact with the instrument weakness to generate the bimodality (p. 33),” although there is no demonstration in RM that this supposed interaction is in fact responsible for the bimodality. This assertion follows from their claim that “in accordance with the consistency of ML, the potential for multimodality in SUR disappears asymptotically if the model is correct—in particular, if the modeling error is normal. (p. 33).”<sup>20</sup> RM claim that the SUR label is appropriate one for the PK ‐ model, stating that “PK’s first stage is a two equation SUR system (complicated by Tobit censoring). (p. 33)” By that standard, any IV model with two or more first-stage equations is a SUR model and must, according to RM, have a globally concave log-likelihood function, asymptotically. The logic of RM is that the (i) PK model is a SUR model; (ii) SUR models have a globally concave log-likelihood as the sample size increases; (iii) the sample size in PK is large; so that (iv) something is wrong in PK since the log-likelihood is not concave; and (v) the source of the problem must be some interaction between instrument weakness and non-normality since there are test statistics that seem to suggest that both characteristics are true.

First of all, the assertion by RM in the paper and in Roodman’s blog that the consistency of maximum likelihood requires that the log-likelihood function have a single mode is just wrong.<sup>21</sup> In theorem 2.5 of the _Handbook of Econometrics_ , Newey and McFadden (1994, p. 2131) set out the conditions for the consistency of maximum likelihood, and having a single mode of the log-likelihood function is not one of the necessary conditions. Newey and McFadden discuss the example of maximum likelihood estimation of the parameters of the Cauchy distribution (in their example 1.1), which is consistently estimated by maximum likelihood even though it is multi-modal for small values of N, and note that in the multi-modal likelihood case in general “consistency results only apply to the global maximum (p.2117).”<sup>22</sup> As there does not seem to be any dispute that the PK parameter estimates correspond to the global maximum, there is no consistency issue that arises with the PK estimates if the likelihood were bimodal. Furthermore, the parameters associated with the local mode identified by RM are neither consistent nor asymptotically normal.

The log-likelihood of the PK model with the PK data is, in fact, bimodal. RM wrongly assert that the bimodality in the PK likelihood means that the PK results are not credible as a consequence of the twin evils of weak instruments and non-normality, and do so without examining the attributes of the PK likelihood.

It might well involve some effort to formally work out the properties of the PK likelihood. It is relatively simple, however, to simulate data sets that have normally distributed

> 20 In contrast to this claim by RM, the consistency of SUR estimated by maximum likelihood does not require that the errors actually be normal when that is the assumption made in constructing the log-likelihood (Davidson and MacKinnon 1993, p.315).

> 21 The claim on the Roodman blog is that “a foundational theoretical finding is that if the model being fit is correct, the probability of there being more than one mountain peak [mode] goes to 0 as the number of data points increases.” (http://blogs.cgdev.org/open_book/2011/12/bimodality-in-the-wild-latest-on-pitt-khandker.php) 22 On estimation of the multi-modal Cauchy distribution, see Ferguson (1978).

17
errors and check if (i) the log-likelihood is bimodal, and (ii) the frequency with which the maximum of the log-likelihood corresponds to the true value of the parameter of interest -- the credit effect, δ. The results of this simulation (Table 4) demonstrate that (i) bimodality was found in all 100 simulations, and is likely to be a “common” feature of PK-type models; (ii) all 100 simulations used data that were constructed with normal errors, so that non-normality is not an implication of the finding of bimodality, as RM suggest; (iii) the global maximum was tightly centered around the true value of δ, so that bimodality is innocuous in the simulations because the true estimate is recovered in every case; (iv) “weak” instruments are not necessary for bimodality, as RM suggest, as the instruments in the simulations were sufficiently “strong” to identify δ with tight bounds; and (v) the inferior mode was always in the direction of the least squares bias, that is, the mode had a smaller credit effect δ. In no case did the mode associated with the smaller value of δ have a higher likelihood. RM do not dispute that the PK estimates are the global maximum. Although the data-generating process of the PK data is, of course, unknown, this is suggestive evidence that the global maximum in PK, the one with a significant and positive credit effect, is the “true” maximum.

The simulated data-generating process has two genders and one program. Forty variables Xi were generated from independent normal (0,1) random variables. The treatment chosen by those who are offered treatment is cf = 0.5∑x - 7μ + 5εf for females, and cm = 0.5∑x - 7μ + 5εm for males, where μ , εf and εm, are independent normal (0,1) random variables. The outcome is y = 1.0*cf + 1.0*cm + 0.5∑x + 7μ + 5ν, where ν is an independent normal (0,1) random variable. Half of the generated sample is female and half male. Treatment was made available to half of each gender-defined sub-sample. One hundred data sets with 5000 observations each were independently generated and the models estimated with maximum likelihood using Roodman’s _cmp_ estimation package to trace the converged likelihood over a range of the credit effect parameters δ = δf = δm starting at -0.3 and incrementing by 0.1 until δf =1.5 . The gender breakdown is not really relevant in this exercise, since the point is to examine the shape of the log-likelihood surface with respect to δ, the parameter on credit.

In Table 4, as noted, the log-likelihood with respect to δ is bimodal in every single simulated dataset. The global maximum was tightly clustered around the true value of δ = 1, and the local maximum (inferior mode) was clustered around δ = -0.1.<sup>23</sup> Figure 1 presents the log-likelihood surface for the 1<sup>st</sup> , 25<sup>th</sup> , 75<sup>th</sup> , and 100<sup>th</sup> simulated data set. The resemblance to the PK log-likelihood surface presented by RM (see also our Figure 2) is striking. One attribute of a bimodal log-likelihood is that selectively dropping observations can “cut off” the mode corresponding to the global maximum, resulting in a discrete (and negative, in this case) jump to the local mode. Indeed, it was not difficult to find data-generating

> 23 When iterated to convergence, the mean value of the estimated δ was 0.998 with a standard deviation of 0.032, and the asymptotic t-ratios on δ were in excess of 30.0 in all simulated datasets.  The data-generating process thus seems to result in a very well-identified model; that is, the instruments are not weak.

18
processes of the type outlined above (having, for example, normally distributed errors) for which the judicious dropping of less than 1 percent of the sample occasionally resulted in the local mode of the complete sample becoming the global maximum of the selected sample. These few observations may be deemed “outliers,” but at least in the simulated case, dropping them from the sample generates incorrect inference about the underlying parameters of the true data generating process.

Bimodal log-likelihood surfaces were easy to obtain – one was found on the first try. Some limited experimentation found that generating datasets with 50,000 observations (instead of the 5,000 used in the simulations) from the same data-generating process still resulted in bimodal likelihoods. If the explanatory power of the instruments in the first-stage is made sufficiently large, bimodality seems to disappear. This occurred when the t-values for the credit parameter δ were pushed to exceed 42.0 simply by increasing the value of the parameters on the instruments in the first-stage so that average t-ratio for the 40 (independent) instruments in the first-stage was around 5.0.

What are the implications of all of this for the PK results, then? Are the instruments weak and the errors non-normal, and does it matter? The “foundational theoretical findings” of RM claiming a lack of consistency for maximum likelihood models with multiple modes are wrong; moreover, maximum likelihood of the broad class of IV models to which PK belongs is consistent in the presence of non-normal errors. Therefore, even if the errors are not normal (and we think it is not too hard to reject normality in large micro datasets such as this), this attribute is neither necessary nor sufficient to bias the results, a proposition supported by our simulation results. RM’s “solution” to the presumed issue of non-normality, linear two-stage least squares by LIML (which they term _linear LIML_ ) regression, which they hold out as the “robust” estimate, is, as we will show, like their test statistics, an approach with a built-in bias to rejecting the PK results as a consequence of the artificial and unjustifiable inflation of the instrumental variable data matrix, making the instruments weak, along with the inclusion of observations with deterministically zero credit program participation in the first-stage.

How important is bimodality for the PK estimates? As we have shown, bimodality is not necessarily a perverse occurrence with the PK likelihood, and in the simulations, at least, the global maximum was always at the true value. What about RM’s claim (p. 32) that the difference in the log-likelihoods between the modes “is slight,” that the “difference probably does not deserve much weight” and “is better viewed not as two peaks but as a single, wide one that straddles 0 and implies that the coefficient is estimated with great imprecision”? It is difficult to reconcile the statement that the female credit coefficient is estimated with “great imprecision” with the asymptotic t-ratios that PK report (4.24, 4.25, and 3.81 for BRAC, Grameen Bank, and BRDB, respectively). For the PK estimates, the log-likelihood was concave around the global maximum and the t-values were computed as clustered (by household), HuberWhite heteroskedasticity-robust standard errors. As the parameter standard error reflects the 19
slope and curvature of the log-likelihood around the maximum, how are such small standard errors consistent with the claim by RM that the log-likelihood is almost flat?

The top panel of Figure 2 presents the log-likelihood surface based upon the PK version of the dataset and restricts credit effects to vary by gender but not by group.<sup>24</sup> If the PK likelihood were really a true likelihood and not a pseudo-likelihood, about which more will be said below, a test of the hypothesis that any value of δ is equal to the maximum likelihood value of δ is given by twice the difference in their respective log-likelihoods and is distributed as χ<sup>2</sup> with one degree of freedom. It is straightforward to calculate the probability associated with this χ<sup>2</sup> statistic. This surface would then represent the probability that any parameter δ (on the horizontal axis) is equal to the value of δ at the global maximum (0.0437). The bottom panel of Figure 2 is the probability surface corresponding directly to the log–likelihood surface in the top panel. The local mode is barely visible in the bottom panel of Figure 2, as it is situated at very low levels of probability. Figure 3 is a close-up that focuses in on the area around the local mode so that one can read the probabilities associated with this mode off of the figure. The local mode corresponds to a probability less than 0.0002.

The complication is that the objective function that PK maximize is not a log-likelihood but a _pseudo_ -log-likelihood, as a consequence of sampling weights, and the “two times the difference in the log-likelihood is χ<sup>2</sup> “ rule does not apply to pseudo-log-likelihoods. The “true” probability surface can be below or above the one drawn in the figure. One simple way to see if the probability surface in the figure is too high or too low is to compare the probability of δ=0 from the clustered Huber-White parameter covariance matrix with the probability obtained from two times the differences in the pseudo-likelihoods. The p-value from the t-statistic is 0.0000013 (corresponding to a t-ratio of 4.83 or a χ2(1) = t<sup>2</sup> = 23.37. The p-value from the pseudolikelihood is 0.0000030, which is higher (by more than 2-fold) than the p-value from the t- ratio,<sup>25</sup> so that the probability surface pictured in Figure 3 and the bottom panel of Figure 2 is likely too high – that is, the probability value associated with the local mode is actually smaller than pictured in the figure and is likely less than 0.0001. The bottom line is that bimodality does not appear to be an issue in PK as (i) there is plenty of curvature in the estimated PK loglikelihood function, (ii) the log-likelihood surface with respect to the single parameter corresponding to the women’s credit effect is not flat by the relevant probability metric, (iii) the probabilities associated with the local mode are extremely small, and (iv) simulation results suggest that the global mode corresponds to the parameters of the true data generating process.<sup>26</sup>

> 24 Differences in credit effects by group are inconsequentially small in PK. To remain consistent with the estimates of Pitt (2011a) and (2011b), we use all three rounds of credit data in estimating the first-stage equation.

> 25 This p-value corresponds to χ2(1) =   21.84.

> 26 RM report on a bootstrapping exercise in which they draw whole villages and then count the number of negative regression parameters.  No implications can be drawn from such a procedure as it lacks any econometric justification, nor do they provide any.

20
# **6.  Due diligence**

Leaving aside their errors in logic – for example, not using choice-based sampling weights in calculating tests of normality – there are serious issues in the RM paper that arise from an apparent lack of attention to the methods and output that RM must have obtained. The former type of error – for example, the unweighted tests of normality – is not directly revealed by the computer output. When Roodman and Morduch computed the skewness/kurtosis test of normality with the Stata _sktest_ command without using sampling weights, the software did not offer a warning that sampling weights should be used. However, things were very different when they estimated their “preferred” linear LIML estimates (their Table 6) of the effects of credit on a measure of household consumption. These estimates form the basis for their tests of instrument weakness and are used to support their contention that credit effects are zero. Rerunning the estimates of RM’s Table 6 using the data and code provided by Roodman and Morduch results in error messages and computer output that is the equivalent of sirens blaring and red lights flashing to proclaim that something is terribly wrong with the estimation. After each first-stage equation estimated as part of the their _ivreg2_ 2SLS command with the LIML option, Stata reports: “ _Warning: estimated covariance matrix of moment conditions not of full rank. model tests should be interpreted with caution_ ” More worrisome than that is that 105 out of 110 slope coefficients in the second-stage output as reported by Stata have a reported t-ratio of exactly 0.01 in absolute value.<sup>27</sup> Regression output like this is an unambiguous and well-known signal something is very wrong. Nonetheless, RM overlook this, and report in their Table 6 the parameters, t-statistics,<sup>28</sup> and Kleibergen-Papp rk rank test statistics exactly as they are reported by Stata. Columns (2) and (3) of Table 5 provide all of the second-stage regression coefficients and t-statistics exactly as generated by Stata’s _ivreg2_ command – just copied and pasted into the table.<sup>29</sup> In addition, while a negative R-squared is possible with 2SLS, it is not conceivable that the (centered) R- squared is -381.4, so that the residual sum of squares is 382 times larger than the sum of squares, as the Stata output reports and which is presented at the bottom of Table 5.<sup>30</sup> There is other evidence that RM’s estimates of their Table 6 column (1) are suspect. The last two columns of our Table 5 provide standard two-stage least squares (as opposed to linear LIML) estimates of the same model. These are from the same data and exactly the same set of variables, the only difference being that RM used the ‘ _liml_ ’ option on i _vreg2_ , and these standard

> 27 Stata reports two digits after the decimal point when reporting t-ratios. The remaining five t-ratios are reported as 0.00, 0.02, 0.03, and 0.04, 0.25 in absolute value.

> 28 RM report three digits after the decimal point so that five of them are 0.012 and one is 0.013.

> 29 Ordinarily, one would not show all of the village fixed effects and other “nuisance” parameters in a table, but they are important here even though the names of the parameters are not.  They are all presented in the table so that the reader can get the full visual impact of 105 coefficients with an absolute t-statistic of 0.01.

> 30 Note as well that the value of the F-test of all of the parameters is F(110,1756)=1.85 in the linear LIML first-stage equation but F(110,1756)=14.88  for the exact same model estimated by standard 2SLS (our Table 5, bottom).

21
2SLS estimates leave out that option. Exactly what LIML means in two-stage least squares estimation is described in detail below. A comparison of the linear LIML estimates of columns (2) and (3) and the standard 2SLS estimates in columns (3) and (4) suggest that they have no relationship to each other at all. For example, the coefficient on the variable _nontarpk_ in the 2SLS is 0.146 with a t-statistic of 3.39, but in the linear LIML it is -2.27 with (of course) a t- statistic of -0.01. RM estimated this model by standard 2SLS and posted it online as part of their response to Pitt (2011a), titled “Response to Pitt’s Response to Roodman and Morduch’s Replication of..., etc.” (available at http://blogs.cgdev.org/open_book/2011/03/response-to-pitts- <u>response-to-roodman-and-morduchs-replication-of-etc.php), and thus should be aware of this</u> discrepancy in the t-statistics. Our Table 6 provides their 2SLS from that March 2011 response to Pitt as well as the linear LIML estimates that appear in this paper, which was distributed as a working paper in December 2011.

Consider as well the second column of parameter estimates in Table 6 of RM. Stata once again warns that the estimated parameter covariance matrix is not of full rank, and 105 out of 106 t-values in the second-stage are less than 0.5 in absolute value.<sup>31</sup> The Stata output of coefficient and t-statistics are copied and pasted into our Table 7. The (centered) R-squared of this regression is -7.26. RM once again ignore this evidence of serious issues with the estimates, and proceed to report parameters, t-ratios, and Kleibergen-Paap underidentification and weak identification tests.

The likely source of the clearly unidentified linear LIML models that RM present in their Table 6 is the extraordinarily artificial nature of the matrix of exogenous variables that they construct for the first-stage equations. To see why this matters, consider the nature of the linear LIML estimator. Leaving aside both sample weights and clustering so as to leave the notation uncluttered, the κ-class estimator of the vector of second-stage parameters **b** = [ **β** δ] is where **Z** is the full set of exogenous regressors in the first-stage, **X** is the set of included exogenous regressors **X1** plus the endogenous credit variables **C** , **X** = [ **X1 C** ], in the secondstage, and **MZ** = **I – Z(Z ́Z)**<sup>**-1**</sup> **Z ́** . The linear LIML estimator sets κ equal to the minimum eigenvalue of **(C ́MZC)**<sup>**-1/2**</sup> **C ́MX1C ́MX1C(C ́MZC)**<sup>**-1/2**</sup> , where **Mx1 = I – X1(X1 ́X1)**<sup>**-1**</sup> **X1 ́** . Exactly what this minimum eigenvalue represents in the RM formulation of the PK model is unclear because (i) the X and Z matrices have been by swelled by a factor of 58; (ii) most of the observations for **C** are deterministically zero but not treated as such; and (iii) 102 exogenous variables are dropped by Stata because of perfect collinearity. Which variables are kept by Stata and which are dropped is likely to be arbitrary, at least from point of view of statistical theory, so that dropping some columns from **X1** rather than from **X2** ( **Z = [X1 X2]** ), the matrix of excluded exogenous variables, may arbitrarily alter the mininum eigenvalue.

> 31 The only exception to this invariance of the t-ratios is for a survey round dummy variable with a t-ratio of 6.2.

22
Even if the minimum eigenvalue used by RM in computing the value of κ lacks the statistical meaning to make it the linear LIML estimator, it can be argued that it does not matter much since the RM estimates can be interpreted as resulting from a _κ-class_ estimator as in Nagar (1959) and others. Many methods of choosing κ have been presented in the literature. As the discussion above makes clear, linear LIML and other κ-class estimators are not the result of iterating on a log-likelihood function until convergence, but rather the result of an algebraic expression with an analytic solution, including the investigators’ choice of κ. However, RM’s assertion that “LIML is more robust to weak instruments” may not hold for choices of κ that are not LIML.

The LIML (κ-class) covariance matrix is computed as where s<sup>2</sup> is the estimated mean squared error, and differs from the standard two-stage least covariance matrix by the eigenvalue κ. However, the usual 2SLS covariance matrix, which sets κ=1, is still valid. In his popular econometrics textbook, Greene (2012, p. 567) notes that "A useful result is that the asymptotic variance of the two-stage least squares (2SLS) estimator, which is yet simpler to compute, is the same as that of the LIML estimator." Imbens (2010, p.17) adds that “Under the standard, fixed number of instrument asymptotics, the asymptotic variance for LIML is identical to that for TSLS, and so in principle we can use the same [2SLS variance] estimator.”<sup>32</sup> What happens if we compute t-ratios for the RM linear LIML model of RM’s Table 6, column (2), in which there are separate gender effects but no disaggregation by credit group, using the perfectly valid 2SLS parameter covariance? Certainly, this is an appropriate exercise because of the problem of ill-conditioned covariance and data matrices, and the large and negative R-squareds. The statistically valid 2SLS t-ratios presented in our Table 7, when combined with Roodman and Morduch’s own parameter estimates, suggest that there are very significant and positive female credit effects (t=10.63) and very small and insignificant male credit effects (t=0.05). On the basis of this 2SLS regression estimated by Roodman and Morduch, one might conclude that the problem with Pitt and Khandker (1998) is that they <u>underestimate</u> the positive effect of women’s credit on household consumption. The point estimate of the female credit effect found by RM is vastly larger than that found by PK -- larger

> 32 These two references are added to refute the contention of Roodman that the 2SLS covariance matrix (κ=1) is inappropriate in this case, a view he stated in an emailed response to an early draft of this paper.  He argues that if the 2SLS covariance matrix was valid for linear LIML, which he (wrongly) doubts, then one should also able to use the OLS covariance matrix since that is also a κ-class estimator (with κ = 0).  The error in his thinking is (1) the asymptotic variance for 2SLS is identical to that for linear LIML, and (2) for LIML, κ is always greater than 1 in finite samples, so that the κ=0 requirement for OLS is not feasible, and κ-class estimators are consistent whenever κ tends to 1 asymptotically at a rate faster than one over the square root of the sample size.

23
by a factor of 10. Of course, as we have shown above, the RM estimates are not interpretable, as they do not correspond either to the PK model or to any formal econometric model of program evaluation that RM or anyone else has ever set out, not to mention that the linear LIML estimation strategy RM adopt just makes the estimates even more strange.

Can one show that, in general, the linear LIML does not “work” in the PK model as RM have formulated it, or is it simply a reflection of some defect in the PK data? In our Table 8 below, we reproduce the simulation results of our Table 1 (above) but add the linear LIML estimates to the set of estimates reported. These are exactly the same simulation datasets described earlier. Recall that the errors are normally distributed and the instruments are not “weak” in these simulations. The linear LIML estimates of column (4) of our Table 8, the only new column in this table as compared to Table 1, have clearly bizarre point estimates (means of 200 simulations are presented) and huge standard deviations. It is no surprise that RM get outlandish parameter values and large standard errors in their Table 6 when they estimate the PK model with their linear LIML setup. Our Table 9 reports the estimates from one of the simulation datasets. The model estimates an effect of female credit from group 1 to be -244.388 when the true value is 0.75, and the male credit effect of group 3 to be 177.334 when the true value is 0.25. This should not be surprising given the large standard deviations reported in column (4) of our Table 9. Almost anything can result when an erroneous econometric model is applied to an artificially altered data matrix.

# **7.  Sources of identification in Pitt and Khandker:  Some new results**

In this section, we further examine two aspects of our instrumental variable approach that have been attacked by RM. The first is the validity of the exclusion restriction underlying their use of interactions between program choice and the set of exogenous variables (including the village fixed effects) as instruments. The second is the application of the “one-half acre” program eligibility rule. We will show that identification does not require both of these, and present new results dropping each assumption in turn.

The PK model is identified even after dropping all of the exogenous variables including the village fixed effects from the first-stage equations given by equation (4). All that is required is a constant term. Consequently, the identifying instrument is whether or not a household had choice to join a female credit program and/or a male credit program. Estimating a model with only a constant term in the first-stage is of interest because it suggests the extent to which the set of exogenous variables X (and village fixed effects), which have been subject to pointed attack by RM, actually drive the results. Note that even if program placement is nonrandom - that is, even if the program availability is correlated with the village effects - dropping village fixed effects from the first-stage does not cause bias in the second-stage as long as the second-stage equation includes the village fixed effects, as it does in PK.

24
Column (1) of our Table 10 presents the baseline PK estimates as re-estimated with Roodman’s _cmp_ program and presented in Pitt (2011a), Table 1, column (3).<sup>33</sup> Column (2) of the table, labeled “No instruments,”<sup>34</sup> presents new estimates dropping all of the slope variables X and all of the village fixed effects from each first-stage equation (4) and (5), leaving only a constant term.<sup>35</sup> What is striking about a comparison of column (2) with column (1) is how little the parameter estimates have changed. Leaving out instruments that RM believe to be troublesome alters the points estimates hardly at all, improves the precision of the estimates, and leaves fully intact the PK conclusion that microcredit provided women positively affects household consumption while microcredit provided men has no statistically discernible effect.<sup>36</sup> In column (2) of our Table 10 we eliminated exclusion restrictions by dropping all of the exogenous variables, X, plus the village fixed effects (leaving only a constant term) from the first-stage. The polar opposite method to eliminate exclusion restrictions is to retain all of these variables in the first-stage equations and just add them to the second-stage equation. To implement this method, we include interactions between the dummy variable indicating a nontarget (ineligible) household with all of the slope variables X plus _thana_ (sub-district) fixed effects in column (3) of Table 10, and, alternatively, we include these variables separately interacted with male and female choice in column (4) of Table 10.<sup>37</sup> Once again, these estimates, labeled “No exclusion restrictions” in Table 10,<sup>38</sup> demonstrate the robustness of the PK results to the exclusion restrictions that RM believe are not credible.<sup>39</sup> To demonstrate that identification in the “no instruments” and “no exclusion restrictions” case does not depend on the Tobit nature of the first-stage equations, we re-estimate these models replacing the Tobit part of the pseudo-log-likelihood function in the first-stage with linear OLS. These estimates are presented in our Table 11. The first column of Table 11 just reproduces the “baseline” PK results of Table 10 and Pitt (2011a). Column (2) of the our Table 11 replaces Tobit with linear OLS with only a slight loss of precision as compared to the model

> 33 To remain consistent with the estimates of Pitt (2011a) and (2011b), we continue to use all three rounds of credit data in estimating the first-stage equation.

> 34 “No Instruments” means that the first-stage equations ((3) and (4)) have no slope terms.

> 35 This looks very much like a Wald estimator as there is a single binary instrument corresponding to each endogenous variable.  Because there are two correlated endogenous variables, the credit effect parameter is not given by the ratio of the difference in the means of the dependent variable over the difference in the means of the endogenous variable for each sub-sample as it is in the single endogenous variable case.

> 36 The log-likelihood function in the “no instruments” case is bimodal.

> 37 There are three villages in a _thana_ (sub-district) in the sample frame.  Using all villages interacted with both female and male choice yields very similar estimates.

> 38 “No exclusion restrictions” means that the second-stage equation contains the full set of slope interactions.

> 39 RM have done something similar in their Table 1 of RM (2011).  They present two sets of estimates, labeled “ _De facto_ ” and “ _De jure_ ” each in which just the choice dummy variables and then all of the instruments are included, in turn, into the second stage.  RM suggest that it is the nonlinearity of the Tobit instrumenting equations that permits model identification in this case.  They are mistaken.  The model is identified even if the first-stage instrumenting equations are linear OLS.  More importantly, even RM report that women’s credit effects are positive and significant under both the _de facto_ and _de jure_ rule when there are “no exclusion restrictions” in their Table 1, and that men’s effects are insignificantly different from zero.

25
with Tobit first-stage equation in column (1). Analogously, column (3) of our Table 11 reproduces the “no instruments” estimates of Table 10 column (2) to enable comparison with the linear OLS estimates of column (4). Once again, the results support the original findings of a statistically significant and positive female credit effect, and a statistically insignificant male credit effect.

Just as we have been able to re-estimate the PK model after dropping all of the instrumental variables consisting of the interactions of choice with all of the exogenous variables (X) and fixed effects, we can also re-estimate the PK model without treating **any** households as having deterministically zero credit as a consequence of the “half-acre” eligibility requirement. That is, we can eliminate the eligibility rule completely and treat all households as endogenously at risk for choosing treatment from a credit program if one exists in their village. As long as the vector of exogenous variables X in equations (4) and (5) have any variables at all beyond a constant, we do not need ineligible (non-target) households in the sample to identify program effects. Identification comes off of variation in program availability by gender across villages.<sup>40</sup> Contrary to the claims of RM, this eligibility rule and how it may have been enforced is not, in principle, required for estimating program effects.<sup>41</sup> However, if PK in their 1998 paper had treated some households as deterministically without choice when those households actually self-selected themselves into or out of credit programs, the PK estimates would be in error. This very issue was addressed at length in Pitt’s 1999 response (Pitt, 1999) to Morduch’s first replication, but seems to have been overlooked by RM in their discussion. It is worthwhile to reiterate that the eligibility rule set by the Grameen Bank is **not** that a household must own less than one-half acre of land of any type or quality to be eligible.<sup>42</sup> In that 1999 response paper, Pitt treats nonprogram households with somewhat more than 1⁄2 acre of land and as if they have 40 In the simplest illustrative case, note that βx in equation (9) can be estimated from the sample of villages without choice. One could then imagine replacing the dependent variable y in equation (9) with = y – X x<sup>, where</sup> x<sup>is</sup> the estimate of βx obtained from villages without program choice. This transformation of y into makes all of the exogenous variables X (except village fixed effects) available as instruments in an instrumental variables regression of on female and male credit.

> 41 ‐ They claim that “in light of the pervasive non enforcement of the rule evident in Figure 2, the eligibility dummy as defined by PK, and thus the key instruments...should be presumed endogenous (RM (2011, p. 21)).

42 According to Mahabub Hossain (1988), the most authoritative source of the time, the rule is that “a person from a household that owns less than 0.5 acre of cultivated land, or assets with a value equivalent to less than 1.0 acre of medium-quality land, is eligible to receive a loan” (Hossain 1988, p.25). Note that the rule about landownership applies (i) only to cultivated land, an important distinction among the land poor whose homestead (uncultivated) land may be a significant share of total landholding; and (ii) allows for eligibility when the quantity of cultivable land **exceed 0.5 acres** as long as the value of all assets is less than 1.0 acres of medium quality cultivable land. As Pitt (1999) makes clear in an econometric analysis of land values and program participation with these data, participating households owning more than 1⁄2 acre of total land at the time of the survey have dramatically lower land values even after conditioning on total land area and its square, participant status and thana fixed effects. Conditional on the other regressors, “mistargeted” households have unit land values of about one-half that of households which are not “mistargeted”, suggesting that they likely fit under the actual _de jure_ eligibility rule. Moreover, the other two microcredit programs examined (BRAC and BRDB) report that they followed the Grameen Bank land and asset eligibility rule in the years prior to our survey. The Grameen Bank, as the first microcredit program in Bangladesh, was the role model for subsequent programs initiated later in the 1980’s.

26
choice. By raising the eligibility cutoff to land ownership greater than 1⁄2 acre, say 1 acre, all households, whether program participants or not, are treated as having the choice to participate. In this way, those who really do have choice but choose not to participate are now treated appropriately. On the other hand, some of the households that are not program participants with land below 1 acre are actually ineligible. Pitt (1999) claimed that this classification error does not alter the consistency of the estimates of program effects since treating a behavior as endogenous when it is in fact exogenous still yields consistent estimates. However, he could have made an even stronger claim – consistent estimates of credit program effects are possible even when every household, no matter their land or wealth status, is treated as having endogenous treatment choice.

Estimates of program effects in which every household in a treatment village is considered eligible are presented in column (5) of our Table 10.<sup>43</sup> A comparison with the “baseline” estimates of the first column of that table reveal that the qualitative results are essentially unaffected – female credit effects are statistically significant and positive and male credit effects are not statistically different from zero. The estimated female credit effects are about 25 percent larger when no households are considered ineligible and the precision of the estimates is greater. The last column of our Table 10 reproduces results reported in Pitt (1999) Table 4, last column. They are almost identical and likely suggests that the relatively few households with more than 2.0 hectares of land in the sample are not the primary source of parameter identification, but rather it is variation in program placement across villages that drives both these estimates and those in the previous column (column (5) of Table 10) in which there are no ineligible households. What is striking in this exercise is not only that the PK results hold up so well, but that the effect of including ineligible households in the sample is to slightly reduce the estimated credit effects, and the effect of dropping exclusion restrictions is to slightly improve the precision of the estimates.

Finally, we add one more robustness check to these new estimates. When sources of parameter identification used in PK are dropped, as they are in the new estimates described above, there may be enhanced concern that an insufficient set of exogenous covariates in the second-stage equation underlies the credit program effects that are estimated. This is particularly acute in the case of “no instruments” where the choice dummies are key components of identification. The issue of the validity of the instruments was a theme in Morduch’s 1998 paper. In particular, Morduch is concerned with the interactions of land with all of the other exogenous regressors because there may be “systematic differences between the landless and landed in, say, the impact of age on income.” In Pitt’s 1999 response to Morduch, he addressed that concern by re-estimating the model with these interactions. Table 6 of Pitt (1999) presents estimates of the effects of program credit, by program and gender, on the log of household per capita expenditure, allowing for interactions between land ownership and all of the exogenous

> 43 As every household is considered eligible, there is no longer a dummy variable indicating target/non-target status in the second-stage equation.

27
regressors and interactions between land ownership and all of the thana fixed effects. All 18 exogenous regressors and the 29 thana dummy variables are interacted with land and are included in the consumption equation. These interactions are the ones most likely to destroy identification if in fact it is the linearity of the consumption function that is driving the identification of credit effects. Based on these estimates, which allowed for interactions with landholding, Pitt (1999) concluded that the “bottom line” qualitative results of PK still hold -- there are positive and statistically significant effects of female credit program participation on household consumption, and much smaller and generally statistically insignificant effects of male credit program participation on household consumption, as in PK. The estimates first presented in Pitt (1999) were presented again in Pitt (2011b), only re-estimated with Roodman’s _cmp_ command. They are reproduced here in column (1) of our Table 12. The “headline results” still hold. Women’s credit has statistically significant and positive effects on household consumption, although these effects are slightly attenuated with the addition of 47 additional control variables.

The question now is whether the addition of the 47 additional independent variables arising from interaction with landholdings will substantially affect the new results in which, in turn, all “instruments” are dropped and all households are considered eligible. The results from this more highly parameterized second-stage equation are presented in columns (3) and (4) of our Table 12. These estimates do differ very little from their counterpart in Table 10 (compare Table 12 column (3) with Table 10 column (2), and Table 12 column (4) with Table 10 column (5)). Once again, we find statistically significant and positive female credit effects and statistically insignificant male credit effects.

In brief, dropping each of the two major sources of identification in turn does nothing to weaken the effects reported in PK. Nor does combining these new estimates with OLS firststage equation rather than Tobits, and heavily parameterizing the second-stage with interactions with landownership. The PK results are strikingly robust.

# **8.  Conclusions**

“The Impact of Microcredit on the Poor in Bangladesh: Revisiting the Evidence,” by David Roodman and Jonathan Morduch, is the most recent of a sequence of papers and postings that seeks to refute the findings of Pitt and Khandker 1998. We find that RM have used testing procedures that necessarily tend to bias the findings in the direction of rejecting the results of PK. A key fault of RM is that it ignores the deterministic nature of choice for a large share of observations. Credit is deterministically zero for a sizable part of the sample and this characteristic is the salient difference between the PK model and the classical two-stage least squares problem on which they focus. RM also make incorrect claims about the causes and implications of bimodality and non-normal errors. Their preferred linear LIML estimates are vastly different from standard 2SLS estimates and are not identified as demonstrated by the 105 out of 110 coefficients reported by Stata to have t-ratios of exactly 0.01 in absolute value.

28
Nonetheless, they report and rely on the test statistics derived from this obviously faulty IV regression. Finally, we further examine two aspects of our instrumental variable approach that have been attacked by RM. These are the validity of the exclusion restrictions underlying the use of interactions as instruments and the importance of the “one-half acre” program eligibility rule. We find that identification does not require both of these. Indeed, the results originally reported in the PK paper hold up extremely well in this new analysis.

The final question is the value of continuing this process of “replication” and rebuttal, a process that can be dated back to Morduch’s (1998) replication paper distributed months before the PK paper was first published. In Section 2 of this paper, we briefly reviewed the recent history of this replication/rebuttal sequence. In this paper, we make the same points about the two-stage setup used by RM that we made in 2011 and in 1999, and we cannot imagine that there is anything to be gained by restating them yet again in the future. Upon discovering errors in their work, RM have always come up with new lines of attack but have never got their econometrics and the 2SLS setup right. Perhaps one reason for the continuation of this process consisting of RM’s triumphal announcement of finding devastating problems with PK, followed by rebuttal by PK, is that RM do not believe in non-randomized studies, deeming them “unreliable” and claiming that their complexity hides a form of “obscurantism.”<sup>44</sup> We note that others have also attempted to replicate PK. Duvendack and Palmer-Jones (2012) have published a paper that is both a “replication” of PK and a replication of an earlier replication of PK by Chemin (2008) in the _Journal of Development Studies_ . Pitt’s reply paper (2012), published in the same issue, lists many serious errors in their code and misrepresentations of method in their paper, and concludes that the Duvendack and Palmer-Jones results provide no credible evidence on the validity of PK or Chemin or on the effectiveness of

> 44 On this, Roodman has written:

My main conclusion about non-randomized quantitative studies is that they are obscurantism, however unintentional, and what their complexity hides, ultimately, is a failure to prove the assumptions needed to demonstrate cause and effect. Warren Buffett’s investment rule—don’t buy what you don’t understand— works in microfinance research too. RCTs are easy to understand, so you should buy them more than nonrandomized studies. _<u>(http://blogs.cgdev.org/open_book/2011/06/rcts-are-people-too.php)</u>_ and The confusing math [referring to some equations describing the model of Pitt and Khandker], _nota bene_ , is from a non-randomized study. The math for randomized control trials (RCTs) could hardly be simpler. You flip a coin a bunch of times to decide who is offered a financial service and who isn’t. Then you come back later, and for each of the two groups—those offered and those not—you compute the average outcome (are you still with me?). Then you subtract one average from the other. Got that? Then you’re done. Even my third grader can understand that math. Well, my third grader derived the formula for the area of a regular octagon...but his classmates can understand RCT math too. So let’s not taint the method with obscurantism. _(http://blogs.cgdev.org/open_book/2011/06/rcts-are-people-too.php)_ 29
microfinance. While we doubt that either side will change its mind, we believe that it is best for all concerned to move on to new research endeavors.

30