factors—including the circumstances of participants, the cultural and political environment, and the administrative context—were not taken into account. For example, a recent evaluation by DECRG of a large World Bank lending operation found low overall impact but also found considerable heterogeneity, with some types of households benefiting more than others. The policy implication is that choosing different beneficiaries would have greatly increased the project’s overall impact. Furthermore, not all sources of heterogeneity are observable, and participants and stakeholders often react to factors unobserved by the researcher, confounding efforts to identify true impacts using standard methods.

Evaluative research must be feasible in the contexts in which the policies and programs are introduced. Since we are typically dealing with the policies of governments, the ethics and political economy of feasibility inevitably constrain research options. For example, in evaluating public safety net programs it is sometimes impossible to set up a baseline survey, since one cannot reasonably delay the program for such a purpose; data collection methods must adapt accordingly, and this can be costly. There is also a potential, but under-explored, role for “evaluative monitoring” of development projects, whereby the tools of project monitoring are modified or extended to allow more rapid feedback to operational staff on the impacts of the program.

# Methodological Research

The second type of cross-cutting research that has increasingly occupied Bank research in the past decade can be termed methodological research. Practitioners carry a set of tools to policies and projects, including data, measures and models. Researchers have played a useful role assessing those tools and helping improve them. They have assessed whether data are sound. They have also assessed whether the measures have a clear theoretical foundation and are the most relevant ones for specific policy problems (i.e., whether the measures used are adapted to the objectives and constraints relevant to each policy problem—which is not as common as one might think). Finally they have also examined whether the assumptions made in modeling are both realistic and internally consistent.

Methodological research can help expand the tool kit routinely employed by policymakers and analysts. This includes the data collected and the methods used to analyze those data, including software. The Bank has become a major producer of development data, and researchers have played a crucial role. A number of the Bank’s most successful data initiatives started as research projects. The best example is probably the Living Standards Measurement Study, which started as a research project in the mid-1980s with the aim of greatly improving household survey data in developing countries. The LSMS database houses over 75 survey datasets with all the relevant documentation. The more recent research projects are designed to improve data collection methods and develop suitable welfare measures to assess poverty.

Researchers have also added a lot of value to the Bank’s operational work by developing useful analytic tools in real applications, often working with colleagues in Operations. For example, DECRG has developed software programs that have facilitated data analysis in client countries, such as the widely used Povcal. Recently-developed software to estimate poverty maps combines household survey data and population census data, a procedure so computationally intensive that standard statistical software packages such as STATA, SAS or SPSS often become unwieldy and sometimes even nonfunctional.

31