metadata: authors: 0: author: Mcmahon,Gary Joseph Raymond
docna: 0: docna: Applying economic analysis to technical
            assistance projects
docty: Policy Research Working Paper
subtopic: Adaptation to Climate Change,Health Care Services Industry,State Owned Enterprise Reform,Public Sector Administrative & Civil
            Service Reform,Educational Sciences,Privatization,Energy Privatization,Democratic Government,Economics and Finance of Public
            Institution Development,Public Sector Administrative and Civil
            Service Reform,De Facto Governments
teratopic: Poverty Reduction
abstracts: cdata!: The author recommends using more
            quantitative economic analysis in appraising technical
            assistance loans and loan components. After giving a brief
            history of tecnhical assistance and the problems commonly
            associated with it, he describes classifications of
            technical assistance, proposes a new typology to be used for
            project appraisal, suggests methods for screening projects,
            and discusses different levels of economic analysis. He
            shows how the typology and economic analysis could be
            applied to 40 projects in the Europe and Central Asia
            region, Although some of the suggested approaches are
            sophisticated and demanding, much of it relies on fairly
            simple techniques. The author estimates that roughly 60
            percent of the technical assistance in his sample was
            suitable for quantitative analysis. Using four case studies,
            he demonstrates how quantitative economic analysis could be
            used more frequently in the appraisal process. The author
            recommends that all technical assistance be classified
            according to the methodology suitable for its appraisal and
            evaluation. In cases in which little or no quantitative
            analysis is used, the staff officer can and should provide a
            justification -in which case it should still be possible to
            conceptualize the problem in economic terms. Essential
            technical assistance should be included when calculating the
            economic rate of return on a project. When technical
            assistance is excluded from this calculation, a strong
            justification for such assistance should be given or it
            should be omitted from the project. In his sample, the
            author estimates that about 80 percent of the technical
            assistance in investment projects was not essential for
            project implementation. In the 12 projects for which an
            economic rate of return was calculated in the appraisal, the
            technical assistance was not included in the estimate -even
            though 53 percent of this technical assistance was deemed
            necessary to the project.
repnme: repnme: Applying economic analysis to technical
            assistance projects
display_title: Applying economic analysis to
            technical assistance projects
listing_relative_url: /research/1997/04/442320/applying-economic-analysis-technical-assistance-projects
topicv3: Education, Skills Development and Labor
            Market m1326135 1484,Education m1327048 470,Privatization m1327236 1283,Climate Change Adaptation m1327767 574,Health m1504661 3480
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Poverty Reduction
origu: Public Economics Division-575
no_of_pages: 46
author_keys: 0: author_key: 123243
model_extractions: PPOLICY RESEARCH WORKING PAPER 1749

# **Summary** findings

McMahon recommends using more quantitative economic analysis in appraising technical assistance loans and loan components.

After giving a brief history of technical assistance and the problems commonly associated with it, he describes classifications of technical assistance, proposes a new typology to be used for project appraisal, suggests methods for screening projects, and discusses different levels of economic analysis. He shows how the typology and economic analysis could be applied to 40 projects in the Europe and Central Asia region.

Although some of the suggested approaches are sophisticated and demanding, much of it relies on fairly simple techniques. McMahon estimates that roughly 60 percent of the technical assistance in his sample was suitable for quantitative analysis.

Using four case studies, he demonstrates how quantitative economic analysis could be used more frequently in the appraisal process.

McMahon recommends that all technical assistance be classified according to the methodology suitable for its appraisal and evaluation. In cases in which little or no quantitative analysis is used, the staff officer can and should provide a justification - in which case it should still be possible to conceptualize the problem in economic terms. Essential technical assistance should be included when calculating the economic rate of return on a project. When technical assistance is exciuded from this calculation, a strong justification for such assistance should be given or it should be omitted from the project.

In his sample McMahon estimates that about 80 percent of the technical assistance in investment projects was not essential for project implementation. In the 12 projects for which an economic rate of return was calculated in the appraisal, the technical assistance was not included in the estimate - even though 53 percent of this technical assistance was deemed necessary to the project.

- - This paper a product of the Public Economics Division, Policy Research Department is part of a larger effort in the department to study the effectiveness of foreign aid. Copies of the paper are available free from the World Bank, 1818 H Street NW, Washington, DC 20433. Please contact Cynthia Bernardo, room N10-055, telephone 202-473-7699, fax 202S22-11S4, Internet address prdpe@worldbank.org. April 1997. (40 pages)

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should he cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the author. They do not necessarily represent the viewof the World Bank, its Executive Directors, or the countries they represent._ Produced by rhe Policy Research Dissemination Center
# **APPLYING ECONOMIC ANALYSIS TO**

# **TECHNICAL ASSISTANCE PROJECTS***

Gary McMahon Consultant Policy Research Department World Bank Washington, DC The author would like to thank Martha De Melo, Shanta Devaraj an, David Dollar, and the participants of the Public Economics workshop for their comments.
# **APPLYING ECONOMIC ANALYSIS TO**

# **TECHNICAL ASSISTANCE PROJECTS**

# Gary McMahon

# **Introduction**

The primary objective of this report is to explore the scope and make concrete recommendations for applying more economic analysis, especially of a quantitative nature, in the a priori evaluation (or appraisal) of technical assistance loans and the technical assistance component of other loans. To meet this objective we begin with a brief history of technical assistance, focusing on the problems that it has encountered and the World Bank response to these difficulties. In Part II of the document we describe different classifications of technical assistance and propose an alternative typology which could be used for project appraisal. A range of different methodologies and suggestions for screening projects are also included in this section. In Part III we look at 40 projects from the ECA region, discuss the methodologies actually used in their appraisals, and see how they fit in the proposed typology. Four of these projects are then analyzed in greater depth and recommendations are made on different methodologies which could have been used in the appraisal process. Part IV contains the conclusions and recommendations. While the examples and specific recommendations mostly concern the World Bank, we believe that the observations and insights may be relevant and useful for all agencies which fund and provide technical assistance.

# **Part** I: Technical Assistance: A **Brief History'**

This section begins with a discussion of the varying definitions of technical assistance and what it actually includes. This is followed by a presentation of the amounts of TA given by donor, recipient countries, regions, activities, and number of personnel involved in delivery. Growth paths and trends in TA are discussed. The third part of this section reviews the major benefits and critiques of TA. Finally, the section ends with a description of the guidelines used for selecting and designing TA in the World Bank, with similar, albeit briefer, descriptions of the process in GTZ and USAID given in Appendix II.

# I(a): Definitions

1 The reader who is quite familiar with the literature on technical assistance may want to go directly to Part **II.**
"Technical assistance is defined as the transfer or adaptation of ideas, knowledge, practices, technologies, or skills to foster economic development. The purposes of Bank technical assistance are classified as follows: (a) Policy development, (b) Institutional development, (c) Capacity building, and (d) Project or programme support." This definition for technical assistance from the World Bank (1991 a) report of the technical assistance review task force is similar, if somewhat more compact, to definitions given by other agencies with one qualification. Many agencies call the (a), (b), and (c) above technical cooperation and only include (d) (or a variation thereof) under the rubric technical assistance. That is, they distinguish between activities whose main contribution is to design or implement a given project or programme and those which are primarily trying to increase the level of knowledge, skills, technological comprehension, or productive aptitudes of residents of a developing country. In this report we shall use the all-encompassing World Bank definition of technical assistance - also used by AID and CIDA - unless otherwise noted.<sup>2</sup> With regards to types of TA, the most commonly used distinction is between freestanding TA and project related TA. Project or investment related TA is meant to support the execution of a capital project. Free-standing TA is anything else, although it is usually related to policy reform, institutional development, or capacity building. Unfortunately, many donors do not statistically differentiate the two, so it is difficult to determnine what portion of TA falls under each broad category.<sup>3</sup> The question of what is actually counted as TA, versus capital assistance, is a more difficult one. In general, however, the following rules of thumb apply. Large capital projects and feasibility studies are not included. However, specific training components attached to large capital projects are TA. On the other hand, small capital projects associated with a free-standing TA, such as renovating a building or supplying computers, are generally included in TA. Unfortunately, some agencies do not make any statistical distinction between TA and capital assistance and others just refer to the financing of personnel (or "experts") as TA and anything else as capital assistance.<sup>4</sup> Nevertheless, our analysis in section III will focus on projects for which the above distinctions are made.

# I(b): Trends in Technical Assistance

While we have some data on technical assistance provided by the World Bank, we only have detailed data on technical assistance in the form of grants. Total current dollar TA grants 2 Berg (1993: 42-47) goes into considerable detail on the definitions of TA and TC used by different agencies.

3 A differentiation is often made between "soft" and "hard" TA, where the latter is for specific investment needs and the former for more general purposes, including institutional development. Most free-standing TA is of the soft variety, while TA attached to projects may be of either sort.

4 Berg (1993: 48-52) discusses the distinction between capital assistance and technical assistance in detail.

2
provided to developing countries, excluding the forner communist countries (or current transition countries), was $US 1.0b in 1970, $US 4.5b in 1980, $US 9.5b in 1990, and $US 11.3b in 1993.5 As a percentage of total ODA, these amounts were 23 percent, 27 percent, 29 percent, and 39 percent, respectively. As these amounts do not include loans for TA, they understate the total share of TA in ODA.

Substantial TA from OECD countries to the fornerly socialist transition countries only began in 1991. If we include these amounts, the total grant TA for 1993 is $US 13.6b versus $US 11.3b if excluded. Therefore, 17 percent of TA went to transition economies in 1993 from less than 3 percent only 4 years earlier. African countries have taken the lion's share of TA, with about two-thirds of this portion going to Sub-Saharan Africa. While Africa's share has generally been greater than 50 percent, the 'rise' of transition countries has brought this amount closer to 40 percent in recent years. While we expected to find TA being increasingly dominated by African and transition countries, this had not happened by 1994, with their combined share being similar to Africa's percentage of TA in the 1980s.

At the beginning of 1996 the World Bank portfolio had 136 free-standing TA loans, with an average size of $US 20m, and TA components in 1300 other projects which ranged from $US 200,000 to $US 50m. Another $US 450m of TA also goes through grants and trust funds each year. Note that 41 percent of TA component lending in 1995 went to only 5 countries: Argentina, Russia, China, Mexico, and India. In the fiscal year 1994-95 there were $US 812m of new free-standing TA loans.<sup>6</sup>

# I(c): Benefits and Critiques of Technical Assistance: A Review

Technical assistance can be used to fill gaps for specific knowledge in development projects, assist in policy reform, and, most importantly, develop local capacity to undertake tasks by themselves either through institutional development or capacity building. While gap-filling TA was once quite common, the ability to foster self-reliance is now considered the main goal of TA. In the context of TA, Cassen (1994: 165) specifies 3 dimensions of self-reliance: (i) to determine knowledge needs that cannot adequately be met domestically, identify where in other countries such needs may be met, know how to acquire this knowledge, and know how to adapt and use it at home; (ii) the ability to undertake domestic research, problem-solving and policy formation; and (iii) the ability to sustain these capacities, which involves the institutional capacities for training successive generations of scientists, technicians, and managers.<sup>7</sup> S Berg (1993: 73) calculates that real TA grants to Sub-Saharan Africa grew by 44 from 1975 to 1989. Given that the African share of TA did not change substantially over that period in our larger, undeflated data set, this figure can be used as a rough approximation of the total real growth of TA in the period.

6 These figures come from Operations Evaluation Department (1996).

7 Note that in the original 1986 edition of his book, the chapter on technical cooperation was written by Robert Muscat. This chapter is substantially unchanged in the 1994 edition.

3
However, despite these praiseworthy goals, technical assistance in virtually all donor or lender institutions has come under a barrage of criticism in the last 15 years from donors, recipients, and independent observers alike. Perhaps the most distressing aspect of the criticism is that general evaluations of technical assistance written in 1995 read much the same as evaluations written in 1983.8 While it would take a book (or at least a thick monograph) to address this problem in full, we will go through some of the most consistent criticisms of technical assistance.<sup>9</sup> The first group of problems of TA comes from the fact that it is often donor (or supply) driven. For various reasons, discussed below, the donor agency wants to provide free-standing TA or add on a TA component to an investment project. It is usually but not always (especially from the World Bank) free, and the recipients take it for that reason or simply because if they want the investment component, they must take it. In addition, free-standing TA often includes substantial equipment support, so the reluctant recipient takes the expert services out of need or desire for the material goods.

There are many potential problems with donor driven TA but by far the most important is that there may be little recipient commitment. Such commitment is particularly essential in the case of institutional development, as otherwise when the foreign experts leave there will be little or no follow-up and the probability of long-term impact is virtually nil. All evaluations of TA concur on the fact that borrower ownership is a prerequisite for success.

The reasons for donor driven TA vary from the well-meaning to the suspect. Often it is realized that without significant institutional development, the probability of success of other initiatives, donor driven or otherwise, is very small. At the other extreme, the TA may be based on the wish to provide work for national consultants, who in turn put pressure on their own govermments to provide this type of foreign aid. In project related TA there may be a wish to keep closer control over the investment or an inability to locate competent local personnel.

However, the criticisms of donor driven TA are more likely the result of the inadequacy of much of the TA rather than the underlying motives. Its frequent failure begins with poor identification and design of the project, a problem which has increased as TA has become "softer" on average. This problem begins with the development of projects without either an institutional development strategy or sectoral analysis to act as guides. A recurring criticism of Bank TA (which would also apply to most other donor agencies) is the lack of expertise on

> 8 A review of a progression of World Bank documents evaluating technical assistance can leave the reader with a feeling of being suspended in time. See, for example, Lethem and Cooper (1983), WAN Task Force (1986), AFTPS (1989), and Jayarajah and Branson (1995).

> 9 The following is based largely, but not exclusively, on evaluations of World Bank technical assistance. However, the materials that we have read on technical assistance of other agencies suggests that the experience of the Bank is quite generalizable.

4
institutional issues within the Bank.<sup>1 0</sup> Consequently, objectives are often unclear, terms of reference for consultants are vague, feasibility is questionable, and follow-up under implementation is half-hearted at best. The capacity of the recipient to undertake the project, or more importantly, to follow-up once the experts have left, is often addressed in a cursory manner. Once underway the lack of proper supervision and/or monitoring is a common complaint in all reviews of Bank TA that we have seen. It is likely that, as in most aid agencies, career development in the Bank is more closely linked to what gets off the ground rather than how it flies later, by which time it is often someone else's responsibility.<sup>11</sup> To counteract this poor incentive structure, in the case of TA loans Klitgaard (1996) argues in favor of a shared conditionality, where repayment depends on the success of the project.

Two criticisms in particular stand out in the discussion of the low success rate of TA. First, inadequate attention is paid to training. Consultants are rarely chosen for their pedagogical skills and usually are much more concerned with fulfilling their technical obligations rather than training successors. In addition, the persons chosen to be trained are often the wrong ones. This is particularly the case in the expert-counterpart model, where the latter is someone from low down in the ranks or outside of the organization who will have no decision making responsibilities once the expert leaves. Second, and related to the first, the short-term goals of operational support are usually given precedence over the long-term goals of institutional development. In fact, the report by the Operations Evaluation Department (1989) of the Bank argues that TA for institutional development should not be attached to SALs or SECALs as they operate on different time horizons and the former will surely suffer under the pressure to implement the latter.

Of course, there are also numerous problems or obstacles on the recipient side. Even if the will is there, the capacity is often lacking. And if the capacity - in the sense of human capital - is there, the institutional structure often is not. Of particular importance is the low motivation of the civil service due to poor working conditions and salaries. The latter is partially remedied by salary supplements but these can create more problems than they solve due to the jealousies and friction that they can generate, not to mention corruption in selection procedures. General problems of a poorly functioning civil service are similarly "resolved" by the creation of project implementation units. However, the usefulness of these enclave like structures seem to rarely survive long once the foreign experts leave.

10 In his review of institutional development in World Bank projects, Paul (1990: 37) states: "The operational staff whose forte is sector technology and economics had an edge in diagnosing and advising on these [implementing] agency-related institutional problems. The wider institutional analysis and design associated with sectoral ID issues, however, are less often within their competence especially if they are relatively inexperienced. For example, the redesign of the management of ministry functions and inter-ministerial linkages, sector-wide identification of institutional altematives for service delivery, redistribution of authority and responsibility between levels of government, and the reorientation of the bureaucracy following deregulation involve different actors, interest groups and inter-organizational behavior problems which they may not have been exposed to in many cases. Thus, there is a clear mismatch between the nature of ID issues being addressed in recent projects and the skills and experience available in the country departments to deal with them." Berg (1993) stresses this point with regards to the Bank.

_5_
Recipients often look on TA as free, not fully understanding its true cost. Most TA grants require direct contributions of the recipient of office space and supplies, support staff, and some professional staff. In addition, there are often significant indirect costs associated with the time of senior management in negotiating and implementing TA projects. These can be very important if the government is the recipient of a large number of uncoordinated TA projects from many donors, as is often the case. In addition, usually more than half of the TA funds go to the salaries and benefits of foreign experts, and it is often questionable whether their contribution equals their costs. This is a significant problem for the recipient if the TA is given on a loan basis.<sup>12</sup> It also brings up the question of whether or not sufficient attention is being paid to the use (or development) of local consultants.

Some commentators - Mosley (1992), Berg (1993) - stress that a fundamental problem of TA is a lack of markets or two separate markets. Mosley (1992: 83) notes: "At present the market for technical assistance is characterised by an enormous gap between need and demand - that is, those who benefit from training or advice in developing countries are not, as a rule, those who pay for those services. In such an environment, the supplier is chasing two separate markets - the ultimate beneficiary who 'needs' the technical assistance and the sponsor who finances it. But neither of these parties, as we have seen, is supporting an evaluation effort of a type which will identify which TC suppliers are meeting a country's development needs in a cost-effective manner and which are not; and as a consequence of this absence of signals, the market for TC is characterised by extreme rigidity, with many training courses having been given for 20 years, at sponsors' request, in substantially the same form. Only if evaluation processes come to supply better information concerning where the critical skill gaps in a country are located and how they can most effectively be filled will aid resources in this area be deployed in anything more than a haphazard way." Similarly, in the case of a TA loan the ministry receiving the assistance is rarely the one that actually pays for it. Therefore, it is rarely refused, especially if much needed equipment is part of the package. Berg (1993: 181) proposes some type of system in which recipient ministries (and countries in the case of grants) are forced to pay part of the cost of TA. The more general point is that for TA to be more effective, it is likely that some type of market mechanism will have to be introduced so that both the donors and recipients will be more aware of its costs and benefits.

Finally, many of the problems of TA begin at the beginning. It is rare that a TA project or component is funded based on in-depth economic analysis. While there are many difficulties with the use of cost-benefit analysis (to be discussed in Part II of this report), it is rare that other procedures such as cost-effectiveness, counterfactual analysis, and alternative projects or project 12 It also can have a negative effect on the behavior of local civil servants who have to work with foreigners earning many multiples of their incomes, and who, moreover, disrupt their organizational structures.

6
design are seriously considered. Even the use of qualitative economic analysis is quite limited, and it is rare that a full conceptualization of the project in economic terms is undertaken.

# I(d). Guidelines for TA Selection and Design

We examined the guidelines for TA selection and design of three institutions primarily to see if there were any standard procedures on the use of quantitative analysis. While in general there was not a great deal, it is worthwhile to look in some detail at the guidelines of the World Bank to see how some of the criticisms in the last section are being met and where further improvements may be necessary. In Appendix II we have brief descriptions of the guidelines for two other institutions - USAID and the GTZ.

If the guidelines and procedures outlined in the Handbook on Technical Assistance of the Operations Policy Department of the World Bank were followed to the letter, many of the problems discussed in the last sub-section would be resolved. This Handbook contains a candid look at many of the difficulties of every aspect of TA and makes detailed recommendations as to how they can be avoided. When coupled with the "Performance Indicators for Technical Assistance Operations," also by the Operations Policy Department (1996), the Task Manager has a comprehensive set of procedures for greatly enhancing the probability of successful design and implementation of a TA project.'<sup>3</sup> However, at the risk of being considered unappreciative of the excellence of these documents, we would like to point out two problem areas. The first potential difficulty is the amount of staff time that strict (or near) adherence to these guidelines would entail. Only time and experience will answer that question but it does not seem impossible that it would double or triple the Bank staff time for a typical institutional development TA project. Then one must consider whether the better quality project justifies this extra time. More importantly, the incentive structure within the Bank must also reflect the reallocation of time towards more intense monitoring and fewer projects per staff officer. Alternatively, it could be argued that what is needed is to distill some rules of thumb from the guidelines which could quickly eliminate many projects. (For example, is there a serious recipient commitment to the project?) Second, while these documents contain a wealth of information on design and implementation, they do not give much advice on selection. That is, once the area of intervention is chosen, adherence to these guidelines and procedures should greatly increase the probability of success, but there is not much advice on how to determine which of several alternative projects should proceed or whether any project should proceed at all or which of several alternative ways of reaching given objectives should be followed. Such decisions are implicitly left to professional (qualitative) judgement.

13 Appendix L.A contains some of the main recommendations of the Handbook to minimnize the difficulties noted above.

7
The above comments aside, a judicious following of the guidelines in the Handbook would eliminate many alternative designs or even entire projects on the grounds of, for example, lack of borrower commitment or capacity. While one does not find an explicit definition of the **fundamentals** in the Handbook, these could include: (a) Adequate assessment of borrower's needs; (b) Country commitment both to project and to follow-up; (c) Adequate competency and skill levels both during project and in follow-up; (d) Adequate legal and paralegal system for the project's needs; (e) Satisfactory incentives for civil servants; (f) Clearly explained connections between inputs, outputs, and outcomes; and (g) Adequate physical and financial resources from both the donor/lender and recipient side.

Nevertheless, if the fundamentals are present in sufficient degree to go ahead with a project, our reservations on Bank staff time and incentives and project selection still hold. More importantly, in some countries - especially in Africa where the bulk of ID/TA is concentrated - it is likely that not all of the fundamentals will be in place or they will be weak. Does this imply that no TA projects should go ahead in such countries?1<sup>4</sup> This brings us to another important question: What does risk analysis mean or how is it undertaken in the context of ID/TA when there is no cost-benefit feasibility study? We argue below that if cost-benefit and costeffectiveness are definitely not possible, risk analysis may have to be undertaken using the results of similar projects in the recipient country or similar countries.

# **Part II: Technical Assistance: Evaluation**

This part begins with a general discussion of institutional development, due to its growing importance among TA activities. We then look at a number of classifications of technical assistance that have been proposed by others. In section three we construct a typology of TA projects and identify relevant methodologies, both quantitative and qualitative. The next section distinguishes between loan and grant TA. In section five we suggest a priori considerations for screening TA activities in order to weed out those unlikely to be justified using economic analysis.

# 11(a): Institutional Development (ID)

Any discussion of technical assistance inevitably turns to institutional development. At times the terms are even used interchangeably. While we believe that ID is only one of many types of technical assistance, it is increasingly the dominant type. In our review of 40 ECA projects in section III(a), we classified ID as the main type of TA in 21 cases. In value terms we classified 71 percent of the TA as institutional development. From an evaluation perspective, a priori and ex post, this could be considered unfortunate, as it is one of the most difficult types of TA to analyze quantitatively. In fact, many analysts say that it is often impossible to quantify 14 Alexander (1990:4), in his response to Operations Evaluation Department (1989), puts this dilemma quite graphically: "the discussion of absorptive capacity is a bit of a chicken and egg problem - if a country has the capacity to handle TA well, it generally needs less TA. The real problem is how can the Bank effectively work through TA and other instruments in countries that have very weak absorptive capacities." 8
and qualitative judgements must be used.<sup>15</sup> Nevertheless, we believe that there are many different sub-categories of institutional development. In our typology of TA activities in section II(c) we divide ID into a number of categories, some of which we believe to be quite amenable to quantification. Before we move on to these classifications, however, we think that a brief discussion of ID in more general terms is helpful in order to have a clearer picture of the problems it engenders, from both implementation and evaluation perspectives.

We will not attempt to review the voluminous literature on institutional economics, but instead we hope to highlight a few key points which in turn will illuminate some of the problems faced by the ex ante and ex post evaluators.1<sup>6</sup> The consensus among economists is that institutions exist primarily for two reasons: (a) to lower transaction costs; and (b) to act as focal points for collective action due to the free rider problem. The quintessential institution is the free market for the exchange of goods and services. From a donor or development bank perspective institutional development is largely concerned with making the free market work better (often by a clearer allocation of property rights) or strengthening the public institutions which are necessary due to high transaction costs and/or free rider problems.'<sup>7</sup> Often it can have both goals simultaneously, as, for example, in the case of judicial reform.

The problems of appraisal and evaluation are quickly apparent. How can one quantify a more efficiently functioning market? Indicators such as changes in growth of GNP or specific sectors are generally going to be much too crude to have any significance. Similarly, the fact that one finds public institutions in cases of imperfect markets implies that they primarily exist in situations where it is difficult to put a price or value on their output. While in some cases shadow prices can be used, in many others, such as the Ministry of Foreign Affairs, there are no obvious output measures. In such situations the best an evaluator can probably do is determine whether this output fulfills the objectives of the ministry and does so at least cost.

The water becomes even murkier if we agree with Arkadie (1989: 169) when he says: "One is left with the awkward conclusion that the public sector is in part the residual legatee of activities that do not lend themselves to effective institutional performance." This statement is closely related to Israel's (1987) hypothesis of specificity and competition (discussed in more detail below) in which he argues that across countries the public institutions which have the most specific functions and tasks and which face the most competition, either directly or indirectly, are those which perform better. This suggests that public institutions which are more akin to industrial firms - and whose inclusion in the public sector is debatable - may be the only ones

**15** See, for example, Operations Policy Department (1996: 143-44) for a discussion of the difficulties of quantifying TA for ID.

16 For overviews of institutional development from different perspectives, see Arkadie (1989), North (1990), Eggertsson (1990), or Lin and Nugent (1995).

**17** Of course, it is likely that TA has often gone to strengthen public institutions which would not have existed if the state had let the market do its job, but this is probably less and less the case given the global emphasis on the renewed importance of free markets.

9
that work well. It is not clear, as Arkadie (1989: 169) argues, that such institutions are appropriate objects of external assistance.

In the analysis below of the ECA projects we make a distinction between institutional development where the output can be measured either in value or quantity terms and those where neither is possible. In essence the former can be considered high specificity institutions while the latter are low specificity institutions. Our (admittedly very crude) analysis shows that 35 percent of institutional development funds went to institutions of low specificity which did not face competitive pressures. Appraisal and evaluation in such cases can be exceptionally difficult, especially if it is clear that no private sector institution would fill the void left by the absence of the public institution. One may be left in a situation in which it is necessary to make the difficult analysis of the situation of the country with and without the institution. Of course, the fall back situation for evaluation purposes could again be whether the institution is cost effective in its delivery.1<sup>8</sup> Finally, there is the issue of path dependence, emphasized by Bardhan (1989) and North (1989) and others. An institution may persist for a long time and even appear to be relatively efficient but it may be the wrong institution. That is, it developed out of an institution which was forned because it suited particular interests at a given time. However, if one could start from scratch right now, it is likely that an alternative (or no) institution would be selected. From an evaluator's perspective, the possibility of path dependence makes her task all the more difficult as it suggests that even cost effectiveness may not be a proper tool of analysis. While we will not discuss this point to any extent in this paper, a strong analysis of whether the institution is properly in the public or private sector should address this question adequately.'<sup>9</sup>

# II(b): Technical Assistance Classifications

It was noted in the first section of part I that technical assistance is typically differentiated between project related and free standing, and, in fact, it is rare that any data exists that goes beyond this simple dichotomy. However, there have been numerous more ambitious attempts to classify different types of TA. We shall review some of these and determine whether any are more suitable to our ultimate goal of classifying TA projects according to their suitability for quantitative analysis.

Berg (1993: 56-57) suggests 5 possible ways of classifying TA, the last of which comes from a World Bank document. The first distinction is between 'hard' and 'soft' overall objectives, where the former is on a specific, concrete task with a measurable output and the latter on 18 An interesting exarnple, given the institutional development effort that is being made in this area, is whether a Central Bank is needed or not. Some countries have toyed with the idea of making the US dollar their official currency. The greatest difficulty in appraisal may be to put a value on national sovereignty.

**19** Devarajan et al (1996) discuss the need for a careful examination of the role of the public sector in the context of the public expenditure review.

10
institutional and human resource development. While based on different criteria, in practice this distinction is very close to his second classification, which is the standard project-related versus free-standing. Nevertheless, our task at hand is to show that different types of 'soft' TA, freestanding or otherwise, are amenable to quantitative analysis. The third classification is from the donor viewpoint and differentiates between skills transfer, control or management of project resources, and TA as a catalyst to breathe life into a dormant project or institution. The fourth distinguishes TA by the type of personnel; ie performner, substitute, teacher or trainer, or mobilizer. Finally, the fifth (World Bank (1991b)) classification consists of foreign experts or high-level policy advisers, gap-fillers, gatekeepers or project managers, and institutional twins.

In addition, Berg (1993: 61-62) lists seven different elements of institutional development, which also could be considered a typology of the goals of ID/TA projects: (i) changing incentives to induce behavioral changes; (ii) enhancing skills by training and education; (iii) strengthening organizational performance; (iv) reforming procedures or systems of coordination between organizations; (v) increasing financial capabilities by more effective resource mobilization; (vi) nurturing societal supports - for example, by encouraging formation of user groups, political reforms that bring greater transparency and accountability, and greater participation of wage eamers in determination of their conditions of employment; and (vii) cultivating new norms and values.

Cassen (1994: 143) lists 8 different types of TA: "(i) pre-feasibility and feasibility studies for capital projects; (ii) engineering design and construction oversight for capital projects; (iii) research; (iv) institution building projects (which may be free-standing or combined with capital projects); (v) policy studies; (vi) individual free-standing fellowships (that is, not part of an institution building or capital project); (vii) individual expert services (consultancies; seconded expatriates); and (viii) short-terrn training (seminars, study and observation tours, equipment familiarization)." In a Bank document Lethem and Cooper (1983: 37-40) distinguish between 4 types of TA, based on the role of the foreign and local personnel. These are: (a) performer or substitute model, (b) prescriptive or expert adviser model, (c) counterpart-adviser model, and (d) collaborative model.

In another Bank study, AFTPS (1989: 14), TA was classified into four categories: "(a) advisory services for supervision, implementation and engineering; (b) feasibility and other technical studies; (c) consultant services for management, economic and financial policy analysis, sectoral studies, organizational and institutional review studies, development of MIS, surveys, research, and collection of other statistics; and (d) training." Operations Policy Department (1996: 3) of the Bank differentiates free-standing technical assistance loans by objectives: (a) public sector capacity building, economic management, tax administration, judicial infrastructure improvement; (b) parastatal reform; (c) sectoral management; (d) privatization; and (e) environmental management.

11
While he does not explicitly define a typology, the work of Israel (1987) on institutional development implicitly contains one. He hypothesizes (with some data support) that the major predictors of the success of an institutional development project are the specificity of the functions and the competition to which they are exposed. Consequently, one finds much more variation across sectors than across countries when looking at the success of institutional development projects. While it is the case that, for example, projects in East Asia are more successful than in sub-Saharan Africa, the similarity of the sectoral rankings in each subset are much more striking. The reasons behind such results, he argues, are that sectors across countries tend to be under similar conditions with respect to both specificity and competition, where specificity is defined as (roughly) the degree to which the objectives, methods, and results of an activity can be defined and, correspondingly, rewarded. High specificity activities usually lend themselves much more easily to competitive pressures, both internal and external to the organization.

11(c): A Technical Assistance Typologv

# IIc.i **Criteria for Establishing Technical Assistance Categories**

Almost all discussions of TA speak of the difficulties of quantitative analysis (especially when it is for institutional development or capacity building), although more recent documents express a need for some attempt. The Handbook of the Operations Policy Department (1993: 143-44) makes a plea for more use of quantitative analysis but concludes: "In the final analysis, judgement serves as the best guide to IDTA." Other observers, such as Cassen (1994: 171) see the evaluation problem as being closely linked to the general problem of analysis of TA: "There is a striking weakness in the intellectual underpinning of institution building, human development, and associated TC compared with the theoretical and quantitative tools used to plan physical investment." Nevertheless, we believe that an increased use of quantitative economic analysis in TA appraisal is not only possible, as will be argued below, but would significantly enhance the quality of the TA. A clear exposition of the costs and benefits of a project is beneficial even if all of the numbers cannot be filled in as it brings out all the elements which will be affected by the project. Similarly, analysis of the fiscal impact will not only clarify the possible effects on the government budget constraint, but it will also elucidate the division of projected costs and benefits between the public and private sectors. Finally, the relatively simple chore of conceptualizing the TA in economic terms can help prevent inconsistencies between the tasks to be undertaken and the project objectives, as well as among the tasks and objectives themselves.

For the most part it is not very difficult to measure the costs of a TA project, as they largely consist of the standard items such as personnel, equipment, communications and supplies, and office space. Some of these are usually provided by the recipient, and there may be some challenge to obtaining their actual value (or opportunity cost), but this should not be insurmountable. The most difficult measurement on the cost side is likely to be the opportunity cost of senior officials in the recipient country in the negotiation and implementation stages. If our guesstimate with respect to the increase in Bank staff time and resources needed to comply 12
with the guidelines and procedures on TA is credible, proper estimation of these costs could also be essential in determining the "profitability" of a TA project.<sup>20</sup> Nevertheless, the true challenge in both a priori and ex post evaluation of TA projects is on the benefit side. In practice, it is rare that any attempt is made at quantitative analysis in project appraisal and it is also uncommon in ex-post evaluation. Thus, it seems likely that the most important prerequisite for a typology of TA projects, given our ultimate purpose, is the measurability of outputs and the ability to assign a monetary amount if measurable. For example, an institution created to train mining engineers has an expected measurable output to which a monetary value can be placed; ie the present value of the expected increase in their earnings. Funding for academic research has a measurable output - number of recipients, publications, etc - but it is difficult to put a monetary value to it. Support for strengthening the operations of a central bank may not have either.<sup>2</sup> 1 Even when a measurable output exists, it may have little value as a quantitative indicator. This is particularly true for training courses where the quality of the course and the selection of the participants is often much more important than the turnover. Accordingly, one must be cautious when assigning values to outputs which have only a tenuous link to impacts.<sup>2 2</sup>

# IIc.ii **Categories of Technical Assistance**

In total we use five categories: investment project, general capacity building, institutional development - output permitting valuation, institutional development - measurable output, and institutional development - non-measurable output. To construct these categories we applied the main criteria discussed in the last sub-section - the ability to value and/or quantify output - to the typology given by Cassen, referred to above.<sup>23</sup> _Investment project TA_ includes his first two categories, feasibility studies and engineering design and construction oversight for capital projects, as well as short-term training needed for the implementation of a specific investment project. Note that the latter is not equivalent to all short-termn training found in investment projects, as a great deal of the TA found in actual 20 In many institutions the opportunity costs on the donor side are far from clear. In general little attempt is made to cost out the staff, supplies, and overhead time devoted to projects unless there is another donor involved. In these instances rules of thumb are usually used which may have no relation to the actual costs incurred for individual projects, even if on average they are close.

21 These examples are only for illustrative purposes, as it is easy to think of caveats to all of them.

22 Moore (1995: 39) is more caustic on this point, referring to training as the 'default mode' of institutional development.

23 Cassen's category "individual expert services" is excluded from this mapping as it is, in effect, captured within all of his other categories.

13
projects is not necessary for implementation.<sup>2 4</sup> All investment project TA can and should be incorporated in the determination of the expected rate of return of the investment.

_General capacity building TA_ is at the opposite end of the spectrum from investment project TA. It consists of research, policy studies, short-term training, and free-standing fellowships. However, we only include the first three if they are not part of an institution building project and also, in the case of short-term training, not necessary for implementation of an investment project.

General capacity building is typically very difficult to appraise using quantitative economic analysis. Research is one of the most difficult areas to appraise. The typical method is to estimate the number of domestic and intemational publications. These could perhaps be monetized by estimating expected salary increases to researchers with more publications, although this would entail a number of difficult assumptions, such as the value of a publication in journal "x" relative to journal "y". Policy studies may be even more difficult to appraise than research as their importance depends on who they influence as much as their content. If they are not published - as is usually the case except for internal use - it is difficult to ascribe an independent value to them. Individual free-standing fellowships are a considerable amount of donor aid and presumably can be evaluated based on expected increases in income. However, this calculation becomes much more difficult (and controversial) when the large amount of brain drain that they often entail is included.

Short-term training for general capacity building can be subdivided into tangible, marketspecific skills such as computer training and more abstract, conceptual knowledge such as courses on structural adjustment. The return to conceptual knowledge is much more difficult to determine, partly due to its nature and partly due to the persons likely to be taking the training. They will generally be part of the public sector, often in middle management or higher, and their human capital will consist of a number of characteristics. It is likely to be difficult or impossible to sort out the effects of one training course on the return to their human capital.

_Institution building TA_ has become the dominant form of non-project related TA. Given its importance, we have sub-divided Cassen's category into three sub-categories, partially based upon the concepts of specificity and competition as defined by Israel (1987). One can first differentiate between projects which are targeted on activities which have a monetary value or are easily monetized versus those which do not - for example, reform of the electric company versus the Central Bank. In the latter group it is then possible to further distinguish between projects centered on easily measurable outputs (although difficult to value) and those whose results are less tangible or, as is more often the case, difficult to disentangle from a number of

> 24 In Part III we estimate that about 80 percent of the TA in non-TA projects in our sample was not necessary for implementation. Of course, the next question is why is the TA there in the first place if it is not necessary for implementation. We will address that question in the conclusions and recommendations.

14
reforms and other developments.<sup>2 5</sup> For example, the output of privatization projects can be measured relatively easily (although it may be difficult to value) while the output from a wideranging project such as Ukranian Institutional Reform is very difficult to quantify. _In sum, institutional development can be separated into activities whose output permits valuation, those which have a measurable output but which cannot be valued, and those which do not have either feature._ In addition to Cassen's institution building category, these three categories will include part of his policy studies, research, and short-term training.

We would like to emphasize that while it is clear that a great deal of institutional development is capacity building, the category of general capacity building only includes items which are not specifically part of an institutional development project.<sup>2 6</sup> Typically, economic analysis of a general capacity building project would focus directly on the return to human capital while economic analysis of an institution building project would focus on the performance of the institution.

# IIc.iii **Appraisal Methodologies**

There are a number of different methodologies that can be used to appraise technical assistance. Those which we believe to be most suitable for World Bank projects are, moving from most quantitative to most qualitative, cost-benefit analysis, cost effectiveness, fiscal impact, lessons from previous projects, qualitative economic analysis, and professional judgement.

While the concept of _cost-benefit analysis_ is well-known, if often difficult to undertake, that of _cost effectiveness_ is often less clear. It is undertaken when costs can be measured although benefits are difficult to value. To be meaningful the outputs must be quantified before and after.<sup>27</sup> In essence, one is asking whether the good or service is being delivered in the most efficient manner. However, it is often difficult to have benchmarks or alternatives, especially in the case of transition economies. One can begin by costing out different ways of production and/or provision. For example, while it may be difficult to value the benefits of a health project, there will often be some estimate of reduced morbidity and mortality. These figures should be compared with the cost of similar reductions with alternative health interventions.

Then one should ask the following two questions. First, is it expected that the good or service will be delivered at a lower cost than before the technical assistance? Second, how will 25 Easily measurable outputs will often be related to what Israel (1987: 49) calls "well-defined tasks"; that is, tasks in which the worker does not have much latitude over the content of the job. For example, in his typology an airplane mechanic has a well-defimed task while an agricultural extension worker does not.

26 For exarnple, mass training of tax administrators would likely be part of an institutional development project while a general scholarship competition would be included in general capacity building. While institutional development could be called (institution) specific capacity building, "institutional development" is the common World Bank termrinology.

27 For example, delivery of pension cheques is likely to be subject to cost effectiveness analysis while delivery of primnary school education may not be, given the importance of quality.

15
the average cost of the good or service compare with similar countries? For example, wiL administrative cost per unit of income tax collected be lower than previously and how will i compare with the per unit cost in other countries?

At times it may be impossible to get a clear picture of cost effectiveness due to significant changes in quality or multiple objectives. Nevertheless, given the budgetary problems facing many borrowing countries, _the fiscal impact_ of the project may be an alternative measure of its viability. By fiscal impact, we simply mean the expected effect on the government's budget balance. For example, in a country with a large fiscal deficit, tax reform may be more concerned with increasing revenues than lowering administrative costs. Thus the anticipated fiscal impact of the project would be more important than the cost effectiveness (defined as average administrative cost per dollar of tax revenue). Note that the analysis of the fiscal impact of a project could be as sophisticated as estimating the present value of the changes in revenues and expenditures or as simple as estimates of the effects on the budget deficit in the short-term. The macroeconomic situation of the country will help determine the relevant time horizon. In a country with severe macroeconomic instability a project which looks quite attractive in the longterm may have to be abandoned if the fiscal impact in the early years is negative.<sup>28</sup> We have already noted that cross-country comparisons can be used in the analysis of the cost-effectiveness of a project. In addition, ex-post _results of other projects_ in the recipient country or similar projects in other countries can be used in the appraisal process. Although it is becoming common practice in the Bank to use lessons from past experience, these could be undertaken in a more rigorous manner. First, when feasible, rates of return should be extracted (or calculated) from previous similar projects and compared with the estimated rate of return for the current project. If these are significantly different, the onus should be on the project appraiser to justify her calculations. Even more important, in the case of non-quantifiable projects which have had a history of failure either at the particular country level or at the similar institutional level, the impetus should again be on the evaluator to explain in detail why this case is "special" if it receives a positive appraisal. This explanation should go beyond statements such as "lessons learned from previous projects are being taken into account" and provide a detailed plan of ways in which past shortcomings will be overcome.

Finally, in some technical assistance projects it is genuinely difficult to measure outputs either due to the nature of the assistance or the fact that it is part of a broad number of difficult to disentangle measures. However, it will almost always be possible to conceptualize the problem in economic terms. In this manner the objectives of the project are at least put in a consistent framework. Accordingly, at a minimum every project should include _qualitative economic analysis_ consisting of the conceptualization of the methods and objectives in order to verify their internal consistency. The go-ahead for the project would then be decided by _professional judgement,_ supported by this qualitative economic analysis. We would suggest that an outside

> 28 Of course, this point will not be relevant if there is a long grace period before repayment begins unless a substantial part of the project is funded directly by the loan recipient.

16
independent analysis also be required for projects which are difficult to quantify and for which there are few past projects with which to compare it.

# IIc.iv **Economic Analysis of Technical Assistance Projects: A Typology**

In our construction of a typology for economic analysis of TA projects, we use a grid system. On the vertical axis we move from the most quantitative to most qualitative techniques. On the horizontal axis we have the type of project based on the measurability and/or monetization of the objectives and impact. This is illustrated in Table 1. Each square contains "x", "xx", or is blank. A blank square indicates that in general the methodology is inappropriate for the type of project. "x" represents what we believe to be an alternative methodology for the type of project if the preferred methodology (or methodologies) cannot be undertaken due to data limitations or cost constraints. It may also be used as a supportive role even when the preferred methodology is feasible. "xx" indicates what we believe will usually be the best methodology for this type of project barring major data limitations.

On the vertical axis of each table we start with cost-benefit analysis, followed by costeffectiveness, fiscal impact, previous projects - quantitative, previous projects - qualitative, and professional judgement. On the horizontal axis of Table 1 the categories are investment project, capacity building, and institutional development. The latter is subdivided into output permitting valuation, measurable output, and non-measurable output. Investment project includes all TA which is necessary for the implementation of the project. These include pre-feasibility and feasibility studies, engineering design and construction oversight, and short-term training for what we referred to as market-specific skills (or what Israel might call well-defined tasks). Capacity building includes training which is neither a crucial part of an investment project nor an integral element of institutional development TA. It also includes scholarships, research, and policy studies, again if the latter are not part of institutional development TA. The different categories of institutional development are as explained in IIc.ii above.

Note that we do not see this typology or our grading system as being definitive. It was constructed partially based on the selection of projects that we examine in section III and would likely need to be modified to fit particular regions. Our main points are that some TA is more susceptible to quantitative analysis than other TA, most TA is susceptible to some type of quantitative analysis, and that there are a variety of methodological techniques to draw upon with varying degrees of rigor. Note also that professional judgement plays a supporting (or "secondbest") role for all types of TA projects. We also believe that quantitative analysis imposes a disciplines on the project appraisal process, forcing the professional to think through the conceptual links of the project as well as the magnitudes to be generated.

# 11(d): Technical Assistance: Loans versus Grants

The bulk of technical assistance from bilateral donors is in grant form while the exact **opposite is true** for the World Bank. In principle there should be no difference in the evaluation 17
of the two - except for perhaps different interest rates - as they both use scarce resources which could be used elsewhere. However, from the country perspective they may have quite different considerations. Oddly enough, a grant could be more costly than a loan if the seemingly free nature of the former results in a neglect of domestic opportunity cost variables, including other uses for the grant. This danger is particularly relevant when the grant is accepted because there is some desired capital equipment attached to it but there is little or no commitment to the TA per se. In section Ic above a number of other problems of TA grants were discussed, such as hidden opportunity costs, lack of coordination of numerous grants, and the political economy and rentseeking problems that can develop when the recipient ministry or institution is not the one that bears the opportunity costs.

On the other hand, the refusal by a country of a grant does not necessarily mean that the rejected funds will be used elsewhere in the country. Projects may be accepted whose funds have a high opportunity cost from a global perspective but a very low opportunity cost from a national perspective. In fact, from the country viewpoint, even a negative rate of return may be acceptable. The grant, no matter how poorly conceived, may simply be perceived as "better than nothing".

# 11(e): Project Screening

We have discussed the problems of quantitative measurement for many TA projects. Even when feasible such analysis requires resources within the donor institution. Therefore, it is recommended that a number of prerequisites are in place even before any quantitative analysis takes place.

First, if the project is in the public sector, the officer in charge should explain why it cannot be undertaken by the private sector. In addition to dealing with the general problem of public versus private sector, this should indirectly partially resolve the problem of path dependency referred to above. That is, by questioning the very existence of the given public sector institution (in a form of zero-based budgeting), the analyst helps resolve the question of whether the institution exists due to historic circumstances which may no longer prevail.

Second, we mentioned above that the most common criticism of TA projects is that there is no commitment from the recipient. Without evidence of such commitment we recommend that the analysis does not continue, unless there is strong reason to believe that an exposition of the costs and benefits is necessary to obtain recipient support. Particular care needs to be taken when the TA contains significant capital support (such as computers and vehicles) that the commitment is to the entire project, particularly but not only when the ministry or institution which received the TA does not have to repay it.

Third, the officials in charge should also review projects in similar sectors in the recipient country or similar countries. If there is a consistent record of failure, the analysis should not continue unless there are new circumstances or provisions which suggest that the current project will not suffer the same fate as its predecessors. While it is now becoming common practice to 18
undertake such comparisons ("past experience and lessons learned"), there is room for a more common treatment across the Bank, with a more rigorous methodology when possible.

Fourth, related to the above the Bank should consider the creation of a group of designated experts within each of the most common categories of projects. At an early stage in the project process, approval from a designated expert and someone with local expertise would be necessary to continue. In this way the common complaint of the shortage of institutional expertise would at least be partially resolved.29 Fifth, all components of a project should be conceptualized in economic terms. If this cannot be done, it is a strong indication that the objectives of the component (or entire project) are vague or inconsistent with the tasks to be undertaken.

Sixth, as noted above in projects where a positive appraisal is largely or entirely based on qualitative professional judgement, an independent expert opinion should be obtained. Such experts could be chosen by the institutional experts.

Seventh, training in TA projects has been especially subject to criticism. It can also be quite difficult to quantitatively measure its effectiveness. We suggest that the Economic Development Institute of the World Bank play a stronger role in screening the design of training.

Eighth, it should be demonstrated that TA in an investment loan is essential to its implementation. Otherwise, it should not be included unless there are other extraordinary reasons arguing for its inclusion.

Finally, while tools such as performance or impact indicators cannot replace a thorough appraisal, an inability to construct concrete measures of performance suggests that objectives may not be well-defined and there may be problems during implementation. Therefore, the current emphasis in the Bank on performance and impact indicators is a positive step in project development. Moreover, the economic analysis and performance indicators should be linked as much as possible. In particular, key assumptions used in the economic analysis could be tracked at the same time as the performance indicators in order to provide "early warning signals" on the quantitative results. Such data could be fed back into the original economic analysis in order to determine if there are significant changes in the expected results and if appropriate action should be taken accordingly.

# **Part III: Preliminary Empirical Analysis and Practical Applications**

We begin this section with a brief examination of 40 projects from the Eastern Europe and Central Asia (ECA) Region of the World Bank, with particular emphasis on the amount of 29 The Sector Boards which are being created in relation to the "Networks for Professional Excellence, Staff Development, and Client Service," could conceivably play this role.

19
TA and how it was appraised. Then we look at 4 of these projects in greater depth and discuss methodologies which might have been used in the pre-approval appraisal of the TA component.<sup>30</sup>

# 111(a): Characteristics of the Large Sample

The 40 projects were all in transition economies except for one in Turkey. One was in Armenia, 2 in Belarus, 4 in Bulgaria, 5 in Hungary, 3 in Kazakhstan, 5 in Poland, 3-in Romania, 11 in Russia, one in Slovenia, one in Turkey, one in Turkmenistan, one in the Ukraine, and 2 in Uzbekistan. Table 2 contains the numbers of loans by type and by year. In sum there were 2 loans in 1992, 15 in 1993, 13 in 1994, and 10 in 1995. From 1993 to 1995 we included all the loans for which we calculated more than $US 5 million in TA. 13 of these were technical assistance loans, 5 were financial intermediary loans, 2 were sectoral adjustment loans, one was a sector investment/maintenance loan, and 19 were specific investment loans. Only one of the loans was the first loan to the country but 13 were one of the first five loans to the country.

The calculation of the amount of TA was often difficult as it was either hard to sort out or the budget was not broken down sufficiently. Moreover, except in the TA loans, training was often separated from the rest of TA. For all 40 loans the average size was $US 101 million, of which $US 24 million was TA. (The average project size, which includes recipient country and other agency contributions, was $US 179 million.) For the 27 non-TA loans the average size was $US 133 million, of which $US 19 million or 14.3 percent was TA. The average TA loan was $US 35 million.

Our original intent was to classify the 40 projects both by type and by methodology used in the appraisal. However, in not a single case was a quantitative technique used in the appraisal, generally without any significant qualitative economic analysis, other than to verify that the project met the criteria of the country assistance strategy. In fact, with one qualification the only method used to appraise the TA was professional judgement. In most of the 1994 and all of the 1995 loans, lessons learned from past Bank experience in the country or similar projects in other countries were taken into account. However, there was never an attempt to make an ex-post quantitative analysis of the other projects reviewed. In almost all cases the design of the new project was said to have been adapted to take into account problems in past projects.<sup>3</sup> ' In addition, in all of the 12 non-TA loans for which an economic rate of return was estimated, the TA portion was excluded from the calculations.

For each of the 40 projects the TA was distributed among the five categories noted in the last section plus a sixth "other" category when we found it difficult to classify. Given time limitations and limited familiarity with the projects, we often had to make "educated guesses" as to where each type of TA would best fit. Nevertheless, we think that the figures give a good 30 Note that the median loan size was $US 70 million and project size was $US 126 million. A few very large loans and projects resulted in substantially higher averages.

31 As we are only looking at loans which were approved, we did not see any projects that were rejected on the basis of lessons of past Bank experience. It would be interesting to know if this is often the case.

20
indication of the types of TA funded by the Bank in ECA. Note that any TA which we deemed to be necessary for the implementation of an investment project was included in the category "Investment Project". Training and studies were allocated to "Investment Project" or "Institutional Development" if they were an integral part of one or the other. If neither was the case, they were categorized as "General Capacity Building".

In the "Institutional Development" sub-category of "Output Permnitting Evaluation" we included any activity which would measurably affect a monetary amount. The two most common examples in our sample were projects in tax administration and institutional strengthening of commercial banks (including the development of regulatory and supervisory institutions). The sub-category "Measurable Output" included any TA which was intended to affect the delivery of a service or good. The most common example in our sample was privatization. Given the complex nature of privatization in transition economies, it would be difficult to estimate a rate of return. Nevertheless, the number and (rough) value of enterprises privatized could be calculated and the cost-effectiveness compared with other countries. In addition, the fiscal impact could often be estimated. The sub-category "Non-Measurable Output" contained a variety of TA, although it was often connected to more general adjustment measures, the effects of which would be very difficult to measure. Much of the TA which fell into this category was part of one of the first five Bank loans to the recipient country.

In sum, of a total TA of $US 973 million, we classified $US 121 million as "Investment Project", $US 107 million as "General Capacity Building", $US 266 million as "Institutional Development - ", $US 180 million as "Institutional Development - Measurable Output", $US 242 million as "Institutional Development - Non-measurable Output", and $US 57 million as other. (See Table 3.) When we only look at non-TA loans, these figures are (in $US million) 110, 42, 139, 59, 102, and 57, respectively. For TA loans, these figures are (in $US million) 11, 65, 127, 121, 140, and 0, respectively.

Note that we only classified about 20 percent of TA for non-TA loans as necessary for the implementation of the project. Therefore, if our classifications are roughly correct, while always in the same sector or "neighborhood", the vast majority of TA attached to non-TA loans was not necessary for the success of the core project. Also, in projects for which an economic rate of return was calculated, 53 percent of TA was classified as necessary for implementation. As noted above, in all such projects the TA was not included in the calculation of the economic rate of return.

In the next sub-section we will look in more detail at 4 loans. However, we would like to note here that if quantitative measures such as cost-benefit analysis, cost-effectiveness, and fiscal impact can often be used for investment projects and institutional development which permit valuation and/or measurement, then for about 60 percent of the TA in our sample there was a strong potential for quantitative analysis.<sup>3 2</sup> 32 This does not mean that there was no potential for quantitative analysis of the other 40 percent. In particular, it is likely that a large portion of the general capacity building efforts could have received some quantitative analysis.

21
# 111(b): Four Case Studies

In this sub-section we will look in detail at 4 of the 40 projects in our sample. Two of the four are TA loans and the other two are specific investment loans. For the TA loans we chose one for which it could, a priori, be considered relatively easy to use quantitative analysis (Hungary Tax Administration Modernization) and one for which it could be considered relatively difficult (Russian Federation Management and Financial Training, including Russia Tax Administration). For one of the specific investment loans an economic rate of return was calculated but only for the non-TA part (Poland Forest Development Support), while for the other it was not calculated at all (Russian Federation Financial Institutions Development).

# IIIb.i **Hungary Tax Administration Modernization** - Loan #36350 (1993)

The size of this TA loan is $US 29m to which the Hungarian govermment added $US 26.6m for a total project size of $US 55.6m. The immediate beneficiary and implementing agency is the Tax and Financial Controls Office (APEH). The economic rate of return was deemed not applicable. The project objective is "to help strengthen tax administration, in line with the ongoing tax reform, in order to maximize tax revenues in a sustainable way." (World Bank, 1993: 2) The three main instruments of the project are: "(a) institutional development through the financing of short-term experts to provide advice in tax administration; (b) development and implementation of a new information technology system, including: (i) technical assistance to guide the development of a tax applications systems software, to operate on new computer systems: (ii) mainframe computers for APEH's headquarters and Budapest branch office, minicomputers and microcomputers for county offices, associated with peripheral equipment; (iii) software development tools; and (iv) administrative software packages; and (c) training in development and the use of application systems." (World Bank, 1993: 2) Note also that a complementary project undertaken by EC-PHARE will finance training in tax administration.

There was no quantitative analysis undertaken for this project. There was a one paragraph statement of lessons from previous Bank involvement, which mentioned "the need for a clear focus and limited scope in the project; commitment at all levels; and a sense of ownership of the project by executing agency staff." (World Bank, 1993: 4) No direct reference was made to the qualitative or quantitative results of other Bank projects in tax administration.

From the documentation that we have seen, the project appears to be a straight forward attempt to improve the capabilities of the tax administration.<sup>3 3</sup> Its main objective is to lower the average and/or marginal cost of tax collection, while its (implicit) secondary objective is to increase revenue; that is, "fiscal improvement must come from improvements in collection 33 In what follows we will make some suggestions about appraisal methodologies for this specific project, while in Box 1 we will discuss the case of tax administration projects more generally.

22
achieved through better administration of the tax system." (World Bank, 1993: 1) There is little if anything that is extraneous in the loan, in the sense of having a larger purpose than just tax administration. It is difficult to estimate a standard economic rate of return for an improvement of tax administration since one would need to calculate welfare gains of more public goods and/or a lower fiscal deficit and contrast this with the losses caused by fewer private goods (if the improvement was revenue increasing). To complicate matters further, there would likely be improved horizontal equity (and perhaps vertical equity) in the tax system, increasing the welfare gains to the administrative reform. In essence you would need a general equilibrium model, requiring significant conceptual development and substantial reliable data. Even if the data was available such models are good tools for reforms which result in equivalent amounts of revenue, but they are not very suitable when the reform is not expected to be revenue neutral.<sup>34</sup> One could, of course, calculate the expected change in the average (and/or marginal) cost of tax collection. Presumably the reform should result in a lower collection cost.<sup>35</sup> In addition, the expected collection costs for various taxes can be compared with those in other countries. The one caveat to this argument is that if revenue enhancement is the driving force of the reform, little change or small increases in collection costs may not be important. However, the second part of the analysis - cross country comparisons - is still valid.

An alternative or back-up analysis would be the fiscal impact of the reform. Stabilization and adjustment in many countries have been threatened by large fiscal imbalances. If the reforn is expected to have a strong effect on the fiscal balance, this could be justification enough in a time of serious macroeconomic imbalances. First, it would be necessary to estimate the expected changes in revenues and costs. Second, if the data was available, it may have been possible to construct a small macroeconomic model, which could be used to estimate the effect of changes in the fiscal deficit on inflation and the balance of payments. Even if the second step was not possible, the simple conceptualization of the problem in such terms would have been useful.

In sum, it seems possible that fairly inexpensive quantitative analysis could have been undertaken in the appraisal of this project. (Of course, if it was decided to build a general equilibrium model, the costs would not be trivial.) It may well be that data limitations would have made even the relatively simple analysis that we have suggested impractical, although for Hungary and this particular project, this seems unlikely. (See Box 1 for a further discussion of the different stages of analysis of tax administration reform.) IIIb.ii **The Russian Federation Management and Financial Training** - Loan #38240 (1995) and **Russia Tax Administration -** Loan #38530 (1995) 34 It is also unlikely that a computable general equilibrium model would be suitable to use for a transition economy where large changes in economic structure are expected to occur.

35 In general changes in collection costs and welfare costs can be quite different. However, as in this project there is no reform of the various types of taxes, the two costs should be quite similar for the revenue neutral scenario.

23
We have combined these two loans as they were developed to be complementary. The loan for the Management and Financial Training (MFTP) Project was $US 40m to which were added $US 10.3m from Russian sources and $US l.lm from the Government of Switzerland. The loan for the Tax Administration Project (TAP) was $US 16.8m to which were added $US O.9m from the IMF and $US 9.7m from the Government of Russia. In both cases the economic rate of return was deemed not applicable.

The MFTP has 3 principal objectives: "(i) to train practitioners in three core fields of management, financial sector, and public finance; (ii) to develop an intermediary institution, the National Training Foundation, to mobilize and channel resources for high priority training investments; and (iii) to implement pilot projects in other key market areas and establish the basis for a broader second-phase investment." (World Bank, 1 995a: i) The main elements of the project are precisely the training needed to meet the above objectives.

The principal objectives of the TAP are: (i) To modernize tax administration in two regions; and (ii) to assist in the institutional development of tax administration by building the capacity of the State Tax Service (the main beneficiary), preparing for nationwide implementation of tax administration reforms, and providing project management support. To achieve these objectives the proposed project would finance expert services, staff training, acquisition of hardware and software, and minor improvements of offices for computerization. (World Bank, 1995b: 4) The tax administration project is very similar to the one discussed above for Hungary; accordingly, we will only mention that the public finance training in the MFTP is very complementary. Most of the arguments given in the Hungarian case also apply here, although given the somewhat more limited scale (due to the size of Russia) it could be difficult to analyze at the macroeconomic level. We would also like to note that despite the lack of quantitative analysis for either project, a significant effort was made to distill and employ lessons from Bank experience in Russia and similar projects elsewhere.

30 percent of the total costs of the MFTP are for management development; ie reformning and upgrading enterprise-based management. A cost-effectiveness analysis could have begun by determining approximately the number of persons that would be trained and the length of training and comparing the cost with that of other transition or similar countries (allowing, of course, for differences in the cost of living and key inputs). A deeper analysis would have estimated the increase in the return to human capital of the various participants and calculated this against the cost of the training. In effect, this is what a typical university student does in a western country (although she would usually only be interested in the private return and not the complete return to society of the education). Of course, the calculations can be difficult if the recipient of the training does not pay for the training (other than in time). It can also be very problematic for short training courses where it is difficult to distinguish the before and after market value of the recipient, in contrast to a high school versus a university graduate. Finally, the data problems could be insuperable in Russia at this time. Still, it may be possible to estimate ex-post the return to similar training courses in countries with better data availability and use these as guidelines.

24
45 percent of the total costs of the MFTP are for training in the financial sector, divided between accountancy, auditing and finance (AAF) versus banking. In the case of AAF, estimates were made of the number of persons to be trained at various levels. These could have been compared with other countries to obtain some idea of cost-effectiveness. As in the case of management development it is possible that the increased returns to human capital could be calculated, although the short-term and very widespread nature of the training may have made this difficult. Given the greater focus of the banking training in this project, it seems likely that the success of the project should depend on a higher rate of return of the commercial banks involved. Therefore, estimates of the effects on profits of the commercial banks receiving the training could have been undertaken in the appraisal.

Finally, about 11 percent of the costs of the MFTP are for the creation of a National Training Foundation. It could be argued that in the calculations of the costs or rates of return of any other part of the project, the costs of the National Training Foundation should be pro-rated in some way. This would have to be handled with great care given the substantial externalities that such a foundation could provide for the Russian economy.<sup>36</sup> In fact, the most appropriate quantitative analysis of the entire project could well have been to undertake a cost-benefit analysis of the National Training Foundation.

# 11Jb.iii **Poland Forest Development Support** - Loan # 36410 (1994)

The loan for the Poland Forest Development Support project was $US 146m to which were added $US 16m from the European Investment Bank, $US 26.9m from bilateral agencies, and $US 146.5m from the Government of Poland, for a total project size of $US 335.4m. The main beneficiary is the General Directorate of State Forests (GDSF). The economic rate of return was estimated for the "investment" part of the project (18 percent) but not for the technical assistance components.

The objectives of the project are: "(i) Assistance in introducing biologically and environmentally sound management practices in the forestry sector and amelioration of environmental damage, particularly by increasing the vitality and productivity of Poland's important forests; (ii) Improvement of the efficiency of forest activities (afforestation, regeneration and restoration of air pollution damaged forests, silvicultural treatments, and harvesting); and (iii) Improvements in the organization, management, and financial structure of the Directorate of State Forests and the Govemment's programs to modemize, restructure, and privatize elements of the GDSF and the downstream wood industry sector." (World Bank, 1994a: 9) The seven components of the project are: (i) intensification of silviculture in existing forests; (ii) restoration of air-pollution damaged forests; (iii) afforestation; (iv) genetic conservation and 36 "The creation of a new training foundation was justified by the massive scale of training requirements, which exceed the scope of any existing agency; the present lack of coordinating mechanisms; the need to involve nongovermnent institutions and employers in the process; the need to address regional requirements flexibly; and the need for objective project selection and strict financial controls. The NTF is an autonomous institution with half government and half private ownership. " (World Bank, 1995a: ii) 25
improvement program; (v) modernization of harvesting, skidding, and transport equipment; (vi) ecosystem conservation in national parks; and (vii) institutional development. An economic rate of return of 18 percent was calculated for the first five of these categories, which comprise 90 percent of the project expenditure.

Our concern in this paper is whether a quantitative analysis could have been undertaken for the last two components. First, we asked whether or not their costs should have been included in the cost-benefit analysis of the first five components; that is, are the main benefits of these components their contribution to the success of the more concrete aspects of the project? In the case of ecosystem conservation in national parks (3.6 percent of total expenditure), the answer is clearly no. In essence, this is a separate project with only a very marginal relationship to the core investment. It appears as if it was packaged with the other components for administrative convenience, given the same beneficiary.<sup>37</sup> The institutional development component (6.2 percent of total expenditure) is focused on the GDSF, the same institution in charge of implementation of the other five components. It has seven sub-components: (i) development of a strategy for environmentally sound forest utilization; (ii) completion and implementation of a forest management information system; (iii) office of project implementation; (iv) section for privatization of forestry; (v) divisions of trade and marketing; (vi) institutional strengthening for GDSF; and (vii) forestry research support to ensure sufficient scientific support for the transition of the sector to a market economy. (World Bank, 1994a: 14) While it seems quite clear that the success of this component would increase the return to the five specific investment components, it could be argued that there are significant externalities associated with this ID package. If it is to be included on the costs side, these other benefits must be measured. However, they would be extremely difficult to measure. Our bias is that the costs of ID should be included in the economic evaluation of the investment component. Some are essential for implementation - (iii), (iv), and (vi) above - while the others are very important. As a final note, cost-effectiveness analysis could have been undertaken for the sub-component on the forest management information system, half of the ID component.

# IIIb.iv **Russian Federation Financial Institutions Development** - Loan # 37340 (1994)

The Bank loan for this project was $US 200m to which was added $US 99m from the European Bank for Reconstruction and Development, a $US 36m grant from the European Union, $US 37.4m in bilateral grants, $US 3m from the Russian government, and $US 14m by participating commercial banks. The total project size is $US 389.4m. The economic rate of return was deemed not applicable.

> 37 In theory the ecosystem component, which focuses on national parks, could be analyzed using environmental valuation methods, but given the fledgling nature of the program this would be quite difficult to undertake.

26
The main objective of the project is to strengthen a core group of banks which will: "(i) Set higher banking standards and create a dynamic for enhancing the quality of banking activities in Russia; (ii) Create the basis for a private clearing system operating at the level of the Federation; and (iii) Qualify as on-lenders of World Bank and other international lines of credit." (World Bank, 1994b: 17) "The project will consist of four components: (i) a commercial banking component, consisting of institutional strengthening programs and systems modernization and automation programs for private commercial banks; (ii) a bank regulation and supervision component, consisting primarily of the development of on-site supervisory capabilities and the provision of legal assistance for the Central Bank of Russia; (iii) a bank accounting component, which will support the modernization of accounting standards and practices; and (iv) a Project Implementation Unit component." (World Bank, 1994b: 17) After a few brief remarks we will concentrate on the first component as it is 95 percent of the total project as well as most of the technical assistance. Both the second and third components (1 percent and 2 percent of the total project, respectively) will likely have widespread repercussions. While it could be argued that they are necessary for the success of the first component and thus should be included in the quantitative analysis, it could just as easily be argued that these components have benefits that extend far beyond those of increasing the likelihood of success of the first component. Given the difficulty of measuring these externalities, it may be better to omit the costs. The last component - the establishment of a Project Implementation Unit - should be included in the quantitative analysis of the first component, assuming such an effort is feasible.

The commercial bank component is divided about 1/3 for institutional strengthening technical assistance, mostly delivered by twinning, and 2/3 for systems modernization and automation. Note that the proceeds from the Bank and EBRD was on-lent to participating banks at a rate on average 2.5 percent above the Ministry of Finance's weighted cost of the two loans. While a good analysis of the qualitative lessons learned from past experience was used in project design, there was no attempt at any sort of quantitative analysis.<sup>3 8</sup> A quantitative analysis could have taken the fairly simple approach in estimating the expected return to the Ministry of Finance, which is ultimately liable for the loan. Based on past experience in other transition countries, it might have been possible to establish an expected default rate.

A preferable approach would have been to look at the expected return to the commercial banks themselves. Given that the institutions receiving the "secondary" loans are profit-making institutions, it could be argued that they would not accept such a loan unless its expected return was higher than the cost of the loan. However, about 19 percent of the total project is in the form 38 In the 40 loans that we reviewed, the section on lessons from past experience was one of the strongest in this project.

27
of grants, suggesting that a more sophisticated analysis was necessary. The quantitative experience of other countries could have been used as a basis for estimating the rate of return to individual banks.

As the TA part of the loan is crucial for its success, it would have to be included in any quantitative analysis. In fact, commercial banks are not eligible for a loan for systems modernization and automation until they successfully go through the ID stage.

Finally, while it is true that there are very large externalities to the development of a modem commercial banking system, this does not necessarily rule out the use of a rate of return centered on the more explicit costs and benefits. All it suggests is that a rate of return less than normal may be acceptable. Moreover, the minimum acceptable level is also likely to be dependent on the expected fiscal impact of the project.

# **Part IV: Conclusions and Recommendations**

Effective delivery of technical assistance continues to be a challenge, one which is only likely to increase if TA becomes more and more concentrated in sub-Saharan Africa and the transition economies in Eastern Europe, Central Asia, and Southeast Asia. In this document we began with a review of the most important criticisms of TA projects. Then we focused on the appraisal process. Our primary objective was to demonstrate the possibility of greater use of quantitative economic analysis in the appraisal process. In the last section we looked briefly at 40 ECA projects with significant TA, following this up with four case studies.

Our main conclusion is that there is a wide scope for the greater use of quantitative methodologies in the analysis of TA projects. While some of this analysis could be sophisticated and quite demanding, a large portion could be done using fairly simple techniques. In the ECA region we crudely estimated that 60 percent of the TA had a high potential for quantitative analysis. Moreover, the loans in the case studies all appeared to be good candidates for such, even though only the Hungary Tax Administration project was selected on that criteria. We recommend that all TA be classified according to the methodology suitable for its appraisal and evaluation. In cases where little or no quantitative analysis is used, the staff officer should provide a justification. In most such cases it will still be possible to conceptualize the problem in economic terms. We recommend that this procedure also be followed.

In our analysis we roughly estimated that 80 percent of the TA in investment projects was not necessary for the implementation of the project. In the 12 investnent projects for which an economic rate of return was calculated in the appraisal, the TA was never included in the estimate, despite the fact that we classified 53 percent of the TA as necessary for implementation of these projects. While essential TA should be included in the calculation of the economic rate of return, we also recommend that either a strong justification is given for the inclusion of nonessential TA or it should be omitted from the project. Our particular concern is that, given the small portion of the total loan of such TA and its non-essential nature, it will not be given sufficient attention in project implementation and monitoring.

28
We also noted that all projects should be screened on a number of important criteria before any in-depth (and costly) analysis begins. In this regard we recommended that for project development to continue past a preliminary phase, there should be: (i) strong recipient interest and commitment; (ii) a clear rationale why the project or the implementing agency is in the public sector whenever that is the case; (iii) in countries or project types with high failure rates, the staff officer should give strong arguments as to why the new project is a "special case"; (iv) approval from a designated expert in the Bank, especially in the case of institutional development projects; (v) review by staff in the Economic Development Institute of the World Bank in the case of training projects; (vi) external independent expert opinion when no quantitative analysis is feasible; (vii) strong justification for the inclusion of a TA component in an investmnent project; and (viii) conceptualization of the project in economic terms and the clear linking of this framework to performance and impact indicators. In addition, the renewed emphasis on accountability within the Bank is particularly important in the case of TA, given its overall less than average performance.

It was noted in the paper that greater use of (quantitative) economic analysis will likely increase the staff time of a project. On the other hand, more transparent screening of projects along the lines just noted could save considerable time. While the establishment and maintenance of a network of institutional networks would require substantial resources, the Bank has already decided to dedicate resources in this direction through the creation of a number of new networks of institutional expertise, the first of which is the human development network. Time of operational staff should be reduced by running project proposals through the designated experts.

Finally, we would like to conclude with a caveat. Some or many of the observations in this document may be the result of the naivete of the outsider looking in. Our last recommendation is that this study go one step further and discuss with task managers in charge of project appraisal the reason for the lack of quantitative analysis.

29