metadata: authors: 0: author: Recanatini, Francesca
1: author: Wallsten, Scott J.
2: author: Xu,L. C
docna: 0: docna: Surveying surveys and questioning questions
            - learning from World Bank experience
docty: Policy Research Working Paper
subtopic: Business Environment,Construction Industry,Common Carriers Industry,Food & Beverage Industry,General Manufacturing,Pulp & Paper Industry,Plastics & Rubber Industry,Labor Markets,State Owned Enterprise Reform,Public Sector Administrative & Civil
            Service Reform,Business Cycles and Stabilization Policies,Democratic Government,Rural Labor Markets,Economics and Finance of Public
            Institution Development,Public Sector Administrative and Civil
            Service Reform,De Facto Governments
teratopic: Finance and Financial Sector Development
abstracts: cdata!: The World Bank has increasingly focused
            on firm-level surveys to build the data foundation needed
            for accurate policy analysis in developing and transition
            economies. The authors take stock of some recent Bank
            surveys, and discuss how to improve their results. Lessons
            on data issues, and hypothesis testing: 1) Use panel data,
            if possible. 2) Have enough information about productivity
            to estimate a production function. 3) Avoid the paradigm of
            "list the severity of the obstacle/problem on a scale
            of 1 to 5". Instead, ask for data on specific
            dimensions of the problem that will shed light on
            alternative hypothesis and policy recommendations. 4) Pick
            particular disaggregated industries, and sample those
            industries in each survey. 5) Identify the most important
            interventions of interest, and consider how you will
            empirically identify specific changes by picking instruments
            useful for doing so. Lessons on questionnaire design: a)
            Incorporate only one idea or dimension in each question. Do
            not ask, in one question, about the "quality,
            integrity, and efficiency" of services, for example. b)
            Consider the costs and benefits of numeric scales compared
            with adjectival scales. Scales in which each point is
            labeled may be more precise than numeric scales in which
            only the end points are labeled. But responses are very
            sensitive to the exact adjective chosen, and it may be
            impossible to translate adjectives precisely across
            languages, making it impossible to compare responses across
            countries. c) Recognize that the share of respondents
            expressing opinions will be biased upward if the survey does
            not include a middle ("indifferent" or
            "don't know") category, and downward if it
            does include the middle category. d) When asking
            degree-of-concern and how-great-an-obstacle question,
            consider first asking a filter question (such as "Do
            you believe this regulation is an obstacle or not?").
            If the answer is yes, then ask how severe the obstacle is.
            e) Be aware of the effects of context. The act of asking
            questions can affect the answers given on subsequent,
            related questions. f) Think carefully about how to ask
            sensitive questions. Consider using a self-administered
            module for sensitive questions. alternatively, a randomized
            response mechanisms may be a useful, truth-revealing mechanism.
repnme: repnme: Surveying surveys and questioning questions
            - learning from World Bank experience
display_title: Surveying surveys and questioning
            questions - learning from World Bank experience
listing_relative_url: /research/2000/03/437914/surveying-surveys-questioning-questions-learning-world-bank-experience
topicv3: State Owned Enterprises m1326323 721,Labor Markets m1326488 855,Manufacturing Sectors m1326521 1281,Jobs m1503931 2670,Business Environment m1586411 2530
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Finance and Financial Sector Development
origu: Development Research Group (DECRG)
no_of_pages: 50
model_extractions: POLICY RESEARCH WORKING PAPER 2307

# Summary findings

The World Bank has increasingly focused on firm-level surveys to build the data foundation needed for accurate policy analysis in developing and transition economies. Recanatini, Wallsten, and Xu take stock of some recent Bank surveys and discuss how to improve their results. Lessons on data issues and hypothesis testing:

* Use panel data, if possible.

* Have enough information about productivity to estimate a production function.

* Avoid the paradigm of "list the severity of the obstacle/problem on a scale of 1 to S." Instead, ask for data on specific dimensions of the problem that will shed light on alternative hypotheses and policy recommendations.

* Pick particular disaggregated industries and sample those industries in each survey.

- Identify the most important policy interventions of interest and consider how you will empirically identify specific changes by picking instruments useful for doing so.

Lessons on questionnaire design:

* Incorporate only one idea or dimension in each question. Do not ask, in one question, about the "quality, integrity, and efficiency" of services, for example.

* Consider the costs and benefits of numeric scales compared with adjectival scales. Scales in which each point is labeled may be more precise than numeric scales in which only the endpoints are labeled. But responses are very sensitive to the exact adjective chosen and it may be impossible to translate adjectives precisely across languages, making it impossible to compare responses across countries.

* Recognize that the share of respondents expressing opinions will be biased upward if the survey does not include a middle ("indifferent" or "don't know") category and downward if it does include the middle category.

* When asking degree-of-concern and how-great-anobstacle questions, consider first asking a filter question (such as "Do you believe this regulation is an obstacle or not?"). If the answer is yes, then ask how severe the obstacle is.

* Be aware of the effects of context. The act of asking questions can affect the answers given on subsequent, related questions.

- Think carefully about how to ask sensitive questions. Consider using a self-administered module for sensitive questions. Alternatively, a randomized response mechanism may be a useful, truth-revealing mechanism.

- - This paper a product of Regulation and Competition Policy, Development Research Group is part of a larger effort in the group to develop consistent cross-country firm level surveys. Copies of the paper are available free from rhe World Bank, 1818 H Street NW, Washington, DC 20433. Please contact Paulina Sintim-Aboagye, room MC3-422, telephone 202473-7644, fax 202-522-1155, email address psintimaboagye@worldbank.org. Policy Research Working Papers are also posted on the Web at www.worldbank.org/research/workingpapers. The authors may be contacted at frecanatini@worldbank.org, wallsten@leland.stanford.edu, or Ixul@worldbank.org. March 2000. (44 pages)

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the view of the World Bank, its Executive Directors, or the countries they represent._ Produced by the Policy Research Dissemination Center
# SURVEYING SURVEYS AND QUESTIONING QUESTIONS: LEARNING FROM THE WORLD BANK EXPERIENCE'

**Francesca Recanatini, Scott J. Wallsten, and Lixin Colin Xu**<sup>**2**</sup>

We would like to thank Philip Keefer, Howard Pack, Mary Shirley and seminar participants at the World Bank for their coniments. Any remaining errors are our own.

**2** The World Bank, 1818 H Street, NW Washington DC, 20433; and the Stanford Institute for Economic Policy Research, 579 Serra Mall at Galvez Street, Stanford, CA, 94305; email: frecanatini@worldbank.org; wallsten@,leland.stanford.edu; Lxul@worldbank.org.

1
I
|I.INTRODUCTION**.....................................................**<br>**2**|
|---|
|II.GENERAL DATA ISSUES AND THE BANK'S EXPERIENCE .....................................................<br>3|
|III. IDEAL DATA VS. EXISTING DATA .....................................................<br>6|
|A. The Firm .....................................................<br>7|
|B. Human Capital .....................................................<br>9|
|C. Technology ....................................................<br>10|
|D. Market Structure .....................................................<br>11|
|E. Transaction Environment ....................................................<br>14|
|F. The Role of the State ....................................................<br>16|
|G. Beyond the firm: "The micro-foundations of macroeconomics" ........................ 19|
|IV. QUESTIONING QUESTIONS ....................................................<br>24|
|General Issues ....................................................<br>24|
|Response scales ....................................................<br>26|
|The "Don't Know" Problem ....................................................<br>28|
|Filters and Branching ....................................................<br>29|
|Order Effects ....................................................<br>31|
|Specific and general questions ....................................................<br>31|
|Recent and less-recent behavior ....................................................<br>32|
|Within-question order effects ....................................................<br>33|
|Sensitive Questions and Truth-Revealing Mechanisms ........................................ 33|
|V.CONCLUSIONS<br>....................................................<br>35|

# **I. INTRODUCTION**

Industrialized nations make rigorous data-collection the foundation of their policy analysis. The paucity of data on firms in developing countries, however, makes policy analysis in those countries more difficult, The World Bank has increasingly focused on firmn-level surveys to help build a data foundation in developing countries and transition economies. The most extensive firn surveys implemented by the World Bank include the Regional Program on Enterprise Development (RPED) survey in 8 African countries, the Industrial Competitiveness Study (ICS) in East Asia, a series of surveys in transition countries, and many surveys on small and medium enterprise (SME) issues in South Asia and South America (see Table 1 for a partial list of World Bank firm-level surveys). This paper takes stock of recent World Bank firm surveys, discussing what we have learned from them and how we could more consistently and efficiently gather data for policy analysis.

2
World Bank surveys are generally done in a decentralized manner, making it difficult to synthesize the lessons that have emerged from these ambitious and often expensive efforts. Still, as the Bank continues its survey efforts, it is worth investigating what we have learned from its experiences to date. We first present a brief overview of World Bank surveys organized by research topic, including technology, incentives, market structure, transaction environment, the role of the state, and the importance of micro data in understanding macro phenomena. We note that to analyze the effects of policy intervention it is often important to collect panel data and to define industries at sufficiently disaggregated levels. Building on our "survey of surveys," we review the literature on survey design, highlighting the point that questions themselves can bias responses and provide some guidelines for recognizing the direction of this bias and minimizing it when possible.

While we address only firm surveys conducted by the World Bank, we believe that this discussion is also relevant to policy analysts outside the Bank. The World Bank has probably conducted more surveys in developing countries than any other single institution. As a result of the large number of surveys and survey topics, the Bank now possesses enormous institutional knowledge not just from the data collected, but from the survey experiences themselves. The Bank should share this knowledge with its member countries. This paper represents such an attempt.<sup>3</sup>

# **II. GENERAL DATA ISSUES AND THE BANK'S EXPERIENCE**

Rigorous policy analysis requires a great deal of data. Ideally, the data are longitudinal (i.e., firm level information collected at discrete points over time) and fairly dissagregated (i.e., industries are defined as narrowly as possible). Only with panel data can one investigate the effects of policy interventions and the process of economic growth. Consider, for example, the

> ' See Blank and Grosh (1999) for a complementary paper on household surveys. They suggest that political will, appropriate resources and careful planning are key requirements for developing analytical capacity. With these prerequisites in place, policy-makers must concentrate on four areas to strengthen the analytical capacity of their country: appropriate training of policy analysts; use of external technical assistance to work closely with local counterparts; establishment of fully staffed policy analysis units; and direct participation of staff members in the development and implementation of the research agenda. Our paper builds on Blank and Grosh by assuming the will and resources are in place, and discusses the best methods for gathering the data.

3
case of new regulations and firn entry and exit. Regulations may affect firm entry and exit in complicated ways, They may impose entry barriers to new finns and impose exit barriers to inefficient (or state-owned) firms. Av "snapshot" of firms at a given point in time (a crosssection) provides limited information on this matter. Only by surveying firms over time and across countries can we begin to understand how new regulations or other policy interventions affect firm entry and exit.

Entry and exit also lead to attrition bias in the sample-another reason to obtain panel data. Properly accounting for both entry and exit of firms is therefore important to understand industry evolution. Attrition bias may arise when firms disappear (exit) because of mergers, acquisition, consolidation, or bankruptcy, or when a firm is included in the sample only if it survives up to the point of interview. Without accounting for attrition, the sole reliance on surviving firms will inevitably bias (usually upward) performance measurements. The additional survey rounds necessary for constructing panel data can help account for this bias by deterrnining which firms have disappeared and why. In addition, panel data can help better understand firn entry.<sup>4</sup> New entrants may differ significantly from surviving firnms. Moreover, a high death rate of firms is not necessarily worrisome if it is accompanied by a high birth rate. To deal with such issues, firm surveys should resample some firms annually and strive to add new entrants.

Surveys should also define industries as precisely as possible and offer detailed informnation of the industry analyzed. When this type of micro data is collected in panel data sets, it becomes possible to compare the technological progress of particular industries in different countries. Such data allow us to better address important unanswered questions, such as why inter-firm dispersion in productivity is greater in industrialized economies than in developing countries (as suggested in Tybout, 1998). A variety of reasons (such as a smaller extent of market and excessive regulation) have been suggested. But there is no consensus among the existing empirical work largely because every data set aggregates industries differently, making comparisons nearly impossible (Tybout, 1998).

> 4 Finding new entrants, however, clearly makes their inclusion easier said than done.

4
Table 1 offers a partial list of _some recent surveys_ done within the Bank. The table reveals four main types of surveys: (1) Comprehensive surveys (RPED, ICS); (2) Surveys on transition (Russia: Economic and Civil Society; Corporate Governance in Poland, Hungary, and Czech Republic; Large Scale Privatization in Mongolia; The Emergence of Private Sector Manufacturing in Poland, in Hungary; Industrial Enterprises and Adjustment in Russia; Private Service Firms in Russia-St. Petersburg.); (3) Surveys of the market environment, including regulation and govemment-industry relations (World Business Environment Survey); and (4) Topic-specific surveys, such as on training (Enterprise Training in Developing Countries, Tan and Batra), or on SME problems (CECPS, "Small-and-Medium Industry Impact Evaluation," 1995; and "Enterprise training in developing countries," Tan and Batra).

How well do World Bank surveys meet the data needs outlined above? We begin to address that question below. One lesson from the Bank experience is that surveys are extremely difficult. Thus, while we make the comments below with the ideal data set in mind, we recognize that in practice surveys may not be able to meet this ideal. Longitudinal surveys are costly and take too long for many Bank operations. Some data, meanwhile, simply do not exist. Nonetheless, it is instructive to consider what would have made these surveys more useful, and use that knowledge when designing future surveys.

**1. Data consistency:** Although World Bank surveys are usually designed to address a particular issue, each could be more broadly useful by consistently including a small set of standard questions on firm performance, such as total output and profitability. The survey should also collect price data on the firm's main products, inputs, and investment goods. In addition, the survey becomes more valuable when it contains enough information to estimate production functions, including value added, number of employees, capital stock, and material inputs. This information would help to isolate the contribution to productivity changes of various production, technological, and institutional factors.<sup>5</sup> Only a few surveys, such as RPED and ICS, include consistent productivity measures.

5Consider the Bank's general interest in Small and Medium-sized Enterprises (SMEs). To understand if SMEs are as efficient as other firms, we must hold constant capital and labor and then compare productivity to that of different types of firms.

5
Similarly, to investigate financial constraints, we need to know not only whether some types of firms (by ownership, size, or the owner's gender) had more difficulty obtaining external finance, but also the consequences of the lack of funds. Estimating production functions would allow us to learn whether, holding other characteristics constant, financially-constrained firms are indeed less productive. If such firns are not less productive, then there is less need to worry about financial constraints.

**2. Importance of panel data:** Most Bank surveys are cross-sectional, leaving the need for panel data unfulfilled (again, with some exceptions, notably, RPED and ICS). As a consequence, we have much less confidence in inferring whether productivity differences reflect unmeasured firm heterogeneity or the variable of interest.

3. **Need to address endogeneity:** Surveys can be most effective when they deal explicitly with the endogeneity of policy interventions. Many researchers find that _ex post_ it is difficult to demonstrate any causal effects because they lack of good instrumental variablesthose variables uncorrelated with the error term but reasonably good predictors of the endogenous variables. It will likely be too late to derive instrumental variables at the data analysis stage. Instead, at the survey design stage researchers should consider the endogenous policy variables of interest and think about potential instruments to identify those variables.

# III. IDEAL DATA VS. EXISTING DATA

In each of the following subsections we give a brief overview of some questions that have yet to be empirically investigated in the context of developing countries, the data that would allow hypothesis testing, and an evaluation of whether existing World Bank survey have addressed these issues. In particular, we discuss research topics such as the firm itself, market structure, the business environment and its impact on firmn's performance, the role of the state, and analysis of the macro-economy through microeconomic data.

6
A. **The Firm**

# **Corporate governance6**

The fast growth of the corporate sector throughout the world makes corporate governance increasingly important. However, we have an especially poor understanding of corporate finance in developing countries, largely because of the lack of firm-level data. For instance, what are the ownership patterns in the developing countries? How are firms financed? Do firms face financial constraints? How well are creditors and shareholders protected in these countries? The answers to these illustrative questions can have important policy implications. The recent East Asian Crisis, for example, has been attributed to the poor protection of minority shareholders rights. The policy implications of these questions call for the collection of such data.

Bank surveys ask many good questions about corporate governance, although the coverage varies greatly. Consider the contrast between what theory suggests we should measure and what Bank surveys actually collect:

1. Theory suggests that competition in the managerial labor markets and reputation concerns force managers to perform well for future career prospects, making it important to measure aspects of managerial labor market, such as turnover, performnance criteria, rewards and punishment. Such questions are hard to find in Bank surveys. _Suggested questions:_ What determines managerial pay (current profits, stock options, relative performance, regulators' evaluations, etc)? If profits increase by one percent, what percentage increase does the managerial wage? Has a manager ever been removed because of poor performance? Can shareholders replace the CEO when performance is bad? What was the worst performance in the past x years? What consequences did the then-CEO suffer (income losses, demotion, or others)?

2. A firn's board of directors can be an important disciplinary force. Theory suggests that because of the free-riding problem associated with monitoring by shareholders, concentrated ownership may emerge to internalize the benefits and costs of monitoring. Thus it is important to uncover the ownership stake of the largest shareholders. Most Bank surveys ask about ownership type, but ask few questions beyond that. _Suggested questions:_ What is the

- **6** This subsection draws heavily on Shleifer and Vishny (1996).

7
composition of the board of directors? What percentage of board members represents management? Major owners (more than 5% shares)? Small shareholders?

3. Theory suggests that debt and equity contracts give shareholders and especially creditors control rights to intervene when a contract is violated. Bank surveys, however, in general do not provide information on the financial structure and the rights of stakeholders. _Suggested questions: (1)_ What is the share of outside equity? Share of inside equity? Share of debt financing? Debt-equity ratio? What type of institutions or individuals are the main shareholders (bank, pension funds, a holding company, a foreign investor, or a domestic investor)? The main creditors? (2) What are the control rights of the equity-holder, of the debt-holder? Would a firm become bankrupt if it defaults interest payment for debts? Does the firm post collateral when borrowing from banks? What is the share of tangible assets? What are the shares of long- and short-term debts in total debts respectively''

# **Internal Incentives**

Internal incentives help align managerial and employee objectives (see Holmstrom and Tirole, 1989, for a survey). There are many ways to motivate workers:

- Promotion, In theory, a large wage differential between workers at different levels improves performance by spurring employees to work hard in order to advance to the next layer, but this benefit has to be balanced against the loss of control and of information. The tournament literature suggests measuring the number of layers within the firm and wage differential between production workers and managers (Lazear and Rosen, 1981; Nalebuff and Stiglitz, 1983; Rosen, 1986).

- The threat of firing. Presumably, internal incentives are stronger when involuntary turnover is a real and credible threat. This may be achieved by allowing a certain percentage of contract workers. A high share of temporary workers, however, may have unintended consequences: The threat cf firing as a disciplinary option spurs worker incentives, but temporary workers face reduced incentives to accumulate firm-specific humlan capital. Since the net effect is unclear, surveys will provide the best source of data to answer this question and provide policy advice.

8
- Linking a worker's pay to firm performance. Supposedly a close link should increase employee incentives.

Understand internal incentives may help policy makers provide better advice on private sector reforms. For instance, suppose surveys reveal little correlation between pay and firm performance in a particular country. Such a finding may imply that the labor force is not properly motivated, perhaps due to poor performance measurement or the many layers of hierarchy. Alternatively, if all employees perceive their jobs as permanent, increasing the share of contract workers may be a valid option to motivate workers.

Many World Bank surveys do not measure internal incentives at all. Some have a minimum number of measures, such as the percentage of work being contracted out, employment benefits, bonuses, payment method (e.g., piece rate or hourly rate), total wages, non-wage costs, and labor turnover rates. The RPED covers internal incentives most extensively. It asks many useful questions, including the payment scheme (piece rate, time rate, or based on firms' sales), the number of people fired, the presence of a union, overtime pay, and detailed wage data by worker category. An especially innovative feature of the RPED is its survey of a subsample of workers in the surveyed firms, which asked workers about issues such as bonuses, wages, and housing allowances.

Despite the strong advances made by the RPED, Bank surveys leave many questions unexplored. For example, we still know little about pay differences between different levels of workers, and between managers and production workers. Likewise, in few countries and industries do we have any sense of the involuntary turnover rate. Without such data we cannot understand the effects and determinants of firm internal incentives, and whether performance improves with better internal incentives in different countries.

# **B. Human Capital**

Human capital theory emphasizes the importance of schooling, labor market experience, seniority, and training in explaining individual productivity (Willis 1986). In the context of firms we should then expect these human capital variables to positively impact productivity. The lack of data has thus far prevented us from learning how education and training affect 9
productivity and earning in countries with different macro environment and protection of property rights.

Another important economic question is how to finance general and specific human capital investments in firms (Willis 1986, Becker 1962, Oi 1962). General human capital raises a worker's productivity regardless of whether she stays at one firm or moves to another. Specific human capital, on the other hand, is lost if she moves to another firm. Efficiency dictates that the employer should sho-ulder much of the costs for specific training, since the worker could not benefit from the training if she moves to another firm. Likewise, there may be little reason for the employer to pay for general training, since the worker could benefit from it even if she moves to another firm. In this case, the worker may pay for the training since she benefits from it wherever she works. It also makes sense for the government to subsidize general training since the economy benefits from more productive workers wherever in the country they work. We have little empirical evidence, however, about the relationship of productivity to labor market experience and how training is financed in different countries. The finding would have policy implications for how governments and firms should approach worker training.

Many Bank surveys cover human capital issues quite well (for instance, RPED and ICS). Most surveys contain the primary measures, although they tend not to explore the distribution of workers in term of labor market experience and seniority structure, and quit rates. Such questions can be important. The turnover rate, for example, can provide an indication of how vigorous a labor market is in one developing country compared to another, or cornpared to industrialized nations. Quit rates may indicate the importance of match-specific human capital and the vitality of a country's labor market (Tybout, 1998). Involuntary turnover rates could indicate the extent to which workers are disciplined or how firms adjust employnient in response to changes in economic conditions.

# **C. Technology**

In the long run, technological progress is largely responsible for growth. It is therefore important to measure it in developing countries. But what determines technological progress and how can we measure it? These questions are exceedingly difficult to answer, even for large firms in industrialized countries. A common input necessary for technological progress is research and 10
development (R&D) expenditures. However, few firms in developing countries categorize any expenditure as R&D, meaning we must look for altemative measures. One determinant is market structure (discussed in the following section), since it directly impacts technological progress.<sup>7</sup> Other aspects of technology can be proxied by equipment vintage, the number of technical personnel and their wages, and purchased foreign inputs and equipment. Sources and vintage of equipment proxy for technology under the assumptions that newer equipment represents a higher level of technology, and that imported equipment is more advanced. In addition, technological know-how is embodied in technical personnel. A higher percentage of the work force with technical knowledge-engineers and employees with college degrees, for example-represents a more advanced state of technology. Finally, the development literature sometimes considers the relationship between purchased foreign inputs and productivity to be an indicator of the firm's level of technology (Tybout, 1998).

Few Bank surveys have attempted to measure technology. In addition, most of these data sets are only cross-sectional, making it difficult to measure technological change. Some surveys contain a minimum amount of information about technology, including questions on the percentage of inputs imported from different foreign countries. The _RPED_ and the _ICS_ surveys serve as nice guides for future firm surveys on technology. These surveys gather data on licensing fees, the presence and amount of foreign technical assistance, the number of expatriates, equipment vintage, level of new investment, imported equipment, the number of scientists and engineers, R&D expenditures, types of R&D, new product or process adoption, and whether the firm sells any technology.

# **D. Market Structure**

Market structure is one of most important determinant of firm behavior and performance. It thus is useful to compare the market structure across countries in the _same_ industries to explore whether competition leads to faster innovation and superior performance. In addition to the 7Scherer and Ross (1990, p. 660) note that "What is needed for rapid technological progress is a subtle blend of competition and monopoly, with more emphasis on the former than the latter, and with the role of monopolistic elements diminishing when rich technological opportunities exist." 11
extent of the market, economists have also focused on concentration, the scope and the extent of the firm (Viscusi, Vernon, And Harrington, Jr., 1990).

**Concentration.** Two hypotheses about the implication of concentration lead to dramatically different policy recommendations. The first hypothesis is collusion. Viscusi, Vernon, and Harrington (1990) note that "the more concentrated an industry is, the less competitive are firms and thus the higher is the price-cost margin." In this case, it is desirable to break up highly concentrated industries. The second hypothesis is Demsetz's superior efficiency hypothesis. That is, superior firms have both higher market shares and larger price-cost margins. Thus, at the firm level, both profitability and price-cost ratios are positively correlated with market shares, and the relationship may be present in a weaker form in the industry level.<sup>8</sup> In this case, the relationship is association, but not causation. The policy implication then is not to break up concentrated industries; the government, after all, does not want to punish firms with superior technology.

The empirical evidence from industrialized countries so far tended to support Demsetz's hypothesis. The evidence is that a firm's profit is strongly correlated with its market share, and the positive association is still observed at the industry level, but in a weaker form (Salinger, 1990; Viscusi, Vernon, And Harrington, Jr., 1990). But what is the evidence in developing countries? To learn the answers and thus give useful policy advice, we need to collect information about concentration ratio, profitability, price/cost ratio, entry barriers as characterized by license fees to entry, sunk costs (the percentage of the value of equipment can be recouped if the firm is to quit), and the existence of exclusive patents.<sup>9</sup>

**The scope of the firm.** The scope of the firm is characterized by its mix of activities (Viscusi, et al., 1990). Several issues of interest to the Bank and to development more generally may be related to fimn scope. For example, firms may be more likely to be vertically integrated in countries with weak contract enforcement. Whether this is true is, again, an empirical question requiring much data to answer. Few World Bank surveys have paid much attention to

> 8 Weaker because one has to average profitability and price-costs ratios with those of the competitors, for which the profitability and price-cost ratios are lower

> **9** This measurement has some problems. It is hard to measure costs, especially because it is subject to accounting methods and may be easily manipulated.

12
this matter. To address this question surveys should ask about the firm's lines of business (defined as precisely as possible), whether the firm is a conglomerate (i.e., engaged in unrelated activities), the extent of vertical integration (i.e., produces its main inputs or has its own distribution channels), and whether the firm has any subsidiaries. With these measures, researchers can link productivity with the scope of the firm to see how firm scope changes under different institutional environments.

**The extent of markets.** Theory suggests that the extent of markets may fundamentally constrain firn development. Some believe that a small market, for instance, may help explain the so-called "missing middle" phenomenon in developing countries-the existence of some large firms and many small firms, but very few middle-sized firms, which are common in industrialized countries. Small markets have also been suggested as a barrier to technological change. Existing Bank surveys tend to neglect this issue. Many surveys can become more useful by asking certain questions about the extent of market: Where firms sell their products, the distance to a major city, and the firm's access to all types of transportation.

Two Bank surveys have made impressive efforts to explore market structure. One, the _RPED,_ asks two especially useful questions: (1) What characterizes a firm's competitors? (None, domestic firms, foreign competitor, or imports). (2) How does the firm set its prices? (Marketprice-taker, markup over costs, in line with imports, price discriminate in different markets, price increases with quality, follow largest competitors, government-set, negotiate-with-the-buyer, or set by a business association). Although the category of "domestic firms" in the first question is rather broad, the second question is especially rich.

The second, _Russia.- Economic and Civil Society,_ asks detailed information about market structure:

- How many Russian enterprises the respondent considers to be direct competitors.

- Whether the respondent's firm's most important products compete with imported products from other countries listed.

- Whether the firm is classified as a monopoly producer.

- The ownership structure of input suppliers.

- Whether distribution channels offer credit assistance.

13
- How the firm sets its prices (e.g., based on production costs, competitors' prices, the target profit level, or willingness-to-pay by the consumers) and how it adjusts price for inflation (e.g., an index based on inflation, own costs, or dollars and exchange rates).

# **E. Transaction Environment**

According to transaction costs economics (Williamson 1989, 1993), incomplete information and the potential for opportunistic behavior prevent contracts from specifying outcomes under every uncertain contingency. Firms thus establish various types of relationships -including using contracts-with suppliers, dealers and customers. Maintaining many types of relationships entails distinct benefits and costs for firms. These myriad relationships require time and productive resources, but allow degrees of specialization and different levels of investments. Researchers therefore must study contractual arrangements that facilitate adaptation and dispute settlement such as the legal system, arbitration, long-term contract, and vertical integration.

Transaction cost economics suggest that to better understand the impact of these costs on firm's performance we should have at least the following information about the principal characteristics of firm's exchanges (Williamson 1989): (1) The frequency with which they occur, (2) The degree and type of uncertainty to which they are subject, and (3) The condition of asset specificity.

When asselt specificity is important, theory suggests that the transacting parties will arrange transaction structures-for instance, vertical integration, merger, long-term contracts-to economize transaction costs. Existing surveys pay only limited attention to such transaction characteristics. Future Bank surveys may become more valuable when this typie of information is collected.

An important determinant of transaction costs is the business and legal environment in which the firn operates. This set of considerations is especially important for transition economies and developing countries, where the legal and business environment is in flux and may seriously impact the firm's decisions. To investigate these issues, surveys should collect detailed information on the firns' exchanges. For example, what are the costs of establishing new trading relationships when old ones are broken? Is the trade between the parties bilateral?

14
Do the parties rely on legal mechanisms to enforce their contracts or on alternative enforcement devices? If a contract dispute arises, does the contract stipulate an arbitration mechanism? Does the contract require "hostages" for deterring opportunistic behavior, such as collateral, and down payments? What devices do the parties involved in the exchange employ to economize on transaction costs? Are organizational forms and the scope of the firm affected by transaction costs considerations-for instance, because of contract enforcement problem, are family firms over-represented? Does the firm become more vertically-integrated to reduce contracting costs?

Bank surveys have collected a significant amount of data on these issues, especially in recent years. Four major surveys-RPED, ICS, Bosnia, and _Survey of Russian Enterprises-all_ gather detailed information on the type of relationships firms had and the devices used to regulate transactions. Together, these surveys yield a rich picture of the enviromnent in which firms operate. The following extracts from existing bank surveys may serve as nice recipes for future surveys on transaction environments:

- The nature of contract: Do contract specifications cover price, methods of adjusting price to rate of inflation, quantity, quality, delivery date, method of payment, warranty, penalty for late payment, penalties for breach of contract, and the method of resolution if a dispute arises? (RPED Survey and the survey of Russian Enterprises).

- The contractual relations with suppliers, with subcontractors, and with clients: ownership, payment method, frequency of purchase, relationship with the firm, whether the other party comes from the same ethnic group, whether there is a trade discount, whether there is subcontracting, the amount of credit purchase, interest, guarantee to supplier (none, physical collateral, third-party checks, witnesses, signed invoices, group guarantees); whether using written contracts more frequently with new trading partners than old trading partners; whether using written contracts more frequently with new private enterprises than other new trading partners? (The survey of Russian Enterprises).

- Informal lending: the amount, maturity, the existence and amount of collateral, and provisions in case of default? (RPED survey)

The data gathered through these surveys has allowed researchers to better understand the role of contracts and legal institutions on firm's performance. In particular, Hendley, Murrell, and Ryterman (1999) focus on the importance of laws and legal institutions for Russian enterprises 15
by asking how laws facilitate transactions. They examine the significance of legal institutions in the transacting process. Their analysis suggests that firms that invest in constructing contracts and have better knowledge of the law have better chances to successfully complete a transaction. To illustrate the potential advances in research on transaction costs, consider the example of credit access. The World Bank's efforts to collect data on this particular type of costs has already allowed researchers to explore more closely the theorized link between credit access and firm performance. Fan, Lee and Schaffer (1996), for example, analyze the impact of the financial sector reform in Russia on firms in the early years of transition. Their data, collected at the end of 1994, shows that a partial reformn of the banking system, with many banks still acting "soft" in their dealings with firms, slowed the enterprise restructuring process amd undermined financial sector reform. Bank loans are not the firm's only source of financing, especially when credit is rationed or the credit market is underdeveloped. A recent study on Zimbabwean manufacturing firms using the RPED data (Fafchamps, 1996) suggests that trade credit is an important, alternative source of short-term liquidity for many firms.

# F. **The Role of the State**

Even in industrialized countries, the state is generally the most importaint single actor in the economy. The policies the government implements can both harm and benefit economic growth. As Stiglitz (1998) notes, "there is a growing consensus that governments can play a vital role in successful development efforts, but we also recognize that the wrong kind of government intervention can be highly detrimental." Surveys of firms can therefore be an important tool in determining which aspects of government involvement are beneficial and which are harmful.

# **Regulation**

Despite its potential importance in explaining different growth rates across nations, little is known empirically about regulation in developing countries. For instance, 'what are the effects of regulation on firm performance, employment, and pricing? What determines the extent and the types of regulation that the government of a developing country adopts? Do regulations change over time? How do regulations affect firms' perception on the likelihood of 16
expropriation? Given the profound implications for firm productivity and innovations, further research should address the impacts and the deterrninants of regulation in developing countries. To accomplish this research agenda it is important to gather panel data on, for example, pricing behavior, service coverage and quality, profitability, market structure, rent distribution (i.e., were employment and wages affected by regulatory changes), and dynamic efficiency (measured by the rate and direction of innovation and productivity).

World Bank surveys probably have focused more on regulation than any other theme, although the variation in coverage is vast across surveys. Those that cover regulation thoroughly _(the Bosnia Survey, the World Business Environment Survey, The Emergence of Private Sector: Hungary,_ and the RPED) ask questions about: the waiting period for goods to arrive, the level of government to deal with in regulation, the main problems in dealing with govermnent agencies, the frequency with which firms are required to meet with government officials, the costs of facilitators necessary for dealing with the government, the burdens of licensing requirements, costs of obtaining licenses and permits, various types of taxes, tax treatments for profits reinvested in company, and incentives for investment in new machinery and equipment.

A general problem with current Bank surveys on regulation, however, is that they focus on the barriers that regulations impose on business. They tend to ask firms "how severe" certain regulations are to firrn operations. It is true that burdensome and often unnecessary regulations are common. However, many regulations are important to the functioning of the economy (e.g., environmental regulations that force firms to internalize all costs of their production). In those cases, the regulation will be an obstacle from the firm's perspective, but efficient from the perspective of the entire economy. Surveyors need to think carefully about how to uncover the true costs of regulation, since estimates will be biased upwards by simply asking the firm whether a regulation is an obstacle.

# **State-Owned Enterprises (SOEs)**

Developing countries claim a higher share of SOEs in their economies than do industrialized countries (Haggarty and Shirley, 1997). While SOEs resemble private firms in many respects, surveys of SOEs must ask some different questions. These differences reflect the different objectives of SOE management and objectives. SOE principals are typically bureaucrats or 17
politicians (whose incentives differ from private shareholders and creditors). In particular, the following are potentially important for understanding SOEs behavior in developing countries.

- Cash flow rights. What tax rate does the SOE face? Does it adjust from year to year? Does the state charge different rates for different SOEs? Since the state may use tax rate as a transfer mechanism, it is useful to see the extent to which the government uses it for this purpose. How do tax rates of SOEs differ from those of comparable private firms? Does the government subsidize or bailout SOEs when they are in financial trouble?

- Control rights. Does the government appoint the top managers? Does the government appoint the middle managers? How were managers selected (bidding, election by workers, appointment by the government)? How long is manager tenure? What is the scope of managers' authority (e.g., decisions regarding production, investment, wages, staffing, and borrowing)? Does the government often interfere in production decisions? Often in wages? In hiring and firing? In new investment? In acquisition and consolidation? In borrowing?

- Incentives. How does the government align the manager's and government's objectives? Does it use state plans, performance contracts that spell out goals and incentives, private managers, who have control rights for a fee, or private parties bidding for concessions to operate and invest in SOEs and keep the profits (World Bank, 1995)?

To illustrate the usefulness of firm-level data to yield insights about policy questions, consider studies based on a 1980s panel data set of Chinese SOEs (Li, 1997; Xu, 1998; Groves et al. 1995a, 1995b). Using these data, researchers have found that decentralizing production decisions, allowing managers to determine wages, and allowing firms to retain a larger share of their profits all improved productivity. Most importantly, increased competition, as measured by lower markup ratios, substantially increased productivity.

18
**G. Beyond the firm: "The micro-foundations of macroeconomics"1 0**

The scope of firm-level surveys goes well beyond testing theories of the firm. As many researchers have suggested, firm-level data has become key to understand the macro economy. Macroeconomic analysis concentrates mainly on understanding the determinants of output, unemployment and inflation. Researchers have thus built macro-economic variables from industry-level data under the assumption of a "representative firm" within each industry. Recent studies in developed economies, however, demonstrate that seemingly similar firms in the same industry exhibit different behavior in terms of output, investment, and employment, suggesting that aggregation may obscure important phenomena. Though these findings do not justify rejecting the representative firm assumption, they highlight how we can more completely understand changes in macro variables by analyzing these variables at the firm level.

Firm-level data is also useful for macro-analysis.since it helps test robustness by complementing aggregate, official, sources of data. Robustness tests are especially relevant for transition economies, where pre-transition data is often not reliable and the process of collection of post-transition data is still under revision. Firm-level surveys can help reconcile seemingly contradictory aggregate results by disentangling contrasting effects and offering a different perspective on the issue in question.

Next, we provide a brief overview of the most important theoretical macro-economic issues, which can benefit from this micro approach to macro-analysis, and of the (limited) empirical testing done using World Bank surveys.

# **Growth and Investment**

An important aspect of macro-economic analysis is the link between growth and investment. In particular, recent studies suggest that the rate of investment is positively linked to the rate of growth. To better understand this link, researchers have focused on the determinants of the investment decision and found that this decision depends on financial constraints, the risks firms face, and the productivity of investment itself (Jenskin, 1998; Gyimah-Brempong and Traynor, 1999). This line of research however has used mainly aggregate (country-level) data.

> **10** This section draws heavily from Haltiwanger (1997) and Campbell (1997).

19
The problem with this approach is that aggregate data fails to capture heterogeneity across firms and non-linearities in investment decisions recently highlighted by a new streamn of works (Doms and Dunne, 1994; Caballero, et al, 1995; Cooper, et al, 1995).

The problems of an "aggregate" approach to the study of investment decisions are even more evident by comparing two recent studies on investment productivity and growth in Africa. The first - Devarajan, et al (1999) - focuses on investment productivity using cross-country data. Contrary to the aforementioned theoretical predictions, the authors find that neither public nor private investment significantly impacted growth and output in Africa over the past three decades. To reconcile the empirical findings with the theoretical predictions, the authors integrate cross-country exploration with firn-level data on investment in the mianufacturing sector in Tanzania. This analysis suggests that the low level of investment can be partially attributed to the low productivity of capital for African firms. The low level of productivity of capital then explains the absence of a cross-country correlation between growth performance and investment.

The second study - Gunning and Mengistae (1999) - on the other hand reaches the opposite conclusion: the productivity of capital for African firms has increasecl over the past ten years. In particular, the authors explore the rate and the productivity of investment using firmlevel panel data collected by the RPED in eight African countries. Their detailed data set contains information on capital stock and market structure and helps document low levels of investment as well as heterogeneity in firm and investment productivity in the African manufacturing industries. They conclude that manufacturing firms are profitable and that investment productivity has improved in the past ten years.<sup>1</sup> These results, in contrast with the findings of Devarajan et al. (1999), suggest that solving the puzzle of the low level of investment and the weak link between investment and growth in Africa may require exploring more closely other determinants of investment, such as the degree of reversibility or risk.

Micro panel data on characteristics of capital stock and firm productivity could improve our understanding of the link between investment and growth. Unfortunately, this type of data is not readily available. Bank coverage has improved recently with the implementation of the three

> 1 l This result is derived by looking directly at the productivity of the firms, and indirectly at the degree of tolerance of the markets toward inefficient investment.

20
waves of the RPED surveys. But we need more detailed survey data to enhance our understanding of investment dynamics. In particular, very limited information is available on the fixed costs of investment, the degree of reversibility or on the perceived risk that firms associate to a certain investment decision.

# **Unemployment and the labor market**

Recently, researchers have investigated the high rates of job creation and job destruction experienced by market economies.'<sup>2</sup> This continuous reallocation of resources, which occurs within industries and is somewhat obscured by the aggregate measures of unemployment, is an important part of economic adjustment and growth. The smoothness of these reallocation processes impacts economic performance by affecting the evolution of firms and industries and thus productivity growth. Reallocating jobs and factor inputs from less efficient to more efficient plants may help improve industry-level productivity in an environment with frictions and imperfect infornation (Baily, Hulten and Campbell 1992, and Olley and Pakes 1996). The same theoretical approach, which allows for frictions in reallocating workers and jobs, helps also to shed light on business cycle behavior of gross worker flows (Mortensen, 1994; Ramey and Watson, 1997).

World Bank coverage of employment dynamics is limited in part because the literature itself is limited. The firm-level data necessary to test hypotheses is not yet fully available even for industrialized economies. Moreover, empirical research on this front is hampered by the lack of micro time-series data. Some notable exceptions include the East Asia Competitiveness Survey, the RPED, and the data collected by Haltiwanger and Vodopivec in Poland and Estonia (1995).

The East Asia survey collects detailed infornation on the workforce, but has limited data on average tenure. The Labor section of the RPED questionnaire focuses on the structure of the workforce, but less on the dynamic aspect of employment flows. The RPED collects informnation on seasonal hires, apprentices, paid and unpaid relatives of the owner, but mainly for the second or third waves of the survey. In addition, the usefulness of the data is limited by consistency

> 12 For excellent surveys of this literature, see Davis and Haltiwanger (1998), Mortensen (1986), Pissarides (1990), and Mortensen and Pissarides (1997).

21
problems across countries in terms of the survey instruments used. The Estonia and Poland data sets document in great detail the employment dynamics experienced by these countries between 1989 and 1995, using both official government data and information collected through a labor force survey.

# **Responses to Crises and Institutional Changes**

Following the transition from a planned to a market economy, Eastern European and fonner Soviet countries experienced a larger-than-expected decline in aggregate activities. Recent important studies have looked for causes of this output collapse. _Ex ante,_ researchers agreed that output may initially decline before resuming its growth as the market reallocated resources from less to more productive uses. Borensztein and Ostry (1995) hypothesize that the sharp fall was mainly the result of a supply side shock: as prices were liberalized, state owned enterprises were unable to pay for inputs. Commander and Coricelli (1992), and Borensztein, et al. (1993) emphasize a demand shock: the increase in prices reduced the domestic demand for final goods. Berg (1994), Atkeson and Kehoe (1996), and Shimer (1995) argue that structural adjustments related to the transition itself were the main cause of the initial decline. Finally, Murphy, et al (1992) and Shleifer and Vishny (1993) focuse on the political economy of the transition process and emphasized the perverse effects of a partial price liberalization reform and government corruption.)<sup>3</sup> The World Bank coverage of output behavior in transition economies is somewhat limited and unsystematic across countries, hampering cross-country analysis. I]n addition, the country-specific analyses are based on firm surveys designed to test specific hypotheses, preventing researchers from evaluating competing hypotheses even within the same country. Nonetheless, existing surveys have provided us with great insights into the effects of transition on output.

> 13 Estrin, Schaffer and Singh (1993) offer new insights on the structural restructuring process that naturally accompanies a change in regime for some Eastern European countries by integrating their industry-level analysis with firm-level data. Their exploration suggests that the process of industrial adjustment across a few Eastern European countries has been remarkably similar, and that the speed at which reforms are introduced affects enterprise profitability. Pinto, Belka and Krajewski (1992) complement Estrin et al. (1993)'s results with a study of the state enterprises in Poland before and after the first round of reforms. Their analysis reveals that firms, helped

22
The transition process did not simply lead to demand and supply shocks. It also led to a deep institutional restructuring. It is plausible that the transition process, by removing the existing coordinating mechanisms, led to an increase in search costs, and in turn to a decline in aggregate activities. Some Bank surveys-for example, Ickes and Ryterman (1994), Qimiao Fan (1994), De Melo (1 997)-have attempted to measure these changes. The data collected however suffer of the limitations emphasized above, focusing mainly on few of the aspects of the transition process.

Another dimension of particular interest to policy-makers is firn responses to macro shocks. Since this type of analysis has a less defined set of theoretical hypotheses and may be more specific in scope, it is difficult to evaluate the coverage of the World Bank surveys. Instead, we discuss the advantages and disadvantages of recent surveys undertaken in response to macro-economic crises.

Research on transition economies mentioned above is a natural starting point since they provide some information on how firns react to shocks or changes in the environment in which they operate. Earle, Estrin and Leshchenko (1996), for example, focus on the privatization process and its impact on firm performance. Using survey data collected in Russia in 1994, the authors analyze the effects of different ownership structures on firm behavior. Their findings suggest that outsider-owned and state-owned firms differ significantly in terms of perforrnance and but less in terms of restructuring behavior. The latter result may be partly a function of the timing of the survey (it was conducted only two years into the transition process) and, consequently, the restructuring process had made only limited progress.'<sup>4</sup> Recanatini and Ryterman (1999a and 1999b) study the reaction of firns to the institutional vacuum left by the removal of government institutions. Their analysis suggests that, because of the sudden loss of information and the underdeveloped legal system, firms reacted by by an underdeveloped banking system, initially postponed the inevitable restructuring process. The decline in real wages and the increase in interest rates appear to be therefore responsible for the collapse in output.

> 1 The timing of these surveys and their "once for all" approach seriously limit the usefulness of these data sets. Especially when focusing on macro-economic issues it is important to analyze cross-country time series. Unfortunately, most of these surveys are not repeated over time, making it difficult to test both the existing alternative theories and the validity of the survey instruments used.

23
attempting to join efforts with each other. This effort translated into the creation of new, infornmal networks, which had a positive effect on firms' performance.

Firm-level survey data has also proven useful in our understanding of the recent East Asian crisis. Detailed information collected on the crisis through surveys administered in 1997 and again in early 1998 offer good examples of how firm-level data can help studying responses to shocks in a way that complements the more traditional macro/aggregate approach. Dollar and Hallward-Driemeier (1998) focus on the adjustment process in the Thai industry following the crisis. Their work highlights the link existing between firms' behavior and (poor) government policies in the year preceding the crisis. As the crisis developed, firms struggled to cope with the existing constraints and to adjust to the new environment.

# **IV. QUESTIONING QUESTIONS**

Having discussed the type of data that would be most useful for firm surveys to gather, we now turn to the subject of questionnaire design itself. Surveys generally ask respondents both about their attitudes towards various issues and their recollection of past factual events. Moreover, in-person surveys pose these questions in a social setting, albeit one with a script and a purpose. Survey designers must realize both that the survey asks the respondent to perform a series of cognitive tasks and that it is an interaction governed by governed by social norms (Sudman, Bradburn, and Schwarz 1996). It is therefore important to consider how the questions themselves can influence responses. This section reviews the literature on questionnaire design. In particular, it seeks to learn how the survey design itself affects responses to questions, and therefore conclusions the researcher may draw from those responses.

# **General Issues**

Rea and Parker (1997) note six general problems to consider when preparing questions: (1) an inappropriate level of wording; (2) ambiguous words and phrases; (3) rnultipurpose questions (i.e., questions that ask two or more opinions simultaneously); (4) rmanipulative information; (5) inappropriate emphasis; and (6) emotional words and phrases.

Additional problems include questions that can be answered differently by people with the same opinion and questions that can be answered identically by people with opposite 24
opinions. Many of these problems are obvious, but some of their manifestations are not. For example, a simple "yes/no" question can accidentally be manipulative or suggest to the respondent that there is a correct response by not including all possible responses in the question itself. A question that simply asks, "Do you do x?" may make the respondent more likely to answer "yes" since the alternative was not presented. One possibility is to ask instead, "Do you do x, or not?" (Fowler 1995). An induced response is more of a concern when one answer to a question is more socially acceptable than another. We discuss issues in asking sensitive questions below.

Asking multipurpose questions-questions that ask about more than one issue or along more than one dimension-turns out to be unfortunately simple and common. For example, the World Business Environment Survey (World Bank 1998) asks respondents to "rate the overall quality, integrity, and efficiency of services delivered by the following public agencies or services." Although quality, integrity, and efficiency are probably highly correlated, they are three separate issues and should be asked separately. Sometimes, however, the problem is not so obvious. "Agree/disagree" questions, for example, usually ask the respondent to rate how much she agrees or disagrees with a statement. The survey asks the respondent whether she "strongly agrees, agrees, disagrees, or strongly disagrees." The problem with this scale is that it combines an emotional component ("strongly") with the more important cognitive component. Instead, the survey should ask whether the respondent "completely agrees, generally agrees, generally disagrees, or completely disagrees" (Fowler 1995).

Careful survey writing can minimize problems of the sort described above. Other problems are more complicated. Small differences in question wording, for example, can generate large differences in responses. A particularly well-known example is from a 1941 experiment in which matched samples of respondents were asked one of two questions (Sudman, Bradburn, and Schwarz 1996). The first group was asked whether they thought the United States "should allow public speeches against Democracy," while the second group was asked whether they thought the United States "should forbid speeches against Democracy." Among the first group, 21 percent favored free speech. Among the second group, 39 percent favored free speech. Either question may be useful in tracking changes in perceptions, as long as the same question is asked consistently across people and over time. That is, whichever question is asked, the exact 25
same question must be asked of every respondent. While it seems obvious to ask all respondents exactly the same question, in practice interviewers often deviate from the survey script. Fowler and Mangione (1990, p.35) note four studies that tracked interviewer-respondent interactions found that interviewers deviated from their script on 20 to 40 percent of the questions on the survey.

It is not possible to ensure that all interviewers follow the script to the letter all the time. Indeed, turning interviewers into little more than programmed question-readers could have unintended consequences by preventing them from interacting properly with the respondent. Nonetheless, survey designers can take steps to minimize this source of error. Specifically, the survey designer should recognize that the more likely a respondent is to interrupt the question, the more likely the interviewer is to change the question wording. The survey writer should design questions that need little additional explanation. Pretests can measure which questions respondents tend to interrupt for clarification or which questions seem most likely to lead to adlibbing by the respondent. The researcher can use the pretest information to change the questionnaire to minimize opportunities for the interviewer to change the written question.

The issues discussed above are general ones to keep in mind when developing and implementing a survey. Many more specific issues are inherent to most surveys. Those issues are discussed in the sections below.

# **Response scales**

Many surveys ask respondents to rate issues on some scale (say, **1** to 5). In World Bank surveys, such questions typically ask about infrastructure, regulation, or competitors. If phrased properly, such questions may tell us what a firm views as problematic relative to other issues presented in the questionnaire. Such methods, however, should be used with caution. First, they are inherently subjective and respondent- (rather than firm-) specific, making interpretation difficult. Second, respondents have little incentive to answer truthfully. Indeed, they may have an incentive to provide answers they perceive as beneficial to themselves. A firm whose production process generates a great deal of pollution, for example, has an incentive to report that 26
environmental regulations are onerous. Third, respondents may have distinct criteria in judging the severity of a problem, making it impossible to divine the cause of the response.

Sometimes it may be impractical or impossible to avoid the rating method. Other times the objective may be explicitly to measure manager opinion, in which case the rating method is appropriate. Often, however, the goal is to determine, from an _economic_ perspective, how such issues affect economic growth. Rather than asking whether something is a problem, ask for objective measures and use that data to determine their economic impacts empirically. For example, rather than asking managers whether labor costs are a problem, ask for wages and productivity measures. Empirical analysis can then determine whether labor costs in a particular industry-country context are, in fact, higher than in other industry-country contexts, or whether those costs have changed over time or in response to some policy intervention.

When one must rely on the rating method, it is important to consider how to interpret scales, the costs and benefits of numeric and verbal scales, and the optimal number of responses. Below we synthesize the recommendations from the survey literature on these issues.

1. The surveyor must remember that ordinal scale measurement is relative. That is, the scale allows us to measure how people feel about an issue _relative to the scale presented,_ but not necessarily how they feel in any absolute sense. Consider a question that provides respondents with the possible answer set A E {good, fair, or poor}. The same question that provides respondents with possible answer set B E {excellent, good, fair, or poor} will not simply divide the "good" responses of set A into subcategories. Instead, the meanings of all the possible answers change in set B relative to set A, so that the meaning of the words "good" or "fair," for example, are not comparable across sets A and B.

_2. Numeric versus verbal scales._ Numeric and verbal scales each have advantages and disadvantages. The main advantage of numeric scales is that they allow greater variation in responses without descriptive adjectives for each category. The main disadvantage is that the anchors of the scale may affect responses. For example, a survey asking "how successful have you been in life" found that on a scale of 0 to 10, 34 percent of respondents answered between 0 and 5, but on a scale of-5 to 5 only 13 percent answered between -5 and 0 (Sudman, Bradburn, and Schwarz 1996). The main advantage of using descriptive adjectives for each category is that each category can be relatively well-defined. The main disadvantage of descriptive adjectives is 27
that it is difficult to come up with meaningful and unambiguous adjectives to use on a multipoint scale. Indeed, "a problem in international research .. . is how to get consistent measurement of subjective states for different cultural groups. In particular, when scales are defined adjectivally, it has been found that it is virtually impossible to have exact translations across languages .... Numerical response scales, with only the ends of the continuum defined, perhaps with some general discussion of how to use the points between the extremes, have numerous advantages and constitute a very good way to have people perform some rating tasks" (Fowler, 1995).

_3. Number of categories._ The researcher also must decide how many categories of responses to include. The number of responses often varies by survey and by question within the survey. Researchers-and econometricians in particular-wish to maximize variation among respondents as much as possible, which suggests a larger number of potential r.esponses. On the other hand, Fowler (1995) notes that studies show that more than 10 categories does not increase variation in responses and that respondents seem to be able to meaningfully differentiate between five to seven categories, but not more.

# **The "Don't Know" Problem**

Every survey wrestles with the problem of whether to include a "don't know" or "indifferent" category. Research on the topic is mixed (Gilljam and Granberg 1993). One view holds that excluding the middle category forces some respondents onto one side of the issue when they either have no opinion or do not have enough information to hold an informed opinion. Excluding the middle category thus biases responses towards expressing preferences even when respondents are indifferent. Another view holds that people who have slight preferences are inclined to pick the middle category if it exists. Including the middle category thus biases responses towards indifference even though, when pressed, many respondents who expressed indifference actually have slight preferences.

Researchers offer differing advice, although they all recognize that the answer is unclear. Gilljam and Granberg (1993) ask and conclude, "Should public opinion researchers take don't know for an answer? Our view is that we don't know for sure, but probably should not." 28
Sudman, Seymour, and Bradburn (1982, p.141), however, note that "[w]hile it is impossible to make any hard and fast rule, our advice would be contrary to general practice: include the middle category unless there are persuasive reasons not to." Fortunately, the positive/negative response ratio seems to remain relatively constant regardless of whether the survey offers a middle option (Sudman, Seymour, and Bradburn 1982).

The important conclusion from the "don't know" debate is that the number of respondents expressing neutrality will be biased upwards if the survey includes a "don't know" or "indifferent" category, and that the number of respondents expressing an opinion will be biased upwards if the survey does not include this option.

# **Filters and Branching**

Filters and branching refer to questions that guide the respondent down a decision tree that first determines whether the respondent has an opinion and then the intensity of that opinion. One common problem with surveys is that respondents typically do not give the same responses to the same questions; that is, there is a great deal of noise in each person's response to any given question. Krosnick and Berent (1993) have shown that filters and branching increases the probability that a respondent will answer questions identically when re-surveyed at a later date."<sup>5</sup> This empirical finding is consistent with the above recommendation to not ask a single question along two dimensions. Questions that ask, "How do you feel about policy x?" are, in effect, asking two questions. First, "do you have an opinion on policy x." Second, "what is the intensity of that opinion?" Filters may be especially useful when the nature of the question mandates that responses can include only one side of the "response distribution." For example, surveys may ask "how concerned" someone is about an issue, or "how problematic" is a particular regulation to firm growth. Allowed responses will likely range from "not concerned" (or "no obstacle") to "very concerned" (or "severe obstacle"). Although it is not obvious, this answer set is biased because it

> 15The phrasing of the filter question itself will impact the results. The filter must be phrased so that it is unbiased. For example, if trying to determine whether the respondent is concerned about an issue, ask, "Are you concerned with x, or do you feel it's not a problem?"

29
presents only half the possible answers. The responses do not allow firms to say that they favor the issue or regulation in question and convey to the respondent that the interviewer believes the answer should be negative. An unbiased response scale, therefore, would include as many positive as negative responses, with "no obstacle" as the middle, indifferent, category. By presenting "not concerned" or "no obstacle" as an endpoint, the positive responses have. in effect, been truncated. Often, however, there is no other obvious way to ask the question as including the positive responses may seem inappropriate. Asking a filter question first can mitigate the bias that these truncated response distributions would otherwise impose.

Sterngold, Warland, and Herrmann (1994) found that including a filter that asks respondents _whether_ they were concerned about an issue before asking them _how concerned_ they were dramatically reduced the percentage of respondents expressing concern. Moreover, the drop in those expressing concern came not just from those who expressed "a little concern," but also among those who were "somewhat" and "very" concerned. The authors concluded that "standard degree-of-concern items may be leading questions that encourage res-pondents to overstate their concerns about issues." Filters and branches could be used effectively in many World Bank surveys, especially when the question itself may bias the respondent's answers. For example, The World Bank often wishes to measure whether institutional features of a country impede firm growth. Such surveys often ask firms to rate regulations or potential production bottlenecks from "no obstacle" to "major obstacle." These questions may suggest to the firm that the interviewer believes that the issues in question are obstacles. In effect, the question leads the respondent to an answer, and thus to biased results. An acceptable, although perhaps unlikely, response is also that the regulations are "very helpful." Presenting these positive responses may make the question seem awkward. Few firms are likely to report that tax inspectors are "very helpful." Not including the positive responses, however, is likely to lead to a strong upward bias in the number of firms reporting that the issue is a problem. World Bank surveys can mitigate this problem by imposing filters on these questions. First ask the firm the binary question of whether the regulation in 30
question is an obstacle or not and then ask how great an obstacle if the firm replies in the affirmative."6

# **Order Effects**

It is well-known that the order in which related questions are asked can impact responses. Order effects probably occur because additional questions about a topic prod the respondent's memory, making certain recollections easily accessible for future questions. Question order matters when asking about related specific and general questions and about the timing of past events. The order in which responses to a question are presented can matter as well, although the literature does not provide any rules as to how they matter.

# **Specific and general questions**

Asking respondents _specific_ questions influences how they will answer related _general_ questions. These "context effects" are termed "contrast" and "assimilation" effects. Contrast effects occur when specific questions cause the respondent to answer a general question in contrast to the preceding specific (but related) questions. Assimilation effects occur when specific questions cause the respondent to answer the general question as a kind of summary of the preceding specific questions.

For example, suppose people are asked first if they have a successful marriage, and then asked how successful their lives are, in general. A contrast effect would cause a respondent to rate his success in life relative to his marriage. An assimilation effect would cause a respondent to include her marriage as one of the factors determining her success in life."<sup>7</sup> 16 Stemgold, Warland, and Herrrnann (1994) also recognize that including a filter for many questions is "a rather time-consuming and burdensome approach that may encourage respondents to give answers that allow them to bypass the follow-up degree-of-concern questions. An alternative is to use filters intermittently during the course of the survey-especially before the first degree-of-concem response item-to communicate to respondents that [not concemed] is a legitimate response. This may have a carryover effect on other degree-of-concern items in the survey that are not preceded by [not concemed filters]. A second altemative is to ask the filter for an entire list of items, and then to cycle back to those iterns respondents said they were concerned about and ask the follow-up questions." ]7As another, example, suppose one asked a random sample citizens whether they believe politicians are, in general, honest. Assume that 20 percent of this sample reported that they do. Next, suppose one asked another random sample first whether they believed their representative to Congress was honest, and then whether they believed politicians, in general, were honest. The results of the general question are likely to change, reflecting 31
Surprisingly, there is agreement about when assimilation and contrast effects emerge. As Schul and Schiff (1993) note, "[r]esearch has shown that the impact of early specific questions have on the responses to a later general question is influenced by the number of specific questions and their positioning in the survey. [Wihen a general question is preceded by a single specific question, responses to the general question show a contrast effect unless the specific and general questions are separated." Sudman, Bradbum, and Schwarz (1996) note that asking the specific questions first yields assimilation effects when there are unrelated questions between the specific and general questions. Even graphically separating the specific and general questions on a self-administered survey or putting the specific and general questions in different sections of an interviewer-led survey generated assimilation effects. The contrast and assim.ilation effects are more intense the more sweeping or ambiguous is the general question.

To sunmnarize, a specific question followed immediately by a general question tends to generate a contrast effect, while a specific question followed by several unrelated questions or a section break and _then_ a general question tends to generate an assimilation effect. A general question asked before specific questions or completely absent specific questions may reflect contrast or assimilation effects, but the researcher has no way to know which. The survey can minimize such problems by avoiding ambiguity as much as possible and by asking the respondent to answer general questions either to reflect his experiences or actions or what he believes the general condition to be relative to his experiences or actions. That is, ask for a contrast or assimilation of specific questions posed first.

# **Recent and less-recent behavior**

Often the researcher is interested in learning about recent and typical behavior, which are not always identical. Surveys frequently ask whether a respondent has done something within the past x weeks (or months or years) and then within the past x + y weeks. It turns out that asking about the shorter time period first tends to lead respondents to over-report that activity (Fowler 1995). This over-reporting may happen because of the respondent's desire to report

> contrast or assimilation effects. Assume 75 percent of the second sample believed its representative was honest. If 10 percent of this second sample believed that politicians in general were honest, it could be an example of a contrast effect, while if 30 percent of this second sample believed that politicians were honest, it could be an example of an assimilation effect.

32
engaging in that activity even if it falls slightly out of the specified time period. Asking about the longer time period first could make the respondent more willing to state that she did not engage in that activity more recently.

# **Within-question order effects**

Order effects occur _within_ questions, as well. That is, the order in which responses to a question are presented can affect the frequency with which those responses are chosen or rated. For example, Boardman, et al. (1996) note "a study that asked some respondents to value preserving seals and then whales, while others were asked to value preserving whales and then seals. Seal values were considerably lower when the seal question was asked after the whale question." Unfortunately, empirical findings on within-question effects are ambiguous. Sudman, Bradbum, and Schwarz (1996) note that "despite the considerable empirical evidence for the emergence of response order effects, we know relatively little about the conditions that determine their emergence and direction, and the area is characterized by a large number of apparently inconsistent findings .... Because of the shortcomings in the available data, it is currently impossible to draw strong conclusions about the processes that underlie response order effects in survey measurement." Survey designers may be able to minimize this type of bias by randomizing across respondents the order the responses are presented.

# **Sensitive Questions and Truth-Revealing Mechanisms**

Often researchers wish to study behavior that society (or the respondent) considers inappropriate or illegal. For these or other reasons, the respondent may have an incentive to be untruthful when responding to questions. Social scientists typically worry about this problem in the context of drug use or sexual behavior (Fowler 1995, Tourangeau and Smith 1996). World Bank surveys face similar problems when they ask about firm behavior regarding issues such as corruption, regulation, and taxation. A firm may understandably be reluctant to admit to engaging in illegal activities or to avoiding regulations. While there is no way to guarantee truthful responses, research has demonstrated that the mode of data collection and the method of asking the question can affect whether respondents admit to embarrassing OT illegal behavior.

33
Unfortunately, these suggestions may be of limited usefulness-or will require some creativity to use-in in-person surveys conducted for the purpose of building a large dataset.

Not surprisingly, respondents are less likely to admit to incriminating behavior in a faceto-face interview than in a self-administered survey. Tourangeau and Smith (1996) note that the "literature on sensitive questions demonstrates that the method of collecting the data can affect the answers that are obtained.... Several of these studies have demonstrated that selfadministration of sensitive questions increases levels of reporting relative to administration of the same questions by an interviewer. Respondents are apparently reluctant to admit to an interviewer that they have engaged in illegal or otherwise embarrassing activities. Studies comparing self-administered questionnaires with conventional paper-and-pencil interviewer administration have shown that self-administration increases reporting of abortions, alcohol consumption, and illicit drug use." Although most World Bank firm surveys are conducted in-person, it may be possible to combine a self-administered component with the in-person interview. For exarnple, the interviewer could hand each respondent a package containing additional questions on sensitive topics and an addressed, starnped envelope that the respondent can use to retuni the survey. The self-administered fbrm would have to be coded so the self-administered section can be matched to the in-person section. This combination raises two issues that are not addressed in the literature. First, would respondents be more likely to be truthful on a self-administered questionnaire if they know that, despite answering the questions in private, thei.r answers will not be anonymous? On one hand, they can answer the questions in private and not admit their behavior verbally.<sup>8</sup> On the other hand, if the answers can be linked back to their firm it may not matter whether the answer is given in written or oral form. This is, however, an open question and could be tested empirically during the survey pretest phase.

_Randomized responses_ are another way to increase reporting of such behavior (Sudman and Bradburn 1982, Fowler 1995). As will be clear, this method is especially useful for learning the percentage of a population engaged in certain activities, but by its nature acids substantial measurement error, making hypothesis testing difficult. The method involves presenting the s It is conceivable that speaking out loud may affect truthfulness, especially since the interv.iewer is likely to be a citizen of that country, and most likely a local resident.

34
respondent with two questions simultaneously: the sensitive or embarrassing question, and a completely innocuous question. For example, question A might be, "Have you ever bribed a government official?" Question B might be, "Is your birthday in December?" The respondent then flips a coin (not revealing the result to the interviewer) and on seeing heads answers question A and on tails answers question B. The surveyor simply records the response. As a result, the respondent can give a completely truthful response knowing that the surveyor has no way of knowing which question she is answering.

Determining the percentage of a sample that has bribed a government official then becomes simple. Assume that 400 of 1200 respondents answered "yes" to the question. We would expect 50 percent of the respondents to answer the birthday question, and 1/12 of those to answer the question affirmatively. That is, 1200(0.5)/12 = 50 of the "yes" responses were to the birthday question. The expected number of people answering "yes" to the bribery question is therefore 400 - 50 = 350. Because 600 (i.e., 1200*0.5) people answered the bribery question we could conclude that 350/600 z 58.3 percent of the sample had bribed a government official. This method complicates hypothesis testing because of the measurement error it introduces, but World Bank surveys intent on measuring particular behavior in populations could potentially use this method to great effect.

# V. **CONCLUSIONS**

This review has attempted to examine existing World Bank firm surveys and the state-ofthe-art in questionnaire design. World Bank surveys have dramatically improved our understanding of firm behavior in developing countries. In addition, most of the data and measurement requirements we spell out in this paper are fulfilled in some surveys, such as the use of panel data, and detailed measurements about various themes. Needless to say, _not_ all Bank surveys should pay heed to the specific themes and hypotheses this paper mentions; we recognize that no survey can address all the concerns raised here given time constraints, _ad hoc_ purposes of specific surveys, and survey costs. We also recognize that our coverage in this survey of surveys is not comprehensive. Ultimately surveyors must decide how they wish to address the question at hand, and design questions accordingly. However, if a firm survey aims 35
to become what LSMS has become for household surveys (that is, an excellent source of data useful for answering a range of questions across countries), it may be worthwhile to take into account the observations made in this paper about data and measurement issues.

Many questions about collecting firm data remain unresolved: How do we check for data quality? How do we know whether the numbers come from a false accounting book? What are the best ways to ask questions? How important is survivor bias and recollection errors? Should we pay interviewees based on the percentage of missing information (Philipson, 1997)?

Our analysis however offers a few important lessons:

1. Data issues and hypothesis testing:

- If possible, use panel data.

- Have enough information on productivity to estimate a production funct'ion.

- 3 Avoid the paradigm of "list the severity of the obstacle/problem on a scale of 1 to 5." Instead, ask for data on specific dimensions of the problem that will shed light on alternative hypotheses and policy recommendations.

- Pick particular disaggregated industries, and sample those industries in each survey.

- Identify the most important policy interventions of interest, and consider how you will empirically identify that change by picking useful instruments.

2. Questionnaire Design:

- Include only one idea or dimension per question. Do not ask in one question about, for example, the "quality, integrity, and efficiency" of services.

- Consider the costs and benefits of numeric versus adjectival scales. Scales in which each point is labeled may be more precise than numeric scales in which only the endpoints are labeled. On the other hand, responses are very sensitive to the exact adjective chosen, and it may be impossible to translate adjectives precisely across languages, making it impossible to compare responses across countries.

- Recognize that the share of respondents expressing opinions will be biased upwards if the survey does not include a middle (i.e., "indifferent" or "don't know") category, and biased downward if the survey does include the middle category.

36
- When asking degree-of-concern or how-great-an-obstacle questions consider first asking a filter question (e.g., "Do you believe this regulation is an obstacle, or not?") and if the answer is yes, then asking how severe the obstacle is.

- Be aware of context effects. The act of asking questions can affect the answers given on subsequent, related questions.

- Think carefully about how to ask sensitive questions. Consider using a self-administered module for sensitive questions. Alternatively, a randomized response mechanism may be a useful truth-revealing mechanism.

37