metadata: authors: 0: author: Islam,Asif Mohammed
1: author: Rodriguez Meza,Jorge Luis
count: World
docna: 0: docna: A Cautionary Tale : An Experiment on The
            Stability of Business Environment Perceptions in a Firm Survey
docty: Policy Research Working Paper
subtopic: Business Environment,Financial Sector Policy,Crime and Society,Energy Policies & Economics
teratopic: Social Development,Private Sector Development,Energy
abstracts: cdata!: Several studies in the literature have
            adopted attitude or perception-based survey questions to
            evaluate the business environment and its effect on firms.
            The Enterprise Surveys of the World Bank are not an
            exception. In the case of the Enterprise Surveys, these
            questions involve rating an element of the business
            environment at the end of each section of the survey
            instrument. Such questions are often used but sometimes are
            inconsistent with responses elicited on the experience of
            the firm over a specific timeframe—experience-based
            questions. The literature is mixed as to whether
            perception-based questions are susceptible to anchoring or
            context effects. In this study, an experiment is set up to
            explore whether perceptions of the business environment are
            stable or vulnerable to the ordering of questions in the
            Enterprise Surveys questionnaire. The experiment entails
            randomizing the placement order of the perception-based
            questions at the end of a section or at the beginning of the
            survey. Significant question-order effects are uncovered
            only for perceptions of corruption and business licensing
            and permits but not the other elements, after accounting for
            a variety of factors. The study recommends that analysis in
            these two areas should go beyond perception-based questions
            and verify their findings with experience-based questions.
repnme: repnme: A Cautionary Tale : An Experiment on The
            Stability of Business Environment Perceptions in a Firm Survey
display_title: A Cautionary Tale : An Experiment on
            The Stability of Business Environment Perceptions in a Firm Survey
listing_relative_url: /research/2021/04/32971700/cautionary-tale-experiment-stability-business-environment-perceptions-firm-survey
topicv3: Investment Climate m1326293 405,Power and Electricity Sector m1327094 515,Financial Sector Development m1327681 1920,Crime m1439217 3041,Business Inspections, Licensing and
            Permits m1504624 2531
publishtoextweb_dt: 2021-04-05T00:00:00Z
dois: 10.1596/1813-9450-9610
historic_topic: Social Development,Private Sector Development,Energy
origu: Global Indicators Group - WB (DECIG)
no_of_pages: 34
author_keys: 0: author_key: 950662
1: author_key: 144470
model_extractions: Policy Research Working Paper 9610

# A Cautionary Tale

An Experiment on The Stability of Business Environment Perceptions in a Firm Survey _Asif M. Islam Jorge Rodriguez Meza_ Development Economics Global Indicators Group April 2021
Policy Research Working Paper 9610

# **Abstract**

Several studies in the literature have adopted attitude or perception-based survey questions to evaluate the business environment and its effect on firms. The Enterprise Surveys of the World Bank are not an exception. In the case of the Enterprise Surveys, these questions involve rating an element of the business environment at the end of each section of the survey instrument. Such questions are often used but sometimes are inconsistent with responses elicited on the experience of the firm over a specific timeframe—experience-based questions. The literature is mixed as to whether perception-based questions are susceptible to anchoring or context effects. In this study, an experiment is set up to explore whether perceptions of the business environment are stable or vulnerable to the ordering of questions in the Enterprise Surveys questionnaire. The experiment entails randomizing the placement order of the perception-based questions at the end of a section or at the beginning of the survey. Significant question-order effects are uncovered only for perceptions of corruption and business licensing and permits but not the other elements, after accounting for a variety of factors. The study recommends that analysis in these two areas should go beyond perception-based questions and verify their findings with experience-based questions.

This paper is a product of the Global Indicators Group, Development Economics. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at aislam@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **A Cautionary Tale: An Experiment on The Stability of Business Environment Perceptions in a Firm Survey**

Asif M. Islam Jorge Rodriguez Meza<sup>1</sup>

**JEL** : C83, D84, D73, L20

**Keywords** : Firm Surveys, Survey Bias, Subjective Response, Question Order

> 1 Islam is a Senior Economist with the Office of the Chief Economist in the Middle East and North Africa Region of the World Bank (aislam@worldbank.org). Rodriguez Meza is the Program Manager for the Enterprise Analysis Unit in the Development Economics Global Indicators Group of the World Bank (jrodriguezmeza@worldbank.org). The authors are grateful for feedback from Aart Kraay, Norman Loayza, Ulrich Zachau, and Donato De Rosa. The paper is largely an exploration of the validity of survey questions of a specific nature. Although the paper is based on an Enterprise Survey undertaken in the República Bolivariana de Venezuela in 2006, its survey methodology findings have general relevance for other country contexts. All errors and omissions are the authors’ responsibility. The opinions expressed in this paper do not represent the views of the World Bank Group, its Board of Directors, or the Governments they represent.
# **A Cautionary Tale: An Experiment on The Stability of Business Environment Perceptions in a Firm Survey**

## **1. Introduction**

The private sector is a fundamental engine of economic growth in an economy. The ability to check the pulse of the private sector is crucial for policy makers to formulate evidence-based recommendations. However, there is a dearth of information on firms across the developing world, thereby generating the need for business surveys such as the World Bank Enterprise Surveys. Questions in the surveys generally come in two forms. The first is what is defined as experience-based questions. Here the respondent, typically the manager or owner of the firm, provides information regarding a specific event, over a defined time frame, on a particular element of the business environment. For instance, “ _Over last complete fiscal year, did this establishment experience power outages?_ ” The second type of question is perception-based. Here the respondent provides a rating, typically using a Likert scale of responses, of a business environment element which is more holistic in nature. For instance, “ _To what degree is electricity an obstacle to the current operations of this establishment?_ ” The choices, consistent across all WBES, are: “no obstacle,” “minor obstacle,” “moderate obstacle,” “major obstacle,” and “very severe obstacle.” Both types of questions address the same element of the business environment, electricity in the example above, but the experiencebased question narrows it down to a specific component of an element of the business environment -usually a key component such as power outages- whereas the perception-based question encompasses all dimensions of that element of the business environment. It is because of this all-encompassing generality that perceptions-based questions may lead to biases or misinterpretations. The aim of this study is to explore the stability of such indicators by evaluating whether the ordering of perception-based questions matters. In this study, we analyze data from a unique experiment conducted in the República Bolivariana de Venezuela in 2006 that randomized the order of perception-based questions. Due to budgetary challenges 2
in the implementation of the República Bolivariana de Venezuela 2006 Enterprise Survey, there were slight deviations in the restrictive criteria (such as the exact length of the survey instrument) typical for World Bank Enterprise Surveys (WBES). This deviation from the WBES afforded the flexibility to run a survey methodology experiment.<sup>2</sup> The paper is focused on methodological, not policy issues. The details of the experiment are as follows. Firms were randomly split into two groups. One group received the standard ES questionnaire with perceptions of the business environment obstacles at the end of every section. The other group received the standard ES questionnaire with one modification - the perceptions of the business environment obstacles lumped together at the very beginning of the questionnaire. We then compare the perceptions of the business environment across the two groups.

Our results indicate that of the 15 business environment obstacle ratings, only 4 show a statistically significant difference across the groups – political instability, corruption, electricity, and business licensing and permits. After using a regression analysis framework to account for several factors, only two business environment elements have a statistically significant difference across the two groups – business licensing and permits, and corruption. The latter finding is consistent with Friesenbichler et al (2017) who uncover answer biases for the corruption perception in firm surveys. We then explore if changes in the order of the perceptions-based questions have any effect on the objective measures in the survey for business licensing and permits, and corruption. We find no significant effects. Thus, objective measures may be unaffected by the placement of the perceptions-based questions. This follows a long literature that has studied the effects of question order in surveys across a wide range of fields (Bradburn and Mason, 1964; Bishop and Smith, 2001; Rohwedder et al., 2006; Holbrook et al., 2007; Lumsdaine and Exterkate, 2013; Bard and Weinstein, 2017).

> 2 The increasing popularity of the use of the WBES has raised questions about the validity of various questions. A program to explore the accuracy of WBES led to the analysis of this experiment. Thus, this experiment was revisited recently, which was not a priority during inception.

3
The stability of perception-based indicators is important given that several studies have opted to use perception-based indicators of the business environment from the WBES. We only sample a few studies here given the large literature. Perception-based indicators of access to finance have been used by Asiedu et al. (2013) to explore the influence of gender. The determinants of access to finance obstacles based on perceptions have been investigated by Beck et al. (2006). The influence of finance, legal and corruption obstacles (perception-based) on firm growth has been explored by Beck et al. (2005). Commander and Svejnar (2011) utilize the effects of nine business environment perception indicators on revenue efficiency and growth. The business environment perception-based indicators used in the study include access to finance, infrastructure, tax rates, business licensing and permits, customs, corruption, crime, anticompetitive practices, and macroeconomic instability. A number of studies have mixed both subjective (perceptions-based) and objective (experience-based) measures of the business environment in the WBES (Kinda et al., 2011; Fowowe, 2017; Reyes et al., 2017). There is some acknowledgement of potential bias in using perception-based measures of the business environment, and thus they should be used only if objective-based indicators are unavailable. A strand of the literature has highlighted the endogeneity of these perception-based measures and have warned against its direct use as covariates to evaluate binding constraints to growth (Carlin et al, 2007; Carlin, Wendy and Mark Schaffer, 2012a and 2012b).

There are several reasons why perception-based indicators are attractive to researchers. First, they tend to be simple to explain. It is much easier to claim that electricity is an obstacle because the manager of a firm indicated so, than to grapple through the experience of a firm in facing various dimensions of electricity challenges including power outages, electricity affordability, use of generators, and presence of rentseeking activity in the power utility sector. This leads to the second advantage of perception-based indicators: they tend to be holistic. Rating of a business environment obstacle by a firm can be used as a catchall indicator for that business environment element, obviating the concerns about the specific features of the business environment element that leads to its lower rating by the firm. Additionally, despite the fact that the business environment obstacle has a time frame by the insertion of the word “current,” it is likely 4
that respondents draw from their experiences over time to arrive at conclusions. Take for example the issue of business licenses and permits. In the ES, only firms that have applied for a construction-related permit respond to the length of time it took to acquire the permit, and if any bribes were solicited in the process. If only a small number of firms engaged in applying for permits, very little inference can be made regarding the time and corruption involved in the process. However, the perception-based indicator is asked from all firms and it is likely that it incorporates, to some extent, the whole experience of a firm, whenever they may have applied for a construction permit. Furthermore, certain elements of the business environment are more amenable to perceptions-based indicators. Take for example political instability. It is a difficult to construct a series of questions of how political instability may have affected firms given that the channels would be very indirect, and thus easily missed in a set of questions. Finally, perception-based indicators of the business environment can have higher responses as they could be cognitively easier to answer than recalling events in a specific timeframe.

The arguments against perceptions-based indicators are however quite strong. Perception-based indicators are dependent on the characteristics of the respondent. Different respondents in a firm can provide differing accounts of the business environment. More importantly, they are susceptible to anchoring or easily influenced by context. Respondents may be drawn towards big events in the past and anchor their answers on that specific event. Similarly, respondents may alter their perceptions based on the previous questions asked in the survey, indicating a lack of stability in the answers. Finally, the subjective nature of perceptions-based indicators makes them a black box. A respondent may indicate access to finance is a very severe obstacle, however this response may be of little help to policy makers as it does not indicate what exactly is hampering access to finance. It could range from difficulty in accessing bank accounts to large collateral requirements for loans, each with different policy implications. Thus, with a loose timeframe reference, and no guidance on specifics, respondents may provide answers that are not robust to several factors.

5
Our study contributes to the literature by examining the stability of perceptions-based evaluations of the business environment using an experiment to uncover question-order effects. This study joins a chorus of voices in urging caution in using perception-based evaluations of the business environment especially for corruption and business licensing and permits. We encourage objective firm-level measures when available. The rest of the paper is constructed as follows. Section 2 provides a detailed discussion of conceptual considerations, section 3 details the data and experiment design. Section 4 provides the empirical strategy while section 5 presents the results. Section 6 presents robustness tests and section 7 concludes.

# **2. Conceptual Considerations**

When a business manager is asked to evaluate an element of the current business environment in a scale of “no obstacle,” “minor obstacle,” “moderate obstacle,” “major obstacle,” and “very severe obstacle,” several processes may be triggered. The assumption is that first, the respondent recalls relevant incidents, makes a judgement as to whether these incidents still apply at that point in time, and then somehow aggregates them up and chooses the option in the rating scale that best corresponds to the aggregate. For instance, to evaluate the degree to which corruption is an obstacle, a respondent recalls several transactions the firm may have had, determines the influence of corruption on those transactions and whether they currently apply, and then paints an overall picture to match with provided answer options. The underlying process can be called a file drawer model (Tourangeau et al, 2000). When asked to evaluate to what degree access to finance, corruption, or crime are obstacles to operations, the respondent looks for files marked _finance, corruption, or crime_ accordingly and reports the evaluations contained in them.

If the above process were to be believed, then respondents would not alter their perceptions without the introduction of any new information to update their evaluations. The stability of perceptions as a reliable source of information has been argued by early strands of the survey methodology literature. To illustrate this stability Allport (1935) argued that perceptions or attitudes that have been formed in childhood can last 6
a lifetime. Petty and Cacioppo (1981) defined attitudes as “an enduring positive or negative feeling about some person, object or issue.” The idea is then to mention the right target of the perception, this would activate the perception and all that remains is to report it (Tourangeau et al, 2000). The implications are that if perceptions are stable, they would provide an accurate evaluation of the business environment by the respondent. Specifically, they would capture exactly what the respondent believes. In this scenario, simply flipping the order of the questions in a survey should have no effect.

However, the stability of perceptions has been called into question. The rationale revolves around the issue of context and anchoring. When responding to perceptions of the business environment, managers must accommodate a large number of considerations in order to provide an answer. These considerations, if extremely large, may induce respondents to change their responses depending on what subset of considerations they evoke. These considerations create anchors and add context to the questions. Thus, any changes in the order of questions would alter the context or the anchoring framework, leading respondents to change their answers. This would suggest perceptions are temporary constructs. One example is priming. Prior questions about an issue affect how quickly respondents respond to succeeding perceptions questions (Tourangeau et al., 1991; Judd et al., 1991). Respondents could assimilate their perceptions to previous questions. If three negative questions on crime preceded a perceptions question on crime, they may be more likely to claim that crime is a severe obstacle. Such assimilation of answers has been found in the literature (Strack et al., 1991). On the other hand, respondents could misread the perception question as implying whether crime is an obstacle excluding the experiences with crime elicited from the previous questions. In this scenario, the respondent may be less likely to state crime is a severe obstacle given the negative experiences have already be divulged in the preceding questions, believing that there are no redundant questions in the survey. This is known as the contrast effect of context (Smith, 1986). Furthermore, it matters if the perception questions or the questions preceding the perception questions are general or specific. A general perception question may seem more natural if followed by a set of specific questions as 7
it would imply that the purpose of the general question is to sum up. In this case assimilation is more likely than contrast effects.

Context or anchoring effects have been uncovered in the larger behavioral and experimental literature. The early work by Kahneman and Tversky (1981) reported that support of a policy can vary if it was presented as “lives lost” versus “lives gained.” Ariely et al., (2003) show that social security numbers of individuals create substantial anchoring effects on their willingness to pay for products that has no relation to their social security number. The effects of an uninformative anchor on a subject’s willingness to pay was also confirmed by Bergman et al., (2010). For a literature review of the robustness of anchoring effects across studies, see Furnham and Boo (2011). In contrast, studies have uncovered no influence of control questions in experimental games (Roux and Thoni, 2015). Using data on self-employed households in coastal India, Gine et al., (2014) find that the order of recall variables in the survey has no effect on self-reported earnings. Furthermore, some studies have indicated that a number of factors can weaken anchors. Implausible anchors may weaken anchoring effects (Sudgen et al., 2013). Anchoring effects may also be minimized by opposing anchors (Sitzia and Zizzo, 2011). Finally, there is some indication that anchoring effects are weaker than those reported in early studies (Maniadis et al., 2014).

To elucidate the likely channels that may influence the stability of perceptions, it is important to delve into the general structure of the WBES questionnaire. The WBES questionnaire evaluates 15 elements of the business environment (list provided in next section). Typically, the perception-based ratings of these elements appear at the end of each section. For instance, after exploring the incidence, frequency and losses due to crime experienced by firms, respondents are asked to rate to what extent crime is an obstacle to operations. Note that for certain business environment elements there are no related preceding experiencebased questions (e.g. political instability, functioning of courts).

8
The question we are interested in is what would change if we switched all these perception-based business environment elements to the beginning of the survey? On one hand, if we see no change, then it implies that these business environment perceptions are stable and cannot be influenced, following the file drawer model. On the other hand, if switching the order of the questions does have an effect, there are a few channels through which this could play out. First, it would show that some level of context is being introduced by the questions preceding the perception-based questions. It may be that this context results in an accurate anchoring of the perception-based questions resulting in more accurate responses. For instance, asking a manager of a firm about their experience of power outages over the last fiscal year may enable the respondent to recall the appropriate considerations to provide an accurate answer. However, the opposite may be true. There is a possibility that there may be incorrect anchoring. Recalling power outages over the last year may result in an overstatement of the effect that power outages have had on the firm resulting in inaccurate responses to the perception-based evaluation of the business environment. Finally, the differences could simply be due to survey exhaustion. Respondents may be tired as they approach the end of the section, and thus provide flippant responses to the perception-based question. If the perception-based questions are at the beginning of the survey, there is no exhaustion and thus more accurate responses.

It is impossible to validate the accuracy of perception-based responses (Torangueau, 2000). Thus, teasing out what mechanisms are at play is beyond the scope of this study. For instance, we cannot tell whether correct or incorrect anchoring is responsible in systematic differences in perception-based responses. What this study can establish is which business environment elements have stable perceptions and which do not. This is important for researchers as they will understand for what topics they need a wide range of indicators beyond perception-based indicators to test their hypothesis or establish their point. In the next section, we provide the details of the experiment and the data used.

**3. Data and Experimental Design**

9
The experiment was implemented in the República Bolivariana de Venezuela 2006 Enterprise Survey (ES). The República Bolivariana de Venezuela ES was conducted between February and June 2006 covering 500 private sector firms.<sup>3</sup> All firms are formal (registered with a government entity) and include 5 or more employees. The sampling methodology was stratified random sampling. Levels of stratification were size, sector, and location. The República Bolivariana de Venezuela ES covered three regions: Caracas, Maracay, and Valencia. The sample is representative of the non-agricultural, non-mining private sector of the economy.<sup>4</sup> The respondents are typically business owners and managers; interviews are administered faceto-face. There were 33 enumerators and 5 supervisors that carried out the fieldwork.

The survey experiment was conducted as follows. The sample was randomly split into half – 250 firms in each group. The first group received the typical Enterprise Surveys where at the end of each section, a question is posed to the respondent to rate an element of the business environment related to the section. The question is structured as follows: “ _To what degree is XX an obstacle to the current operations of this establishment?_ ” where “XX” refers to the specific business environment element referenced at the end of the section. The respondent has 5 options: “no obstacle,” “minor obstacle,” “moderate obstacle,” “major obstacle,” and “very severe obstacle.” The business environment elements covered in the survey include: (1) Tax administration, (2) Access to land, (3) Access to finance (availability and cost), (4) Corruption, (5) Crime, theft and disorder, (6) Electricity, (7) Inadequately educated workforce, (8) Functioning of the courts, (9) Political instability, (10) Labor regulations, (11) Business licensing and permits, (12) Practices of competitors in the informal sector, (13) Customs and trade regulations, (14) Tax rates, and (15) Transportation. The second group of 250 firms received the business environment elements at the very

> 3 Note that in 2006 the Venezuelan economy had a GDP per capita growth rate of 8 percent.

> 4 Note that certain sectors are not covered by the Enterprise Surveys including agriculture, mining, and some service sectors such as education, health, and financial intermediaries. More information on the Enterprise Surveys global methodology is available on the website http://www.enterprisesurveys.org/.

10
beginning of the survey. Thus, each of the specific business environment element questions were removed from the end of each section and repositioned to the beginning of the survey. The purpose of the experiment is to compare the business environment perceptions questions between the two groups so as the ascertain whether the question-order influences the responses.

# **4. Empirical Strategy**

For estimation purposes we convert the business environment element rating variables into binary Yes/No variables. The variable takes a value of 1 if the business environment element is a “major obstacle,” or a “very severe obstacle.” The variable takes a value of zero if business environment element is “no obstacle,” a “minor obstacle,” or a “moderate obstacle.” This is the standard construction of Enterprise Surveys indicators based on the ratings variable.<sup>5</sup> We compute the percentage of firms in each group that finds the business environment obstacle to be a major or severe obstacle to operations, and then compare this across the groups for each business environment element. The mean differences between the groups of the business environment elements are tested for statistical significance.

We first test for business environment elements that are statistically significantly different across both groups, and then we explore if the difference is robust after controlling for other covariates. To achieve this, we estimate the following using a Linear Probability model.<sup>6</sup> B 0 1 E E 2 R 3 4L L 5M 6 7 8 9 10 11E ii ii ii ii ii ii ii = αα + ββ BBE 12 + ββ RREER RREE13+ ββ EES EE + ββ (1)EE + ββ MMRRSSBBR RRM SSRRh + ββ BBEERRE RR ii ii ii jj ss + ββ BBLLBBEERRMM + ββ IILLIIE EEMMEEIIEE + ββ IILLRRLLEEL RRh + ββ LLEEIIR SSEELL + ββ IIRRE ii ii ii + ββ IILLRRE EESSEEIIE + ββ EEMMRRE EESSRRE + εε

> 5 We explore defining the business environment obstacle in alternative ways in section 5 of the study.

> 6 We chose the linear probability model given the number of fixed effects in our estimation. We are concerned about the incidental parameters problem. However, our main results that show an effect of question-order on corruption and business, licensing and permits perceptions stands when using a Probit model.

11
The dependent variable is the binary business environment indicator (B ) for firm _i_ . A dummy variable is included that takes a value of 1 if the firm belongs to the group with the standard ES questionnaire, and takes a value of 0 if the firm belongs to the group that received the questionnaire with the business environment perception variables at the beginning ( E E ). Standard errors are robust and clustered at the sector, size, and location levels. The estimation also accounts for the response rate for every section of BBE the questionnaire R . The response rate captures two effects. First, the higher the response rate, the more likely context has had an influence on the respondent. Second, a higher response rate is likely to (RREER RREE) be correlated with survey exhaustion. Response rates are included for the 10 sections. There are certain firm characteristics that may influence the evaluation of the business environment. The estimation accounts for size that include two dummy variables capturing whether a firm is a small or medium-sized firm. The log of firm age (EES EE) (L L is also accounted for as a startup may face a different set of constraints than a long-established firm. Firms that are part of a larger firm may have ways to circumvent certain challenges, EE) and thus a dummy variable is included that takes a value of 1 if the establishment is part of a larger firm and 0 otherwise (M . Firms exporting status is also accounted for given that exporting firms may shoulder the burden of trade permits and face greater exposure to corruption via their need to interact MMRRSSBBR RRM SSRRh) with customs officials . (BBEERRE RR) The survey also includes questions answered by the enumerator. Enumerators do report whether they believe the respondent was truthful. We include two dummy variables capturing whether the enumerator believes the firm was truthful or somewhat truthful, with the omitted category being “not truthful” . Only 12 firms in the survey were identified as such. Financial figures may be obtained from (BBLLBBEERRMM)official books or estimated. We include two dummy variables, one capturing whether the figures are from books, and another capturing if they were estimated with precision ( . The omitted category is “Arbitrary and unreliable numbers,” which 33 firms were tagged under. The estimation also accounts for IILLIIE EEMMEEIIEE) whether the interview was completed in one sitting or through several visits. Two dummy variables are included – one capturing whether the interview was taken in one visit with one respondent, and the second 12
dummy variable captures whether the interview consisted of one visit but with multiple respondents . The omitted category is when the interview was undertaken over “Several visits.” Location fixed effects (IILLRRLLEEL RRh) and sector fixed effects (E ) are included in the estimation. To account for interviewer or supervisor bias, interviewer fixed effects (LLEEIIR SSEELL) IIRRE ) and supervisor fixed effect ) are also included in the specification. (IILLRRE EESSEEIIE (EEMMRRE EESSRRE

# **5. Results**

_Mean Differences in Business Environment Evaluation_ Table 1 presents the mean differences for all the 15 elements of the business environment. Only 4 elements show a statistically significant difference in means due to changes in the question order of the business environment perception variables, indicating that perceptions for these elements may not be stable. These are corruption, political instability, electricity, and business licensing and permits. For corruption, a standard WBES questionnaire order format results in an 9-percentage point higher perception that corruption is a major or severe obstacle than the alternative format with business environment perception questions at the beginning of the survey. The difference is statistically significant at the 5 percent level. Similarly, for political stability, the standard WBES question order format results in an 8.7 percentage point higher perception that political instability is a major or severe constraint than the alternative format. This difference is statistically significant at the 5 percent level. Political instability is the one category among the four where it does not have a full section in the questionnaire dedicated to it. Electricity has the highest difference across the two questionnaire formats, 10.8 percentage points higher for the standard WBES question order format than the alternative format. The difference is statistically significant at the 1 percent level. Finally, Business, Licensing and Permits is the only business environment element where a higher percentage of firms declared it a major or severe constraint in the alternative format than then standard 13
WBES question order format. However, the difference is only statistically significant at the 10 percent level.

For the remaining 11 business environment elements, there is no statistically significant difference across the two groups. This may indicate that perceptions regarding these elements are stable, which is an important finding by itself. There is heterogeneity in these 11 business environment elements in terms of questionnaire structure. Some of them have extensive sections dedicated to them (access to finance, crime), while others hardly have a question (access to land). Some of them have sections that appear early in the questionnaire (transportation, customs and trade) while others appear late (access to finance, labor regulations). Thus, there is no obvious pattern that could explain why perceptions of the four business environment elements vary across the question order groups.

# _Regression Results Accounting for Additional Covariates_

Using equation (1), we now explore whether the differences of the 4 business environment elements disappear once several factors are accounted for in a regression analysis framework. The results are presented in table 2. The difference across the groups for electricity and political instability are no longer statistically significant as shown in table 2. The statistical significance remains for corruption and business, licensing and permits, at the 1 percent and 5 percent levels respectively – higher than the statistical significance of the corresponding mean differences. The difference across the groups is larger for corruption with the addition of controls (13.3 percentage points), but about the same for business, licensing and permits (7.7 percentage points).

There are some interesting results for both business environment elements regarding the controls. Firms with higher response rates in the crime and regulation sections are more likely to perceive corruption as a concern. While firms with higher response rates in the customs and trade section are more likely to perceive 14
business, licensing and permits as a concern. Older firms are more likely to perceive corruption as a concern while in the interviews where the enumerator indicated the respondent seem truthful, corruption appeared to be less of a concern. For both corruption and business, licensing and permits, if the interview took place with multiple respondents, they were both likely to be rated as a higher concern.

# **6. Robustness**

# _Objective measures_

A valid question is whether more objective measures of corruption and business, licensing and permits are also affected by the order of the business environment perception questions. Note that in the standard WBES question order, the objective questions would be placed before the business environment perception-based questions. In the alternative ordering, the business environment perception questions appear at the beginning of the questionnaire, and thus before the objective questions. Therefore, if the perception-based questions at the beginning of the questionnaire in the alternative format can influence the respondent, this may be reflected in the responses provided for the objective questions.

In table 3 we explore the effect of question order on three objective measures of corruption – bribery depth, bribery incidence, and whether a firm is expected to give gifts in meetings with tax inspectors. Bribery depth is the percentage of public transactions where a gift or informal payment was requested. Bribery incidence is whereby a firm experienced at least one request for a gift or informal payment out of all the transactions. As shown in table 3, the question order has no statistically significant effect on the objective measures of corruption. Thus, the perceptions of the business environment have no effect on the objective responses. However, one interesting finding is that if the enumerator believed the respondent was not truthful, then the respondent is more likely to indicate corruption was high. This could be an argument for controlling for interview evaluation in survey data analysis.

15
In table 4 we repeat the same exercise for business licensing and permits. Three measures are included: the percent of senior management time spent dealing with requirements of government regulations, average number of visits or required meetings with tax officials, and number of days it takes to obtain an operating license. Note that we have only 60 observations for the measure on days to obtain an operating license as only 60 firms applied for an operating license over the preceding two years of the survey. As indicated in table 4, the question order has no statistically significant effect on the objective measures of business licensing and permits. Thus, the perceptions of the business environment have no effect on the objective responses.

# _Obstacle Ranking measures_

We explore if the question-order has any effect on another perception-based question on corruption and business licensing and permits. At the end of the questionnaire, firms are asked to pick one of the 15 business environment elements that represents the biggest obstacle faced by this establishment. In table 5 we show that the question-order rating of corruption has no effect on whether a firm cites corruption as a top obstacle. However, for business licensing and permits, the standard ES question order results in firms being less likely to cite corruption as a top obstacle. Although, the finding is barely significant given the statistical significance of 10 percent.

# _Alternate Obstacle Rating Measures_

It has been the standard in the ES to use major or severe ratings of an obstacle to document the perception of the business environment. Other options could also be pursued. For instance, one could adopt a more stringent definition of the obstacles by only considering business environments that report severe obstacle ratings. Alternatively, one could adopt a more lenient definition by exploring business environments that receive any obstacle rating of minor, moderate, major or severe. In table 6 we explore these possibilities for the corruption and business licensing and permits obstacles. We also employ the Ordered Probit estimation to explore the effects of obstacle ratings placement on the full range of the business environment 16
rating. The findings indicate that the question-order of both corruption and business licensing and permit business environment ratings have a statistically significant effect on the perceptions of severe obstacles, but not on any obstacle (minor, moderate, major, and severe altogether). The implication is that more severe perceptions of corruption and business licensing and permits are susceptible to bias. The findings using the Ordered Probit are mixed. We find a statistically significant effect of question order for Corruption but not for Business Licensing and Permits.

# **7. Conclusion**

Firm-level data is scarce in developing economies. The ideal data ecosystem is one where researchers have access to comprehensive low frequency census data with short questionnaires, complemented with high frequency firm survey data with more detailed questionnaires to obtain a full picture of the private sector. However, such situations are rare. In many cases national statistical offices are understaffed and underfunded. This is evident in the difficulty of meeting the data demands of the SDG indicators (Desai et al, 2018). However, policy makers cannot wait for availability of perfect data and have to take decisive action on urgent concerns. The World Bank Enterprise Surveys (WBES) have been extensively used by policy makers, governments, and academics alike to understand the formal private sector, especially for poorer economies with limited or no data ecosystems. The use of survey questions that capture the perceptions of business owners and managers in the WBES are seductive as they provide broad statements on the business environment that are easy to grasp. This study advises some caution on how they are utilized, with the understanding in some cases that they are the only available option.

The study set up an experiment to explore whether perceptions of the business environment are stable or whether they are vulnerable to the ordering of questions in the Enterprise Surveys questionnaire. For corruption and business licensing and permits, the question order seems to matter. The findings stand even after accounting for a host of control variables. The swing in the percentage of variables identify corruption 17
as a constraint is 13.3 percent, with the corresponding figure for business licensing and permit being 7.7 percent. Given that it is not possible to evaluate the accuracy of perception-based questions, all the analysis can indicate is the presence or lack of stability in the measures, but not the direction of the bias. Question order seems to have no statistically significant effects on experience-based or objective measures of corruption and business licensing and permits.

A key insight from this study is that wherever possible researchers should complement the use of questions on business environment perceptions in the WBES with more experience-based measures that trace the specific situations and transactions the firm undergoes. This is particularly the case of corruption and business licensing and permits. On a more positive note, 11 of the 15 business environment elements do indicate some level of stability in their perceptions. We cannot state with certainty why perceptions of corruption or business, licensing and permits are susceptible to change depending on their placement in the survey instrument, but we provide some conjectures. Corruption is a highly sensitive topic. It is quite likely that a respondent has strong priors about it before the survey. Furthermore, one can imagine situations where businesses are nervous about the government institutions in their economies. Such anxiety and perceptions could evolve through the questionnaire as the respondent is queried about different aspects directly and indirectly related to transactions with the public sector.

This study brings into light the question on how robust or stable perception-based or attitude questions are for firm-level surveys. Future work should explore ways by which the stability of perception-based measures can be increased. We also acknowledge that the findings may only apply to the Venezuelan private sector. Although the firms surveyed were not in the extractives sector, the nature of the Venezuelan economy may be unique, and its dependency on extractives may influence several aspects of the private sector including its size and influence. Thus, further survey methodology experiments in other countries would help ascertain the external validity of these findings.

18