metadata: authors: 0: author: Blancas Mendivil,Luis C.
1: author: Oliveira,Alexandre Borges De
2: author: Chioda,Laura
3: author: Cordella,Tito
4: author: Vardy,Felix Johannes J.
count: Brazil
docna: 0: docna: Do procurement rules impact infrastructure
            investment efficiency ? an empirical analysis of inversao
            das fases in Sao Paulo state
docty: Policy Research Working Paper
subtopic: Social Policy,Legal Products,Judicial System Reform,Sanitation and Sewerage,Engineering,Labor Markets,Labor Policies,Public Sector Economics,Water and Human Health,Hydrology,Legal Reform,Water Supply and Sanitation Economics,Sanitary Environmental Engineering,Small Private Water Supply Providers,Environmental Engineering,Legislation,Regulatory Regimes,Energy Demand,Energy and Environment,Energy and Mining,Public Finance Decentralization and
            Poverty Reduction,Town Water Supply and Sanitation,Rural Labor Markets
teratopic: Law and Development,Public Sector Development,Private Sector Development,Macroeconomics and Economic Growth,International Economics and Trade
abstracts: cdata!: As a means to reduce delays in public
            works implementation, a number of Brazilian states have
            recently reformed their procurement rules allowing
            contractor price proposals to be assessed before the
            technical evaluation of submitted bids is undertaken (in a
            procedure known as inversao das fases). In order to evaluate
            the effects of such reform, this paper adopts a
            difference-in-differences methodology to compare the
            procurement performance of Sao Paulo state (a reformer
            state) and Minas Gerais' (a non-reformer state) largest
            water and sewage utility along three efficiency dimensions:
            (i) procurement process duration; (ii) likelihood of
            complaint resolution litigation; and (iii) prices paid to
            contractors. The analysis finds that the reform is
            associated with a 24 day reduction in the duration of
            procurement processes for large projects and a 7 percentage
            point drop in the likelihood of court challenges
            irrespective of project size. Although both effects are
            economically important, only the latter is statistical
            significant. Finally, the paper finds no evidence of an
            effect of the procurement reform on prices paid.
geo_reg_key: 119228,517191
repnme: repnme: Do procurement rules impact infrastructure
            investment efficiency ? an empirical analysis of inversao
            das fases in Sao Paulo state
display_title: Do procurement rules impact
            infrastructure investment efficiency an empirical analysis
            of inversao das fases in Sao Paulo state
listing_relative_url: /research/2011/01/13439171/procurement-rules-impact-infrastructure-investment-efficiency-empirical-analysis-inversao-das-fases-sao-paulo-state
topicv3: Public Works m1326041 1665,Government Procurement m1326093 957,Water m1326522 137,Energy and Extractives m1326595 507,Wastewater Collection m1327591 148
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Law and Development,Public Sector Development,Private Sector Development,Macroeconomics and Economic Growth,International Economics and Trade
no_of_pages: 33
author_keys: 0: author_key: 809592
1: author_key: 458354
2: author_key: 954626
3: author_key: 879396
4: author_key: 1004239
model_extractions: WPS5528 Policy Research Working Paper 5528 Do Procurement Rules Impact Infrastructure Investment Efficiency?

An Empirical Analysis of _Inversão das Fases_ in São Paulo State _Luis Blancas Laura Chioda Tito Cordella Alexandre Oliveira Felix Várdy_ The World Bank Procurement Unit of the Operations Service Department Brazil Country Management Unit & Latin American and the Caribbean Region Office of the Chief Economist January 2011
Policy Research Working Paper 5528

# **Abstract**

As a means to reduce delays in public works implementation, a number of Brazilian states have recently reformed their procurement rules allowing contractor price proposals to be assessed before the technical evaluation of submitted bids is undertaken (in a procedure known as _inversão das fases_ ). In order to evaluate the effects of such reform, this paper adopts a difference-in-differences methodology to compare the procurement performance of São Paulo state (a reformer state) and Minas Gerais’ (a non-reformer state) largest water and sewage utility along three efficiency dimensions: (i) procurement process duration; (ii) likelihood of complaint resolution litigation; and (iii) prices paid to contractors. The analysis finds that the reform is associated with a 24 day reduction in the duration of procurement processes for large projects and a 7 percentage point drop in the likelihood of court challenges irrespective of project size. Although both effects are economically important, only the latter is statistical significant. Finally, the paper finds no evidence of an effect of the procurement reform on prices paid.

This paper is a product of the Brazil Country Management Unit, the Procurement Unit of the Operations Service Department, and the Office of the Chief Economist, Latin American and the Caribbean Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at lblancas@worldbank.org, lchioda@worldbank.org, tcordella@worldbank.org, and aoliveira@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
**Do Procurement Rules Impact Infrastructure Investment Efficiency? An Empirical Analysis of** **_Inversão das Fases_ in São Paulo State**

By: <u>Luis Blancas</u><sup>1</sup> <u>, Laura Chioda</u><sup>1</sup> <u>, Tito Cordella</u><sup>1</sup> <u>, Alexandre Oliveira</u><sup>1</sup> <u>, Felix Várdy</u><sup>2,3</sup> <mark>T</mark>

> 1 The World Bank, 1818 H Street N.W., Washington DC 20433.

> 2 University of California Berkeley Haas School of Business, 2220 Piedmont Avenue, Berkeley, CA 94720. 3 The authors gratefully acknowledge participants at seminars at SABESP, COPASA, the Brazilian Ministry of Planning, and the state administration of Minas Gerais for helpful comments and discussion. They also thank those institutions for their generosity in sharing data and answering to a number of queries The are deeply indebted to Rafael Arantes, Estelle Cantillon, Carlos Carvalho, Aline Dieguez, Loreni Foresti, Francisco Gaetani, Ana Hirle, Carlos Henrique, Atsushi Imi, Álvaro Mendes, Alessandro Nirino, Loçanny Silva, Karen Silverwood-Cope and Maurício Soares for useful comments and constructive criticism; and to Jivago Ximenes for his outstanding research assistance.
# **_1._** **_<u>Introduction</u>_**

Timely investment in infrastructure remains a key enabler of Brazilian economic growth (Ferreira and Gonçalves, 2005). In practice, however, the execution of public works—which are the backbone of infrastructure investment—is a protracted and delayprone endeavor in the country (Frischtak, 2008). Among the various stages involved in the public works execution chain, one of the most heavily regulated (through federal- and state-level laws) is the procurement of construction services. As a result, procurement tends to be seen as a unique area of opportunity to drive efficiencies in the infrastructure investment cycle through regulation reform. To what extent are procurement rules responsible for delays in the implementation of infrastructure projects in Brazil? Is it possible to improve the timeliness of Brazilian public investments by changing procurement rules? What are the effects of different procurement rules on the prices paid to contractors for the delivery of public works?

In order to shed some light on these issues, in this paper we assess the effects of a prominent reform to the Brazilian open competitive bidding procurement method for public works. Commonly referred to as _inversão das fases_ , or bid assessment stage inversion, the reform has recently been implemented in four Brazilian states (and is under consideration by Congress at the federal level). The main innovation brought about by _inversão das fases_ vis-à-vis the “conventional” Brazilian public works procurement method is that it allows bid price proposals to be opened and taken into consideration before conducting the bids’ technical evaluation. Such additional information disclosure, in principle, should reduce the amount of time spent on the technical evaluation of submitted bids, as well as the time spent in legal delays arising from the handling of complaints associated with the procurement process. Furthermore, the combination of price and technical information as inputs to the bid evaluation process may influence the prices paid by government agencies for the construction of works.

The main contribution of this paper is the evaluation of a public procurement reform utilizing rigorous empirical analysis. Specifically, we study the impact of _inversão das fases_ along three dimensions of procurement efficiency: (i) procurement process duration; (ii) likelihood of procurement process complaint resolution litigation; and (iii) prices paid to contractors.

To do so, we utilize a new dataset describing public works procurement processes carried out by the largest (and majority state-owned) water and sewage utilities in two Brazilian states—SABESP, of the state of São Paulo, and COPASA, of Minas Gerais— for the years 2007 and 2009. Since 2008 marks the year in which the state of São Paulo reformed the procurement of public works through _inversão das fases,_ we are able to evaluate the reform’s impact at SABESP utilizing a difference-in-differences methodology relative to COPASA in Minas Gerais, a state where no such reform took place.

Page | 2
Our empirical analysis shows that, while the adoption of _inversão das fases_ led to a 24 day reduction in the average duration of the procurement process for large works, such effect is not statistically significant. The reform, however, does appear to have reduced the likelihood of court challenges, by a statistically-significant 7 percentage points. Finally, the reform does not seem to be associated with any significant change in the cost of public works to taxpayers, as measured by prices paid to winning contractors.

The paper is organized as follows. The next section describes the Brazilian procurement process for public works. In particular, it discusses the process’s virtues and shortcomings, and the rationale by which some states have decided to reform it through the adoption of _inversão das fases._ Section 3 briefly discusses the theoretical literature on procurement methods, and provides some testable implications on the possible effects of _inversão das fases_ on our three efficiency dimensions of interest: procurement process duration, incidence of process-triggered legal complaints, and prices paid to contractors _._ Section 4 presents the methodology used to evaluate the procurement reform in São Paulo state and further illustrates the data utilized. Section 5 presents the paper’s empirical results, while section 6 provides some tentative conclusions and discusses possible extensions to the analysis.

# **_2._** **_<u>Government procurement in Brazil: Why it matters</u>_**

In January 2007, the Brazilian federal government launched an economic stimulus package—referred to as _Programa de Aceleração do Crescimento_ (PAC), or Growth Acceleration Program—that would spend R$504 billion (about $270 billion) over 4 years, primarily on expanding or updating the country’s infrastructure stock (about 65% of planned PAC funds were explicitly allocated to infrastructure projects, with the bulk of the balance going to housing finance). Launched well in advance of the onset of the global financial and economic crisis of 2008-2009, the PAC’s primary objective was to promote faster long-term growth on an equitable and fiscally responsible basis.

Due to PAC’s magnitude and the political prominence it has mustered, public sector efficiency in implementing PAC projects has been closely observed by a broad list of stakeholders, from political parties to oversight agencies, government contractors, the media, and, of course, the taxpayer. There are two performance indicators that these interest groups track with particular care: (i) the program’s infrastructure project completion rate; and (ii) the amount spent in completed infrastructure projects as a percentage of the program’s overall infrastructure budget. Yet, according to PAC’s own progress reports, implementing agencies at various levels of the Brazilian public sector have experienced delays—as measured by both of these indicators. Specifically, as of April 2010 (just 8 months prior to PAC’s phase-out), a full 43% of planned infrastructure projects had not been completed. Moreover, the completed projects at that point in time Page | 3
represented R$144 billion, or merely 34% of the total PAC budget for infrastructure<sup>4</sup> (Programa de Aceleração do Crescimento, 2010).

Most major public works funded by PAC (primarily in transport and energy infrastructure) are procured and contracted under Brazilian law through open competitive bidding. Since conducting bidding processes of this kind in Brazil is procedurally time consuming, it is hardly surprising that over the past three years public procurement has received a great deal of attention by stakeholders preoccupied with the timely and efficient use of stimulus money. To some of them—including the federal government— the bottlenecks that have plagued many PAC-originated bidding events are a key reason why disbursements and project deliveries have experienced delays. The significance of public procurement efficiency in the country has only been amplified by the March 2010 unveiling of PAC 2, a second and bigger public investment program, which will spend R$959 billion (about $512 billion) over 2011-2014.

_2.1 The Brazilian process for the procurement of public works_ The Brazilian government procurement system is governed by a two-pronged regulatory framework. The first prong is that of law 10.520 of 2002 (10.520/02), which regulates the procurement of off-the-shelf goods and non-consulting services (the latter also exclude construction services). Under this law, goods and non-consulting services are to be procured through the reverse auction method, commonly known to Brazilians as _Pregão_ .<sup>5</sup> The second prong in the regulatory framework is that of law 8.666 of 1993 (8.666/93) which, since the 2002 adoption of reverse auctions, has primarily been used for the procurement of public works (such as those funded by PAC) and consulting services. In the particular case of public works procurement (as opposed to consulting services), law 8.666/93 stipulates that project contracts shall be awarded on the basis of the lowest offered price among those proposals previously deemed compliant with the bidding documents. In order to determine whether a bid complies with the bidding documents (referred to as a “responsive bid”), and eventually determine how pricecompetitive a responsive bid is, public works procurement, as dictated by law, is to be carried out via a sequential “two-envelope” bidding process.

The Brazilian two-envelope procedure for the procurement of public works takes place as follows. Having carefully reviewed the bidding documents, which describe in

> 4 PAC’s total budget was expanded since its inception to R$656.5 billion (as of April 2010), of which R$427.8 billion was allocated to infrastructure investments in transport/logistics, energy, and social/urban development, with the rest allocated to housing and domestic sewage.

> 5 A reverse auction is a procurement mechanism that utilizes the lowest bid price as the critical award criterion. During the auction, bidders compete for the right to sell a given item to one or more contracting agencies by offering decreasing prices through successive, dynamic bidding, and for a pre-specified period of bidding time (say, 1 hour). When the bidding time expires, the winning bidder is the one that offered the lowest price—provided that, upon subsequent inspection, the item(s) and terms offered by the bidder are substantially responsive to the requirements set forth in the bidding documents. In Brazil, the vast majority of reverse auction events are carried out online (for example, in 2009, 97% of Federal Government reverse auctions took place over the internet).

Page | 4
detail the project in question (including, for example, engineering designs), interested contractors submit their bids in two separate envelopes. The first envelope contains the bidder’s legal, fiscal, technical, and financial qualification documents. Such qualification documents are meant to certify that the contractor is (i) legally eligible to win the contract, and (ii) able (technically and financially) to successfully execute it (per the detailed engineering design and estimated construction costs provided in the bidding documents). The second envelope solely contains the contractor’s price proposal.

After receiving paired sealed envelopes from all bidders, a technical committee appointed by the implementing agency (say, a federal- or state-government ministry) evaluates the bids in two stages. In the first stage, only the envelopes containing the qualification documents are opened and their contents assessed in order to determine which bidders are qualified to execute the contract (in strict accordance with bidding document criteria). This assessment is conducted for all bids received, regardless of what the corresponding price envelope might contain. Since reviewing bidder qualifications (particularly the technical component) can be a highly specialized (and therefore time consuming) task, the need to review all proposals may result in time bottlenecks.

Bidders deemed not in compliance with the bidding documents are disqualified at the first stage and their price envelopes returned unopened. In effect, the first stage acts like a filter through which only qualified bidders will advance to the second stage, where award decisions are made. It is in this sense that the two-envelope mechanism is sometimes referred to as a pre-qualification procurement method. During the second stage, the price envelopes of qualified bidders are opened, and the qualified bidder that submitted the lowest price proposal is awarded the contract.

- _2.2 Distinction between evaluating project quality and evaluating bidders’ qualifications_

In order to correctly assess the efficiency of the Brazilian two-stage procurement procedure for public works as defined by law 8.666/93, it is important to emphasize that its objective is to award a given contract to the qualified bidder that offers the lowest price. In this context, being a “qualified” bidder means, as mentioned earlier, being “able to execute” a given (i.e., pre-defined, per detailed engineering designs) project. Since a bidder can only be deemed either able or unable to execute a project, qualification decisions are strictly made on a pass/fail basis.

A project’s intrinsic characteristics—such as its physical layout, the materials with which it is built, the layers of reinforcement that it calls for, the project’s location— encompass what is defined as the project’s “quality” dimension. These are critical, since they define how the project will ultimately fulfill the service delivery needs that led to its creation in the first place. However, such characteristics are not the object of assessment in the two-envelope procedure for the construction of public works that this paper is concerned with. Such characteristics are defined by a project’s engineering design and Page | 5
estimated costs, which are an input to—not a result of—the two-envelope competitive bidding process for the procurement of public works construction.<sup>6</sup>

- _2.3 Complaint resolution mechanisms in the procurement of public works_

According to Brazil’s Constitution, all government documents must be made publicly available, including those used for government procurement. Therefore, any bidder—in fact, any citizen—is allowed to inspect each and every bid received by the contracting agency during the bid opening ceremony. Equipped with that knowledge, bidders in disagreement with first-stage evaluation decisions may file complaints, either to have their bids reinstated for the second stage or to request the rejection of someone else’s bid.

Interested bidders have three readily available channels to voice complaints: (i) an administrative channel, implemented by the agency carrying out the procurement process; (ii) the nation’s Supreme Auditor (known as _Tribunal de Contas da União_ , or TCU), which reports to the Legislative branch of power; and (iii) the country’s regular courts of law.

Even though a formal administrative complaint resolution channel is provided in the procurement law, the process in practice is perceived as ineffective with regard to settling complaints definitively (as it is carried out by the very agency whose decision originated the complaint). As a result, unsatisfied bidders tend to either file directly with the courts to settle complaints, or do so after utilizing the administrative complaint resolution process as a matter of course.

In recent years, the number of complaints filed with TCU has risen considerably, to a point where TCU-filed complaints may now outnumber the cases referred to courts of law. A primary reason for TCU’s popularity is that, according to clause 113 of law 8.666/93, any citizen can refer a case to TCU to be reviewed free of charge, whereas regular lawsuits involve potentially costly legal fees. Moreover, even after an unfavorable decision by TCU, bidders can still file a lawsuit at regular courts if they remain unsatisfied.

Importantly from a time bottleneck perspective, neither TCU nor the courts of law are bound by decision-making timeframes. Whenever a complaint is filed for review with these entities, procurement processes are interrupted for unspecified periods of time (several-month long reviews are not uncommon). As a result, the large number of lawsuits that exists in practice leads to potentially long delays in the conduction of procurement events. In addition, the fact that TCU reviews complaints at no charge to the

> 6 There are cases in which the design and construction of works are bundled into a single procurement process, in a procedure known as “design and build.” In Brazil, the design and build approach is mainly used for awarding large concession contracts (e.g., for power plants). Design and build, however, represents but a fraction of procurement transactions; most transactions follow the more traditional twostep approach of preparing designs first (whether in-house or on a contractual basis) and subsequently launching a bid solely for the construction stage. Design and build contracts are not analyzed in this paper.

Page | 6
complainant has proven to be an incentive for frivolous (i.e., otherwise avoidable) complaints.

- _2.4 The concept of inversão das fases_

As noted earlier, reviewing all technical proposals in the first stage of the evaluation process—particularly for large scale, complex public works—is time- and resourceconsuming. Such an evaluation procedure is especially onerous if one takes into account that the ultimate award criterion under which contractors are compared is their offered price (in other words, submitting a responsive bid is a necessary but not sufficient condition for award). It then follows that if price proposals were known upfront, only bids that are competitive on price (say, the three lowest) could initially undergo (the resource-intensive) technical evaluation, while high bids, which holding everything else constant would have little chance of winning the contract, could be safely ignored. Opening the price proposal envelopes upfront could therefore shorten the bid evaluation period (and thus make works procurement more cost-effective).

But the technical evaluation is not the only time- and resource-intensive activity that takes place in the pre-qualification bidding process: complaint resolution also plays a role. When prices are known only after the qualification documents are reviewed, procurement processes are particularly prone to legal complaints during the first stage— especially frivolous ones—since first-stage rejected bidders do not know how likely they are, if reinstated, to eventually win the award. Conversely, if price proposals were opened first, only bidders with a real chance of winning (i.e., only price-competitive bidders) would have incentives to file a (potentially costly and protracted) complaint. Were it in fact to lead to fewer complaints per procurement process, switching the order in which bid envelopes are opened could result in shorter (and cheaper) legal delays and therefore shorter (and more resource-efficient) overall procurement processes.

As it turns out, the potentially costly time and administrative resource impact of using pre-qualification in public works procurement is becoming increasingly apparent to Brazilian authorities at all levels of government. To many of them, the (highly resourceefficient and internationally emulated) reverse auction mechanism for the procurement of off-the-shelf goods has over the past eight years provided tangible evidence of the benefits of opening a bid evaluation process with prices rather than with qualification documents, as is the practice under _Pregão_ .

As a result, several states—including such investment powerhouses as São Paulo— have modified their public works procurement laws to allow for switching the bid evaluation stages, in a procedure known as _inversão das fases_ (which literally translates as “phase inversion”). Such a method provides the legal basis for (i) opening and assessing the contents of price envelopes before evaluating qualification documents; and (ii) conducting technical reviews of only those bids that are price-competitive (typically, it is the three lowest bids that are automatically evaluated, leaving the fourth and subsequent bids to be evaluated if, and only if, the three lowest bids are _substantially_ unresponsive to the bidding documents). Because under _inversão das fases_ technical Page | 7
evaluations are conducted after price proposals have been compared, the mechanism can be defined as a _post-qualification_ procurement procedure. Besides São Paulo, which passed _inversão das fases_ legislation in 2008, the sate administrations of Bahia (2005), Sergipe (2006), and Paraná (2007) have all enacted sub-national laws allowing phase inversion in the context of public works procurement.

Notwithstanding the successful passage of these reforms at the sub-national level, phase inversion in the procurement of public works remains a highly controversial issue in Brazil. Specifically, the topic has galvanized government officials on one side (arguing for faster and more efficient works procurement implementation) and large contractors on the other (claiming that phase inversion may result in awarding contracts to unqualified bidders, with potentially risky consequences). Those against phase inversion assert that less qualified contractors could deliver low-quality construction, request amendments to the contract price during project execution (having originally secured the contract by offering the lowest price), or altogether default on a contract mid-execution. This paper will try to shed light on the first of these opposing positions. Namely, is phase inversion really associated with faster (and therefore more efficient) works procurement implementation?

# _2.5 The role of prices paid_

Aside from its impact on the duration (and therefore on time-driven costs) of public works procurement processes (both from a bid evaluation and from a complaintresolution standpoint), _inversão das fases_ may impact procurement efficiency through a second dimension: the use of taxpayer money in the form of prices paid to contractors.

All phase inversion laws that have been enacted by state legislatures have a provision known as _fase saneadora_ , or “cleanup phase.” The role of the cleanup phase is to legally allow bid evaluation committees to either seek clarification on, or altogether waive, small, non-material deviations—vis-à-vis the bidding documents—in the qualification documents of price-competitive and otherwise responsive bids. In that way, a competitive bid (say, the lowest offered price) can be “cleaned up” of non-material deviations (say, lack of page numbering, or a missing, but retrievable, financial statement) that otherwise would have either triggered an outright rejection by the evaluation committee or significantly increased the likelihood of legal (and potentially binding) complaints by competing bidders.

The federal public procurement law (8.666/93) does not provide for a _fase saneadora_ . As a result, the approach taken in practice by bid evaluators is overly legalistic, by which any deviation from the requirements set forth in the bidding document—material or otherwise—will likely result in outright bid rejection. From a prices paid perspective, the lack of a cleanup phase may lead to highly competitive (i.e., comparatively priceattractive) bids being rejected for non-material deviations—without the evaluation committee ever knowing of the competitive nature of the rejected bids (as price envelopes are returned unopened after a first-stage rejection).

Page | 8
The presence of a _fase saneadora_ in the practice of phase inversion can thus potentially lead to lower prices paid by implementing agencies (relative to those that would have been attained without switching the order of the evaluation stages) on the basis legally more flexible (due to the cleanup phase) decisions made by evaluation committees. On the other hand, the potential for lower prices paid created by these two procedural tools can in practice be partially or entirely offset by post-award contract amendments, which may be negotiated by a successful “low-ball” bidder during project execution.

But even taking only the initial award price into account, the absolute magnitude of the savings in contractor fees that might result from practicing phase inversion to award a given project is likely to be small. This is due to homogeneity in the cost structure among contractors, especially those within a state, as well as the fact that contractor bids are prepared with full knowledge of the implementing agency’s estimated project cost (which tends to be highly accurate). Both of these elements constrain the variability in price proposals for a given project.

Still, added across thousands of procurement processes in (currently) four states— particularly during the implementation of a large, infrastructure-oriented stimulus package—overall savings to the taxpayer associated with prices paid under phase inversion could be substantial.

Nevertheless, no effort has yet been made in the literature to measure the presence and magnitude of a potential phase inversion impact on prices paid. This paper will attempt to do that, as explained in subsequent sections.

- _2.6 Potential implications for different bidder invitation procedures under inversão das fases_

The more time- and resource-consuming it is to assess the qualifications of a group of bidders vying for a particular public works contract, the bigger the impact _inversão das fases_ is likely to have on both the overall duration of the procurement process and the number and duration of legal complaints associated with it. Procurement process timeand resource-intensiveness, in practice, tend to be closely associated with a project’s price tag: large packages (say, the construction of a bridge or a water plant) are typically more resource intensive, from a procurement perspective, than smaller packages (say, a small rural road project). The primary reason for this is that larger projects carry higher complexity in assessing bidder qualifications, as well as higher financial and engineering risk.

Under law 8.666/93, even though the two-envelope bid evaluation mechanism applies to the procurement of all public works, regardless of their size, a procedure’s openness to competition does depend on size (specifically, on the estimated contract size, which implementing agencies must provide to bidders as an input to bid preparation). For public works with an estimated contract size of up to R$150,000 (about $90,000), public agencies can use an invitation procedure known as _convite_ , or limited bidding, for which Page | 9
the direct invitation of at least 3 interested bidders (whether or not they are registered in the official list of government suppliers) is enough for launching the bidding process.<sup>7</sup> For works above R$150,000 and up to R$1.5 million, public agencies must utilize an invitation procedure known as _tomada de preços_ , or price inquiry, where the bidding process is open to an unlimited number of interested bidders, provided they are properly registered in the official list of government contractors.<sup>8</sup> Finally, all public works with an estimated contract size above R$1.5 million must be tendered under an invitation procedure called _concorrência_ , or fully-open bidding, where any bidder, registered or not, can participate.<sup>9</sup> Because projects under _tomada de preços_ (TP) are smaller in contract size relative to those under _concorrência_ and because, as suggested above, larger-sized projects tend to be more procurement resource intensive, it is for _concorrência_ procurement processes for which _inversão das fases_ would likely have the bigger efficiency impact. However, no factual evidence exists to date to corroborate the existence of such a differentiated _inversão das fases_ impact across tender invitation procedures.

# **_3._** **_<u>Some theoretical considerations and testable implications</u>_**

The foundations of modern auction theory were laid by Vickrey (1961), in his seminal paper on _Counterspeculation, Auctions, and Competitive Sealed Tenders_ . Other milestones in the literature are Myerson (1981) and Riley and Samuelson’s (1981) papers on optimal auctions and the revenue equivalence theorem, and the paper by Milgrom and Weber (1982) on auctions with affiliated valuations. Krishna (2002) gives an excellent overview of the very extensive theoretical literature, while the book by Milgrom (2004) takes a more applied approach and considers practical issues in auction design.

In standard auctions, the principal is a monopolistic seller who only cares about the price he obtains in the auction. In a procurement auction, on the other hand, the principal is a monopsonistic buyer of goods and services who cares both about price and quality. In such an environment, bids are necessarily multidimensional in nature. The existing literature on auctions with multidimensional bids is more limited. Some references are Che (1993), Zheng (2000), and Asker and Cantillon (2008). The common approach in these papers is to study bidding under various “scoring rules” which assign scores to each aspect of a multidimensional bid that are then added up to determine the winner of the auction.

> 7 The procedure is still considered open competitive bidding, since uninvited bidders may themselves request participation in a _convite_ event up to 24 hours prior to the deadline for bid submission.

> 8 Non-registered suppliers interested in a _tomada de preços_ event may request registration up to 3 days prior to the event’s bid submission deadline.

> 9 Besides limits on the number of bidders invited and restrictions regarding official registration, these three invitation procedures may also differ in terms of the broadness with which they are advertised (for example, in a newspaper of local circulation versus a newspaper of national circulation, the official gazette, and/or online government procurement portals).

Page | 10
Public works procurement systems in Brazil (both in the case of pre- and postqualification) could be modeled as a scoring auction in which the auctioneer cares about both quality and prices. As appealing as it can be from a theoretical perspective, such specification, however, would not reflect well the Brazilian system in which the qualification criteria are set by law. Thus, in our “quasi formal” analysis we model the Brazilian public works procurement system as a first-price sealed bid auction with minimum quality standards. We think that this is a better characterization of the system, despite the fact that it might not be an optimal choice. Indeed, it is well-known (Che, 1993) that scoring auctions that use the auctioneer’s actual preferences are better than price-only auctions with minimum quality standards.

In what follows, we describe a simple set-up that can shed some light on the effect of pre- versus post-qualification. Since the arguments we develop to compare the different types of auctions are straightforward and the equivalence results follow directly from Riley and Samuelson (1981) or Milgrom and Weber (1981), we opted to proceed in an informal way.

# _3.1 Sketch of a model_

In order to “model” _inversão das fases,_ we assume that bidders (participating in a first price sealed bid auction to supply some kind of good to the government) differ in two dimensions: (i) whether or not they are qualified to produce the good; and (ii) the cost they face in producing it.

We further assume that, before bidding, each bidder gets to observe her own cost but not whether or not she is qualified; and that if the winning bidder is unqualified, the good produced (and paid for) is worthless to the government.

Under pre-qualification, the agency first chooses which bidders qualify observing a signal of their type. Then, in the second stage, it is committed to awarding the project to the lowest bid among the qualified bidders. Under post-qualification, the agency first ranks the bidders by price from low to high and, initially, only assesses whether the lowest bidder qualifies. If she does, the project is awarded to her and the auction ends. Else, the agency moves to the second-lowest bid. This in turn implies that:

**_Result 1:_** _The expected number of evaluations that the implementing agency needs to undertake under post-qualification is smaller than under pre-qualification._

Bidders can challenge in court any decision taken by the agency. If a decision as to (dis)qualify a bidder is challenged, the court gets to observe the type and, if warranted, overrules the agency. Note that, if the cost of suing is sufficiently low (which indeed seems to be the case in Brazil as appealing to the court of auditors is free), then each and every decision taken by the agency will be challenged in court.

This in turn implies that under pre-qualification the agency takes a decision as to the qualification of every bid. When the costs of suing are sufficiently small, every decision Page | 11
will be challenged in court: not only bidders who have been disqualified will challenge their disqualification, but bidders who have been qualified have an incentive to challenge the qualification of every other bidder who has also been qualified, in the hope of reducing the number of competitors. Under post-qualification, instead, there is a strictly positive probability that the court decides that the lowest bidder is qualified, and if this is the case only one lawsuit takes place. We thus have that:

**_Result 2:_** _When the costs of suing are sufficiently low, the expected number of lawsuits under post-qualification is strictly smaller than under pre-qualification._

Looking at the bidding profiles, it is important to notice that under both pre- and postqualification a bidder wins the auction if and only if she offers the lowest bid among all qualified bids. Since, under both systems, all of the agency’s qualification decisions are challenged in court and the court perfectly observes the qualifications of each bidder, the probability of winning remains unchanged, for every strategy profile. As a bidder’s payoff is fully determined by the probability of winning the auction, and the profit conditional on winning, pre- and post-qualification are strategically equivalent and, hence, a bidding profile is an equilibrium under pre-qualification if and only if it is an equilibrium under post-qualification. Accordingly,

**_Result 3:_** _If the bidders’ costs (and the likelihood of bids being accepted) under preand post-qualification are the same, then a bidding profile is an equilibrium under prequalification if, and only if, it is an equilibrium under post-qualification._

Finally, consider the case in which—for instance, as a consequence of the introduction of the _fase saneadora—_ in the post-qualification system the likelihood that a bid (of a given qualification responsiveness) is accepted is higher than in the prequalification system. This would in turn imply that, for any given bidding strategy, the probability of winning (conditional upon the bid being accepted) is lower and bidding is more aggressive under post-qualification than under pre-qualification. Notice that this effect would be further magnified if the probability that a bid is accepted depends (negatively) upon the price offered. We thus have that:

**_Result 4:_** _If in the post-qualification system the likelihood that a bid (of a given qualification responsiveness) is accepted as responsive is higher than in the prequalification system, then the equilibrium prices are lower under the post-qualification system than under the pre-qualification system._

Notice that Propositions 1 and 2 trivially imply the following testable implication:

# **_Testable Implication 1: The procurement process should be faster under postqualification than under pre-qualification;_**

Moreover, Proposition 3 and 4 imply that: Page | 12
# **_Testable Implication 2: One should not expect a significant difference in prices paid between pre- and post-qualification rules, unless under the latter system it is more likely that (competitive) bids qualify._**

In what follows we evaluate whether these testable implications hold true in our analysis of the effects of shifting from pre- to post-qualification in the practice of works procurement in Brazil.

# **_4.      Methodology_**

We assess the impact of _inversão das fases_ on three procurement outcomes: (i) average duration of a procurement process for public works; (ii) likelihood that one or more complaints are filed during a procurement process; and (iii) prices paid to contractors, as measured by the average size of awarded contracts. To do so, we adopt a difference-in-differences (DD) strategy, wherein changes in outcomes for a contracting entity located in a reforming state—São Paulo’s SABESP—pre- relative to post-reform are compared to those of a contracting entity located in a non-reforming state—Minas Gerais’ COPASA. In essence, COPASA serves as a control entity, such that its trend in outcomes pre- and post-reform is differenced out of the trend observed at SABESP, the treatment entity. Since São Paulo’s phase inversion law was enacted in 2008, we use procurement processes conducted in 2007 as our pre-reform sample, and processes conducted in 2009 as our post-reform sample.

# _4.1 Description and justification of the DD approach_

Formally, we estimate models of the following form: where _Yst_ is the outcome—either process duration (measured as the time, in days, between the opening of the first envelope and the granting of the award), complaint incidence, or award size—in state _s_ in year _t_ ; _SPs_ is an indicator for whether the project takes places in the treatment state, São Paulo; _Y2009t_ is an indicator for whether the project takes place in 2009 (i.e., post-reform); and **X** st is a vector of project characteristics, such as an indicator for the procurement process invitation mechanism ( _tomada de preços_ or _concorrência_ ), month of the year, and project type fixed effects. All _reais_ -denominated quantities are deflated to January 2007 _reais_ by the Brazilian construction price index.

The parameter _β_ captures any permanent differences in outcomes between São Paulo and Minas Gerais. For instance, _β_ includes any differences in the nature of public works or “productivity” levels at SABESP relative to COPASA. In turn, _γ_ embodies the trend in outcomes between 2007 and 2009 that is common to both entities. In turn, _δ_ —the difference-in-differences parameter of interest—measures the change in outcome at SABESP from pre- to post-reform above and beyond that observed at COPASA, net of Page | 13
changes that are correlated with project characteristics. With rich enough **X** st, and assuming that systematic differences across the two states remain fixed from 2007 to 2009, ordinary least squares estimation of [1] will deliver causal estimates of the reform’s impact on works procurement outcomes.

# _4.2 Data description_

As mentioned throughout the paper, we utilize works procurement data from two water and sewage utilities operating in Brazil: SABESP, a company that serves most of the São Paulo state market, and COPASA, the largest water utility in the state of Minas Gerais.

SABESP and COPASA are similar water and sewage operators in many respects. Both companies are publicly traded and majority state-owned (and therefore subject to public procurement laws). Each provides water to at least 10 million people and had 2009 revenues in excess of $1 billion.<sup>10</sup> Importantly from a works procurement perspective, both companies have announced sizable capital expenditures for the next several years. For 2009-2010 alone, the combined capital expenditures of both companies will stand at about $3 billion ($2 billion by SABESP, $1 billion by COPASA). They also operate in similar business and socioeconomic environments within their states, as São Paulo and Minas Gerais (a) have consistently ranked among the 10 Brazilian states with the highest per capita income over the past several years, and (b) register a Human Development Index score, as measured by the UN, that is markedly higher than the national average.

The data gathered comprise the full universe of procurement processes for works carried out by each company in 2007 and 2009 under both the _tomada de preços_ and _concorrência_ invitation mechanisms (i.e., the largest projects by size). For each bidding process, we observe data on the type of work being bid on (which makes possible segregating projects by their nature or segment), the duration (in days) of the bidding process (from the opening of the first envelope to the award of contract), the presence (if any) and duration (in days) of legal complaints filed by bidders during the bidding process, the estimated project cost (in R$) provided by each company to the bidders during the bid preparation period, and the actual contract amount (in R$), which is the end result of the procurement process. Note that the period of study purposely excludes 2008, as this was the year that São Paulo state introduced sub-national legislation allowing phase inversion for the procurement of works.

More specifically, data collected from SABESP are organized in two separate datasets: the first describing all bidding processes, the second containing data specifically related to those processes impacted by procurement complaints. Table 1 describes the fields contained in each SABESP dataset. For COPASA, a single dataset was collected, whose detailed contents are described in Table 2.

> 10 Specifically, SABESP provides water to 24 million people and generated 2009 revenues of $4.6 billion; COPASA serves a population of some 13 million, with 2009 revenues of $1.2 billion.

Page | 14
Table 3 presents summary statistics on the outcomes of interest, broken down by company/state and year. In both 2007 and 2009, COPASA conducted almost twice as many procurement processes for works under TPs and _concorrências_ as SABESP. The average duration of a procurement process at SABESP went from 85 days in 2007 to 68 days in 2009, while similar numbers at COPASA went from 37 to 32 days. In 2007, the likelihood of a complaint being filed for a particular project at SABESP (16%) was 75% higher than that observed at COPASA (9%). In 2009, however, the incidence of litigation was virtually the same between the companies, with a slightly lower likelihood of complaints at SABESP (13%) than at COPASA (15%). COPASA projects were roughly 60% the size of their SABESP counterparts in 2007, as measured by the median contract award size. The ratio falls to 41% in 2009.

# **_5.        Empirical analysis: Results_**

# _5.1  Results_

As a first pass, unadjusted estimates of _δ_ in equation [1] can be constructed for all 3 outcomes of interest from the figures in Table 3. For the mean duration of a procurement process (in days), the unadjusted D-in-D estimate of the effect of reform is (68.27 – 85.15) – (31.95 – 37.19) = -11.64. According to this result, _inversão das fases_ was associated with a 12 day reduction in the average duration of a SABESP procurement process for works. Analogous calculations suggest _inversão das fases_ is associated with a reduction in complaint incidence of about 9 percentage points and a reduction in contract award size of about R$7.3 million.

# _5.1.1 Adjusted impact on procurement process duration_

As laid out in the sketch of the model, the theory provides one sharp prediction: that the procurement process should be faster under post-qualification (i.e., under _inversão das fases_ ) than under pre-qualification. We test this prediction by studying the number of elapsed days between the opening of the first envelope and the signing of the award. As mentioned above, an unadjusted benchmark estimate of the effect of _inversão das fases_ (based on Table 3) on procurement process duration is that it indeed sped up the average process by about 12 days.<sup>11</sup> Table 4 reports estimates of _δ_ , _β_ , and _γ_ in equation [1], as well as the estimated coefficients of several other controls (standard errors are robust to heteroskedasticity of unspecified form). The first column ([a]) of the upper panel provides estimates for a model that controls only for seasonal indicators and procurement event type ( _concorrência_ or TP). The model suggests that at baseline SABESP’s projects lasted on average 42 days longer than COPASA’s. It also suggests that the reform reduced event

> 11 We also estimated parametric duration models. While we noticed a gain in efficiency, the point estimates were sensitive to the underlying parametric assumption, highlighting the relevance of the robustnessefficiency tradeoff. Hence, we opted to report only OLS results, which are statistically conservative and robust estimates, imposing minimal assumptions on unobservables.

Page | 15
durations by about 13 days, consistent with theory (although only the difference at baseline is statistically significant at conventional levels). It is possible, however, that such reduction arises from smaller-scale projects going up for bid at SABESP in 2009 relative to 2007. This possibility is explored in column [b], where we additionally control for each company’s estimated costs for projects. Controlling for the company’s “reservation price,” which carries information about project scale and scope, indeed reduces the differences reported in column [a].

Even this (statistically insignificant) gain in process speed may be overstated if complaints on average lengthen durations and post-qualification generates fewer bidder complaints (which we test below). Column [c] therefore reports results of specifications that add an indicator of whether any litigation was pursued. These suggest that the reform increased delays by one statistically insignificant day and that, on average, litigated (i.e., complaint-delayed) events are 47 days longer (which is statistically distinguishable from zero at the 1% level).

One concern regarding these estimates is that the composition of company-specific projects may have changed in either firm from 2007 to 2009, which may confound the effects of the reform. In the lower panel of Table 4, we address this concern by controlling for the general category of projects, adding indicators for project segment. However, all conclusions are qualitatively unchanged, with point estimates remaining relatively stable from the top to the bottom panel, suggesting that composition effects do not appear to confound the conclusions.

In sum, there is no evidence that _inversão das fases_ sped up the process when we look at the full universe of projects (that is, including _concorrências_ and TPs). Such a result is consistent with our original conjecture that the phase inversion impact on process duration is not likely to be significant when smaller-sized (and therefore less complex) TP projects are included in the analysis. As will be shown later in detail, when restricting the sample to _concorrência_ -only projects, the reform is associated with a 30% drop in the duration of the procurement process. While economically significant, the effect is not statistically significant—possibly due to limited sample size (there are 223 large scale projects in our sample).

# _5.1.2 Adjusted impact on the likelihood of complaints_

We then test the proposition that the likelihood of lawsuits or complaints under postqualification is smaller than that under pre-qualification, as hypothesized above. We estimate probit versions of equation [1], where the outcome is an indicator for whether a complaint was filed, and report average marginal effects in Table 5. Although there is no statistically discernible difference in the incidence of complaints between the two companies after controlling for covariates and estimated project costs, the table’s lower panel (which includes segment fixed effects) suggests that litigation is approximately 7 percentage points less prevalent post-reform. This figure is both statistically (at the 5% level) and economically significant, since 13% of projects in our sample are the subject of litigation.

Page | 16
# _5.1.3 Adjusted impact on prices paid_

We end this section by estimating versions of equation [1] where the award size (in real terms) is the outcome. Table 6 reports estimates of _δ_ , _β_ , and _γ_ . The upper panel of column ([a]) indicates that, at baseline, SABESP’s projects were on average R$7 million costlier (i.e., carried a larger award size by that amount) than those at COPASA. The project cost gap was reversed by the reform, which was associated with a R$7.6 million reduction in award size (both numbers are statistically significant at conventional levels). However, column [b], which controls for each company’s a priori estimated project costs, illustrates that this reduction resulted from smaller scale projects put up for bid at SABESP in 2009 relative to 2007: holding constant each company’s estimated project costs eliminates the statistically significant difference in award size reported in column [a].<sup>12</sup> To confirm that the scale of projects changed at SABESP between 2007 and 2009, we estimate the D-in-D model on the estimated project costs and report estimates in column [e]. According to this specification, the SABESP-COPASA gap in project budgets at baseline (which averaged +R$8.5 million), was entirely closed in 2009, as hypothesized.

In column [c], we additionally control for whether a complaint was filed over the procurement process. Doing so barely alters the estimates from column [b], though the point estimate on the dummy indicating the presence of one or more complaints implies that such projects are associated with roughly R$ 0.6 million lower estimated costs.

Finally, in column [d] we explore the possibility of a structural break in the relationship between actual and estimated costs, by controlling for the interaction of the 2009 year indicator and the estimated cost. No such structural break is observed.

As mentioned above, the possibility of company-specific projects to change segment composition from 2007 to 2009 may confound the effects of the reform. In the lower panel of Table 6 we therefore control for the general characteristics of projects by adding indicators for project segment. However, this hardly alters conclusions or point estimates relative to the top panel, suggesting that the composition of projects, as embodied by these indicators, did not change materially between 2007 and 2009 for both SABESP and COPASA.

As a robustness check, and to explore the possibility that the zero effect at the mean does not mask heterogeneity in the reform’s effect elsewhere on the conditional distribution of contract prices, we estimate quantile models analogous to equation [1] for the 5<sup>th</sup> , 10<sup>th</sup> , 50<sup>th</sup> , 90<sup>th</sup> , and 95<sup>th</sup> centiles of project costs. The idea being that effects that go undetected at the mean may still be observed if the reform either compressed or flattened the conditional distribution of the outcome. The quantile regression estimates are reported in Table 7 and show little effect at any of the conditional quantiles, even at the 10% level.

12 The addition of this control trivially increases the explanatory power of the model: the R2 rises from 19% in column [a] to 87% after controlling for estimated costs.

Page | 17
# _5.2 Concorrência-only projects_

In 2007 and 2009, 29% of all projects in our sample were launched as _concorrências_ (based on their estimated cost), but represented 92% of the almost R$3 billion in contracts awarded across both companies. As noted earlier, these projects are of special interest from a procurement efficiency perspective, due to their relatively higher procedural complexity. We thus repeat the above analyses for this subset of projects. In addition, this exercise allows us to indirectly explore the heterogeneity of the effects of the reform as a function of the economic scale of the underlying projects.

Table 4’ studies the effect of the reform on the procurement process duration among these larger scale projects (which take on average 20 days longer to implement across both companies). Conditional on segment fixed effects and estimated project costs, _concorrência_ events took 64 additional days to be carried out at SABESP relative to COPASA. Looking at column [c] we find a statistically insignificant but economically large impact—a 24-day reduction—of post-qualification on the duration of procurement processes.

Table 5’ provides no statistically significant evidence that the reform affected the likelihood of complaints among _concorrência_ projects, though the point estimate of the difference-in-differences marginal effect precisely matches the one observed for the full sample.

Finally, Table 6’ reports the results of the difference-in-differences analysis on prices paid for _concorrência_ projects. At baseline, such public works were R$22 million costlier at SABESP than at COPASA (column [a]), but this gap was more than offset by the procurement reform (which is associated with a change of -R$26 million). However, as in the pooled sample case, these differences are entirely accounted for once we control for estimated project costs, such that the estimates of _β_ , _γ_ , and _δ_ are all indistinguishable from zero.

These results emphasize a tradeoff between economic and statistical significance. When we restrict the analysis to _concorrência_ projects, our estimates are often larger in magnitude; however the more limited number of observations does not grant enough power to identify the effects with generally-accepted precision levels.

Our estimation methodology relies on a local identification strategy (i.e., in the neighborhood of the policy change) that enables us to estimate the parameter of interest without imposing structural assumptions beyond a minimal (semiparametric) linear specification. Given the limited amount of data, an alternative avenue would be to exploit more carefully the economic structure of the bidding problem to inform the estimating equation and improve the precision of our estimates (see for instance Athey and Haile, 2006), thereby relaxing the data requirement imposed by semiparametric methods.

Page | 18
# **_6._** **_<u>Conclusions</u>_**

According to economic theory, the _inversão das fases_ procedure adopted by SABESP in São Paulo state should have led to a faster procurement process, due to both a procedurally simpler process and a reduction in the number of decision complaints. In addition, one should have expected that, if the introduction of the _fase saneadora_ increased the number of competitive bids, then more competitive prices would also result from the reform. In order to test such theoretical predictions, we utilized a difference-indifferences methodology that allowed us to isolate and quantify the impact of the reform on works procurement efficiency.

While our results suggest that _inversão das fases_ might have sped up the procurement process, mainly through the reduction in the number of complaint filings, the results we find are not statistically significant. This may well reflect the limited number of observations with which we worked, and we hope that in future new data will allow us, as it becomes available, to improve on the significance of our estimates and get a cleaner sense of the effects of the reform.

Looking at the reform’s impact on the prices paid to winning bidders, we find no evidence of a reduction effect. However, such finding should be taken as very preliminary. The information available in our dataset allowed us to measure the reform’s impact on the size of the award, as opposed to the impact on actual fees ultimately paid to contractors from project inception through delivery (some projects can be highly prone to contract amendments during implementation, which may materially impact the fees charged by contractors). Since we have not (yet) been able to collect data on amendment requests during project execution, our conclusion on prices paid should be seen as an approximation to the impact of the reform on “true” (i.e., ultimately incurred) prices.

We hope that new data will in future allow us to improve on the significance of our estimates. Nevertheless, even with the data limitations faced we believe this paper provides a useful contribution to the procurement literature by deriving testable implications from economic theory and applying a statistically-sound methodology to the evaluation of a particular procurement reform.

Page | 19