metadata: authors: 0: author: Fazekas,Mihály
1: author: Blum,Jurgen Rene
count: World
docna: 0: docna: Improving Public Procurement Outcomes :
            Review of Tools and the State of the Evidence Base
docty: Policy Research Working Paper
subtopic: Legal Reform,Social Policy,Regulatory Regimes,Judicial System Reform,Legal Products,Legislation,Public Sector Economics,Public Finance Decentralization and
            Poverty Reduction,Public Financial Management,Public Sector Administrative & Civil
            Service Reform,Public Sector Administrative and Civil
            Service Reform,Democratic Government,De Facto Governments,ICT Applications
teratopic: Social Development,Law and Development,Governance,Information and Communication Technologies,Public Sector Development
abstracts: cdata!: Considering that about 15 percent of
            global gross domestic product flows through public
            procurement systems, the lack of systematic evidence on what
            works in this field is a major challenge for effective
            policy making. Hence, this paper systematically reviews the
            state of the evidence on major public procurement reforms
            and their impact on value for money and open access to
            public tenders. It discusses the reliably identified costs
            and benefits and systematically evaluates the quality of the
            evidence base, relying on academic and policy literature.
            The quality of evidence on the impact of public procurement
            interventions is mediocre, with reliable evidence
            established in multiple countries using diverse analytical
            methods only for selective, typically narrow tools. Although
            there is a range of policy tools with global policy interest
            and extensive implementation record, these have received
            little to no evaluation. As high-quality research uses
            different outcome measures, comparing intervention
            effectiveness is only possible for a very narrow outcome:
            savings. Comparing intervention types according to their
            effects on savings, centralized procurement and framework
            agreements stand out with the largest effects, over 50
            percent. Most other intervention types were documented to
            achieve about 5-10 percent price savings if they were well
            implemented. Given the estimated US$11 trillion spent on
            procurement annually around the world, even savings of 1
            percent amounts to US$110 billion annually. This systematic
            review points out that research on e-procurement and its
            variants, transparency portals, civil society supervision,
            and opening up the black box of public management, among
            others, would deserve considerably more research going forward.
repnme: repnme: Improving Public Procurement Outcomes :
            Review of Tools and the State of the Evidence Base
display_title: Improving Public Procurement
            Outcomes : Review of Tools and the State of the Evidence Base
listing_relative_url: /research/2021/06/33152601/improving-public-procurement-outcomes-review-tools-state-evidence-base
topicv3: Education, Skills Development and Labor
            Market m1326135 1484,Public Finance Management m1326316 716,Big Data m1327333 661,e-Government Procurement m1327708 1880,Public Administration m1327723 723
publishtoextweb_dt: 2021-06-08T00:00:00Z
dois: 10.1596/1813-9450-9690
historic_topic: Social Development,Law and Development,Governance,Information and Communication Technologies,Public Sector Development
origu: EFI-EAP-GOV-FM & PS-1 (EEAG1)
no_of_pages: 41
author_keys: 0: author_key: 997228
model_extractions: Policy Research Working Paper 9690 Improving Public Procurement Outcomes Review of Tools and the State of the Evidence Base _Mihály Fazekas Jürgen René Blum_ Governance Global Practice June 2021
Policy Research Working Paper 9690

# **Abstract**

Considering that about 15 percent of global gross domestic product flows through public procurement systems, the lack of systematic evidence on what works in this field is a major challenge for effective policy making. Hence, this paper systematically reviews the state of the evidence on major public procurement reforms and their impact on value for money and open access to public tenders. It discusses the reliably identified costs and benefits and systematically evaluates the quality of the evidence base, relying on academic and policy literature. The quality of evidence on the impact of public procurement interventions is mediocre, with reliable evidence established in multiple countries using diverse analytical methods only for selective, typically narrow tools. Although there is a range of policy tools with global policy interest and extensive implementation record, these have received little to no evaluation. As high-quality research uses different outcome measures, comparing intervention effectiveness is only possible for a very narrow outcome: savings. Comparing intervention types according to their effects on savings, centralized procurement and framework agreements stand out with the largest effects, over 50 percent. Most other intervention types were documented to achieve about 5–10 percent price savings if they were well implemented. Given the estimated US$11 trillion spent on procurement annually around the world, even savings of 1 percent amounts to US$110 billion annually. This systematic review points out that research on e-procurement and its variants, transparency portals, civil society supervision, and opening up the black box of public management, among others, would deserve considerably more research going forward.

This paper is a product of the Governance Global Practice. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://www.worldbank.org/prwp. The authors may be contacted at jblum@worldbank.org.

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent._ Produced by the Research Support Team
# **Improving Public Procurement Outcomes: Review of Tools and the State of the Evidence Base**<sup>**1**</sup>

Mihály Fazekas<sup>2</sup> and Jürgen René Blum<sup>3</sup> JEL codes: H54, H57, H83, L14 Keywords: systematic review, public procurement, contracting, value for money, open access

> 1 The authors would like to express their gratitude for the financial and intellectual support of the World Bank for compiling this review. We are also grateful to Isabelle Adam for outstanding research assistantship and the numerous colleagues who have commented on earlier versions of the draft, especially the participants at the World Bank brownbag lunch seminar in August 2018. Our special thanks goes to Alexandre Borges de Oliveira, Anna L. Wiegolorska, Jachym Hercher, Marko Klasnja, Shaun O. Moss and Vinay Sharma who have provided in depth comments and suggestions. The views expressed here by no means reflect the official position of the World Bank, they correspond to the views of the authors. 2 Central European University and Government Transparency Institute 3 World Bank
# **A. Introduction**

This review aims to take stock of the evidence on major public procurement reforms and policy interventions which have been pursued around the globe by governments of developing as well as developed economies in the last decade or so. It is conceived as a “living document” that gets updated regularly in order to keep track of new insights regarding the different interventions. In particular, this review discusses the reliably identified costs and benefits as well as unintended consequences for major intervention types. In addition, it also systematically evaluates the quality of evidence on the impacts of major intervention types in order to identify evidence gaps. In this regard, some chapters discussing interventions where the evidence base is very sparse effectively serve as placeholders for future research. As it is the first ever systematic review of public procurement interventions’ impacts and much of the accumulated evidence lies in diverse sources from academic literature to policy reports, it merely aspires to serve as a basis for further discussion with experts of the field.

Reviewing the evidence base for effective interventions in public procurement has been long overdue given how much public money flows through such systems and how often mismanagement leads to wastage. Public procurement represents on average 29% of total general government expenditure in OECD countries, ranging in 2011 from 12% in Greece to 45% in the Netherlands (OECD, 2013). Developing countries’ governments spend typically even larger shares of their GDPs on public contracts, up to 25% (Asian Development Bank, 2011; Bosio et al., 2020).<sup>4</sup> Such a high share of GDP makes public procurement indispensable for the very functioning of government as well as for pursuing most development outcomes. For example, major investment expenditures, such as transport and utility infrastructure, or schools and hospitals, are typically executed through public contracts. In addition, a significant share of recurrent administrative expenditures are also procured, take for example office equipment, electricity, or maintenance services. By improving how governments plan and design procurement contracts, how they select contractors and how they supervise them, they can save cost and time, pursue wider economic or social goals such as the development of small and medium-sized enterprises (SMEs) through providing them with access to government contracts, or sustainable development through green procurement (OECD, 2015). The promise of effective interventions is substantial, and only a small percentage of cost savings translates into millions of dollars staying in state coffers.

Development agencies and international financial institutions such as the World Bank’s support to governments over the last decade has traditionally mostly focused on (i) institutional reform and (ii) capacity building interventions in procurement. These comprise assistance in (a) drafting of procurement regulations, (b) determining the institutional setup and functions of central procurement units or agencies, (c) establishing cross-cutting procurement systems (procedures, guidelines, standard bidding documents) with a focus on e-procurement systems, (d) promoting transparency of procurement processes (for example through web platforms), and (e) establishing procurement training centers and technical assistance to build capacity.

For the purposes of this review, public procurement reforms and policy interventions (interventions in short) are narrowly understood. They refer to all those changes to public procurement regulations or implementing institutions which modify a limited set of features within the public procurement framework. This implies that fundamental reconfigurations of the whole public procurement system are not considered here, simply because of the methodological difficulty of evaluating the impacts of such complex interventions. Instead, mid-range theories and interventions are more feasible to test and evaluate, leading to more reliable policy advice (Merton, 1967). Moreover, interventions which modify regulations and implementing institutions outside the public procurement system while having strong links to procurement outcomes (e.g. civil service meritocracy, political party funding) are only briefly discussed mainly to provide an external yardstick to

> 4 It may partially explain these differences in the total procurement spending estimates that the OECD does not include procurement spending by state-owned enterprises in its calculations.

2
the public procurement-specific reforms reviewed (see section C.2 and F). Public procurement is broadly defined for the purposes of the review, including all purchases of goods and services by governments which include traditional tenders and auctions, but also direct contracting, while also including long-term complex contracts such as framework contracts and public-private partnerships.

The goals of the reviewed interventions are diverse reflecting their varying character and different contexts, with strategic procurement functions such as sustainability or social goals increasingly gaining prominence. However, we selected the below two overarching policy objectives to focus on because they are shared by most reforms as key objectives while they are also analyzed as key outcome variables in most impact evaluations:

- improving value for money and

- promoting fair and open access to public contracts.

Measuring and conceptualizing both of these outcomes have been hotly debated in recent years which can only very partially be reflected here. In addition, they are linked to each other through multiple channels, sometimes giving rise to trade-offs. Value for money is a broad concept encompassing the effective, efficient, and economic use of resources in public procurement tendering and contract implementation, hence the price alone may not necessarily represent value for money (World Bank, 2016). However, for the purposes of this review, we adopt a narrower concept in order to enable comparisons across studies and interventions. We will mainly look at cost-efficiency achieved through administrative procedures and competition for contracts; in short, the total cost of achieving a pre-determined outcome of public procurement such as the successful completion of the contract. This implies that the adequateness of project design and cost-benefit ratio of alternative project designs are taken for granted. This vastly simplifies the analysis and allows for comparisons to be made between highly divergent markets as well as focusing the analysis on the public procurement system itself. By implication, value for money boils down to measuring costs of administering tenders, contract awards and contract implementation, and the prices paid for the products procured giving rise to a series of straightforward indicators (World Bank, 2014).

The idea of fair and open access to government contracts is closely associated with the more generic concept of limited access orders developed in institutional economics (North et al., 2009). In the context of public procurement, it refers to the allocation and performance of public procurement contracts by bending prior explicit rules and principles of good public procurement in order to benefit a closed network while denying access to all others. Such a concept is akin to political science concepts recently gaining prominence such as quality of government as impartiality (Rothstein & Teorell, 2008) and universalistic governance regimes (Mungiu-Pippidi, 2015), that is a situation in which the state treats all citizens equally irrespective of their connections to power holders or social status. In practical terms, these translate into actions which steer the contract to the favored bidder by, for example, avoiding competition through unjustified sole sourcing or direct contracting awards; and favoring a certain bidder by tailoring specifications or sharing inside information (World Bank, 2009). Measuring limited access in public procurement then directly follows from the idea of unjustified restriction of access and delivers a set of objective proxies increasingly used in the literature (Charron et al., 2017; Chong et al., 2015; Coviello & Gagliarducci, 2017; Fazekas et al., 2014, 2016; Fazekas & Kocsis, 2020; Klasnja, 2016).

This review has to follow a practical approach to identifying and evaluating the evidence base. On the one hand, it aims to be as comprehensive as possible by reviewing academic as well as policy literature, quantitative as well as qualitative assessments. On the other hand, when assessing the quality of evidence,<sup>5</sup> it puts emphasis on high quality quantitative research papers using randomized controlled trials, natural experiments and other methods reliably establishing the causal link between the intervention and the

> 5 For the exact assessment category definitions see the summary section (section F).

3
outcomes in question. Theoretical arguments necessarily provide the basis for interpreting and evaluating empirical findings as well as for identifying evidence gaps. The availability of rigorous qualitative research tracing causal pathways and mediating factors is also considered essential for a high quality evidence base. As the evidence is scant for many interventions, there is only rarely robust quantitative as well as qualitative research, making the findings of this review tentative in many respects unfortunately.

The rest of the review is structured as follows: first, the broader background of public procurement interventions is spelled out in terms of new developments brought about by Bid Data and the data revolution. In addition, those broader groups of interventions are also briefly enlisted, which lie outside the public procurement system, but bear strong influence on procurement outcomes. Second, each of those public procurement interventions relating to a particular phase of the procurement process is reviewed. Third, public procurement interventions with a broader impact are reviewed. Fourth, conclusions are drawn, in particular by highlighting the need for further research.

# **B. Methodology**

In order to take stock of the evidence on major public procurement reforms and their impact on value for money and open access, we implemented a structured search and assessment strategy building on both standard online search and the expert knowledge of the authors and reviewers of earlier drafts of the manuscript. First, we defined the main areas of policy interventions both for those which target specific procurement phases and those which target the whole procurement process. As the choice of intervention groups defined both the search strategy and the structure of the review, we sought extensive validation of these choices at the outset. Second, we aimed at identifying as many potentially relevant academic publications as well as policy reports for each intervention area. We built on the authors’ knowledge of the field as a key starting point, while we also applied standard keyword search to each intervention group using specific keywords such as “e-procurement”, “impact”, “value for money”, etc. Once a broad body of potentially relevant documents was identified we followed up on widely cited authors’ publication list and also explored widely cited documents’ citation network.

Third, we screened and selected studies in each intervention group. Given the small number of high-quality studies in most areas, we could not apply strict selection criteria, in essence retaining most relevant studies. We excluded only those studies and reports which were not clearly linked to one of our intervention groups and the methodology was of particularly low quality (e.g. ) or narrow in scope (e.g. only discussing a few case studies whose representativeness is questionable). Fourth, the selected studies were assessed in-depth, classified and prioritized. We aimed to prioritize studies which applied the highest quality empirical methods combined with sound theoretical framework. Experiments and quasi experiments were of particular interest for example studies exploiting sharp thresholds of applying new rules or randomized controlled trials in real life settings (i.e. field experiments). We also considered observational studies, that is studies relying on observing behavior without intervention or manipulation, even though they are less reliable in identifying causal effects. Given the size of public procurement markets (often consisting of tens of thousands of contract awards even in a small country per year), we prioritized large-N studies (i.e. studies analyzing tens or hundreds of thousands of observations) while also occasionally taking into account smallN quantitative studies. We only very rarely took into consideration qualitative studies and case study methods as their results typically do not easily compare to the high quality quantitative studies composing the bulk of evidence reviewed and their findings are often not representative. Furthermore, we aimed at considering a geographically balanced set of studies in the review, in particular including studies from both developed and developing economies.

Finally, we carried out an in-depth appraisal and synthesis of the selected body of knowledge. This included both conceptual synthesis and the summary of empirical findings. Given the small number of directly comparable studies in each area, we could not systematically compare effect sizes across many studies, no 4
attempt at a meta-analysis was made, even though with the availability of more evidence we hope to be able to do so.

# **C. The context of public procurement reforms**

# 1. <u>The promise and challenge of Big Data in public procurement</u>

Big Data in public procurement, if appropriately harnessed, holds the promise of fundamentally transforming how procurement performance is understood and it can provide a vastly superior guide to effective policy decisions and implementation compared to our current knowledge. Such benefits are not automatic however, they also harbor considerable risks and require some investment.

Due to extensive regulations, the presence of multiple actors, and a demand for public security, public procurement has long been a data rich area of public spending. However, with the increasing use of electronic and online procurement tools, this rich set of administrative records have become more readily and more extensively available – giving rise to a data revolution experienced across many domains of social life (UN Global Pulse, 2012). This enables real-time data analysis using data sets tracking individual actions such as bids submitted to a tender, evaluation scores assigned, or invoices paid.

What fundamentally reconfigures our capacity to understand and govern public procurement systems is the move from individual records to a structured database.<sup>6</sup> This means that on top of the ability to identify and analyze a small set of tendering documents; governments, businesses, and citizens are becoming increasingly capable of systematically analyzing large swaths of procurement activities. In spite of the obvious informational basis and wide ranging benefits, making this move to a data rich approach has proven to be surprisingly challenging with some governments and international organizations even decreasing their ‘Big Data readiness’ rather than improving it (Fazekas & Saussier, 2018). On the one hand, building integrated data systems even if they only encompass the already collected information requires considerable investment. Linking disparate procurement data systems, standardizing data formats, and delivering reliable data management infrastructures are typically far from straightforward exercises with many frontline civil servants opposing change. Capturing past data, typically recorded in semi-structured text files, and building state-of-the-art data infrastructures going forward both require IT expertise and the understanding of complex data systems typically lacking in public sectors. On the other hand, systematizing data collection and publication had frequently revealed that reporting requirements are grossly neglected, making the most essential bits of data erroneous, missing or incomprehensible. The demand for using such data across the globe, as examples from Canada,<sup>7</sup> Czech Republic,<sup>8</sup> the European Union,<sup>9</sup> and Hungary reveal, makes the challenge of improving data quality even more pressing (Czibik et al., 2015). In addition, linking contract data to databases holding related information is also challenging but promises considerable benefits, most notably the ability to trace development outcomes back to procurement performance. Potentially linked databases include for example public financial management systems, treasury accounts of public bodies, company registry, financial and ownership data (including beneficial ownership registries gaining prominence recently), and information on sectoral outcomes such as student achievement or mortality (Fazekas & Tóth, 2016, 2014).

Big Data in public procurement also gives rise to the need for new, more advanced indicators which help diverse users in making sense of the often daunting variety of data; recall, public procurement includes purchases for anything ranging from nuclear power plants to school meals. Such new indicators of value for money and open access can potentially complement or in some cases replace traditional indicators of

> 6 - <u>http://www.open contracting.org/iodc15_open_contracting_reflections</u>

> 7 <u>https://sites.google.com/site/do101mtl/seao/iqd-1</u>

> 8 <u>http://www.profily.info/</u>

> 9 <u>http://www.open-contracting.org/digiwhist_big_data_meets_the_concerned_citizen</u> and <u>http://ec.europa.eu/internal_market/scoreboard/performance_per_policy_area/public_procurement/index_en.htm</u>

5
governance, by providing actionable and more objective insights (Knack et al., 2003). While there is a long way to go, a body of literature has been emerging which develops, tests, and applies objective proxies of open access, corruption, and favoritism around the globe (Trapnell, 2015). These initial innovations, nevertheless, already amply demonstrate the increasing capacity of indicators and impact evaluations to inform policy decisions and the wider public. Parallel to exciting data and indicator developments, users have to adapt their expectations and skills in order to effectively navigate in the new information landscape. High frequency rich data sets can become part of daily practice right across the public sector and civil society, if a new data culture is accommodated and data is easily accessible (e.g. through public APIs). Given the complex reality captured by Big Data, it requires careful thinking about and experimenting with different information systems in order to avoid information overload while keeping precise and interpretable information readily available to every level of decision making.

# 2. <u>The broader context matters for procurement outcomes</u>

Public procurement has long been considered a government function best left to narrow specialists without much regard to its broader governance implications and the crucial role it plays for effective governments (OECD, 2012; Thai, 2009b). This was a mistake which is increasingly recognized by policy makers as well as researchers across the globe leading to a re-evaluation of public procurement and what it means for governments and societies. As public procurement has the capacity to bring complete governments down even in well-established democracies such as Italy it is hard to underestimate its importance (della Porta & Vannucci, 1999). Building on newly available data and the increasing recognition of public procurement’s importance, there have been a range of research projects and impact evaluations placing public procurement and procurement reform in the broader governance context such as:

- National and local political economy such as electoral competition or acceptance of corruption as a norm;

- Quality of public administration as determined by civil service pay and meritocratic recruitment;

- Quality of oversight institutions such as courts and audit institutions; and

- Public financial management framework.

Each of these are reviewed briefly here to paint a broader picture of procurement process and outcomes and also to start exploring the ways in which procurement interventions’ impact may be mediated by the governance contexts they are embedded in.

The **political economy** of public procurement is probably one of the most crucial contextual factors determining procurement outcomes (Fazekas et al., 2015). This involves political party competition, electoral system, or party finances, the degree of separation between public and private spheres; citizens’ tolerance of corruption and inefficiency in public institutions; and civil society’s and business groups’ capacity to protect their interests. Public procurement is closely linked to the political economy of the country or locality as it allows for extracting a large amount of rents by a small elite (e.g. ruling family winning many large contracts), while it can also be very effectively used to distribute rents among supporters (each supporting local ‘strong men’ getting a few contracts as a reward for loyalty) (D’Souza & Kaufmann, 2013; Piga, 2011).

Lack of political competition either because there is no one standing to compete or because the incumbent has a strong grip on power, has found to increase corruption in public procurement in Italy (Coviello & Gagliarducci, 2017), Romania (Klasnja, 2016), Sweden (Broms et al., 2019), and the UK (Fazekas, 2015). Political party financing has also been linked to suspected corruption and distorted spending structure in public procurement in a range of contexts such as Brazil where companies’ campaign contributions translate 6
into additional contracts won worth 14 times more than the contributions (Boas et al., 2014), or the US where the same figure is only 2.5 times (Bromberg, 2014). Unfortunately, neither of these studies could establish how much actual performance is expected in return for these contracts from suppliers making the true amount of corrupt rents earned unknown. In the Russian Federation, companies with at least 5% revenue from procurement contracts increase their illicit political party financing transfers by about half a few weeks before elections and gain substantially more procurement contracts than their non-donating peers afterwards (Mironov & Zhuravskaya, 2016). In Latvia, companies whose campaign contributions were not diversified, i.e. only contributed to the governing party unexpectedly losing office in 2002, lost roughly 30% of their revenues compared to the control group arguably to a large degree due to lost procurement income (Dombrovsky, 2008). Emerging micro-level tendering evidence from Czech Republic and Romania suggests that red flags such as single bidder contracts more typically accompany firms donating to political parties than their comparable peers (Doroftei & Dimulescu, 2015; Počarovský, 2014).

Bidding companies’ direct political connections in terms of hiring ex politicians and top bureaucrats (i.e. revolving door) or using family and friendship ties to political officeholders have also been shown to be related to corruption and inefficient spending. Some scholars considered short as well as long term direct benefits to the connected companies (1-4 years) (Goldman et al., 2013; Luechinger & Moser, 2014) while others considered ties either to specific individuals or political parties (Akey, 2013; Straub, 2014). Unfortunately, most studies look at individual countries with only partially comparable research questions, data, and analytical tools. In the US, the largest predictor of company procurement contract value from before to after the 1994 change in the controlling majority of the House and the Senate is to which party the company was connected to (Goldman et al., 2013). Surprisingly, in Denmark which is one of the least corrupt countries of the world, direct family ties between companies and politicians increase company profitability, especially in sectors dominated by public procurement spending (Amore & Bennedsen, 2013).

More broadly, the degree to which citizens tolerate corruption and inefficiency in public institutions influences the functioning of the state and its capacity to control corruption in public procurement (Rothstein & Torsello, 2014). Similarly, the capacity of civil society and business groups to organize themselves in protection of their interests in opposition to predatory elites has fundamental ramification to favoritism, corruption, and prices in public procurement (Mungiu-Pippidi, 2015).

The **quality of public administration** designing and administering public procurement contracts is also fundamentally important for public procurement outcomes. On the one hand, the quality of administrative procedures and bureaucrats’ skills determine the efficiency of administering procurement contracts (Rasul & Rogger, 2015; Thai, 2004). On the other hand, shielding career pathways of bureaucrats from political interference gives them the opportunity to design tenders in pursuance of public rather than privateregarding goals of top politicians (Charron et al., 2017).

One key determinant of the skills and motivation of bureaucrats is civil service pay. In diverse countries such as Argentina (Di Tella & Schargrodsky, 2003) or Romania (Klasnja, 2016), evidence suggests that higher wages for those who can meaningfully influence public procurement tender design, award decisions, and contract performance monitoring contribute to lower corruption and lower prices. For example, in Buenos Aires hospitals the wage elasticity of standard product prices was 0.2 throughout 1996-1997 (Di Tella & Schargrodsky, 2003), similarly a 13% jump in mayors’ salary led to a decrease in standard product prices of 0.34 standard deviation and a 0.75 standard deviation decrease in missing infrastructure stock (Klasnja, 2016). Furthermore, meritocratic appointment and promotion of civil servants at the regional level across the European Union was found to decrease corruption risks as well as prices in 2009-2013 (Charron et al., 2017). In particular, a 3 standard deviation increase in bureaucrats’ self-reported civil service meritocracy across European regions leads to a 0.6-1.3% price savings or 14-31 billion EUR per year.

The **quality of oversight bodies** , in particular courts and audit organizations, is widely understood to influence their capacity to monitor and punish misconduct independent of political influence or influence 7
purchased through bribes. Oversight bodies contribute to procurement outcomes by directly uncovering corruption and mismanagement hence protecting public funds as well as indirectly by influencing actors’ expectations. Efficiency of courts, as measured by average trial duration, negatively correlates with procurement outcomes such as delays in contract completion, share of larger suppliers, and the incidence of postponed payments in Italy (Coviello et al., 2014). Publicly released audits and the probability of being audited was also found to decrease the incidence of corruption-related irregularities in municipalities in Brazil in 2009-2010: a 20% increase in audit probability was associated with 17% decrease in irregularities (Zamboni & Litschig, 2016).

**Public financial management frameworks** and systems for planning and selecting investment projects have obvious close relations with public procurement. Making sure that the investment projects selected for procurement are needed and not excessive in character limits potential for corrupt abuse at diverse tendering phases (OECD, 2007; Piga, 2011).

# **D. Interventions targeting specific procurement phases**

In this section, those interventions are discussed which target specific phases of the procurement process (Figure 1) which are likely to have a more identifiable impact mechanism than broader interventions. However, targeting only one phase also risks generating spillover effects to other phases. Probably, the most widely discussed such spillover goes from the intensity of competition during the advertisement and award phases to the incidence of renegotiation and incomplete delivery at the contract execution phase (e.g. David-Barrett & Fazekas, 2018; Decarolis, 2014). In short, the tender preparation and advertisement phase starts with the decision to procure a defined product and ends with bidders submitting their bids or the submission deadline passing. Hence, it includes preparing tendering documents, advertising the tender, and handling company expressions of interest and questions. The bid evaluation and contract award phase starts with the submission of bids or the submission deadline passing and ends with the contract awarded and signed with the successful bidder(s). The contract execution stage starts with contract signature and ends with the successful or unsuccessful completion of the contracts including all aspects of contract performance, monitoring and any potential renegotiation.

**Figure 1. Phases of the procurement process**

<!-- Start of picture text --> tender bid evaluation<br>contract<br>preparation & & contract<br>execution<br>advertisement award<br><!-- End of picture text --> While a lot of interventions could be identified in the literature, only those are discussed in detail which have received considerable policy as well as research interest. These have been grouped into 9 main categories of which 3 belong to the widely used umbrella term: electronic-procurement (e-procurement) (Table 1). E-procurement is split into different intervention types because its different variants and functionalities treat different phases of the procurement process and they are also expected to have different impacts.

8
**Table 1. Overview of intervention groups discussed in detail**

|**#**|**Public procurement phase**|**Intervention group**|
|---|---|---|
|1|tender preparation & advertisement|e-procurement: e-notification, e-access, e-attestations and e-submission|
|2|tender preparation & advertisement|framework agreements|
|3|bid evaluation & contract award|e-procurement: e-auction and e-evaluation|
|4|bid evaluation & contract award|preferential treatment of bidder and product classes|
|5|bid evaluation & contract award|award mechanism and auction design|
|6|contract execution|e-procurement: e-invoicing, e-payment, and e-contract monitoring|
|7|contract execution|civil society supervision of contract execution|

1. <u>E-procurement: e-notification, e-access, e-attestations and e-submission</u>

# _a) Theory_

E-procurement refers to the use of electronic communications and transaction processing by public organizations when procuring public works, goods and services including any phase of the public procurement process (Buyse et al., 2015). Four functionalities influencing the tender preparation and advertisement phase are most pronounced: e-notification, e-access, e-attestations and e-submission. E- notification refers to the official electronic publication of announcements, such as call for tenders or contract award announcements on a publicly accessible website. E-access denotes tender documentation’s electronic publication and availability for download.<sup>10</sup> E-attestations refer to the electronic submission and storing of qualification documents such as proofs of company registration or prior experience. E-submission allows for submitting tenders to procuring bodies by electronic means in a purpose-built IT system which also enables the electronic opening of tenders received. Of these, the first two are by far the most widely implemented tools among OECD countries (Figure 2) as well as among EU Member States (Buyse et al., 2015); in fact, they are frequently taken as synonyms for e-procurement blurring terminological boundaries.

> 10 Tender documents are those documents which are prepared by the contracting entity, not those which are submitted bidders such as attestations.

9
<!-- Start of picture text --> E-procurement functionalities in OECD countries<br>100%<br>50%<br>80%<br>70%<br>60%<br>50%<br>40%<br>30%<br>20%<br>10%<br>0%<br>E-notification: E-access: E-submission _E-reverse Notification of E-submission Online<br>Announcing Provision of of bids auctions award of invoices catelogue<br>tenders tender<br>documents<br>@ Ina national central e-procurement system B Not<br><!-- End of picture text -->
hypothesized cost savings crucially hinge upon the assumption that public procurement administrators, bidding firms, monitoring organizations and other actors using e-procurement tools develop the necessary computer literacy and system-specific knowledge. If they fail to make full use of e-procurement functionalities (e.g. bureaucrats manually re-entering information which could be automatically filled in) or record incomplete or inconsistent data, the expected savings might fail to materialize. In addition, organizational capacity is likely linked to e-procurement readiness with SMEs and low capacity public organizations facing relatively higher adoption costs (Croom & Brandon-Jones, 2007). This potentially puts low capacity organizations at a further disadvantage (e.g. decreasing instead of increasing the number of bidders where competition was weak to start with).

# _b) Evidence_

In spite of the political salience of e-procurement during the tender preparation and advertisement phase in developed as well as developing countries,<sup>11</sup> it has barely been systematically evaluated in either contexts (Luijken & Martini, 2014). Two non-experimental and largely descriptive reports on e-procurement in the EU find that e-procurement decreases administrative costs both for public and private organizations due to two key factors (Buyse et al., 2015; Strand et al., 2011).<sup>12</sup> First, it automatizes and simplifies a range of administrative procedures (e.g. it is enough to enter the supplier’s address only once in the system which then is automatically copied to all relevant documents). Second, it can safeguard against a number of typical errors leading to failed tenders (e.g. automatically disallowing certain procedure types if contract value and product type conditions are met). While the methods in these studies are ill-suited to reliably establish causality, the straightforward theoretical frame and rich qualitative evidence lend some support to the findings.

Two studies exploiting natural experiments establish the effect of e-procurement on increasing access to government contracts by increasing the participation of a wide set of potential bidders, many of which are less likely to have particularistic connections to the government, at least to start with. A high-quality regression discontinuity design in Italy is used to compare tender advertisement in local newspapers to advertisement on the national public procurement portal (i.e. e-notification) which yields a positive effect of online advertisement on the number of bidders, prevalence of non-local winners, and price discounts (Coviello & Mariniello, 2014).<sup>13</sup> The price effect for example amounted to a 17% increase average winning rebates.

In a quasi-experimental impact evaluation of infrastructure e-procurement in India and Indonesia, LewisFaupel et al. (2014) find that e-procurement increases the number of bidders, the prevalence of non-local winners, and contract implementation quality, but finds no evidence of lower prices. A further, nonexperimental study assessing the impact of e-procurement in Chile suggests that the increase in the number of bidders and the corresponding decrease in bid prices is the key driver in cost savings to the Chilean central procurement agency (ChileCompra) amounting to 2.65% of total spending (Singer et al., 2009). This contrasts with a number of government reports claiming price savings of a magnitude of 20% in Brazil, Mexico and Romania (Auriol, 2006). While the above studies cannot be directly compared, they partially reinforce the claim that e-notification improves fair and open access through transparency and lower transaction costs hence improving value for money. However, evidence from Paraguay raises a fundamental barrier to such a positive outcome, namely there have to be companies who can potentially enter the market

> 11 - <u>http://standard.open contracting.org/</u>

> - - - <u>http://ec.europa.eu/growth/single market/public procurement/rules implementation/new/index_en.htm http://www.worldbank.org/en/news/feature/2015/05/13/reforming-the-banks-procurement-framework</u> 12 In addition, the annual report of the Albanian Public Procurement Agency states that the administrative costs associated with administering tenders through the e-procurement amounted to 15% in 2009, 12% in 2010 and 20.1% in 2011 of the comparable manual tendering administrative costs (Luijken & Martini, 2014). Due to lack of information on the reliability of the data, this information should be used only carefully.

> 13 In addition, the amount and nature of information publication associated by the e-procurement system could have further effects on bidding outcomes and company performance. In the US (Oklahoma), the release of detailed cost estimates ahead of public procurement auctions for highway contracts decreased bid prices and increased long-term success of entrants (De Silva et al., 2008, 2009).

11
once access is widened; if there are none to very few such companies short to mid-term positive effects are null (Straub, 2014).

The slow adoption of various e-procurement tools across OECD as well as EU member states (Buyse et al., 2015; OECD, 2011b) suggest that the cost of implementing new tools, even if they only add to existing e- procurement functionalities can be substantial, potentially stalling reform (Thai, 2009a). Given the lack of systematic analysis of implementation barriers, we are left with guessing a crucial aspect of e-procurement reforms. It is certain, however, that introducing e-procurement systems in any public administration requires far more than a simple purchase of a software, it almost always requires changing administrative procedures, shifting duties and controls within public organizations; all of which suggests implementation being a high risk and potentially derailing process on its own (Schapper, 2007).

In sum, a considerable amount of further work is needed until enough evidence is gathered for guiding policy advice. Given that procurement systems are responsible for the purchase of enormously different products, they are likely to work differently in different environments, and that such systems come in very many forms with different functionalities, a lot more rigorous impact evaluations in this area may be needed. First, more experimental evidence needs to be gathered for a lot more countries, developed as well as developing, which cover multiple markets, in particular services and goods. Second, distinct functionalities of e-procurement systems should be evaluated on their own and in conjunction with each other as their impact mechanism may differ and there might be synergies between them. The recent emergence of e- procurement portals providing specialized services to bidding firms<sup>14</sup> are of particular interest as evaluating their impact on bidding behavior can shed light on the added value of information aggregation and packaging for bidding firms as ways of further decreasing transaction costs. Third, straightforward administration cost implications of e-procurement tools should be systematically evaluated. Fourth, the barriers and enablers of e-procurement reform need to be better understood if the gathered evidence is to be turned into actual reforms.

# 2. <u>Framework agreements</u>

# _a) Theory_

Framework agreements represent a procurement procedure type which typically comprises two stages: in the first stage, a competitive tender takes place (typically a reverse auction) to select one or more suppliers as framework agreement winners whereby they commit to providing products within a given time frame, price and conditions. In the second stage (the buying stage), the government agencies may buy the products from the framework agreement winners as they wish under the already set conditions in a simplified and speedier procedure (Gur et al., 2015). There may or may not be a competition at the second stage among winners of the first stage for the actual contracts (Albano & Sparro, 2010). While framework agreements are often jointly discussed with centralized and collaborative procurement, they are distinct; hence reviewed separately in this paper.

Framework agreements are expected to bring down costs of procuring goods and services in two principal ways. First, as the framework agreement sets the terms of purchase for a time period following the first stage of the process, costly auctions and negotiating terms are conducted only at the beginning of the process, bringing down overall transaction costs drastically. Second, framework agreements give considerable flexibility to public buyers to purchase on-demand rather than following the rigid time-scales of most alternative procedure types.

However, framework agreements entail risks for the suppliers as the exact amount of products to be supplied is unknown potentially driving prices up. Moreover, as framework agreements set product specifications

> 14 Some examples are: www.govini.com/ and <u>http://smartprocure.us/</u> in the US or <u>https://spendnetwork.com/index/ in the UK.</u>

12
for a time period, some flexibility in purchasing decisions is lost which can diminish the value of the framework agreement to end-users with diverse requirements.

As framework agreements are frequently coupled with demand aggregation by centralized purchasing bodies, there are a number of further pros and cons for framework agreements which are best discussed in section 8 on centralized and collaborative procurement.

# _b) Evidence_

Evidence on framework agreements is limited, there is only one quasi-experimental study touching on the effects of framework agreements in Italy (Bandiera et al., 2009), a few simulation studies exploring different design options (Gur et al., 2015), and a few government reports reviewing achieved savings (National Audit Office, 2010, 2013). Albeit ongoing scholarship is likely to add considerably to the evidence base (Albano & Nicholas, 2016).

A high quality quasi-experimental study aiming to estimate the amount of passive and active waste (that is low administrative capacity versus corruption) in Italian standardized goods purchases also touches on the effect of central framework agreements compared to individual contracts by government bodies (Bandiera et al., 2009). The authors find that average prices paid for standardized goods such as office stationery and electricity widely differ across the Italian public sector: the procuring body at the ninetieth percentile of the price distribution pays on average 55% higher prices than the procuring body at the tenth percentile. More interestingly, different types of public bodies such as central government or municipalities pay 34-78% higher prices for identical goods than their price in the central framework agreement. The UK National Audit Office in an audit of public procurement has found that national framework agreements have achieved savings of 426 million GBP in FY2011-2012 for central government only (National Audit Office, 2013). The amount of savings achieved was limited by failing to fully implement the framework agreements across the public sector, insufficient progress with standardizing products bought and overlap between various framework agreements. In addition, end user needs have not been sufficiently communicated to the managers of the framework agreements diminishing the demand for products in the frameworks. Taken together, these two studies underline the capacity of framework agreements to lower prices for standardized goods in low as well as high integrity environments if enabling conditions are met (e.g. user needs are harmonized).

In sum, the evidence base is limited in many respects calling for a range of fresh research to be conducted. First, prior research exclusively focuses on framework agreements in conjunction with centralized procurement or collaborative procurement, while understanding what effects framework agreements have over standard forms of contracting when purchased volumes are comparable is essential. Second, we lack experimental studies on the impact mechanisms and design variants of framework agreements. For example, managing uncertainties about the amount and timing of actual purchases may have considerable influence on the performance of framework agreements. Similarly, the size of frameworks and lots allocated to individual companies are likely to influence bidding patterns (e.g. SME participation) hence performance. Third, better understanding which products with which degree of homogenization are most suitable to be part of framework agreements balancing diverse end-user needs and economies of scale benefits is a question searching for answers. Fourth, making procuring bodies purchase through framework agreements has proven to be challenging in Italy as well as the UK, suggesting that reform impediments can be considerable which requires thorough research especially in developing countries.

13
# 3. <u>E-procurement: e-auction and e-evaluation</u>

# _a) Theory_

Among the many e-procurement functionalities discussed in this paper, two are closely related to the bid evaluation and contract award phases: e-auction and e-evaluation. E-auction<sup>15</sup> refers to the repetitive process for the presentation of prices, typically revised downwards (reverse auction), making use of a structured electronic platform. E-evaluation denotes the partial or complete automation of the assessment of tenders as well as the full tracking of the decisions made throughout the evaluation process. These two tools are rather different in their impact mechanisms and the literatures discussing them are thus separate too.

**E-auction** is expected to increase transparency as well as the intensity of competition contributing to better value for money as well as more open access. Usually, e-auctions imply the transparent publishing of key bidding information such as prices at each stage of the process. This creates a high degree of transparency both for the buyers and sellers a) limiting the buyers’ room for manipulation and b) increasing bidders’ trust in the fairness of the process. Intensity of competition is expected to increase c) because the number of bidders is higher due to transparency and d) because bidders are informed about all the other submitted prices and given the chance to lower their own prices for winning the contract (Soudry, 2004).

However, such desirable impacts may not materialize or even turn into negative depending on users’ computer literacy and SMEs’ ability to access online services, as discussed in section D1 already. Second, high intensity competition at the bidding stage may well bring prices down, but could lead to the so-called winner’s curse whereby the lowest price bidder is compelled to renegotiate the contract after the award (Soudry, 2004). But similarly, a sophisticated corrupt network spanning through the public and private spheres could make sure a corrupt company wins with the lowest price while also guaranteeing a watering down of contractual conditions during contract execution. Third, corrupt politicians and bureaucrats may actually prefer a seemingly transparent and fair e-auction mechanism which lends them the appearance of integrity and allows them to shift blame should corruption be discovered (Yakovlev et al., 2014).

Unfortunately, there is only scant literature discussing the theoretical expectations regarding the impact of **e-evaluation** on value for money and open access. Nevertheless, a few tentative claims can be formulated. First, e-evaluation increases the transparency of the evaluation process within the public administration (i.e. managers reviewing their employees’ evaluation decisions) and for the wider public including bidding firms and NGOs (e.g. open scrutiny of detailed records of the evaluation decision). Whether such transparency increases accountability hence value for money is an open question as it can be used to strengthen control within a corrupt network as well as strengthening horizontal accountability to citizens. Second, e-evaluation decreases the administrative burden of evaluators and administrators involved in assessing the bids, for example by automatically checking some of the submitted certificates or calculating overall scores based on prices and other quantitative criteria. This however crucially hinges upon the computer literacy and skills of bureaucrats just like in the case of other e-procurement tools.

# _b) Evidence_

The empirical evidence on e-auctions is very limited, there are only two correlational studies with narrow geographical as well as market focus. No empirical test of e-evaluation could be identified. In addition, the literature on e-auctions frequently conflate and bundle e-procurement more broadly and centralized framework agreements with e-auctions, hence the empirical evidence quoted here is tentative at best.

> 15 Please note that the discussion of e-auctions partially overlaps with the review of award mechanisms. Here the focus is predominantly on the electronic means of auctioning, in the later section the emphasis is more on the design features of the auction electronic or traditional paperbased.

14
A small N correlational study looking at public works and IT purchases in the Slovak Republic in 20072009 suggests that electronic reverse auctions decrease prices as measured by discounts (final price / original estimated price) through increasing the number of bidders (Pavel & Sičáková-Beblavá, 2013). However, conditional on the number of participating bidders, the authors find no further price effect, i.e. no confirmation of higher intensity competition. The overall price effect attributed to the use of e-auctions compared to standard open auctions is estimated to be 2.4% lower price than the originally estimated contract value. Another correlational study using a considerably larger sample from Russian sugar purchases in 2011 finds a considerably larger effect correlated with e-auction use of 28.0%-28.7% additional increase in discounts (Yakovlev et al., 2014). This corresponds to 5.8%-6.7% lower price per kilogram compared to the regional average price of sugar. While the authors do not reach a clear conclusion regarding the mechanism bringing about price savings, they suggest it is due to the number of bidders and contract characteristics such as contract size and contract duration.

Given the very limited amount of evidence available to test the impacts of e-auction and e-evaluation tools, there is a general need to expand the evidence base in terms of more advanced analytical methods, multiple countries, and other product markets. As electronic reverse auction is currently used in only few countries (OECD, 2011b) but is increasingly promoted, a more solid evidence base seems indispensable.

# 4. <u>Preferential treatment of bidder and product classes</u>

# _a) Theory_

The clearest sign that public procurement is more and more treated as a strategic rather than administrative function is that it is increasingly used to pursue broader policy objectives through special procurement rules and procedures. These socio-economic and environmental objectives either relate to specific bidder classes or types of products. In the former case, policy objectives typically concern the support of small- and medium-sized enterprises (SMEs), minorities, and women-owned or run businesses. Targeting specific products - what is often called horizontal policy objectives - aim to promote sustainability, foster innovation, or support disadvantaged regions. In both of these cases, public procurement can pursue these goals by offering preferential treatment through adjusting scoring rules, reserving some contracts, or facilitating access to tenders (OECD, 2015). Such preferential policies have become widespread among OECD countries with about 60% of countries having special support for SMEs and green procurement while 40% supporting innovative goods and services (OECD, 2015).

One notable example is the US Small Business Act which positively discriminates SMEs in public procurement through i) set-asides for SMEs, i.e. reserving contracts to be awarded solely to them, with a target of 23% of direct contracts and 40% of subcontracts to SMEs. Other common measures include ii) carrying out trainings and workshops for SMEs and iii) making documentation or guidance focused on SMEs available online. In addition, iv) simplified administrative procedures are used to facilitate the participation of SMEs in tenders. Another example is Chile’s Women Supplier Certification scheme which aims to identify and recognize women-led businesses as well as those with more than 50% of women workforce. Moreover the inclusion of gender-specific criteria in public tenders has also been fostered across the country by the procurement agency.

In general, assigning additional points for designated bidder and product groups such as SMEs (preferences), preserving some tenders only for such bidders (set-asides), or lowering tendering costs for them (targeted capacity development) should lead to both higher participation and a higher success rate of such bidders and offers. These policies may carry costs in terms of value for money if the positively discriminated bidders are otherwise less productive or products are of a lower value/price ratio. However, if preferential policies lead to wider access and stronger competition, the net effect may also be positive on value for money (Marion, 2007). Applying any threshold, monetary or product content-related, is also likely to distort firm investment decisions potentially carrying longer-term costs with it (e.g. limiting company 15
growth to remain under the privileged regime). Probably the greatest risk of any such discriminatory policy is that it can lead to undue discrimination and corrupt groups can capture or abuse it. Any such cost and risk, however, should be set against the declared policy goals of increasing access for some bidder groups which unfortunately is typically not systematically done (OECD, 2012).

# _b) Evidence_

There is high quality evidence for developed countries on the effect of SME preferences; however, there is only scant evidence for developing countries or on the other major policy goals such as gender, green and innovative procurement.

In a series of high quality quantitative research papers on US and Japanese data, authors directly model the effect of SME-preferences on bidders’ choice of whether to bid as well as the probability of winning with sophisticated analytical techniques for estimating company profitability too (Krasnokutskaya & Seim, 2011; Marion, 2007; Nakabayashi, 2013). This literature equivocally finds that under optimal policy design and sufficient bidder pool production costs go up due to SME preferences, however this efficiency loss is more than offset by increasing competition lowering prices. For example, in the Japanese construction sector in 2005-2009, the SME preference program decreased overall procurement costs by about 0.10%0.23% according to diverse model estimates (Nakabayashi, 2013). Such positive overall effect depends on the productivity differences between SMEs and large firms, the two groups’ propensity to bid as a function of SME preferences, capacity constraints in the industry, and allocative efficiency losses due to knock-on effects to auctions where no preferences apply. Most relevantly for policy makers, even under sub-optimal preference regime scenarios, the overall cost of preferences seem to be low, for example in Californian road construction tenders a 5% bid preference for SMEs (i.e. higher evaluation scores for SMEs) resulted in a 3.6% economic loss due to less productive bidders winning (Marion, 2007).

In sum, while the quality of evidence is high for SME preferences in developed economies, more research needs to be done in developing countries where institutional and market conditions substantially differ. In addition, none of the discussed studies could estimate the long-term effects on firm investment decision and market dynamics which may decrease or increase overall costs. Finally, the lack of noticeable evidence on preferences for sustainable and innovative products leaves policy makers wonder about whether such policies do more good than harm.

# 5. <u>Award mechanism and auction design</u>

# _a) Theory_

The group of policy interventions under the heading “award mechanism and auction design” encompass the decision rule applied (e.g. lowest bid) and the types of criteria taken into account (e.g. price or past performance) when comparing bidders and determining the winner. They may also include more detailed award mechanism characteristics for example whether prices or technical specifications are assessed first in the scoring process (Blancas et al., 2011). Procurement systems have a wide variety of policies in place in these regards with significant shifts occurring in the last decades, for example, the move from average bid auction (i.e. the bid closest to the average of the submitted bid wins) to lowest bid auctions in Italy in the 2000s.

Award mechanisms received extensive theoretical and empirical discussion with two particular designs most frequently discussed: average bid auctions and first price sealed bid auctions (i.e. bidders simultaneously submit their bids without knowledge of other bids and the lowest price bidder wins) (Lengwiler & Wolfstetter, 2006 part IV). The argument for average bid auctions is that they soften price competition and select the bidder which most accurately estimated the production costs of the procured products which in return decrease cost overruns and delays in the contract execution stage. However, the 16
promise of lowest price auction is that it encourages price competition and selects the best price bidder after eliminating all the unreasonably low bids hence maximizes value for money with little to no cost to cost overruns and delays.

Taking into account different types of company and bid characteristics can have a fundamental impact on procurement outcomes. The rule of thumb appears to be that efficiency is maximized if all characteristics relevant for delivery according to the buyer’s needs are taken into account, ideally even those which are not easily observable and contractible. The use of price only criteria has been considered as safeguarding against corruption, with the EU recently increasingly discouraging its use in favor of economic efficiency and for more adequately taking into account the full set of buyer needs. Price only – lowest price auctions can minimize discretion in determining the winning bidder, but may not be suited for selecting the highest value-for-money, in particular due to the so-called “winner’s curse” (Soudry, 2004). In addition, such auctions may invite fraudulently low bids by bidders that bet on cost-overruns, if these are frequent and insufficiently sanctioned. There are differences in the use of past performance as a basis of selecting winning bidders with the EU discouraging it while the US doing the opposite. The argument is that using past performance as a scoring criteria discourages market entrants hence limits access and diminishes value for money (Spagnolo, 2012).

# _b) Evidence_

There is high quality evidence on many of the above instruments and impact mechanisms in particular for the two major award mechanisms (i.e. average versus first price) applying natural experiments, laboratory experiments and a range of regression techniques (e.g. Albano et al., 2006; Decarolis, 2014).

Evidence regarding the choice of average versus first price auctions, by and large confirm that under most reasonable environmental conditions, first price auctions are preferable as they decrease prices at the bidding stage even though some of the price advantage is lost due to ex post renegotiations (Albano et al., 2006; Chang et al., 2014; Decarolis, 2014). For example, in a natural experiment in Italy switching from the average bid method to the first price auction increased the average discount by 8%-13% points while also increasing cost overruns by 6% of the reserve price and increasing delayed delivery by 28% of the original contractual terms. The combined effect of these is still positive, with procuring bodies’ increased effort to screen unreasonably low bidders having a decisive influence: lengthened decision making time leads to lower initial discounts but manages to counteract two-thirds of the negative effect on cost overruns and delayed delivery (Decarolis, 2014). While there are strong theoretical as well as empirical arguments against using past performance for evaluating bidders claiming that it discourages market entry, under some conditions the downsides can be avoided (Spagnolo, 2012). When some characteristics of the procured product are not easily contractible or the contract is large with many uncertainties past performance can provide a crucial cue as to how the supplier will perform. Unfortunately, there is little systematic evidence on the relative merits of using price-only rather than price plus quality scoring criteria. Correlational evidence comparing EU member states in 2009-2014 reveal a mixed picture with most corrupt countries exploiting the use of non-price criteria for corruptly restricting competition (Fazekas & Kocsis, 2020).

In sum, there is high quality evidence underpinning the impacts and optimal design of some tools in this intervention group, most notably the use of first price auctions. Nevertheless, there is a great need for evaluating further specific interventions especially the corruption susceptibility of different scoring criteria such as different price plus quality criteria. In addition, the almost complete lack of evidence on developing countries calls for extending research geographically.

# 6. <u>E-procurement: e-invoicing, e-payment, and e-contract monitoring</u>

Among the reviewed e-procurement tools, e-invoicing, e-payments, and e-contract monitoring are the least widely used as well as studied (Buyse et al., 2015; OECD, 2011b). Nevertheless, they are briefly discussed 17
as they are generally considered as integral and indispensable parts of a comprehensive e-procurement system and a number of recent high impact reforms entail their implementation (e.g. World Bank STEP e- procurement system<sup>16</sup> ). E-invoicing implies the automated process of issuing, sending, receiving and processing invoices and billing for procurement contracts through electronic means. E-payment means that financial payments between a contracting body and a supplier for a public procurement transaction are made electronically as an integral part of the e-procurement system. E-contract monitoring means the electronic submission and approval of documentation pertaining to contract execution progress and monitoring such as evidence of delivery.

In the absence of any noticeable theoretical and empirical research on impacts of these e-procurement tools, only tentative theoretical arguments are presented here. First, e-procurement tools pertaining to the contract execution stage are likely to decrease administrative costs increasing value for money; in a similar way to other e-procurement tools with the caveats already mentioned. Second, displacement effects between the different stages of public procurement make contract implementation an important locus of public scrutiny with transparency brought about by e-procurement tools being crucial. Strategic bargaining during the contract implementation stage influences bidder behavior and can inflict considerable costs on public budgets (Bajari et al., 2014; Decarolis & Palumbo, 2015). Furthermore, if e-procurement tools used in the bidding and award phases are effective in improving value for money and widen open access, it is highly likely that some degree of mismanagement is displaced to the contract implementation phase, although prior research is mixed on this point (David-Barrett & Fazekas, 2018; Lewis-Faupel et al., 2014).

Empirical work should be carried out regarding each of the identified mechanisms for adequately underpinning policy advice.

# 7. <u>Civil society supervision of contract execution</u>

# _a) Theory_

Civil society or community supervision of procurement contract execution is understood here as any sort of direct involvement of local communities or civil society groups in the monitoring of contract execution next to or on top of traditional horizontal accountability mechanisms (e.g. audits). Civil society or community participation in monitoring development projects more broadly has seen a surge of interest as well as disappointment in the last decades, while it has been much less frequently subject to thorough evaluation delivering robust conclusions (Mansuri & Rao, 2013). Building on general theories of social accountability and citizen monitoring in development projects (Fox, 2015), civil society supervision is expected to increase the probability of detecting low quality delivery as well as its punishment regardless of whether it is due to corruption or simple neglect. This is because local communities directly benefitting from contracts, for example in the case of a road construction or school meal delivery, are incentivized to monitor and try to prevent low quality delivery (Olken, 2007). This strengthens the independent supervision of those managing contract execution who are potentially benefitting from low quality delivery (e.g. extraction of bribes), but avoiding (most of) its costs (e.g. failing local roads).

However, such positive effects depend on the incentive structure of the local community and its capacity to act collectively in pursuance of its goals (Kenny, 2010). Most procurement contracts amenable to local community monitoring create local public goods prone to free-rider problems and the grassroots monitoring process can also be captured by local elites (Bardhan & Mookherjee, 2006). In addition, the observability of ill-delivery of a contract depends on the complexity and technological nature of the procured products limiting the applicability of community supervision. For example, insufficient materials built in a road may not be readily observable to non-experts only visiting construction sites intermittently (Olken, 2007).

> 16 <u>http://www.worldbank.org/en/news/video/2015/05/13/step-procure-better-and-faster-to-achieve-results</u>

18
# _b) Evidence_

While the evidence base for civil society or community monitoring of development projects is much broader, in the specific domain of government contract execution, we could identify only two high-quality field experiments. One looks at village road construction projects in Indonesian in 2003-2004 (Olken, 2007). Community supervision was increased in two ways: inviting villagers to community review meetings where officials had to account for project implementation and distributing anonymous feedback forms which were summarized and publicly discussed. While both interventions increased participation, their effect on road quality as measured by input costs claimed minus inputs spent according to independent engineers’ assessment (i.e. missing infrastructure inputs) was small and statistically insignificant. However, the overall effect masks variation relevant for policy makers: first, missing labor expenditure considerably decreased due to civil society monitoring, while missing material inputs didn’t change suggesting the observability of progress and local interests (i.e. local labor force participating in constructing roads) matter. Second, bypassing local governments in the distribution of comment forms decreased missing expenditure considerably more than in the case of local government officials distributing the forms, probably to their supporters (i.e. local elite capture).

The second field experiment on civil society supervision compares 200 district governments implementing small-scale infrastructure projects, half of which received the intervention of an extended audit treatment by a civil society organization, the other half being the control group (Lagunes, 2017). Crucially, the civil society organization was supported by the national anti-corruption agency officially confirming to the mayors of the targeted district governments that it cooperated with the organization. The experiment therewith combined centralized and citizen-led modes of oversight. The findings indicate that the rate of contract execution did not differ significantly between the groups, but the cost of the public works implemented by district governments in the treatment group were considerably lower than in the control group with 51% cost reduction, saving on average 243,000 Peruvian soles ($75,000) per public works project. This suggests that the collaboration of a civil society with the relevant authorities can be a very cost-efficient tool for supervising contract execution.

Given the narrow, but high quality evidence on the potential effects of civil society monitoring of contract execution, a number of similar exercises should be conducted which could shed light on the best ways of engaging local communities, product and contracts characteristics’ mediating role, and the impact of local elite capture and community power structure more broadly. The relative neglect of this public procurement intervention is particularly troubling as World Bank experience suggests that mismanagement during the bidding and evaluation phases are much less harmful than mismanagement during the contract execution phase for development outcomes (Kenny, 2010).

# **E. Interventions targeting the whole procurement process**

In this section, those interventions are discussed which influence the whole procurement process having a broad impact throughout the procurement cycle. By implication, their impact mechanism might be more complicated and more difficult to adequately measure. On the bright side, targeting the whole procurement process avoids the usual pitfalls of displacing mismanagement from one stage to another. In spite of the promise of a more holistic approach, many impact evaluations reviewed here use indicators bound to only one procurement phase, decreasing the value of evidence for guiding policy.

While a lot of interventions could be identified in the literature, only those are discussed in detail which have received considerable policy as well as research interest. These have been grouped into 7 major groups which are discussed below in turn (Table 2). Some of these interventions more closely intervene in the procurement process itself such as discretionary decision making; while others concern directly linked policies such as watchdog portals.

19
**Table 2. Overview of intervention groups discussed in detail**

|**#**|**Interventiongroup**|
|---|---|
|8|centralized and collaborativeprocurement|
|9|transparencyand watchdog portals|
|10|rule-bound or discretionarydecision making|
|11|Professionalization and capacitydevelopment|
|12|audits and supervisions|
|13|performancepayand incentives|

# 1. <u>Centralized and collaborative procurement</u>

# _a) Theory_

Centralized and collaborative procurement are related practices even though they are typically discussed by different literatures. Nevertheless, both of them are increasingly popular among policy makers and a range of such institutional arrangements have been set up across the globe (OECD, 2015). Under the collaborative procurement model horizontal collaboration is established between two or more procuring bodies to jointly carry out some or all steps of the procurement process (Bakker et al., 2008). A number of successful examples of voluntary collaborative procurement arrangements have been identified in the UK for example (National Audit Office, 2010). Whereas centralized procurement entails the establishment of a central purchasing unit which makes bulk purchases directly from suppliers; then individual procuring entities, or end-users in this case, can purchase from the central unit under set conditions, typically using a central framework agreement (OECD, 2011a). Probably the most widely researched example of centralized procurement is the Italian central purchasing body, CONSIP.<sup>17</sup> From these definitions, it is clear that the hypothesized impact mechanism in these two related cases may be somewhat different.

Collaborative procurement is expected to improve value for money as well as widen access to public contracts due to three main channels, partially overlapping with centralized procurement impact mechanisms (Bakker et al., 2008; Walker et al., 2013). First, collaborative procurement induces learning among procuring entities increasing efficiency of administrative procedures and more efficient purchasing choices. Second, it allows procuring bodies to combine their skills and resources effectively reducing the cost of purchasing. Third, economies of scale may result from demand aggregation. However, these positive impacts can be diminished if the collaborating organizations are not able to effectively align their purchasing activities for example due to incompatible organizational needs or cultures.

While central purchasing units can often take on further rules such as regulation, advice and training, here only their core function of purchasing on behalf of other public sector bodies is discussed. On the upside, centralization can yields three main forms of synergies (OECD, 2000, 2011a): it can yield (i) _economies of scale_ , i.e. price reductions due to volume bundling / market power; (ii) _economies of process_ , i.e. reduction of duplicated efforts in several of the purchasing processes and (iii) _economies of information_ , i.e. better quality information (on market structures and suppliers) can be obtained, is bundled and more easily accessible, and procurement officials can develop more specialized skills. In addition, (iv) introducing a third player at arm’s length can make it harder for buying agencies to collude with suppliers ( _cutting favoritism_ ).

But centralizing purchasing may not deliver these expected improvements in value for money and open access. First, central bodies can become a bottleneck themselves if they are poorly staffed. Added layers of

> 17 <u>http://www.consip.it/en/about_us/</u>

20
oversight can add red tape, rather than cutting it. It can further a compliance- rather than performanceattitude among staff. Second, the entry of a third player can undermine line-agencies ownership of the procurement process and dilute responsibility. When contract implementation and monitoring quality of delivered products are in the hands of the end-user public bodies efficiency gains achieved by central framework agreements can be undermined as end-users may not be willing or able to fully enforce centrally set contractual terms (Albano & Zampino, 2012). Third, standardization can imply that the needs of end users are not fully met. Inflexible standard tender documents may cause complications if they do not fit the needs of a particular contract. Fourth, demand aggregation and the implied larger contracts also concentrate potential corrupt rents raising the attractiveness of agency capture. Fifth, larger contracts may prevent smaller companies from bidding which is likely to drive prices up if there are not enough large companies willing to bid.

# _b) Evidence_

Evidence on collaborative procurement is very limited with a handful of qualitative or small-N quantitative studies with the exception of a UK government report (National Audit Office, 2010), while centralized procurement, especially on the national level, received somewhat more thorough scholarly interest. Unfortunately, neither streams of literature allow for drawing solid conclusions for policy makers.

Qualitative and quantitative studies looking at collaborative procurement practices in the UK and Australia have found that different organizational forms of collaborative procurement perform strikingly differently in distinctive contexts for different types of procuring bodies (Bakker et al., 2008; Barbosa & Fiuza, 2012; Ey et al., 2014; Walker et al., 2013). For example, the level of standardization of the products purchased and user needs define the scope for savings achievable with collaborative procurement: e.g. office stationery versus defense procurement (National Audit Office, 2010).

A high quality quasi-experimental study aiming to estimate the amount of passive and active waste (that is low administrative capacity versus corruption) in Italian standardized goods purchases also touches on the effect of central framework agreements. It finds that centralized purchasing of standard goods such as office stationery achieved a 34%-78% price reduction according to the invoices paid by procuring bodies reaping the benefits of demand aggregation as the central agency can save on transaction costs and has more buyer power that can be exploited to obtain lower prices (Bandiera et al., 2009). Considerably dampening the positive expectations, another Italian study looking at the implementation of the centralized framework agreements by receiving public bodies found that roughly one-third of procuring bodies (30.75%) received products in substantial breech of contractual terms (Albano & Zampino, 2012). More alarmingly, a negligible portion of these procuring bodies have actually issued a penalty to the supplier according to the contract (2.49%). Taken together, such findings highlight the potential for savings through centralized procurement with the risks carried by dissociating tendering and contract management.

Examining the case of CONSIP in Italy, Decarolis (2018), using procurement data on all public contracts awarded between 2015 and 2017, finds that administrations expecting to lose their ability to contract independently game the centralization requirements in three ways. In the short run, they anticipate their purchases to avoid delegating to a central body. In the longer run, they both manipulate contract values, breaking down purchases into smaller lots of amounts below the thresholds driving centralization requirements, and, when given the option, aggregate into the smallest types of centralized purchasing bodies. These three distortions partially offset the potential benefits of the centralization reforms (Decarolis, 2018).

Another study using a unique dataset on tender prices of selected drugs for hospital usage provided by a sample of 52 Italian local health service providers between 2009 and 2012 tests which procurement system (centralized, decentralized or hybrid) performs better. Controlling for several covariates, including measures of institutional quality and corruption, it finds that centralized and hybrid procurers pay lower 21
prices than decentralized units. Moreover, the results show that in areas in which institutional quality is lower or corruption is higher, the effect of centralization in terms of negotiating lower prices is much stronger, with savings of up to 60% of the price paid by local health service providers that procure independently (Baldi & Vannoni, 2015).

In sum, there are strong theoretical arguments for collaborative and centralized procurement (Albano & Sparro, 2010), however the empirical basis is rather scattered and in general of insufficient quality. The organizational and product characteristics which define the savings potential and user satisfaction with such practices need a lot more systematic study. Furthermore, better understanding the interaction between central purchasing bodies, public organizations buying through central framework agreements and supplier opportunistic behavior require more comprehensive study for identifying trade-offs between savings at the bidding stage and difficulties of contract implementation.

# 2. <u>Transparency and watchdog portals</u>

# _a) Theory_

As noted earlier, e-procurement generates a large amount of structured or semi-structured information which can inform bidding decisions, civil society oversight, as well as within-government management decisions. Such rich datasets are almost always publicly released on transparency portals which report announcement data in line with national legislation while allowing for some limited search functions. Typically, there is no way to download the full dataset and providing aggregate statistics is not possible (e.g. total value of contracts won by any particular company in a certain period) (Cingolani et al., 2015). Classic examples of such portals are the EU’s Tenders Electronic Daily<sup>18</sup> or the more recently introduced Bangladeshi government procurement portal<sup>19</sup> to name a few. Watchdog portals, however, offer additional functionalities, typically using the same source data as transparency portals run by governments. These portals allow for monitoring governments from the viewpoint of corruption risks and spending efficiency by providing novel indicators which help making sense of the diverse and often hard-to-interpret data and also allow for calculating summary statistics. Widely publicized, early examples are the Slovakian public procurement portal<sup>20</sup> or its Georgian twin,<sup>21</sup> both run by Transparency International local chapters; examples from developing countries can be found in Nigeria,<sup>22</sup> Indonesia,<sup>23</sup> or Mexico<sup>24</sup> to name a few.

Transparency and watchdog portals are expected to improve value for money and access through influencing bidder behavior as well as underpinning accountability mechanisms vis à vis the society at large and within the government. The impact on bidding behavior is not discussed here in detail as it has been extensively covered in section D.1. Transparency and watchdog portals are expected to support (predominantly) vertical government accountability mechanisms by providing information on bidding processes and actor behavior to civil society. While a lot has been assumed about the transformational impact of transparency and sheer information provision, it is clear by now that they have to be combined with sufficient motivation and capacity to act on the revealed information, in particular civil society monitoring (Center for Global Development, 2014; Kenny, 2010). In addition, the impact of transparency in complex fields such as government procurement hinges on the actual usability and accessibility of the data, e.g. reaching the relevant information in two to three clicks rather than half an hour of digging in diverse websites. Nevertheless, information of government contracts can have a broader, albeit less measurable, impact on public discourse and citizen awareness of public spending (i.e. tax knowledge and

> 18 <u>http://ted.europa.eu/</u>

> 19 <u>http://www.eprocure.gov.bd/</u>

> 20 <u>http://tender.sme.sk/en/reports?cut=contract_date:2013,12</u>

> 21 <u>http://tendermonitor.ge/en</u>

> 22 <u>http://tendermonitor.ge/en</u>

> 23 <u>http://opentender.net/content/database</u>

> 24 - <u>http://mexico.procurement analytics.org/</u>

22
awareness). In addition, public procurement data combined with performance indicators can nurture the spread of best practices in the public sector.<sup>25</sup> In spite of such promises a range of caveats apply. First, information available on official government and watchdog portals is typically on high level of aggregation (e.g. containing total contract value, but no information on the prices of individual inputs of deliverables) and often contain missing or erroneous information limiting its use for civil society monitoring. Especially as information on project execution progress and payments is typically missing from procurement portals, transparency may achieve little on the ground (Cingolani et al., 2015). Second, effective monitoring of complex transactions and projects, typical of public procurement, requires direct observability of performance and mismanagement. This may well be the case in construction, but in services such as education and training provision, lack of easy observability of mismanagement can render transparency ineffective (Banerjee et al., 2010). Third, costs of making data publicly available on transparency and watchdog portals may be substantial with oversight and monitoring costs even higher, significantly reducing net benefits or even turning them into negative.

# _b) Evidence_

Evidence on the effects of transparency and watchdog portals is very limited, it by and large incorporates non-systemic collections of suggestive facts and perception survey data (Šípoš et al., 2015). No experimental or similar study could be identified which would reliably establish causal relationships.

While no rigorous economic analysis underpins the above theoretically sound arguments, some data from the Slovakian watchdog portal provides suggestive evidence (Šípoš et al., 2015). The reach of public procurement information across the Slovakian population has increased dramatically from practically null to 11% of the total population claiming to have checked at least one contract online since 2011. In addition, media reporting of public procurement related scandals increased often making explicit use and reference to the watchdog portal: from 877 articles in 2003-2006 to 1765 articles in 2011-2014. Furthermore, where estimates exists, the additional cost of running transparency and watchdog portals is negligible not only for the managers of the portal but also for the organizations imputing data (Center for Global Development, 2014). These costs appear unimportant in particular when set against the billions of USD spent through them annually.

Given the high and quickly growing number of transparency and watchdog portals around the globe – Czech Republic, Hungary, Indonesia, Nigeria, Philippines, the Slovak Republic, Ukraine, and Vietnam, to name a few – rigorous evaluations are indispensable. In particular, understanding the conditions under which such portals can have a positive effect on access to public contracts is a key concern.

# 3. <u>Rule-bound or discretionary decision making</u>

# _a) Theory_

The dichotomy of rule-bound and discretionary decision making in public procurement is frequently reduced in the literature to the dichotomy of using (open) auctioning versus negotiation procedures in the bidding and award phases (Bajari et al., 2009).<sup>26</sup> However, it has broader ramifications than that implying impacts at the contract execution stage (Rasul & Rogger, 2015) as well as more generally referring to bureaucratic behavior. Hence, these set of tools are more broadly defined: one the one hand, discretionary or autonomous decision making in public procurement implies decision making freedom and flexibility of bureaucrats and procurement administrators throughout the procurement cycle bound only by general principles of good administration such as civil service code of conduct and organizational culture (Kelman,

> 25 One watchdog site with explicit purpose of spreading best practices in the Czech Republic: http://zindex.cz/

> 26 While this implies the general neglect of the issue of discretion in the contract execution phase which would place this discussion in section D; the general character of the issues rather leave it among interventions of more holistic character.

23
1990). On the other hand, rule-bound decision making in public procurement denotes the severely restricted decision making freedom and flexibility of bureaucrats and procurement administrators throughout the whole procurement cycle (Rasul & Rogger, 2015).

On the one hand, rules binding bureaucrats to carry out contracting in an open, fair, and competitive manner (e.g. using fully open auctions) are expected to reduce corruption, increase competitiveness, drive prices down, and improve quality. Such positive outcomes are predominantly down to rules forcing bureaucrats to conduct competitive procedures as opposed to corruption-ridden or convenience-driven (i.e. suing the same-old supplier again and again) contracting. Furthermore, highly regulated bureaucratic procedures carry the image of objectivity and fairness which can make politicized bureaucracies to opt for such procedures even though they have considerable costs in terms of value for money (Coviello et al., 2018). On the other hand, discretionary decision making is expected to produce very similar positive outcomes based on the presumption that rigid rules prevent bureaucrats to run contracting efficiently in a manner best suited to the particular circumstances of each tender. Discretion can be used to select the most productive suppliers to bid considerably decreasing transaction costs and room for negotiation with reputable suppliers resulting in better formulated and more enforceable contracts (Chever et al., 2017).

While in the economics literature, contract complexity and completeness are the key characteristics defining whether rule-bound decision making (i.e. auctions) or discretion (i.e. negotiated procedure) produce better results (Bajari et al., 2009). However, it is apparent that neither rules nor discretion are homogenous treatments. There are good and bad rules, hence the impacts crucially depend on matching the right rules to the right tenders and contracts (Parrado, Dahlström & Lapuente, 2018). Moreover, discretion can be used to many ends depending on incentive structures set by organizational management practices and norms, politicization of the bureaucracy, or informal power relations. Unfortunately, with the notable exception of (Rasul & Rogger, 2015) neither of these considerations are taken seriously bringing considerable theoretical opacity to the debate.

# _b) Evidence_

There is a good number of high-quality research papers comparing auctions and negotiated procedures as a proxy for rule-bound versus discretionary decision making in public procurement, some exploiting natural experiments some others explicitly modelling the selection of procedure type to better identify causal links. Unfortunately, only one paper could be identified which opened up the black box of bureaucratic decision making and management in order to address the discretion versus rule dichotomy at its core.

The literature on the effects of using negotiated procedure or auction is divided on their relative merits. For complex products, negotiations appear to produce better outcomes. Exploiting a discontinuity in the rules governing procedure type selection in Italy, negotiated procedures lead to a higher likelihood of the same firm winning repeatedly while projects are delivered faster and larger firms win more often (albeit the latter two effects are present in a smaller neighborhood around the discontinuity) (Coviello et al., 2018). While the number of bidders decrease this does not decrease discounts. Research on social housing construction in Paris, France, come to similar conclusions regarding fewer bidders invited in negotiated and nonformalized procedures, however the authors also find lower prices (Chever & Moore, 2012). While the analyzed construction contracts could be considered as complex, further evidence from France using tendering data on small and simple purchases point at the same direction (Chever et al., 2017). Contrary to these findings, the analysis of German passenger rail line auctions, come to the conclusion that auctions led to 16% more trains for 25% lower prices compared to lines awarded in negotiated procedures (Lalive & Schmutzler, 2011). What drives these differences is unclear unfortunately. Regarding corruption risks, the analysis of an Asian trading firm’s internal records of bribery and data on procurement auction participation suggests that the mandatory implementation of auctions in the public sector led to a significant decrease in bribery, albeit at the cost of allocative efficiency (i.e. less productive firms winning) (Tran, 2008).

24
The only paper to our knowledge which explores managerial practices within public agencies delivers highly policy relevant results worth further exploring in other contexts (Rasul & Rogger, 2015). Using a survey of Nigerian civil servants, bureaucratic autonomy and the use of incentives/monitoring within agencies has been established and linked to project performance in the social sector. Surprisingly, 1 standard deviation increase in bureaucrats’ self-reported discretion led to 18% higher project completion rate whereas one standard deviation increase in the use of incentives/monitoring resulted in 14% lower project completion rate. Project complexity, ambiguous project design, and organizational IT facilities mediated these observed relationships. While the findings cannot be interpreted as causal, they do point at the significant and non-trivial impact of public sector management practices on value for money and corruption.

Overall, the limited evidence on the role discretion and rule-bound decision making play in producing public procurement outcomes calls for significantly more research. First, the impact of different procedure types clearly varies with contract as well as organizational characteristics which need to be more directly addressed if seemingly contradictory findings are to be reconciled. Second, the black box of bureaucracies must be opened if we are to understand procurement outcomes, for example analyzing the impact of diverse management practices, politicization of the bureaucracy, and organizational control mechanisms could lead to policy advice with high impact.

# 4. <u>Audits and supervision</u>

# _a) Theory_

Our discussion of audits and supervision is restricted to those specifically designed for public procurement rather than broader processes such as financial audits concerning entire public organizations (for the discussion of such broader institutions’ impact see section C.2). Audits and supervision can be carried out by any organ of the state which is to some degree independent of the contracting body, that is we focus on horizontal accountability mechanisms.

Audits and monitoring by higher-level or independent state organs are expected to increase the risk of detecting misconduct, corruption in particular, and the threat of punishment as a result. Increased risk of punishment, in turn, contributes to higher levels of compliance with rules and lower corruption. This basic model resting on elementary microeconomic theory assumes that the monitoring body is non-corrupt (Becker & Stigler, 1974). If this is not the case, monitoring can result in simply reallocating rents from one organ of the state to another (Olken, 2007). If audits and monitoring are uncertain and even rule-abiding bureaucrats can be found guilty, they can generate a culture of fear which stifles innovation and creativity (Kelman, 1990).

# _b) Evidence_

There is a small number of high-quality research papers investigating the impact of audits and supervision in public procurement (Fazekas & Tóth, 2017; Lagunes, 2017; Olken, 2007) with some further correlational studies (Albano & Zampino, 2012); however, overall the issue has received only modest interest in the research and policy community.

A randomized controlled field experiment looking at village road construction projects in Indonesia in 2003-2004 found that the increase of audit probability from 4% to 100% led to the reduction of missing infrastructure spending of 8% points (i.e. lower corruption) (Olken, 2007). This is a substantial reduction from a baseline of 24% missing spending (materials and labor combined). Interestingly, the main channel of influence was not criminal proceedings which was quite rare, rather publicly reading the audit results on open village meetings. The author further found that parallel to reduced missing expenditure the project jobs given to family members also increased suggesting substitution between different forms of corruption and patronage.

25
A comparable study looked at the prices of homogenous hospital inputs such as ethyl alcohol as a proxy for corruption in Argentine in 1996-1997 (Di Tella & Schargrodsky, 2003). It exploited the exogenous variation in monitoring activities, that is the introduction of full monitoring of some input prices and a public campaign again corruption led by the city hall. The increased monitoring regime led to a 14.6% decrease in input prices while a loosening of this regime lowered the positive impact to 11%. In a rather different setting, the EU’s single public procurement market in 2009-2014, a study looked at the impact of the European Court of Justice’s decisions striking down anticompetitive practices on the basis of the EU Public Procurement Directives (Fazekas & Tóth, 2017). Comparing procuring body behavior (e.g. use of exceptional procedures) as well as bidding outcomes (i.e. number of bidders) from before to after the decisions entered into force suggest that monitoring by EU courts – arguably not captured by local elites – decrease the incidence of corruption-related anticompetitive practices by 5%-30% depending on the country-group studied. Similar to the long-term results identified in Argentina, the evidence from the EU confirms that once monitoring efforts decrease positive effects fade away.

Using a natural experiment in a large public sector organization in Russia, Tkachenko, Yakovlev, and Rodionova (2017) examined the impact of increased procurement monitoring in two different organization types: income earning (commercial) units (IEU) that earn from paid services and operate under hard budget constraints, and non-commercial units (NCU) that provide services mostly free of charge and receive financial support from the government. In the middle of the period under review (2008–2013), the top management of this public sector organization decided to expand procurement monitoring from a focus on the supplier selection stage to all stages of the procurement process. Similar to the work of Di Tella & Schargrodsky (2003), empirical analysis showed that under standard monitoring, which was focused only on the supplier selection stage, procurements of IEU were characterized by higher effectiveness, measured by the number of bidders that applied and were admitted to the auction and the length of delays in contract executions in days in two comparable groups of services that were actively procured by both types of divisions, namely printing and data collection services. However, after the intensification of centralized monitoring, which covered all stages of the procurement cycle, the differences between IEU and NCU became insignificant. At the same time, a slight increase in procurement effectiveness was registered in the second period for NCU, and a relative decrease was registered for IEU. These findings show that stricter monitoring appears to be efficient for organizations with soft budget constraints, while for organizations with hard budget constraints it is preferable to use more flexible regulations.

Overall, the potentially positive effect of audits and supervision by higher levels of government or courts is confirmed, however, there are a number of open questions: first, what are the impact channels in different contexts: strengthening socio-political accountability (Indonesia case) or relying on purely bureaucratic channels (EU case)? Second, what are the long term effects of monitoring, especially when public attention and detection probability drops. Third, how to set up institutions where the monitoring and monitored bodies are sufficiently independent of each other to result in the expected accountability relationships. Furthermore, a lot more studies need to explore the very same mechanisms in different context in order to arrive at a robust evidence base underpinning effective policies.

# 5. <u>Performance pay and incentives</u>

Performance pay and other incentives provided to public bureaucrats on the individual or group levels have been one of the great fads of the new public management literature while receiving considerable attention in economic studies too (Hood, 1991; Hood & Dixon, 2015; Shah, 2007). However, very few of the central tenets have been applied and evaluated in public procurement. There is a very recent impact evaluation in Pakistan combining performance incentives with increased discretion in the frontline procuring bodies. It has begun unpacking the crucial research and policy questions in this area, suggesting that autonomy leads to lower prices while performance pay does not (Bandiera et al., 2020) 26
In the absence of sufficiently broad theoretical and empirical research on the impacts of performance pay and other incentives, only tentative theoretical arguments are presented here. Linking procurement administrators’ pay to procurement outcomes such as discounts achieved, number of bidders, or project completion on time and budget can potentially increase their effort hence contribute to better value for money and counter corruption. This impact pathway nevertheless crucially depends on a number of conditions: first, bureaucrats’ effort should be able to meaningfully and measurably influence outcomes which may not be the case if for example inter-bidder collusion puts competition off. Moreover, if bureaucratic action is over-regulated to the degree that even the best effort and skill cannot push a tender beyond the required minimum process, again the link between bureaucrats’ effort and outcomes is broken. Second, public bureaucracies are often governed by informal rules and power relations, especially in developing countries which implies that it is very hard to incentivize those who are really in charge. For example, incentivizing a front-line procurement administrator to widen access to tenders while his boss is corruptly linked to a particular bidder is likely to create frustration rather than better outcomes. Third, material rewards have proven to crowd out intrinsic motivation in a number of contexts including the public sector and procurement project management (Rasul & Rogger, 2015).

Empirical work should be carried out regarding each of the identified mechanisms for adequately underpinning policy advice.

# 6. <u>Professionalization and capacity development</u>

As shortage of staff and lack of capability has been identified as one of the primary barriers to effective procurement reform (Telgen et al., 2016), professionalizing and improving the capacity of the public procurement workforce has been one of the key reform avenues sought by developed as well as developing countries in the last decade (OECD, 2012; World Bank, 2007). It is acknowledged that these reforms need to involve providing training for procurement staff advancing their skills and knowledge of procurement systems and management techniques, offering specialist advice, and supporting effective leadership of and knowledge sharing among procurement bodies (Telgen et al., 2016; UK Office of Government Commerce, 2007).

# _a) Theory_

Unfortunately, only one systematic assessment of public procurement professionalization and capacity development programs could be identified which provides some pointers for impact mechanisms, potential barriers and enablers (Telgen et al., 2016). Hence, the theoretical considerations enlisted here remain preliminary. First, given that public procurement regulations and IT systems are highly complex and dynamically change, capacity development is essential for increasing and updating comprehension among procurement administrators even for maintaining compliance with the most basic rules (OECD, 2009). Second, increasing the performance rather than compliance orientation among staff, for example through the spread of state-of-the-art supply chain management techniques, is a key promise of capacity development trainings. Third, establishing a procurement code of conduct or code of ethics and promoting integrity can also be pursued through training and workshops (Telgen et al., 2016). All these point at the potential of capacity development for improving value for money as well as widening access to public contracts. However, in organizations plagued by systemic corruption, it will only achieve more sophisticated corruption strategies (i.e. making corrupt transactions look legal) if anything. Similarly, if it has an insufficient mandate or when motivations, whether intrinsic or extrinsic, are low, more information and knowledge will achieve little more than a tick-box exercise.

# _b) Evidence_

The aforementioned meta-study by Telgen et al. (2016) examines the existing evidence on the impact of interventions that aim to improve transparency in public procurement. It uses an evidence base consisting 27
of 48 studies of varying quality with the geographic scope focusing on developing countries, including countries in the Middle East and North Africa, South Asia, and Sub-Saharan Africa. Overall, the sum of the studies reveals that a lack of capacity and knowledge is one of the main problems deteriorating procurement outcomes. The review finds that the development of procurement courses is delivered positive results consistently across 20 studies, rendering sufficiently trained staff at procuring bodies one of the most important criteria for successful reform. More broadly, the quality of bureaucrats has been found to account for about 20% of variation in prices paid for standardized goods in the Russian Federation (Best et al., 2017).

These positive results however are circumscribed by post-reform challenges such as the insufficient mandate and conflict of interests in authorities or inefficiency due to unclear, lengthy tendering procedures. Three institutional conditions are found to be supportive of successful and sustainable reform. First, the provision of appropriate rewards for procurement staff (including an appropriate financial reward system and a career path) can help attract and retain quality staff and make staff less susceptible to bribery. Second, the implementation of a procurement code of conduct or code of ethics can provide much needed guidance in situations of conflicts of interest or corruption. Third, not only the training of government staff, but also of private sector actors is crucial for successful training interventions. Training contractors and bidders may involve information on how to work with the new and changed documentation and expected procurement behaviors (Telgen et al., 2016).

The evidence base of this review consists of studies with mainly qualitative data, which do not provide sufficient hard evidence to assess the effect size of public procurement interventions. This is due to the lack of comparable data available from before and after interventions, or the difficulties of measuring an intervention that is part of a bigger reform package. In consequence, further empirical work should be carried out regarding each of the identified mechanisms for adequately underpinning policy advice.

# **F. Summary of evidence on interventions**

This section briefly reviews both the identified knowledge gaps and the main substantive conclusions from reviewing public procurement interventions’ impact on value for money and access to public contracts.

Overall, the quality of evidence on public procurement interventions’ impact is mediocre with some selective, typically very narrow tools boosting reliable evidence established in multiple countries using diverse analytical methods (Table 3). However, overall, there is a range of policy tools with global policy interest and extensive implantation record receiving little to no evaluation. Research priorities and policy agendas are detached to a considerable extent which is probably due the difficulties of accessing reliable data and conducting high-quality publishable research. If governments and international organizations want to rely on better evidence in public procurement policy making and implementation, they have to provide the access and means for research targeting their needs. Considering that about 15% of global GDP flows through public procurement systems, the amount of research available is alarming; if such spending is to be better controlled considerably more systematic evidence needs to be gathered and synthesized at regular intervals. The good news is that there is an emerging vocabulary, research community, and analytical arsenal laying the foundations for systematic knowledge accumulation.

New research, considering current policy directions, savings potential, and research gaps, could concentrate on:

- e-procurement and variants, with particular focus on contract execution monitoring which should also address the general lack of data on final contract performance across the globe;

- centralized framework agreements which have high savings potential, but estimates should simultaneously consider prices, quality and user satisfaction;

28
- transparency and accountability portals whose proliferation is hard to miss while our understanding of how and under which conditions they work is very limited;

- exploring the dichotomy of tight administrative rules and bureaucratic discretion in public procurement administration in order to understand why and when rules stifle good management and how discretion can be used for public good rather than private gain;

- opening up the black box of public administration and management practices which fundamentally determine procurement outcomes; and

- training and capacity development programs.

29
**Table 3. Summary of evidence and the assessment of its quality by intervention group**

|**nr.**|**intervention name**|**procurement**<br>**h**||<br>|**evidence**<br>|||
|---|---|---|---|---|---|---|---|
|||**pase**|**key references**|**countries**|**markets**|**periods**|**quality**|
|1|e-procurement: e-notification,<br>e-access, e-attestations & e-<br>submission|tender preparation<br>& advertisement|Lewis-Faupel et al (2014)<br>Buyse et al (2015)<br>Strand et al (2011)<br>Coviello-Mariniello(2014)|EU, Italy,<br>India,<br>Indonesia|general procurement,<br>infrastructure|2000-2015|limited<br>systematic<br>evidence|
|2|framework agreements|tender preparation<br>& advertisement|Bandiera et al (2009)<br>NAO (2010 2013)<br>Gur et al (2015)<br>Albano-Sparro (2010)|Italy, UK|general procurement,<br>standard goods|2000-2005 and<br>2010-2013|limited<br>systematic<br>evidence|
|3|e-procurement: e-auction & e-<br>evaluation|bid evaluation &<br>contract award|Pavel-Sicakova-Beblava (2013)<br>Yakovlev et al (2014)<br>Soudry (2004)|Russian<br>Federation,<br>Slovak<br>Republic|public works, IT,<br>sugar|2007-2009 and<br>2011|limited<br>systematic<br>evidence|
|4|preferential treatment of<br>bidder and product classes|bid evaluation &<br>contract award|Marion (2007)<br>Nakabayashi (2013)<br>Krasnokutskaya-Seim (2011)<br>Albano et al (2008)|Italy, Japan,<br>USA|general procurement,<br>roads, construction|1996-2009|high quality<br>systematic<br>evidence|
|5|award mechanism and auction<br>design|bid evaluation &<br>contract award|Blancas et al, 2011<br>Albano et al (2006)<br>Chang et al (2013)<br>Decarolis (2014)<br>Butler et al (2013)<br>Spagnolo (2012)<br>Fazekas-Kocsis(2017)|Brazil, EU,<br>Italy|general procurement,<br>construction, social<br>spending|2000-2014|high quality<br>systematic<br>evidence|
|6|e-procurement: e-invoicing &<br>e-payment & e-contract<br>monitoring|contract execution|-|-|-|-|no<br>systematic<br>evidence|
|7|civil society supervision of<br>contract execution|contract execution|Olken (2007)<br>Fox (2015)<br>Mansuri-Rao (2013)<br>Lagunes (2017)|Indonesia,<br>Peru|road construction|2003-2004 and<br>2015-2016|limited<br>systematic<br>evidence|

30
|8|centralization of procurement<br>functions and collaborative<br>procurement|all|Bandiera et al (2009)<br>Barbosa-Fiuza (2011)<br>NAO (2010 2013)<br>Bakker et al (2013)<br>Walker et al (2013)<br>Ey et al (2014)<br>Albano-Sparro (2010)<br>Albano-Zampino (2012)<br>OECD-SIGMA (2000 2011)<br>Decarolis (2018)<br>Baldi & Vannoni (2015)|Australia,<br>EU, Italy,<br>UK|general procurement,<br>homogenous goods|2000-2005<br>2008-2009<br>2013-2014|high quality<br>systematic<br>evidence|
|---|---|---|---|---|---|---|---|
|9|transparency and watchdog<br>portals|all|Sipos et al (2015)<br>Kenny (2010)<br>Center for Global Dev. (2014)<br>Huter-Chaturian(2014)|Georgia,<br>Slovak<br>Republic|general procurement|2003-2014|limited<br>systematic<br>evidence|
|10|rule-bound or discretionary<br>decision making|all|Coviello et al (2014)<br>Tran (2008)<br>Bajari et al (2009)<br>Rasul-Rogger (2015)<br>Chever et al (2012 2013)<br>Lalive-Schmutzler (2011)<br>Kelman (1990)<br>Parrado, Dahlström & Lapuente<br>(2018)|China,<br>France,<br>Germany,<br>Italy,<br>Nigeria, US|general procurement,<br>construction, social<br>spending|1995-2009|limited<br>systematic<br>evidence|
|11|professionalization and<br>capacity development|all|Telgen et al (2016)|Low- and<br>middle<br>income<br>countries|General procurement|2005-2016|limited<br>systematic<br>evidence|
|12|audits and supervisions|all|Olken (2007)<br>Fazekas-Tóth (2017)<br>Di Tella-Schargrodsky (2003)|Argentine,<br>EU,<br>Indonesia|general procurement,<br>hospital supplies,<br>roads|1996-1997<br>2003-2004<br>2009-2014|limited<br>systematic<br>evidence|
|13|performance pay and<br>incentives|all|-|-|-|-|no<br>systematic<br>evidence|

31
The public procurement research is not yet ready for drawing firm conclusions as sometimes different outcome measures are used and data and analysis are of vastly varying quality, still as a tentative synthesis, it is possible to enumerate the biggest identified effect on prices by intervention area (Table 4). This comparison, of course, hides the debates within areas such a different studies’ finding opposing effects, but very well demonstrates the promise of different tools and reforms. While the baseline is drastically varying in each study ranging from high income countries such as the UK to developing countries such as Indonesia, using percentage changes still gives a rough idea of which effects to expect from successful interventions.

Looking at the roughly comparable effect sizes per intervention group, centralized framework agreements stand out (rows 2 and 8) with a significantly larger effect. This is hardly a surprise as all other interventions basically keep most of the institutional and human capital intact while modifying one aspect of public procurement administration. However, central framework agreements create an institution and purchasing system outside the traditional channels which fundamentally reconfigures procurement delivery. Each of the other intervention groups can hope to achieve about 5%-10% price savings if well implemented which on its own is very substantial, especially given the huge amounts of public money flowing through procurement systems around the world. Considering that about the estimated 11 trillion USD spent on procurement annually around the world, 1% saving on it amounts to 110 billion USD annually. Small reforms can take governments far.

However, even the most successful reforms achieving price savings may be susceptible to corruption and capture with corrupt actors flexibly adjusting to new circumstances and new regulations. Hence, no intervention can be expected to maintain its performance without regular checking and adjustments.

Interestingly, some interventions look merely technical fixes to underlying governance weaknesses such as changing the way the bidders’ scores are calculated or moving thresholds for applying auctioning mechanisms. Still, the evidence shows that they can deliver as substantial results as more fundamental reconfigurations of the procurement system such as introducing e-notifications. As a reference, we also considered some even broader reforms outside of public procurement such as improving civil service meritocracy or electoral competition (Table 4, rows A1 and A2). Crucially, these broad reforms often achieve just as high price reductions as the much narrower reforms limited to public procurement. Of course, more encompassing reforms achieve a lot more than just improving procurement outcomes the effects of which have not been considered here. In addition, weighing reform implementation costs with potential impacts is a calculation which is partially already possible with the available evidence, but with the addition of new results could become standard practice.

32
**Table 4. Summary of largest identified effect size on prices by intervention group**

|**nr.**|**intervention name**|**effect size (% savings**<br>**from baseline)**|**country**|**market**|**period**|**source**|**intervention**|
|---|---|---|---|---|---|---|---|
|1|e-procurement: e-notification, e-<br>access, e-attestations & e-submission|1.1.-5.3%*|Italy|public works|2000-<br>2005|Coviello-<br>Mariniello<br>(2014)|publishing in regional official gazette and 2<br>provincial newspapers<br>vs local administration noticeboard|
|2|framework agreements|34-78%**|Italy|homogenous<br>goods|2000-<br>2005|Bandiera et al<br>(2009)|central framework agreement<br>vslocalprocurement contract|
|3|e-procurement: e-auction & e-<br>evaluation|5.8-6.7%***|Russia|sugar|2011|Yakovlev et al<br>(2014)|e-auction<br>vs request for quotation procedure|
|4|preferential treatment of bidder and<br>product classes|0.10-0.23%|Japan|public works|2005-<br>2009|Nakabayashi<br>(2013)|60% of budget set aside for SMEs<br>vs no set-asides|
|5|award mechanism and auction design|7-8%*|Italy|road<br>construction<br>and repair|2000-<br>2006|Decarolis<br>(2014)|first price auction<br>vs average bid auctions|
|6|e-procurement: e-invoicing & e-<br>payment & e-contract monitoring|n.a.|-|-|-|-|-|
|7|civil society supervision of contract<br>execution|51%|Peru|small-scale<br>infrastructure<br>projects|2015-<br>2016|Lagunes (2017)|civil society audit (supported by the audit<br>general) vs no audit|
|8|centralized and collaborative<br>procurement|60%|Italy|pharmaceuticals|2000-<br>2005|Baldi &<br>Vannoni (2015)|central purchasing<br>vs local procurement contract|
|9|transparencyand watchdog portals|n.a.|-|-|-|-|-|
|10|rule-bound or discretionary decision<br>making|25%|Germany|train services<br>licences|1994-<br>2004|Lalive-<br>Schmutzler<br>(2011)|auctions<br>vs negotiation procedure|
|11|professionalization and capacity<br>development|n.a.|-|-|-|-|-|
|12|audits and supervisions|11%|Argentine|homogenous<br>hospital inputs|1996-<br>1997|Di Tella-<br>Schargrodsky<br>(2003)|100% monitoring<br>vs no monitoring|
|13|performance pay and incentives|n.a.|-|-|-|-|-|
|A1|weak electoral competition|1-4%*|UK|general<br>procurement|2009-<br>2013|Fazekas (2015)|same party control 66%+ of seats continuously<br>in 2006-2015<br>vs more competitive elections|
|A2|civil service meritocracy|0.6-1.3%*|EU-27|general<br>procurement|2009-<br>2013|Charron et al<br>(2017)|3 standard deviation increase in public sector<br>meritocracy|

Notes: * using discounts compared to the original price estimate; ** centralized procurement & framework agreements jointly estimated; *** % change in unite price compared to regional sugar price 33