outcomes in question. Theoretical arguments necessarily provide the basis for interpreting and evaluating empirical findings as well as for identifying evidence gaps. The availability of rigorous qualitative research tracing causal pathways and mediating factors is also considered essential for a high quality evidence base. As the evidence is scant for many interventions, there is only rarely robust quantitative as well as qualitative research, making the findings of this review tentative in many respects unfortunately.

The rest of the review is structured as follows: first, the broader background of public procurement interventions is spelled out in terms of new developments brought about by Bid Data and the data revolution. In addition, those broader groups of interventions are also briefly enlisted, which lie outside the public procurement system, but bear strong influence on procurement outcomes. Second, each of those public procurement interventions relating to a particular phase of the procurement process is reviewed. Third, public procurement interventions with a broader impact are reviewed. Fourth, conclusions are drawn, in particular by highlighting the need for further research.

# **B. Methodology**

In order to take stock of the evidence on major public procurement reforms and their impact on value for money and open access, we implemented a structured search and assessment strategy building on both standard online search and the expert knowledge of the authors and reviewers of earlier drafts of the manuscript. First, we defined the main areas of policy interventions both for those which target specific procurement phases and those which target the whole procurement process. As the choice of intervention groups defined both the search strategy and the structure of the review, we sought extensive validation of these choices at the outset. Second, we aimed at identifying as many potentially relevant academic publications as well as policy reports for each intervention area. We built on the authors’ knowledge of the field as a key starting point, while we also applied standard keyword search to each intervention group using specific keywords such as “e-procurement”, “impact”, “value for money”, etc. Once a broad body of potentially relevant documents was identified we followed up on widely cited authors’ publication list and also explored widely cited documents’ citation network.

Third, we screened and selected studies in each intervention group. Given the small number of high-quality studies in most areas, we could not apply strict selection criteria, in essence retaining most relevant studies. We excluded only those studies and reports which were not clearly linked to one of our intervention groups and the methodology was of particularly low quality (e.g. ) or narrow in scope (e.g. only discussing a few case studies whose representativeness is questionable). Fourth, the selected studies were assessed in-depth, classified and prioritized. We aimed to prioritize studies which applied the highest quality empirical methods combined with sound theoretical framework. Experiments and quasi experiments were of particular interest for example studies exploiting sharp thresholds of applying new rules or randomized controlled trials in real life settings (i.e. field experiments). We also considered observational studies, that is studies relying on observing behavior without intervention or manipulation, even though they are less reliable in identifying causal effects. Given the size of public procurement markets (often consisting of tens of thousands of contract awards even in a small country per year), we prioritized large-N studies (i.e. studies analyzing tens or hundreds of thousands of observations) while also occasionally taking into account smallN quantitative studies. We only very rarely took into consideration qualitative studies and case study methods as their results typically do not easily compare to the high quality quantitative studies composing the bulk of evidence reviewed and their findings are often not representative. Furthermore, we aimed at considering a geographically balanced set of studies in the review, in particular including studies from both developed and developing economies.

Finally, we carried out an in-depth appraisal and synthesis of the selected body of knowledge. This included both conceptual synthesis and the summary of empirical findings. Given the small number of directly comparable studies in each area, we could not systematically compare effect sizes across many studies, no 4