Supplementary material The tables and figures provided here offer brief overviews of the selected 59 articles that characterize the effectiveness literature between 1996 and 2016. Tables 1, 2, 3, 4, and 5 categorize the studies first by procedural, substantive, transactive, normative, and multidimensional effectiveness dimensions, respectively. Next, within each effectiveness dimension table, studies are grouped by their respective methods with a brief study description, methods used, strengths, and weaknesses. Figures 1, 2, and 3 highlight geographic, thematic, and temporal trends, respectively. Note that the totals for the figures below may differ due to studies having no or multiple location foci or one or more effectiveness dimension themes. Table 1 Procedural Effectiveness Methods Description Method(s) Strengths Weaknesses References National EIA systems: provide historical and legal context; focus on EIA process with occasional descriptions of EIA systems; some descriptions of EIA practice via interviews Case study or comparative case studies; interviews; workshops Provide various frameworks for highlighting the procedural and structural peculiarities of EIA systems; high internal validity Lower external validity; difficulty in repetition; cannot evaluate EIA outcomes other than efficiency of process (Androulidakis and Karakassis 2006; Appiah-Opoku 2001; Arts et al. 2012; Badr et al. 2011; Bassi et al. 2012; Clausen et al. 2011; El-Fadl and El-Fadel 2004; Glasson and Bellanger 2003; Glasson and Salvador 2000; Haydar and Pediaditi 2010; Heinma and Põder
2012; Guerra et al. 2015; Toro, Requena, and Zamorano 2010) History tracing case studies with evolutionary model proposed Case study; history tracing High internal validity; rich case narrative; impact of external events on EIA systems; long term focus of how EIA processes, EIA systems, and actors’ behaviors change No or low external validity (Gibson 2002; Ogunba 2004) EIS documental reviews (aggregated or disaggregated) ESRP (1999); EU EIS review guidelines (2001); Oxford Brookes University Environmental Impact Statement Review Package (2005); Index systems Easily applied to various EISs; provides common evaluation framework Highly subjective; reliable repetition is difficult; EIS has questionable impact on decision-making; cannot evaluate EIA outcomes other than efficiency of process (Lee et al. 1999; EC 2001; Glasson et al. 2005; Naser 2015; Chang et al. 2012) Table 2 Substantive Effectiveness Methods Description Method(s) Strengths Weaknesses References
EIA report length, time, and cost measures; results are perceptual SEA case study Case study; interviews; checklists Identifies link with substantive dimension; delineate criteria for substantive and transactive effectiveness Results are perceptual; does not address (Theophilou et al. 2010) Project analysis of 88 EIAs Case study Provides empirical evidence for timeliness of EIA steps Unclear how project characteristics affect timeliness; does not address cost-effectiveness; does not address timeliness of individual EIA process steps (Middle and Middle 2010) Analysis of 15 EIAs in Switzerland Comparative case study; qualitative comparative analysis; configurational analysis Identify necessary and sufficient conditions for a timely and effective EIA process; can account for conditions’ interactions Limited context descriptions, clear causality cannot be established; limited diversity of empirical cases; overly deterministic; does not address (Befani and Sager 2006) Table 4 Normative Effectiveness Methods
Description Method(s) Strengths Weaknesses References Discourse theory studies Case studies; surveys; interviews Problems of passive participation; challenges of integrating stakeholders’ objectives; identifies the asymmetrical power relationships of stakeholders Perceptual results; assumed link between stakeholder discourses and policy outcomes; often rely on one stakeholder group; unclear how to steer plurality towards sustainable development (Rozema and Bond 2015; Runhaar et al. 2013; Wiklund 2005) EIS Analysis Case study, document analysis, and interviews Identifies the points where environmental concept narrows Causes of narrowing unclear; policy recommendations are vague; focuses exclusively on EIA process and not on wider EIA system (Kørnøv et al. 2005) Analyzes 12 EIAs operationalizing aspects of actor capacity Comparative case study Identifies links between procedural and normative effectiveness dimensions Insufficient number of EIAs for representative samples (Kolhoff et al. 2016) Table 5 Multidimensional* Methods
Description Method(s) Strengths Weaknesses References Compare biomass crop planning in São and in England Comparative case study; checklist based on effectiveness dimensions Evaluates procedural, substantive, transactive, and normative dimensions Unclear how various dimensions are connected; transactive effectiveness measured only temporally (Gallardo and Bond 2011) Case study of SEA for city planning Case study; fuzzy analytical hierarchy process Evaluates procedural, substantive, and normative effectiveness; compares importance of dimensions; integrate stakeholder values into one model Relies on one stakeholder group (Wang et al. 2012) Theoretical of effectiveness dimensions for SA Theoretical discussion and literature review Evaluates links of procedural, substantive, transactive, and normative dimensions; explicitly incorporates pluralism and learning; identifies central role of procedural dimension Unclear how SA should become more iterative (Bond et al. 2013a; 2013b)