Appendix 4: Methods for assessing effects of options Qualitative methods / techniques Something for which measurability has not been established is called a quality or a characteristic of the object (eg, opinions about an object). Method/technique When best used or avoided Strengths Weaknesses Assumptions, risks Inputs, resources, skills required Consultation/focus groups/workshops/hui Consultation of stakeholders and communities affected by a proposal helps to draw in different forms of knowledge, highlight debates over the nature and scale of future effects, including who is affected and how significant these effects are (perceived or actual). Consultative methods can take many forms, including meetings, open invite or specific invite meetings, and requests for written feedback. All RMA plan-making processes should involve consultation as part of preparing new provisions. For more complex issues, consultation may need to be in multiple stages (ie, issues and options; assessment of options; preferred direction). Recommended when community buy-in is important; cultural effects are anticipated; if large numbers of stakeholder groups are affected; and there is wide geographic spread of effects. Particularly useful if there is not good existing information about what is ‘valued’ (ie, the significance of effects). Helps highlight debates over the value of resources and the nature and scale of existing and future effects. Sometimes, new effects are identified, while in other cases, the consultation reveals different assessments of what is more or less important. Risks are often identified. For more than an “off the cuff” assessment of future effects, people being consulted need good information and understanding of the process that they are involved in. May not reach consensus on effects (and their scale and significance). There is risk of gaps in the combined knowledge of participants, where important effects may be missed or underestimated. Group situations which are predominantly verbal may result in views/knowledge of more vocal participants getting undue weight. May result in a large set of articulated ‘effects’ if views are not categorised. For practical reasons, consultation might be broken down by stakeholder group/location or by area of effect (ie,. social effects dealt with separately from environmental effects). Can be time and if more than a one-off exercise. Care is required to accurately record verbal discussions. Councils are usually well resourced to run or facilitate consultation type methods.
natural areas. This helps to identify the current ‘baseline’ or status quo. and resources are identified as being of national importance. an integrated system within which landscape and ecological values may vary considerably. other areas and is easily understood and credible. Impact assessments Assessment of future effects of a proposal on a resource or a sector of the community. These typically involve a combination of both qualitative and quantitative methods, but in some areas the qualitative aspects and use of expert opinion are especially important, such as social impact assessment, cultural impact assessments, heritage impact assessments, urban design assessment, landscape assessment, visual impact assessment, archaeological assessment etc. Best used when there are specific effects or resource issues that need to be examined in-depth (and are likely to lead to hearings later in the plan change process). Also when a defined project is proposed, for example, a Systematic identification of future effects under current conditions, as well as in relation to changes in policy/methods. Results in an assessment relevant to particular resources and is likely to determine nature and scale of changes (positive and negative). Identifies nature and scale of effects in terms of the individual resource, but not necessarily across resources or areas. Relies upon a range of inputs, including assessments of value, consultation and case studies. Usually needs expert involvement. Often involves stakeholder consultation. Draws upon a variety of methods, guides and data bases. Qualitative surveys/questionnaires Surveys of communities and/or specific sectors to obtain primary data on knowledge, opinions, values, priorities and so on. Applicable to many resource management issues, where community or sector attitudes and values are important. Can be used in conjunction with more focussed consultation or collaborative processes. Assuming well-designed survey instruments, and adequate coverage, these are a good source of primary data, which offers generally accurate material from the persons or businesses covered. Reliability of results depends on representativeness of the sample, and the influence of non-response rates. Assumes survey instruments are adequate to capture the relevant information, and that respondents are well enough informed to provide valid feedback. May require significant resource and expert input in survey design and administration if the issue is complex and the affected community is wide-spread.
Surveys using instruments which elicit quantitative information from respondent, directly through reporting on factual information (such as household size) and interactively using intelligent survey tools. attitudes and values are important, and the current attitudes are not well understood; and where the scale and significance are sufficient to require new or updated information. businesses covered. the influence of rates. and possible limits to survey depth when specific matters are covered within a more general survey. administration. Projections/forecasts/ extrapolations A range of methods to identify the future base environment and expected future effects based on current trends, and existing relationships among different factors. Common methods include trend extrapolation (past patterns continuing into the future), and use of models which take into account key influences and project future outcomes based on those known relationships - eg, population forecasts based on demographic drivers (age-sex structures, birth, death and migration rates) and information on capacity for residential growth. Most situations in RMA, since assessment is predominantly and it is important to understand (at least) the business as usual future as a baseline for assessment of proposals. Helps to identify the future under the 'without' change option – that is, under current policy settings. Commonly combined with other tools, especially scenario analysis, and information sets to identify different future outcomes. Most projections are based on a combination of historical actual patterns, and assumptions about the key drivers of those patterns into the future. Hence, they must rely on the robustness of core assumptions. May not take account of changes in management responses. Projections are usually more reliable at regional or district level, as short-term variations are more evident at the local level. May be based on scenarios. Sensitivity analysis may be required. May need to use in conjunction with discount rates to show present value. Can be simple or complex, depending on circumstances. Need data inputs in terms of trend/time series, to track historic patterns as basis for trend or modelling. A range of national and regional indicators is available to set context for district or localised projections. Non-market valuation methods Non-market valuation techniques are essentially concerned with measuring the extent to which changes brought about by proposed provisions either subtract from, or add to, values experienced by individuals and these values are not traded in a market place.
Method/technique When best used or avoided Strengths Weaknesses Assumptions, risks Inputs, resources, skills required Revealed preference methods – These methods estimate the value of a non-market good or service based on observed behaviour toward some closely connected marketed good or service. For example, using travel costs to infer the value of recreation or the experience of cultural/heritage sites, based on the cost of travel to those sites. Hedonic pricing infers the value of individual attributes of a market commodity. e.g., the value people implicitly pay for housing with reduced earthquake risk or expansive views. Useful to evaluate losses (eg, loss of recreation value in a river) or justify the replacement of public goods and services. Can be used when suitable market data (closely connected to the good or service) is available. Useful when there are close comparators of similar patterns or processes which may reliably be drawn on. Sensitive to model specification, including choice of functional forms. Limited/narrow application. Can only be applied when relevant market data is available and consumer/producer behaviour can be observed. Assumes values can be reasonably approximated by some relevant market behaviour. May be undertaken by a range of persons with appropriate background information. Stated preference methods – These methods utilise surveys or experiments to elicit what would hypothetically be paid for values which are not traded on markets. They are simulated approaches and are used in the absence of market data to portray consumer intentions. contingent valuation uses surveys to elicit the monetary value people are willing to pay to avoid a decrement of some type of non-market good or service, or the value they are willing to accept in exchange for its deterioration. In choice Best used when there is no suitable or proxy market data to portray consumer intentions. Capable of capturing current and future use values as well as non-use values (bequest and existence values). Wide applicability. Choice modelling has gained popularity over contingent valuation in recent years. Choice modelling in particularly can provide a rich data set of information directly applicable to policy objectives. Studies can be very difficult to implement and are complex. Must be carefully designed to avoid biases. Respondents can misrepresent real preferences in an attempt to anticipate the outcome of the study/process. Requires specialist expertise in state preference techniques. Primary surveys must be undertaken with comprehensive design and pre-testing.
modelling, willingness to pay is elicited from experiments in which respondents are asked to rank preferences for different attributes of non-market goods or services. Benefits transfer methods – Takes values from existing study site(s) and then applies these to the policy site using either a direct or function transfer method. Benefits transfer is used to inform policy and the decision-making process at various stages, including whether the extra expense of a primary survey is warranted. Useful if there are budget and time constraints that would make stated preference surveys (above) unfeasible. Best used when study site characteristics are very similar to the policy site characteristics. Can draw upon value estimates from one or multiple other studies. Sometimes viewed as a controversial method. Large transfer errors are a common problem (differences in experimental design, data collection, and econometric models make comparison difficult). May be limited NZ based studies to draw upon. Errors in the primary study are compounded. Adjustments of transferred values may be needed (including inflation or exchange rates). Function transfer methods generally perform better than direct transfer methods as they allow for site specific adjustments to be made. Primary studies need to be carefully screened for critical factors (site characteristics, population characteristics, how the research was framed). A sufficiently similar study or group of studies must have already been undertaken that can then be drawn on. Modelling Method/technique When best used or avoided Strengths Weaknesses Assumptions, risks Inputs, resources, skills required
Agent-based modelling simulates the behaviour of multiple agents acting together within a system. It is concerned with identifying patterns and trends that emerge as a result of the complex interactions of the agents as a whole. Examples of when agent-based modelling is well suited includes (micro simulation) traffic modelling or modelling of shopping behaviour, modelling of facility/resource use behaviour. Best used when there are lots of agents (ie, modelling the behaviour of all individuals). Useful if there are a large number of stakeholders. Not suitable for small budget evaluations. Able to simulate complex, non-linear, non-equilibrium behaviour of real world systems. Ability to study emergence of behaviour patterns. Relatively few experts of this technique. Can show quite erratic results, ie, sensitive to initial conditions. Difficult to validate. Same as for simulation models. May be difficult to understand underlying causes of behaviour due to complexity. Same as for simulation models. Requires specialist software packages. Integrated assessment models take models from a variety of disciplines (eg, scientific, engineering and economic) and combine these into a coherent whole by creating feedback structures between the models. These models tend to be spatial in nature, applying GIS to facilitate data entry and recording. Suitable for complex/big issues. Often applied in regional or large city contexts. Ideal for strategic policy evaluation, particularly considering transport, economic and land-use issues together. Not suitable for small budget evaluations. Broadens the boundary of investigation. Collaborative approach. Allows understanding of multiple effects of proposed policy provisions simultaneously. Once set up, speeds up strategic policy development. Subject to limitations of the underlying models. Huge overheads to set up. Difficult to validate. Same as for simulation models. Same as for simulation models. Requires specialist software packages. Requires a range of people with different knowledge/ backgrounds to come/work together (ie, a steering group). Economic impact assessment Economic impact assessment (EIA) is an approach used to understand (or often justify) the economic impact that projects, programmes or policies have on an economy. There are two core methods, described below.
identified through technical assessment, and the contribution of these effects, progressively, to elements, components and finally the overall objectives. The criteria themselves are weighted to reflect overall (for example, district-wide) objectives. This enables options to be scored according to their contribution to objectives. The best option is that with the highest score, across the relevant criteria. This method can address both efficiency and effectiveness. It identifies both the negative and positive aspects (costs and benefits) and combines them in a structure where outcomes are able to be compared as to benefits relative to costs (both in aggregate, rather than summed), and the extent of objectives achieved (effectiveness). therefore enables feedback loops to develop. Weightings and scores can be modified to reflect evolving understanding of trade offs. there is a long list of options to consider. higher score. Thus low costs score highly, while large benefits also score more highly. This may mean the reversal of normal measurement processes. Weighting of criterion needs to address the relative importance of the associated objective to which each criterion relates. need some form of independent (third party with no interest in the final outcome) check, review of the scores. See the