metadata: authors: 0: author: Ouchi,Fumika
count: World
docna: 0: docna: A literature review on the use of expert
            opinion in probabilistic risk analysis
docty: Policy Research Working Paper
subtopic: Information Technology,Power & Energy Conversion,Nuclear Energy,Energy Demand,Energy and Environment,Energy and Mining,Global Environment
teratopic: Education
abstracts: cdata!: Risk assessment is part of the decision
            making process in many fields of discipline, such as
            engineering, public health, environment, program management,
            regulatory policy, and finance. There has been considerable
            debate over the philosophical and methodological treatment
            of risk in the past few decades, ranging from its definition
            and classification to methods of its assessment.
            Probabilistic risk analysis (PRA) specifically deals with
            events represented by low probabilities of occurring with
            high levels of unfavorable consequences. Expert judgment is
            often a critical source of information in PRA, since
            empirical data on the variables of interest are rarely
            available. The author reviews the literature on the use of
            expert opinion in PRA, in particular on the approaches to
            eliciting and aggregating experts' assessments. The
            literature suggests that the methods by which expert
            opinions are collected and combined have a significant
            effect on the resulting estimates. The author discusses two
            types of approaches to eliciting and aggregating expert
            judgments-behavioral and mathematical approaches, with the
            emphasis on the latter. It is generally agreed that
            mathematical approaches tend to yield more accurate
            estimates than behavioral approaches. After a short
            description of behavioral approaches, the author discusses
            mathematical approaches in detail, presenting three
            aggregation models: non-Bayesian axiomatic models, Bayesian
            models, and psychological scaling models. She also discusses
            issues of stochastic dependence.
repnme: repnme: A literature review on the use of expert
            opinion in probabilistic risk analysis
display_title: A literature review on the use of
            expert opinion in probabilistic risk analysis
listing_relative_url: /research/2004/01/3169814/literature-review-use-expert-opinion-probabilistic-risk-analysis
topicv3: Nuclear Energy m1326166 509,Energy and Extractives m1326595 507,Environment, Natural Resources and Blue
            Economy m1327391 566,Psychiatry, Psychology m1504686 3613,Radioactive Waste m1582481 4286
publishtoextweb_dt: 2010-07-01T00:00:00Z
historic_topic: Education
no_of_pages: 17
author_keys: 0: author_key: 126704
model_extractions: WPS 3201 A Literature Review on the Use of Expert Opinion in Probabilistic Risk Analysis Fumika Ouchi

# World Bank Policy Research Working Paper 3201, February 2004

_The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the view of the World Bank, its Executive Directors, or the countries they represent. Policy Research Working Papers are available online at http://econ.worldbank.org._
A Literature Review on the Use of Expert Opinion in Probabilistic Risk Analysis

# **Introduction**

Risk assessment is part of the decision-making process in many fields of discipline, such as engineering, public health, environment, program management, regulatory policy, and finance. There has been considerable debate over the philosophical and methodological treatment of risk in the past few decades; and yet the discussion continues today on many basic issues ranging from the definition and classification of risk, to approaches to its assessment and management, to societal risk acceptance (Klinke and Renn 2002; Hatfield and Hipel 2002; Jenni and Fischhoff 1997; The U.S. Presidential/Congressional Commission on Risk Assessment and Risk Management 1997; Skjong and Eknes 2002).

Bedford and Cooke (1996) characterize risk with two particular elements: hazard (a source of danger) and uncertainty (quantified by probability). Probabilistic risk analysis (PRA) specifically deals with events represented by low probabilities of occurring with a high level of consequence. Expert opinion is one of the key research areas in PRA. The use of expert judgment is critical, and often inevitable, when there are no empirical data or information available on the variables of interest. In reliability analysis, for example, expert opinion has been extensively examined by Singpurwalla and others (Campodonico and Singpurwalla 1995; van Noortwijk et al. 1992; Singpurwalla 1988; Singpurwalla and Song 1988; Lindley and Singpurwalla 1986).

Because of the complex, subjective nature of expert opinion, there has been no formally established methodology for treating expert judgment. In recent years, there has been increasing effort in establishing a more systematic approach to eliciting expert opinion (Winkler et al. 1992; von Winterfeld 1989). Cooke and Goossens (2000) provide formal protocols, comprehensive procedures and guidelines on the elicitation process and handling of such data in uncertainty analysis.

This paper presents a review of the literature on the use of expert opinion, specifically, the approaches to aggregating different experts’ probability assessments. There are numerous studies on how to reconcile multiple experts’ assessments, many of which are extensively reviewed by Cooke (1991), Bedford and Cooke (1996), and Clemen and Winkler (1999). Genest and Zidek (1986) also provide a useful annotated bibliography of more than 90 studies on this subject. Drawing results from these reviews, the present paper attempts to provide a short summary of key aggregation methods and discussions. The paper also makes reference to new approaches proposed in recent articles on the subject.

# **Methods for Eliciting and Aggregating Expert Opinion**

Clemen and Winkler (1999) classify the elicitation and aggregation processes of expert assessments in two groups: mathematical and behavioral approaches. In mathematical approaches, experts’ individual assessments on an uncertain quantity are expressed as subjective 2
probabilities. They are combined through various mathematical methods by the decision-maker after their elicitation. Behavioral approaches aim at producing some type of group consensus among experts, who are typically encouraged to interact with one another and share their assessments.

One of the most well-known behavioral approaches is the Delphi technique, which was developed in the 1950s. In this method, experts are asked to anonymously judge the assessments made by other experts in a panel. Each of the experts is then given a chance to reassess his/her initial judgment based on the others’ review. Typically, the process is repeated several rounds until a smaller spread of experts’ opinions is achieved. The Delphi method later incorporated a self-rating mechanism, allowing experts to rate their expertise (Cooke 1991; Parenté and Parenté 1987; Sackman 1975). The Nominal Group method is another well-known behavioral method, in which experts are allowed to discuss their estimates directly with one another in a controlled environment (Delbecq et al. 1975). This method is considered more favorably than other group methods, particularly the Delphi method (Gustafson et al. 1973).

While a group consensus method may help identify experts’ errors and misunderstandings during the process, there are no formal rules to apply to reconcile differences when the consensus is difficult to achieve among different experts. Conformity induced by the group interaction is a major concern with such an approach. Mosleh et al. (1988) note that the group interactive process can suffer from, for example, the tendency for less confident experts to limit their participation, the influence of dominant personalities, and a tendency to reach speedy conclusions. Genest and Zidek (1986) warn of potential “strategic manipulation, bluffing, intimidating tactics and threats” if unrestricted dialogue is permitted. Cooke (1991) points out that the group interaction tends to produce more extreme probability estimates, potentially making the participants overconfident. Issues on group polarization are discussed by Plous (1993), Sniezek (1992), and Phillips (1987). There is a wide range of literature on the method for estimating a group consensus (Zahedi 1986; Goicoechea et al. 1982; Eliashberg and Winkler 1981).

It is generally agreed that mathematical approaches yield more accurate results than do behavioral approaches in aggregating expert opinions (Clemen and Winkler 1999; Mosleh, Bier and Apostolakis 1988). In the next section, three well-established mathematical modeling approaches to aggregating expert opinion are discussed: non-Bayesian axiomatic models, Bayesian models, and psychological scaling models (paired comparisons). The general framework of the present review is based on Cooke (1991), with reference to additional reviews and discussions from other relevant literature.

# - <u>Modeling Approach 1:  Non Bayesian Axiomatic Models</u>

An axiomatic-based approach to combining expert opinion is discussed in many of the earlier studies (Genest and Zidek 1986). In this approach, axioms or certain properties and regularity conditions for combining probability distributions are established, based on which the 3
form of combination rules is derived<sup>1</sup> . The approach requires that the relationship between experts’ opinions and the aggregated opinion must satisfy a certain set of axioms. Most axiomatic models proceed to identify some form of ‘weights’ as parameters to be estimated<sup>2</sup> .

# **General mathematical principles**

Bedford and Cooke (1996) provide a fundamental mathematical concept behind the process of combining probability distributions. General principles are reproduced here. Using their notations, suppose we have experts 1, ..., e and let _P_ be a set of probability measures over some fixed but unspecified probability space ( _P1, ... , Pe_ ∈ _P_ ). Then a combination rule that depends only on _P1, ... , Pe_ is a function G: _P_<sup>e</sup> → _P_ , i.e. G( _P1, ... , Pe_ ) = p. A set of axioms or constraints on G determines the form of G. Assuming that the probability space contains a finite number of events _a1 , ... an_ , let _pij_ be the expert _i_ ’s probability for event _aj_ . Then a vector of probabilities given by expert _i_ is _Pi = ( pi1, ... , pin )_ where _pij_ ≥ 0 and Σ _j pij_ = 1 for _j_ = 1, ..., n and _i_ = 1, ..., e. Let _wi_ be a set of non-negative weights, _wi_ ≥ 0 and Σ _i wi_ = 1. Then for any real _r 1/r_ number _r_ ∈ _R,_ the elementary r-norm weighted mean probability of _aj_ is _Mr (j) = (_ Σ _i wi pij )_ , _i_ = 1,... e, and the r-norm probability of _aj_ is _Pr(j) = Mr (j) /_ Σ _k Mr (k)_ , _k_ = 1, ..., n. (Note: for _r_ =1, _Pr_ is the weighted arithmetic mean of _Pi_ ; for _r_ = 0, _Pr_ is the weighted geometric mean, and for _r_ = -1, _Pr_ is the weighted harmonic mean of _Pi_ .) The interpretation of _Mr_ is the following (Bedford and Cooke 1996, Chapter 10): i) _Mr (j_ ) → Π _i pijwi_ as _r_ → 0 ( _i_ = 1, ..., e) ii) _Mr (j_ ) → max _i_ =1,..., e ( _pij_ ) as _r_ → ∞ iii) _Mr (j_ ) → min _i_ =1,..., e ( _pij_ ) as _r_ → -∞ iv) If _r_ < _s_ then _Mr (j_ ) ≤ _Ms (j_ ) with equality if and only if _pij_ = _pkj_ for 1 ≤ _i , k_ ≤ e v) Define _Mr (j+k_ ) = _(_ Σ _i wi (pij+ pik_ )<sup>r</sup> _)_<sup>_1/r_</sup> , _i_ = 1,..., e, and assume that _pij/pik_ is not constant in _i_ . Then the following (in)equalities hold: _r_ > 1 ⇒ _Mr (j+k_ ) < _Mr (j_ ) + _Mr (k_ ) _r_ = 1 ⇒ _Mr (j+k_ ) = _Mr (j_ ) + _Mr (k_ ) _r_ < 1 ⇒ _Mr (j+k_ ) > _Mr (j_ ) + _Mr (k_ ) The value for _r_ should be chosen so that the combination function G( _P1, ... , Pe_ ) satisfies the following three properties.

- i) _Strong set-wise function property_ :  For every subset _A_ ⊆ { _a1 , ... an_ }, the decision maker’s probability depends only on the experts’ judgments of _A_ , so if Qi = ( _Pi(A), 1- Pi(A)),_ then _G(P1, ... , Pe )(A) = G(Q1, ... , Qe )(A)_ .

- ii) _Marginalization property_ : The probabilities are unaffected by refinements of the partition of alternatives _a1 , ... an_ .

- iii) _Independence preservation_ : For all _A, B_ ⊆ { _a1 , ... an_ } such that _Pi(A_ ∩ _B)= Pi(A)* Pi(B), i_ =1, ..., e, _G(P1, ... , Pe )(A_ ∩ _B) = G(P1, ... , Pe )(A)* G(P1, ... , Pe )(B)_ .

> 1 For example, Morris (1983, 1986) provides an axiomatic approach to expert aggregation, which created a series of reactions and debate by Lindley (1986), French (1986), Schervish (1986), and Winkler (1986).

> 2 A departure from the traditional axiom approach is given by Myung et al. (1996), who use the principle of maximum entropy in their aggregation method.

4
# **A notion of opinion pools**

Decisions on many political and public policy issues rely on the opinions of experts. An opinion pool is a method of combining a number of different opinions about some unknown quantity θ to generate a single pooled opinion about θ. The two most widely used opinion pool methods are linear opinion pools and logarithmic opinion pools. Suppose we have n experts, and let _pi(_ θ _)_ represent expert _i_ ’s probability distribution for unknown quantity θ, i =1, ..., n, and _wi_ be expert _i_ ’s weight. Then the combined probability distribution _p(_ θ _)_ is a weighted linear combination of the experts’ probabilities (weighted arithmetic mean model) in a linear opinion pool, whereas _p(_ θ _)_ in a logarithmic opinion pool is expressed as multiplicative averaging (weighted geometric mean model): Linear opinion pool _p(_ θ _)_ = ∑ _i wi * pi(_ θ _)_ Logarithmic opinion pool _p(_ θ _)_ = _k_ Π _i pi(_ θ _)_<sup>_wi_</sup> ( _k_ is a normalizing constant) The problem of opinion pools generally reduces down to determining the optimal weights _wi_ for experts. Various methods for finding the optimal models are explored, for example, in DeGroot and Mortera (1991), Bordley (1982), and DeGroot (1974). Apparently, the simplest choice of weights is assigning all experts an equal weight, _wi = 1 / n_ . A simple arithmetic averaging of experts’ assessments is used in many studies, including a U.S. Nuclear Regulatory Commission study on the frequency of accidents at nuclear reactors (NRC 1989). Cook (1991) discusses that while there are some efforts in compensating such a simplistic method by improving the elicitation procedure itself, such as those witnessed for the NRC document (Honano et al. 1990; Wheeler et al. 1989), this type of method is less than optimal as it lacks any attempt to evaluate the quality of experts’ estimates.

# **Cooke’s classical performance-based weight model**

The method proposed by Cooke (1991) is a performance-based weighted averaging model using properties of scoring rules, known as the classical model<sup>3</sup> . He emphasizes that the fundamental goal of science is to build rational consensus and, therefore, the process of collecting expert assessments must be subjected to the following five basic principles (the first and second principles are later combined as a scrutability/accountability principle (Cooke and Goossens 2000)):

1. _Reproducibility_ :  All results must be reproducible, with calculation models and data being clearly specified and made available.

2. _Accountability_ : The source of data (name and institution) must be identified, and data must correspond to the exact source from which the data are elicited.

3. _Empirical Control_ : Experts’ assessments must be, in principle, physically observable. 4. _Neutrality_ : The elicitation process must ensure that the actual beliefs of experts be collected (e.g. no punishment or rewords through a self-rating system).

5. _Fairness_ : All experts must be regarded equally before the aggregation process.

> 3 The term classical comes from the calibration measure’s close association with classical hypothesis testing.

5
The classical model is designed to satisfy all these principles of rational consensus. In case of continuous variables, the model requires experts to provide a set of fixed quantiles _qr_ , _r = 1, ..., R_ , for some unknown variables (seed variables) _X1 , ..., XN_ . The decision-maker then determines the intrinsic range (lower and upper bound, [ _ql , qh_ ]) of each variable for each expert. The weights for the experts are established by two measures of performance: calibration and information scores. Calibration of expert _e_ , _C(e)_ , is the statistical likelihood that an expert’s quantile assessment corresponds to a set of experimental results.

Let _p = p1, ..., pn_ be the hypothesized probability distribution of a variable over a set of alternatives {1, ..., n}, and _s = s1, ..., sn_ be the empirical distribution from N independent samples (i.e. N is the number of seed variables) from the distribution _p_ . The discrepancy between _p_ and _s_ can be measured by the relative information of _s_ with respect to _p_ , _I(s, p)_ = Σi _si ln(si/pi), i=1, ..., n,_ where we note that P(2N _I(s, p)_ ≤ x) → χ<sup>2</sup> n-1<sup>(x) as N →∞. Then</sup><sup>_C(e),_which</sup> is the probability of obtaining a relative information score worse than what has been observed, is obtained as _C(e)_ = 1 - χ<sup>2</sup> n-1<sup>(2N</sup><sup>_I(s, p)_).</sup> The information score, or informativeness, measures the degree to which an expert’s distribution is concentrated, relative to some background measure (usually of uniform or loguniform distribution over an intrinsic range for each variable). The relative information for expert _e_ on a given variable is obtained as _I(e)_ = Σi _pi ln(pi/ri), i=1, ..., n,_ where _ri_ is the background measure for interval _i_ . The overall information score of each expert is the average of the information scores over all variables.

The weights for the experts (which satisfy the asymptotically strictly proper scoring rules) are then determined as: _we_ = _we’ / W_ , where _we’_ = _C(e)_ x _I (e)_ x _1_ α _(C(e)_ ), and _W_ = Σ _we’_ (note: for the level of significance α, _1_ α _(C(e))_ = 1 if _C(e)_ ≥ α, and = 0 if _C(e)_ < α, α∈(0, 1)). Given _we_ and each expert’s cumulative distribution function _Fe_ , the combined expert distribution is Σ _e we Fe._

# <u>Modeling Approach 2:  Bayesian Models</u>

Perhaps the most robust technique in combining expert opinion is the Bayesian method<sup>4</sup> . In this method, the decision maker uses experts’ probability assessments as data to update his own prior belief about the distribution of an unknown quantity of interest, according to Bayes’ Theorem. The early framework of the Bayesian aggregation method was provided by Winkler (1968) and Morris (1974, 1977). Winker (1968) discusses his natural conjugate theory, in which prior and posterior distributions belong to the same parametric family of distribution.

Let _P(x)_ be the decision-maker’s prior probability distribution for some unknown quantity _x_ , and _P(D|x)_ be the likelihood of some observational data _D_ given _x_ . Then the decision-maker’s posterior distribution is _P(x|D) = [P(D|x)*P(x)] / P(D)_ via Bayesian update.

> 4 Winkler (1986) notes, for example, that all contributors to Morris’ (1983, 1986) axiomatic approach, i.e. Lindley, French, and Schervish, would in principle agree that the modeling approach with Bayesian principles is the most appropriate way to combine experts’ assessments.

6
Suppose the decision-maker uses the beta distribution to model the prior uncertainty about _x_ and then updates that beta prior on the basis of an observation, i.e. information from each successive expert, then the posterior distribution of _x_ also will be a beta distribution. Knowledge of _D_ and parameters of the prior distribution will immediately lead the decision-maker to his posterior distribution.

Morris’ work (1974, 1977) fully establishes the foundations for the Bayesian paradigm in the analysis of expert judgment. He presents a straightforward aggregation method for the single expert case and the multi-expert case. In general, suppose the decision-maker wishes to make an inference about an unknown quantity _X_ , and experts _1, ..., e_ provide their estimates of _X, x1_ , ... _xe_ . Let _p(x)_ be the decision-maker’s prior probability density for _X_ , and _p(x1_ , ... _xe_ | _x_ ) be his beliefs about the experts’ assessments as estimates of _X_ , i.e. the likelihood function. The primary goal of the decision-maker is to find his posterior density, _p(x | x1_ , ... _xe)_ :

Morris assumed that the experts are independent, and their density functions are represented by a normal distribution with the mean and variance. When independence among the experts is assumed, the likelihood term is simply the product of _p(xi_ | _x)_ , _i=1, ..., e_ :

Building on Morris’ method, Mosleh and Apostolakis (1986) suggest two practical models for determining the above product.

# **Additive error and multiplicative error models**

Mosleh and Apostolakis (1986) express each of the experts’ assessments using the true value of unknown quantity and an error term.

1. Additive error model. The expert _i_ ’s assessment, _xi_ , is expressed as the sum of the _2_ true value of _X_ , _x_ , and an error ξi : _xi_ = _x_ + ξ _i_ where ξ _i_ ∼ _N (_ μ _i ,_ δ _i_<sup>_)_and independent</sup> (μ _i_ and δ _i2_ are the mean and variance determined by the decision-maker to reflect his belief about the expert’s bias and accuracy). The likelihood _p(xi_ | _x)_ of obtaining _xi_ when the true value is _x 2_ is the value of the normal density with mean _x +_ μ _i,_ and variance δ _i_ . Suppose the decision- _2_ maker acts as the e+1<sup>st</sup> expert and his prior distribution _p(x)_ ∼ _N (xe+1 ,_ δ _e+1_<sup>_)_. Then the</sup> decision-maker’s posterior _p(x | x1_ , ... _xe)_ is normal with mean and variance given as follows:

2. Multiplicative error model. The expert _i_ ’s assessment, _xi_ , is expressed as the product _2_ of the true value _x_ and an error ξi : _xi_ = _x_ * ξ _i_ where ln ξ _i_ ∼ _N (_ μ _i ,_ δ _i_<sup>_)_and independent.</sup> 7
_2_ Thus, _ln(xi ) = ln(x) + ln(_ ξ _i )_ ∼ _N ((ln(x) +_ μ _i ,_ δ _i_<sup>_)_i.e.</sup><sup>_x_</sup> _i_<sup>is lognormally distributed. Given the</sup> _2_ decision-maker’s prior distribution _p(x)_ ∼ _Lognormal (xe+1 ,_ δ _e+1_<sup>_)_,</sup> Thus the decision-maker’s posterior expectation and variance are given as:

# **Issues on stochastic dependence**

Winkler (1981) stresses the importance of taking into account the possibility of stochastic dependence in modeling expert opinion. Dependence among experts’ assessments arises when, for example, the experts selected by the decision-maker have worked in the same field or shared similar training and techniques. Dependent opinions are not only redundant, but they can cause a significant impact on the decision-maker’s assessment. The sensitivity of the decision-maker’s posterior density distribution in the presence of high correlations among experts is discussed by Winkler and Clemen (1992) and Clemen and Winkler (1985). Chhibber and Apostolakis (1993) also provide a sensitivity analysis on the use of dependent information sources.

Because of its practicality, an independent normal distribution is assumed for experts’ assessments in many studies that followed Morris’ (1974, 1977) Bayesian approach (French 1980, 1981; Winkler 1981; Lindley 1983, 1985). Identifying a joint likelihood distribution for experts’ probability assessments is considered as one of the difficulties in using a Bayesian method. Some of the recent studies indicate numerous attempts to tackle the issue. Clemen and Winkler (1993) provide a procedure for constructing the joint likelihood function as a product of marginal and conditional likelihood functions. Lipscomb et al. (1998) propose a hierarchical approach to incorporate dependencies among experts, in which experts’ variation is assumed to follow a normal distribution and a sample of experts can be drawn from the second-order distribution. Saaty and Vargas (1998) use the analytic hierarchy process to address dependence symptoms.

8
The use of copulas is another approach to specifying a likelihood function that seems to have gained some attention in the recent literature (Jouini and Clemen 1996; Yi and Bier 1998; Clemen and Reilly 1999; and Clemen et al. 2000). A copula is a function that is used to join a set of particular marginal distributions, thereby generating a multivariate distribution that carries those marginals. The approach discussed by Jouini and Clemen (1996), for example, shows the following: Let _fi(_ θ _)_ be the continuous density for unknown θ provided by expert _i_ , _Fi(_ θ _)_ be the corresponding cumulative distribution function, and _C_ be the copula density function. Then under some conditions, the decision-maker’s posterior distribution is shown as Mendel and Sheridan (1989) use the Bayesian method to recalibrate and combine experts’ assessments. Their model does not require experts to follow any particular distribution, and thus offers more flexibility and an ease of implementation. In this model, each expert provides _R_ quantiles for a variable of interest, which defines _R+1_ cells that could be hit by the true value. Given _n_ experts, an _(R+1)_<sup>_n_</sup> array is formed, which is considered as a random variable. The decision-maker uses the theory of exchangeability to update his probability distribution for this random variable, before receiving the experts’ advice on the current variable. Information from past assessments and outcomes provides the likelihood of hit in each cell.

# <u>Modeling Approach 3:  Psychological Scaling Models (Paired Comparisons)</u>

The psychological scaling models assume that every expert has some internal value associated with a variable of interest and he/she can only provide his or her qualitative input (no numerical estimates). The decision-maker asks experts to state their preference or views on pairwise comparisons. This approach originated from the study of estimating intensities of physical stimuli, which later developed into the study of estimating relative intensities of psychological stimuli among experts<sup>5</sup> . Using simulation, experts’ assessments lead to a consensus with confidence bounds. Their inputs are measured for their consistency and concordance (Cooke 1991).

Suppose there are _n_ experts, and each expert is asked to express his/her preference for one of two events. Let _A(1), ..., A(t)_ be events to be compared, _V(1), ..., V(t)_ be the true probabilities of events. Let _V(i, e)_ denote the internal value of expert _e_ for event _i_ . If expert _e_ prefers event _A(i)_ to event _A(j)_ , (i.e. _A(i) > A(j)_ ), then _A(i)_ is judged more probable than _A(j)_ by _e_ , and _V(i, e) >V(j, e)_ . Three models are presented below.<sup>6</sup>

# **Thurstone model (1927)**

Assumptions: Normal distribution of internal values over the population of experts _V(i, e)_ ∼ _N(_ μ _i,_ δ _i)_

> 5 An important contribution to the study is the assessment of human error probabilities.  See Kirwan (1994), Humphreys (1988), USNRC (1983), and Swain and Guttmann (1983).

> 6In addition to the three models shown here, see Pulkkinen (1994) for his various paired comparison methods.

9
Then the probability that expert _e_ prefers _A(i)_ to _A(j)_ is: _P((V(i, e)-V(j, e))>0) = P{ (V(i, e) - V(j, e)-_ μ _ij)/_ δ _ij_ > - μ _ij/_ δ _ij } = P(X >_ - μ _ij/_ δ _ij) =_ Φ _(_ μ _ij/_ δ _ij)_ Note: By setting _xij_ = Φ<sup>-1</sup> (the percentage of experts who prefer _A(i)_ to _A(j)_ ) ≈ μ _ij /_ δ _ij_ , and δ _ij_ = 1 ⇒ μ _i_ - μ _j_ ≈ _xij._ Using the least-squares method, the estimate of μ _i_ is obtained as:

# **Bradley-Terry model (1952, 1953)**

Assumptions: Each event _A(i)_ is associated with a true scale value _V(i)_ , and the probability that event _A(i)_ is preferred to event _A(j)_ , _r(ij)_ , is given by: Experts’ judgments are treated as independent coin-toss trials with the probability of head = _r(ij)_ . The proportion of experts preferring _A(i)_ over _A(j)_ is used as an estimate of _r(ij)_ .

Using the maximum likelihood method (David 1963), _V(i)_ , _i_ = 1, ..., t, is given as: where _a(i)_ is the number of times event _A(i)_ is preferred to other events by experts. The value of _V(i)_ is obtained by iteration, which converges to a unique solution.

# **Negative exponential lifetime (NEL) model**

This model is used specifically for estimating constant failure rates. For each pair of independently-operating components, the decision-maker asks experts which one of the two components will fail first. Component _A(i)_ is assumed to have an exponential life distribution with failure rate _r(i)_ , and all components are assumed as good as new at time t =0. Thus, the probability that _A(i)_ fails before time T is: And the probability that _A(i)_ fails before _A(j)_ is _P(A(i) < A(j))_ = _r(i)_ / [ _r(i)+r(j)_ ] . The solution is obtained in a similar manner as in the Bradley-Terry Model.

The paired comparison models are appealing in that experts are not required to be familiar with numerical assessments and the overall elicitation process is relatively simple. They have, however, disadvantages such as requiring a large number of experts and forcing stringent 10
assumptions about experts’ underlying assessment mechanisms. Cooke (1991) notes that the paired comparison approach has a difficulty in satisfying the principle of empirical control. The goodness-of-fit tests used in the paired comparison models examine the fitness of experts’ assessments against modeling assumptions, but not against the empirical values. The confidence intervals obtained through the models reflect uncertainty due to the choice of experts, and do not reflect uncertainty due to modeling assumptions.

# **Conclusions**

The three modeling approaches presented in the paper have all been used in some reallife situations. Each approach clearly has advantages and disadvantages, for example, as seen in the case of the Bayesian modeling approach that has mathematical sophistication but suffers from limitation in its practical application. A general agreement appears to be that there is no single all-purpose aggregation method for expert opinion.

In their critical review of several case studies involving the use of expert opinion, Mosleh et al. (1988) note that while there is evidence that expert opinion can be highly beneficial in probabilistic risk analysis, little attention has been paid to structuring the elicitation process. Although empirical evidence indicates that mathematical methods of aggregation generally yield better results than behavioral methods, the latter methods are often perceived appealing, particularly when experts have knowledge in different areas and the synthesis of their expertise is needed. Based on a case study that successfully assessed seismic hazard rates using expert opinion, Mosleh et al. (1988) show the use of a multiple-team approach with mathematical aggregation as one of the most promising methods in dealing with such a problem.

As decision-makers in general tend to use the most convenient aggregation methods of their choice (and not necessarily the most appropriate), Cooke and others call for formalizing the elicitation process and using expert opinion by ensuring the basic principles of rational consensus (i.e. satisfying reproducibility, accountability, empirical control, neutrality, and fairness in resulting assessments), which seems truly timely and appropriate.

In order to support the practical application of expert analysis, two relevant software programs have been developed by the Department of Mathematics at Delft University of Technology. _EXCALIBUR_ (from Expert CALIBRation) is a Windows program for expert judgment analysis. It allows the user to input experts’ quantile assessments and parameters, and combine their assessments based on equal weights, user weights, and expert performance-based weights (Cook 2001, 1991). _UNICORN_ is a software package for uncertainty analysis. It is designed for dependence modeling with high dimensional distributions, including graphic features, such as cobwebs (Cooke 1995). Both programs are designed to facilitate the formal procedure of expert analysis, which is based on the principles of rational consensus.

11
References on Expert Opinion (*Cited in the paper) Bedford T. and Cooke, R.T. (2001), “Probabilistic Risk Analysis: Foundations and Methods,” Cambridge University Press.* Bier, V.M. and Yi, W. (1995), “A Bayesian Method for Analyzing Dependencies in Precursor Data,” _International Journal of Forecasting_ , Volume 11, Issue 1, pp.25-41.* Bordley, R.F. (1982), “A Multiplicative Formula for Aggregating Probability Assessments,” _Management Science_ , Volume 28, Issue 10, pp.1137-1148.* Campodonico, S. and Singpurwalla, N.D. (1995), “Inference and Predictions from Poisson Point Processes Incorporating Expert Knowledge,” _Journal of the American Statistical Association_ , Volume 90, Issue 429, pp.220-226.* Chande, P.K. and Tokekar, S. V. (1998), “Expert-Based Maintenance: A Study of Its Effectiveness,” _IEEE Transactions on Reliability_ , Volume 46, No. 1, pp.53-58.

Chhibber, S. and Apostolakis, G. (1993), “Some Approximations Useful to the Use of Dependent Information Sources,” _Reliability Engineering and System Safety_ , Volume 42, Issue 1, pp.6786.* Clemen, R.T. (1986), “Calibration and the Aggregation of Probabilities,” _Management Science_ , Volume 32, Issue 3, pp.312-314.

Clemen, R.T., Fischer, G.W., and Winkler, R.L. (2000), “Assessing Dependence: Some Experimental Results,” _Management Science_ , Volume 46, Issue 8, pp.1100-1115.* Clemen, R.T. and Winkler, R.L. (1999), “Combining Probability Distributions from Experts in Risk Analysis,” _Risk Analysis_ , Volume 19, Issue 2, pp.187-203.* Clemen, R.T. and Reily, T. (1999), “Correlations and Copulas for Decision and Risk Analysis,” _Management Science_ , Volume 45, Issue 2, pp. 208-224.* Clemen, R.T. and Winker, R.L. (1993), “Aggregating Point Estimates: A Flexible Modeling Approach,” _Management Science,_ Issue 39, No. 4, pp. 501-515.* Clemen, R.T. and Winker, R.L. (1985), “Limits for the Precision and Values of Information from Dependent Sources,” _Operations Research,_ Volume 33, pp. 427-442.* Cooke, R.M. and Slijkhuis, K.A. (2003), “Expert Judgment in the Uncertain Analysis of Dike Ring Failure Frequency,” Appearing in Case Studies in Reliability and Maintenance, pp.331-350.

Cooke, R.M. (2001), “EXCALIBUR – Windows version of EXCALIBR: Software for performance based combination of expert judgments,” Department of Mathematics, Delft University of Technology.* 12
Cooke, R.M. and Goossens, L.H.J. (2000), “A Procedures Guide for Structured Expert Judgment,” EUR 18820, European Commission Report.* Cooke, R.M. and Goossens, L.H.J. (2000), “Procedures Guide for Structured Expert Judgment in Accident Consequence Modeling,” Radiation Protection and Dosimetry, Vol. 90, No. 3, pp.303309.* Cooke, R.M., and Goossens, L.H.J. (2000), “Expert Judgment Elicitation in Risk Assessment,” Delft University of Technology (“Abstract NATO workshop, Lisbon, 1-4 October 2000)* Cooke, R.M. (1995), “UNICORN: Methods and Code for Uncertainty Analysis,” AEA Technologies.* Cooke, R. M. (1991), “Experts in Uncertainty: Opinion and Subjective Probability in Science,” Oxford University Press, Oxford.* David, H.A. (1963), “The Method of Paired Comparisons,” Charles Griffin, London DeGroot, M.H. (1974), “Reaching a Consensus,” _Journal of the American Statistical Association_ , Volume 69, pp. 118-121.* DeGroot, M.H. and Mortera, J. (1991), “Optimal Linear Opinion Pools,” _Management Science_ , Volume 37, Issue 5, pp 546-558.* Delbecq, A., Van de Ven, A., and Gusstafson, D. (1975), “Group Techniques for Program Planning,” Glenview, III, Scott-Foresman.* Eliashberg, J. and Winkler R.L. (1981), “Risk Sharing and Group Decision Making,” _Management Science_ , Volume 27, Issue 11, pp.1221-1235.* French, S. (1986), “Calibration and the Expert Problem,” _Management Science_ , Volume 32, Issue 3, pp 315-321.* French, S. (1981), “Consensus of Opinion,” _European Journal of Operations Research_ , 7, pp. 332-340.* French, S. (1980), “Updating of Belief in the Light of Someone Else’s Opinion,” _Journal of the Royal Statistical Society: Series A_ , Volume 143, pp. 43-48* Genest, C. and Zidek, J.V. (1986), “Combining Probability Distributions: A Critique and an Annotated Bibliography, _Statistical Science_ , Volume 1, Issue 1, pp. 114-135.* Goicoechea, A., Hansen, D.R. and Duckstein, L. (1982), “Multiobjective Decision Analysis with Engineering and Business Applications,” Wiley.* 13
Goossens, L.H.J., Harper, F.T., Kraan, B.C.P. and Metivier, H. (2000), “Expert Judgment for a Probabilistic Accident Consequence Uncertainty Analysis,” Radiation Protection and Dosimetry, Vol. 90, No. 3, pp.295-303.

Goossens, L.H.J., Cooke, R.M. and Kraan, B.C.P. (1998), “Evaluation of Weighting Schemes for Expert Judgment Studies,” _Probabilistic Safety Assessment and Management_ (Proceedings of PSAMA 4), pp.2389-2396.

Gustafson, D., Shulka, R., Delbecq, A., and Walster, A. (1973), “A Comparative Study of Differences in Subjective Likelihood Estimates Made by Individuals, Interacting Groups, Delphi Groups, and Nominal Groups,” _Organizational Behaviour and Human Performance,_ Volume 9, pp. 280-291.* Hatfield, A.J. and Hipel, K.W. (2002), “Risk and Systems Theory,” _Risk Analysis_ , Volume 22, No. 6, pp.1043-1057.* Honano, E.J., Hora, S.C., Keeney, R.L., and von Winterfeldt, D. (1990), “Elicitation and Use of Expert Judgment in Performance Assessment for High-level Radioactive Waste Repositories,” NUREG/CR-5411.* Humphreys, R. (1988), “Human Reliability Assessors Guide,” Safety and Reliability Directorate, United Kingdom Atomic Energy Authority* Jenni, K. and Fischhoff, B. (1997), “Attributes for Risk Evaluation,” 1997 Annual Meeting of Society for Risk Analysis.* Jouini M. N., and Clemen, R. T. (1996), “Copula Models for Aggregating Expert Opinions,” _Operations Research_ , Volume 44, Issue 3, pp. 444-457.* Kirwan, B. (1994), “A Guide to Practical Human Reliability Assessment,” Taylor and Francis, London.* Kline, A. and Renn, O. (2002), “A New Approach to Risk Evaluation and Management: RiskBased, Precaution-Based, and Discourse-Based Strategies,” _Risk Analysis_ , Volume 22, No. 6, pp.1071-1094.* Kraan, B.C.P. and Cooke, R.M. ( 2000), “Processing Expert Judgment in Accident Consequence Modeling,” Appearing in Radiation Protection Dosimetry (Expert Judgment and Accident Consequence Uncertainty Analysis; special issue), Vol. 90, No. 3, pp.311-315.

Lindley, D.V., and Singpurwalla, N.D. (1986), “Reliability (and Fault Tree) Analysis Using Expert Opinions,” _Journal of the American Statistical Association_ , Volume 81, Issue 393, pp. 87-90.* Lindley, D.V. (1987), “Using Expert Advice on a Skew Judgmental Distribution,” _Operations Research_ , Volume 35, Issue 5, pp 716-721.

14
Lindley, D.V. (1986), “Another Look at an Axiomatic Approach to Expert Resolution,” _Management Science_ , Volume 32, Issue 3, pp 303-306.* Lindley, D.V. (1985), “Reconciliation of Discrete Probability Distributions,” _Bayesian Statistics_ 2, pp.375-387.* Lindley, D.V. (1983), “Reconciliation of Probability Distributions,” _Operations Research,_ Volume 31, Issue 5, pp.866-660.* Lipscomb, J., Parmigiani, G., and Hasselblad, V. (1998), “Combining Expert Judgment by Hierarchical Modeling: An Application to Physician Staffing,” _Management Science_ , Volume 44, Issue 2, pp 149-161.* Mendel, M. and Sheridan, T. (1989), “Filtering Information from Human Experts,” _IEEE Transaction Systems, Man and Cybernetics_ , Volume 36, pp. 6-16. * Morris, P.A. (1986), “Observations on Expert Aggregation,” _Management Science_ , Volume 32, Issue 3, pp 321-328.* Morris, P.A. (1983), “An Axiomatic Approach to Expert Resolution,” _Management Science_ , Volume 29, Issue 1, pp 24-32.* Morris, P.A. (1977), “Combining Expert Judgments: A Bayesian Approach,” _Management Science_ , Volume 23, Issue 7, pp 679-693.* Morris, P.A. (1974), “Decision Analysis Expert Use,” _Management Science_ , Volume 20, Issue 9, Theory Series, pp 1233-1241.* Mosleh, A., Bier, V.M., and Apostolakis, G. (1988), “A Critique of Current Practice for the Use of Expert Opinions in Probabilistic Risk Assessment,” _Reliability Engineering and System Safety_ , Volume 20, pp. 63-85.* Mosleh, A., Bier, V.M., and Apostolakis, G. (1987), “Methods for the Elicitation and Use of Expert Opinion in Risk Assessment,” NUREG/CR-4962, PLG-0533, US Nuclear Regulatory Commission (micro fiche) Mosleh, A. and Apostolakis, G. (1986), “The Assessment of Probability Distributions from Expert Opinions with an Application to Seismic Fragility Curves,” _Risk Analysis_ , Volume 6, No. 4, pp 447-461.* Myung, I.J., Ramamoorti, S., and Bailey, A.D.Jr. (1996), “Maximum Entropy Aggregation of Expert Predictions,” _Management Science_ , Volume 42, Issue 10, pp 1420-1436.* Parenté, F.J. and Anderson-Parenté, J.K. (1987), “Delphi Inquiry Systems,” _Judgmental Forecasting_ .* 15
Phillips, L.D. (1987), “On the Adequacy of Judgmental Forecasts,” _Judgmental Forecasting_ , pp. 11-30.* Plous, S. (1993), “The Psychology of Judgment and Decision Making.” New York, McGrawHill.* Pulkkinen, U. (1994), “Gaussian Paired Comparison Models,” _Reliability Engineering and System Safety_ , Volume 44, Issue 2, pp. 207-217.* Pulkkinen, U. (1994), “Bayesian Analysis of Consistent Paired Comparisons,” _Reliability Engineering and System Safety_ , Volume 43, Issue 1, pp. 1-16.* Pulkkinen, U. (1993), “Methods for Combination of Expert Judgments,” _Reliability Engineering and System Safety_ , Volume 40, Issue 2, pp. 111-118.

Saaty, T.L. and Vargas, L.G. (1998), “Diagnosis with Dependent Symptoms: Bayes Theorem and the Analytic Hierarchy Process,” _Operations Research_ , Volume 46, Issue 4, pp 491-502.* Sackman, H. (1975), “Delphi Critique: Expert Opinion, Forecasting and Group Processes,” Lexington, MA, Lexington Books.* Schervish, M.J. (1986), “Comments on Some Axioms for Combining Expert Judgments,” _Management Science_ , Volume 32, Issue 3, pp 306-312.* Singpurwalla, N.D. (1988). “An Interactive PC-Based Procedure for Reliability Assessment Incorporating Expert Opinion and Survival Data,” _Journal of the American Statistical Association_ , Volume 83, Issue 401, pp. 43-51* Singpurwalla, N.D., and Song, M.S. (1988), “Reliability Analysis using Weibull Lifetime Data and Expert Opinion _,” IEEE Transactions on Reliability_ , Volume 37, No. 3, pp.340-347.* Skjong, R. and Eknes, M.L. (2002), “Societal Risk and Societal Benefits,” _Risk Decision and Policy_ , Volume 7, pp. 57-67.* Sniezek, J. (1992), “Groups under uncertainty: An Examination of Confidence in Group Decision Making,” _Organizational Behavior and Human Decision Processes_ , 52, pp.124-155.* Swain, A.D., and Guttmann, H.E., (1983), “Handbook of Human Reliability Analysis with Emphasis on Nuclear Power Plant Applications,” NUREG/CR-1278.* The U.S. Presidential/Congressional Commission on Risk Assessment and Risk Management (1997), “Risk Assessment and Risk Management in Regulatory Decision-Making,” Final Report, Volume 2.* Thurstone, L. (1927), “A Law of Comparative Judgment,” _Psychological Review_ , Volume 34, pp.273-286.* 16
U.S. Environmental Protection Agency (2003), “Framework for Cummulative Risk Assessment,” Risk Assessment Forum, EPA (EPA/630/P-02/001F, April 2002).

U.S. Nuclear Regulatory Commission (1989), “Severe Accident Risks: An Assessment for Five US Nuclear Power Plants,” U.S. NRC, NUREG/CR-1150* U.S. Nuclear Regulatory Commission (1983), “PRA Procedure Guide,” U.S. NRC, NUREG/CR-2300* Van Noortwijk, J.M., Dekker, R., Cooke, R. and Mazzuchi, T.A. (1992), “Expert Judgment in Maintenance Optimization,” _IEEE Transactions on Reliability_ , Volume 41, No. 3, pp.427-432.* Von Winterfeld, D. (1989), “Eliciting and Communicating Expert Judgments: Methodology and Application to Nuclear Safety,” Joint Research Centre, Commission of the European Communities.* West, M. (1988), “Modeling Expert Opinion,” _Bayesian Statistics_ 3, Oxford University Press, pp.493-508.

Winkler, R.L. (1986), “Expert Resolution,” _Management Science_ , Volume 32, Issue 3, pp.298303.* Winkler, R.L. (1981), “Combining Probability Distributions from Dependent Information Sources,” _Management Science_ , Volume 27, Issue 4, pp.479-488.* Winkler, R.L. (1968), “The Consensus of Subjective Probability Distributions,” _Management Science_ , Volume 15, Issue 2, pp. B61-B75.* Winkler, R.L. and Clemen, R.T. (1992), “Sensitivity of Weights in Combining Forecasts,” _Operations Research,_ Volume 40, pp.609-614.* Winker, R.L., Hora, S.C., Baca, R.G. (1992), “The Quality of Experts’ Probabilities Obtained Through Formal Elicitation Techniques,” Center for Nuclear Waste Regulatory Analyses CNWRA, CNWRA T. Rep.* Wheeler, T.A., Hora, S.C., Cramond, W.R., and Unwin, S.D. (1989), “Analysis of Core Damage Frequency from Internal Events: Expert Judgment Elicitation,” NUREG/CR-4550, Volume 2, Sandia National Laboratories. * Yi, W. and Bier, V.M. (1998), “An Application of Copulas to Accident Precursor Analysis,” _Management Science_ , Volume 44, Issue 12, Part 2 of 2, S257-S270.* Zahedi, F. (1986), “Group Consensus Function Estimation when Preferences are Uncertain,” _Operations Research_ , Volume 34, Issue 6, pp. 883-894.* 17