A Statistical Approach to Anaphora Resolution

This paper presents an algorithm for identifying pronominal anaphora and two experiments based upon this algorithm. We incorporate multiple anaphora resolution factors into a statistical framework specifically the distance between the pronoun and the proposed antecedent, gender/number/animaticity of the proposed antecedent, governing head information and noun phrase repetition. We combine them into a single probability that enables us to identify the referent. Our first experiment shows the relative contribution of each source Of information and demonstrates a success rate of 82.9% for all sources combined. The second experiment investigates a method for unsupervised learning of gender/number/animaticity information. We present some experiments illustrating the accuracy of the method and note that with this information added, our pronoun resolution method achieves 84.2% accuracy. 1 I n t r o d u c t i o n We present a statistical method for determining pronoun anaphora. This program differs from earlier work in its almost complete lack of hand-crafting, relying instead on a very small corpus of Penn Wall Street Journal Tree-bank text (Marcus et al., 1993) that has been marked with co-reference information. The first sections of this paper describe this program: the probabilistic model behind it, its implementation, and its performance. The second half of the paper describes a method for using (portions of) t~e aforementioned program to learn automatically the typical gender of English words, information that is itself used in the pronoun resolution program. In particular, the scheme infers the gender of a referent from the gender of the pronouns that 161 refer to it and selects referents using the pronoun anaphora program. We present some typical results as well as the more rigorous results of a blind evaluation of its output. 2 A P r o b a b i l i s t i c M o d e l There are many factors, both syntactic and semantic, upon which a pronoun resolution system relies. (Mitkov (1997) does a detailed study on factors in anaphora resolution.) We first discuss the training features we use and then derive the probability equations from them. The first piece of useful information we consider is the distance between the pronoun and the candidate antecedent. Obviously the greater the distance the lower the probability. Secondly, we look at the syntactic situation in which the pronoun finds itself. The most well studied constraints are those involving reflexive pronouns. One classical approach to resolving pronouns in text that takes some syntactic factors into consideration is that of Hobbs (1976). This algorithm searches the parse tree in a leftto-right, breadth-first fashion that obeys the major reflexive pronoun constraints while giving a preference to antecedents that are closer to the pronoun. In resolving inter-sentential pronouns, the algorithm searches the previous sentence, again in left-to-right, breadth-first order. This implements the observed preference for subject position antecedents. Next, the actual words in a proposed nounphrase antecedent give us information regarding the gender, number, and animaticity of the proposed referent. For example: M a r i e Giraud carries historical significance as one of the last women to be ezecuted in France. S h e became an abortionist because it enabled her to buy jam, cocoa and other war-rationed goodies. Here it is helpful to recognize that "Marie" is probably female and thus is unlikely to be referred to by "he" or "it". Given the words in the proposed antecedent we want to find the probability that it is the referent of the pronoun in question. We collect these probabilities on the training data, which are marked with reference links. The words in the antecedent sometimes also let us test for number agreement. Generally, a singular pronoun cannot refer to a plural noun phrase, so that in resolving such a pronoun any plural candidates should be ruled out. However a singular noun phrase can be the referent of a plural pronoun, as illustrated by the following example: "I think if I tell Viacom I need more time, they will take 'Cosby' across the street," says the general manager ol a network a~liate. It is also useful to note the interaction between the head constituent of the pronoun p and the antecedent. For example: A Japanese company might make television picture tubes in Japan, assemble the T V sets in Malaysia and extort them to Indonesia. Here we would compare the degree to which each possible candidate antecedent (A Japanese company, television picture tubes, Japan, T V sets, and Malaysia in this example) could serve as the direct object of "export". These probabilities give us a way to implement selectional restriction. A canonical example of selectional restriction is that of the verb "eat", which selects food as its direct object. In the case of "export" the restriction is not as clearcut. Nevertheless it can still give us guidance on which candidates are more probable than others. The last factor we consider is referents' mention count. Noun phrases that are mentioned repeatedly are preferred. The training corpus is marked with the number of times a referent has been mentioned up to that point in the story. Here we are concerned with the probability that a proposed antecedent is correct given that it has been repeated a certain number of times. 162 In effect, we use this probability information to identify the topic of the segment with the belief that the topic is more likely to be referred to by a pronoun. The idea is similar to tha t used in the centering approach (Brennan et al., 1987) where a continued topic is the highest-ranked candidate for pronominalization. Given the above possible sources of informar tion, we arrive at the following equation, where F(p) denotes a function from pronouns to their antecedents: F(p) = argmaxP( A(p) = alp, h, l~', t, l, so, d~ A~') where A(p) is a random variable denoting the referent of the pronoun p and a is a proposed antecedent. In the conditioning events, h is the head constituent above p, l~ r is the list of candidate antecedents to be considered, t is the type of phrase of the proposed antecedent (always a noun-phrase in this s tudy), I is the type of the head constituent, sp describes the syntactic structure in which p appears, dspecifies the distance of each antecedent from p and M" is the number of times the referent is mentioned. Note that 17r ", d'~ and A~ are vector quantities in which each entry corresponds to a possible antecedent. When viewed in this way, a can be regarded as an index into these vectors that specifies which value is relevant to the particular choice of antecedent. This equation is decomposed into pieces that correspond to all the above factors but are more statistically manageable. The decomposition makes use of Bayes' theorem and is based on certain independence assumptions discussed below. P( A(p) = alp, h, fir, t, l, sp, d~ .Q') = P(alA~)P(p,h, fir, t,l, sp,~a, 2~) (1) P(p, h, fir, t, t, sp, diM ) o¢ PCalM)P(p, h, fir, t, l, sp, ~a, .Q') (2) = P(a[:Q)P(.%, ~a, :~'I) P(p,h, fir, t, l la ,~ ,sp , i) (3) = P(all~)P(sp, d~a,.Q ) PCh, t, Zla, ~'0", so, i) PC.. ~ la , .~', so, d, h, t, l) (4) oc P(a]l~)P(So,~a,M') P(p, 14tin, ]Q, s o, d, h, t, I) (5) = P(al.Q)P(sp, d~a, 3~r) P(ffrla, I~, s o, d, h, t, I). (6) P(pla. l~, sf,, d. h, t, l, l~) cx P(a163P(dtt la)P(f f ' lh , t, I, a) P(plw°) (7) Equation (1) is simply an application of Bayes' rule. The denominator is eliminated in the usual fashion, resulting in equation (2). Selectively applying the chain rule results in equations (3) and (4). In equation (4), the term P(h. t, lla, .~, So, d) is the same for every antecedent and is thus removed. Equat ion (6) follows when we break the last component of (5) into two probability distributions. In equation (7) we make the following independence assumptions: • Given a particular choice of the antecedent candidates, the distance is independent of distances of candidates other than the antecedent (and the distance to non-referents can be ignored): P(so, d~a, 2~) o¢ P(so, dola , IC4) • The syntnctic s t ructure st, and the distance from the pronoun da are independent of the number of times the referent is mentioned. Thus P(sp, dola, M) = P(sp, d.la) Then we combine sp and de into one variable dIt, Hobbs distance, since the Hobbs algorithm takes both the syntax and distance into account. The words in the antecedent depend only on the parent consti tuent h, the type of the words t, and the type of the parent I. Hence e(ff'la, M, sp, ~, h, t, l) = P ( ~ l h , t, l, a) • The choice pronoun depends only on the words in the antecedent, i.e. P{pla, M, sp, d, h, t, l, ~ = P(pla, W) 163 • If we treat a as an index into the vector 1~, then (a, I.V') is simply the a th candidate in the list ffz. We assume the selection of the pronoun is independent of the candidates other than the antecedent. Hence P(pla, W) = P(plw,~) Since I~" is a vector, we need to normalize P(ff ' lh, t,l, a) to obtain the probability of each element in the vector. It is reasonable to assume tha t the antecedents in W are independent of each other; in other words, P(wo+llwo, h , t , l ,a ) = P(wo+llh, t , l ,a}. Thus,