linguist us
extern knowledg
human interest comput
new corpu
question topic
inform nugget
sentenc fragment
human reader
interest
interest nugget
uniqu qualiti
surpris factor
lexic pattern
manual labor
baselin system
definit question answer
human interest

Interesting Nuggets and Their Impact on Definitional Question Answering Kian-Wei Kor Department of Computer Science School of Computing National University of Singapore dkor@comp.nus.edu.sg Tat-Seng Chua Department of Computer Science School of Computing National University of Singapore chuats@comp.nus.edu.sg ABSTRACT Current approaches to identifying definitional sentences in the context of Question Answering mainly involve the use of linguistic or syntactic patterns to identify informative nuggets.
This is insufficient as they do not address the novelty factor that a definitional nugget must also possess.
This paper proposes to address the deficiency by building a Human Interest Model from external knowledge.
It is hoped that such a model will allow the computation of human interest in the sentence with respect to the topic.
We compare and contrast our model with current definitional question answering models to show that interestingness plays an important factor in definitional question answering.
Categories and Subject Descriptors H.3.3 [Information Search and Retrieval]: Retrieval Models; H.1.2 [User/Machine Systems]: Human Factors General Terms Algorithms, Human Factors, Experimentation 1.
DEFINITIONAL QUESTION ANSWERING Definitional Question Answering was first introduced to the TExt Retrieval Conference Question Answering Track main task in 2003.
The Definition questions, also called Other questions in recent years, are defined as follows.
Given a question topic X, the task of a definitional QA system is akin to answering the question What is X?
or Who is X?
.
The definitional QA system is to search through a news corpus and return return a set of answers that best describes the question topic.
Each answer should be a unique topic-specific nugget that makes up one facet in the definition of the question topic.
1.1 The Two Aspects of Topic Nuggets Officially, topic-specific answer nuggets or simply topic nuggets are described as informative nuggets.
Each informative nugget is a sentence fragment that describe some factual information about the topic.
Depending on the topic type and domain, this can include topic properties, relationships the topic has with some closely related entity, or events that happened to the topic.
From observation of the answer set for definitional question answering from TREC 2003 to 2005, it seems that a significant number of topic nuggets cannot simply be described as informative nuggets.
Rather, these topic nuggets have a trivia-like quality associated with them.
Typically, these are out of the ordinary pieces of information about a topic that can pique a human reader``s interest.
For this reason, we decided to define answer nuggets that can evoke human interest as interesting nuggets.
In essence, interesting nuggets answer the questions What is X famous for?
, What defines X?
or What is extraordinary about X?
.
We now have two very different perspective as to what constitutes an answer to Definition questions.
An answer can be some important factual information about the topic or some novel and interesting aspect about the topic.
This duality of informativeness and interestingness can be clearly observed in the five vital answer nuggets for a TREC 2005 topic of George Foreman.
Certain answer nuggets are more informative while other nuggets are more interesting in nature.
Informative Nuggets - Was graduate of Job Corps.
- Became oldest world champion in boxing history.
Interesting Nuggets - Has lent his name to line of food preparation products.
- Waved American flag after winning 1968 Olympics championship.
- Returned to boxing after 10 yr hiatus.
As an African-American professional heavyweight boxer, an average human reader would find the last three nuggets about George Foreman interesting because boxers do not usually lend their names to food preparation products, nor do boxers retire for 10 years before returning to the ring and become the world``s oldest boxing champion.
Foreman``s waving of the American flag at the Olympics is interesting because the innocent action caused some AfricanAmericans to accuse Foreman of being an Uncle Tom.
As seen here, interesting nuggets has some surprise factor or unique quality that makes them interesting to human readers.
1.2 Identifying Interesting Nuggets Since the original official description for definitions comprise of identifying informative nuggets, most research has focused entirely on identifying informative nuggets.
In this paper, we focus on exploring the properties of interesting nuggets and develop ways of identify such interesting nuggets.
A Human Interest Model definitional question answering system is developed with emphasis on identifying interesting nuggets in order to evaluate the impact of interesting nuggets on the performance of a definitional question answering system.
We further experimented with combining the Human Interest Model with a lexical pattern based definitional question answering system in order to capture both informative and interesting nuggets.
2.
RELATED WORK There are currently two general methods for Definitional Question Answering.
The more common method uses a lexical patternbased approach was first proposed by Blair-Goldensohn et al. [1] and Xu et al. [14].
Both groups predominantly used patterns such as copulas and appositives, as well as manually crafted lexicosyntactic patterns to identify sentences that contain informative nuggets.
For example, Xu et al. used 40 manually defined structured patterns in their 2003 definitional question answering system.
Since then, in an attempt to capture a wider class of informational nuggets, many such systems of increasing complexity has been created.
A recent system by Harabagiu et al. [6] created a definitional question answering system that combines the use of 150 manually defined positive and negative patterns, named entity relations and specially crafted information extraction templates for 33 target domains.
Here, a musician template may contain lexical patterns that identify information such as the musician``s musical style, songs sung by the musician and the band, if any, that the musician belongs to.
As one can imagine, this is a knowledge intensive approach that requires an expert linguist to manually define all possible lexical or syntactic patterns required to identify specific types of information.
This process requires a lot of manual labor, expertise and is not scalable.
This lead to the development of the soft-pattern approach by Cui et al. [4, 11].
Instead of manually encoding patterns, answers to previous definitional question answering evaluations were converted into generic patterns and a probabilistic model is trained to identify such patterns in sentences.
Given a potential answer sentence, the probabilistic model outputs a probability that indicates how likely the sentence matches one or more patterns that the model has seen in training.
Such lexicalosyntactic patterns approach have been shown to be adept at identifying factual informative nuggets such as a person``s birthdate, or the name of a company``s CEO.
However, these patterns are either globally applicable to all topics or to a specific set of entities such as musicians or organizations.
This is in direct contrast to interesting nuggets that are highly specific to individual topics and not to a set of entities.
For example, the interesting nuggets for George Foreman are specific only George Foreman and no other boxer or human being.
Topic specificity or topic relevance is thus an important criteria that helps identify interesting nuggets.
This leads to the exploration of the second relevance-based approach that has been used in definitional question answering.
Predominantly, this approach has been used as a backup method for identifying definitional sentences when the primary method of lexicalosyntactic patterns failed to find a sufficient number of informative nuggets [1].
A similar approach has also been used as a baseline system for TREC 2003 [14].
More recently, Chen et al. [3] adapted a bi-gram or bi-term language model for definitional Question Answering.
Generally, the relevance-based approach requires a definitional corpus that contain documents highly relevant to the topic.
The baseline system in TREC 2003 simply uses the topic words as its definitional corpus.
Blair-Goldensohn et al. [1] uses a machine learner to include in the definitonal corpus sentences that are likely to be definitional.
Chen et al. [3] collect snippets from Google to build its definitional corpus.
From the definitional corpus, a definitional centroid vector is built or a set of centroid words are selected.
This centroid vector or set of centroid words is taken to be highly indicative of the topic.
Systems can then use this centroid to identify definitional answers by using a variety of distance metrics to compare against sentences found in the set of retrieved documents for the topic.
BlairGoldensohn et al. [1] uses Cosine similarity to rank sentences by centrality.
Chen et al. [3] builds a bigram language model using the 350 most frequently occurring google snippet terms, described in their paper as an ordered centroid, to estimate the probability that a sentence is similar to the ordered centroid.
As described here, the relevance-based approach is highly specific to individual topics due to its dependence on a topic specific definitional corpus.
However if individual sentences are viewed as a document, then relevance-based approaches essentially use the collected topic specific centroid words as a form of document retrieval with automated query expansion to identify strongly relevant sentences.
Thus such methods identify relevant sentences and not sentences containing definitional nuggets.
Yet, the TREC 2003 baseline system [14] outperformed all but one other system.
The bi-term language model [3] is able to report results that are highly competitive to state-of-the-art results using this retrieval-based approach.
At TREC 2006, a simple weighted sum of all terms model with terms weighted using solely Google snippets outperformed all other systems by a significant margin [7].
We believe that interesting nuggets often come in the form of trivia, novel or rare facts about the topic that tend to strongly cooccur with direct mention of topic keywords.
This may explain why relevance-based method can perform competitively in definitional question answering.
However, simply comparing against a single centroid vector or set of centroid words may have over emphasized topic relevance and has only identified interesting definitional nuggets in an indirect manner.
Still, relevance based retrieval methods can be used as a starting point in identifying interesting nuggets.
We will describe how we expand upon such methods to identify interesting nuggets in the next section.
3.
HUMAN INTEREST MODEL Getting a computer system to identify sentences that a human reader would find interesting is a tall order.
However, there are many documents on the world wide web that are contain concise, human written summaries on just about any topic.
What``s more, these documents are written explicitly for human beings and will contain information about the topic that most human readers would be interested in.
Assuming we can identify such relevant documents on the web, we can leverage them to assist in identifying definitional answers to such topics.
We can take the assumption that most sentences found within these web documents will contain interesting facets about the topic at hand.
This greatly simplifies the problem to that of finding within the AQUAINT corpus sentences similar to those found in web documents.
This approach has been successfully used in several factoid and list Question Answering systems [11] and we feel the use of such an approach for definitional or Other question answering is justified.
Identifying interesting nuggets requires computing machinery to understand world knowledge and human insight.
This is still a very challenging task and the use of human written documents dramatically simplifies the complexity of the task.
In this paper, we report on such an approach by experimenting with a simple word-level edit distance based weighted term comparison algorithm.
We use the edit distance algorithm to score the similarity of a pair of sentences, with one sentence coming from web resources and the other sentence selected from the AQUAINT corpus.
Through a series of experiments, we will show that even such a simple approach can be very effective at definitional question answering.
3.1 Web Resources There exists on the internet articles on just about any topic a human can think of.
What``s more, many such articles are centrally located on several prominent websites, making them an easily accessible source of world knowledge.
For our work on identifying interesting nuggets, we focused on finding short one or two page articles on the internet that are highly relevant to our desired topic.
Such articles are useful as they contain concise information about the topic.
More importantly, the articles are written by humans, for human readers and thus contain the critical human world knowledge that a computer system currently is unable to capture.
We leverage this world knowledge by collecting articles for each topic from the following external resources to build our Interest Corpus for each topic.
Wikipedia is a Web-based, free-content encyclopedia written collaboratively by volunteers.
This resource has been used by many Question Answering system as a source of knowledge about each topic.
We use a snapshot of Wikipedia taken in March 2006 and include the most relevant article in the Interest Corpus.
NewsLibrary is a searchable archive of news articles from over 100 different newspaper agencies.
For each topic, we download the 50 most relevant articles and include the title and first paragraph of each article in the Interest Corpus.
Google Snippets are retrieved by issuing the topic as a query to the Google search engine.
From the search results, we extracted the top 100 snippets.
While Google snippets are not articles, we find that they provide a wide coverage of authorative information about most topics.
Due to their comprehensive coverage of a wide variety of topics, the above resources form the bulk of our Interest Corpus.
We also extracted documents from other resources.
However, as these resources are more specific in nature, we do not always get any single relevant document.
These resources are listed below.
Biography.com is the website for the Biography television cable channel.
The channel``s website contains searchable biographies on over 25,000 notable people.
If the topic is a person and we can find a relevant biography on the person, we include it it in our Interest Corpus.
Bartleby.com contains a searchable copy of several resources including the Columbia Encyclopedia, the World Factbook, and several English dictionaries.
s9.com is a biography dictionary on over 33,000 notable people.
Like Biography.com, we include the most relevant biography we can find in the Interest Corpus.
Google Definitions Google search engine offers a feature called Definitions that provides the definition for a query, if it has one.
We use this feature and extract whatever definitions the Google search engine has found for each topic into the Interest Corpus.
Figure 1: Human Interest Model Architecture.
WordNet WordNet is an well-known electronic semantic lexicon for the English language.
Besides grouping English words into sets of synonyms called synsets, it also provide a short definition on the meaning of words found in each synset.
We add this short definition, if there is one, into our Interest Corpus.
We have two major uses for this topic specific Interest Corpus, as a source of sentences containing interesting nuggets and as a unigram language model of topic terms, I. 3.2 Multiple Interesting Centroids We have seen that interesting nuggets are highly specific to a topic.
Relevance-based approaches such as the bigram language model used by Chen et al. [3] are focused on identifying highly relevant sentences and pick up definitional answer nuggets as an indirect consequence.
We believe that the use of only a single collection of centroid words has over-emphasized topic relevance and choose instead to use multiple centroids.
Since sentences in the Interest Corpus of articles we collected from the internet are likely to contain nuggets that are of interest to human readers, we can essentially use each sentence as pseudocentroids.
Each sentence in the Interest Corpus essentially raises a different aspect of the topic for consideration as a sentence of interest to human readers.
By performing a pairwise sentence comparison between sentences in the Interest Corpus and candidate sentences retrieved from the AQUAINT corpus, we increase the number of sentence comparisons from O(n) to O(nm).
Here, n is the number of potential candidate sentences and m is the number of sentences in the Interest Corpus.
In return, we obtain a diverse ranked list of answers that are individually similar to various sentences found in the topic``s Interest Corpus.
An answer can only be highly ranked if it is strongly similar to a sentence in the Interest Corpus, and is also strongly relevant to the topic.
3.3 Implementation Figure 1 shows the system architecture for the proposed Human Interest-based definitional QA system.
The AQUAINT Retrieval module shown in Figure 1 reuses a document retrieval module of a current Factoid and List Question Answering system we have implemented.
Given a set of words describing the topic, the AQUAINT Retrieval module does query expansion using Google and searches an index of AQUAINT documents to retrieve the 800 most relevant documents for consideration.
The Web Retrieval module on the other hand, searches the online resources described in Section 3.1 for interesting documents in order to populate the Interest Corpus.
The HIM Ranker, or Human Interest Model Ranking module, is the implementation of what is described in this paper.
The module first builds the unigram language model, I, from the collected web documents.
This language model will be used to weight the importance of terms within sentences.
Next, a sentence chunker is used to segment all 800 retrieved documents into individual sentences.
Each of these sentences can be a potential answer sentence that will be independently ranked by interestingness.
We rank sentences by interestingness using sentences from both the Interest Corpus of external documents as well as the unigram language model we built earlier which we use to weight terms.
A candidate sentence in our top 800 relevant AQUAINT documents is considered interesting if it is highly similar in content to a sentence found in our collection of external web-documents.
To achieve this, we perform a pairwise similarity comparison between a candidate sentence and sentences in our external documents using a weighted-term edit distance algorithm.
Term weights are used to adjust the relative importance of each unique term found in the Interest Corpus.
When both sentences share the same term, the similarity score is incremented by the two times the term``s weight and every dissimilar term decrements the similarity score by the dissimilar term``s weight.
We choose the highest achieved similarity score for a candidate sentence as the Human Interest Model score for the candidate sentence.
In this manner, every candidate sentence is ranked by interestingness.
Finally, to obtain the answer set, we select the top 12 highest ranked and non redundant sentences as definitional answers for the topic.
4.
INITIAL EXPERIMENTS The Human Interest-based system described in the previous section is designed to identify only interesting nuggets and not informative nuggets.
Thus, it can be described as a handicapped system that only deals with half the problem in definitional question answering.
This is done in order to explore how interestingness plays a factor in definitional answers.
In order to compare and contrast the differences between informative and interesting nuggets, we also implemented the soft-pattern bigram model proposed by Cui et al. [4, 11].
In order to ensure comparable results, both systems are provided identical input data.
Since both system require the use of external resources, they are both provided the same web articles retrieved by our Web Retrieval module.
Both systems also rank the same same set of candidate sentences in the form of 800 most relevant documents as retrieved by our AQUAINT Retrieval module.
For the experiments, we used the TREC 2004 question set to tune any system parameters and use the TREC 2005 question sets to test the both systems.
Both systems are evaluated the results using the standard scoring methodology for TREC definitions.
TREC provides a list of vital and okay nuggets for each question topic.
Every question is scored on nugget recall (NR) and nugget precision (NP) and a single final score is computed using F-Measure (see equation 1) with β = 3 to emphasize nugget recall.
Here, NR is the number of vital nuggets returned divided by total number of vital nuggets while NP is computed using a minimum allowed character length function defined in [12].
The evaluation is automatically conducted using Pourpre v1.0c [10].
FScore = β2 ∗ NP ∗ NR (β2 + 1)NP + NR (1) System F3-Score Best TREC 2005 System 0.2480 Soft-Pattern (SP) 0.2872 Human Interest Model (HIM) 0.3031 Table 1: Performance on TREC 2005 Question Set Figure 2: Performance by entity types.
4.1 Informativeness vs Interestingness Our first experiment compares the performance of solely identifying interesting nuggets against solely identifying informative nuggets.
We compare the results attained by the Human Interest Model that only identify interesting nuggets with the results of the syntactic pattern finding Soft-Pattern model as well as the result of the top performing definitional system in TREC 2005 [13].
Table 1 shows the F3 score the three systems for the TREC 2005 question set.
The Human Interest Model clearly outperform both soft pattern and the best TREC 2005 system with a F3 score of 0.303.
The result is also comparable with the result of a human manual run, which attained a F3 score of 0.299 on the same question set [9].
This result is confirmation that interesting nuggets does indeed play a significant role in picking up definitional answers, and may be more vital than using information finding lexical patterns.
In order to get a better perspective of how well the Human Interest Model performs for different types of topics, we manually divided the TREC 2005 topics into four broad categories of PERSON, ORGANIZATION, THING and EVENT as listed in Table 3.
These categories conform to TREC``s general division of question topics into 4 main entity types [13].
The performance of Human Interest Model and Soft Pattern Bigram Model for each entity type can be seen in Figure 2.
Both systems exhibit consistent behavior across entity types, with the best performance coming from PERSON and ORGANIZATION topics and the worst performance from THING and EVENT topics.
This can mainly be attributed to our selection of web-based resources for the definitional corpus used by both system.
In general, it is harder to locate a single web article that describes an event or a general object.
However given the same set of web-based information, the Human Interest Model consistently outperforms the soft-pattern model for all four entity types.
This suggests that the Human Interest Model is better able to leverage the information found in web resources to identify definitional answers.
5.
REFINEMENTS Encouraged by the initial experimental results, we explored two further optimization of the basic algorithm.
5.1 Weighting Interesting Terms The word trivia refer to tidbits of unimportant or uncommon information.
As we have noted, interesting nuggets often has a trivialike quality that makes them of interest to human beings.
From this description of interesting nuggets and trivia, we hypothesize that interesting nuggets are likely to occur rarely in a text corpora.
There is a possibility that some low-frequency terms may actually be important in identifying interesting nuggets.
A standard unigram language model would not capture these low-frequency terms as important terms.
To explore this possibility, we experimented with three different term weighting schemes that can provide more weight to certain low-frequency terms.
The weighting schemes we considered include commonly used TFIDF, as well as information theoretic Kullback-Leiber divergence and Jensen-Shannon divergence [8].
TFIDF, or Term Frequency × Inverse Document Frequency, is a standard Information Retrieval weighting scheme that balances the importance of a term in a document and in a corpus.
For our experiments, we compute the weight of each term as tf × log( N nt ), where tf is the term frequency, nt is the number of sentences in the Interest Corpus having the term and N is the total number of sentences in the Interest Corpus.
Kullback-Leibler Divergence (Equation 2) is also called KL Divergence or relative entropy, can be viewed as measuring the dissimilarity between two probability distributions.
Here, we treat the AQUAINT corpus as a unigram language model of general English [15], A, and the Interest Corpus as a unigram language model consisting of topic specific terms and general English terms, I. General English words are likely to have similar distributions in both language models I and A. Thus using KL Divergence as a term weighting scheme will cause strong weights to be given to topicspecific terms because their distribution in the Interest Corpus they occur significantly more often or less often than in general English.
In this way, high frequency centroid terms as well as low frequency rare but topic-specific terms are both identified and highly weighted using KL Divergence.
DKL(I A) = t I(t)log I(t) A(t) (2) Due to the power law distribution of terms in natural language, there are only a small number of very frequent terms and a large number of rare terms in both I and A.
While the common terms in English consist of stop words, the common terms in the topic specific corpus, I, consist of both stop words and relevant topic words.
These high frequency topic specific words occur very much more frequently in I than in A.
As a result, we found that KL Divergence has a bias towards highly frequent topic terms as we are measuring direct dissimilarity against a model of general English where such topic terms are very rare.
For this reason, we explored another divergence measure as a possible term weighting scheme.
Jensen-Shannon Divergence or JS Divergence extends upon KL Divergence as seen in Equation 3.
As with KL Divergence, we also use JS divergence to measure the dissimilarity between our two language models, I and A. DJS(I A) = 1 2 centsDKL I I+A 2 ¡+ DKL A I+A 2 ¡# (3) Figure 3: Performance by various term weighting schemes on the Human Interest Model.
However, JS Divergence has additional properties1 of being symmetric and non-negative as seen in Equation 4.
The symmetric property gives a more balanced measure of dissimilarity and avoids the bias that KL divergence has.
DJS(I A) = DJS(A I) = 0 I = A > 0 I <> A (4) We conducted another experiment, substituting the unigram languge model weighting scheme we used in the initial experiments with the three term weighting schemes described above.
As lower bound reference, we included a term weighting scheme consisting of a constant 1 for all terms.
Figure 3 show the result of applying the five different term weighting schemes on the Human Interest Model.
TFIDF performed the worst as we had anticipated.
The reason is that most terms only appear once within each sentence, resulting in a term frequency of 1 for most terms.
This causes the IDF component to be the main factor in scoring sentences.
As we are computing the Inverse Document Frequency for terms in the Interest Corpus collected from web resources, IDF heavily down-weights highly frequency topic terms and relevant terms.
This results in TFIDF favoring all low frequency terms over high frequency terms in the Interest Corpus.
Despite this, the TFIDF weighting scheme only scored a slight 0.0085 lower than our lower bound reference of constant weights.
We view this as a positive indication that low frequency terms can indeed be useful in finding interesting nuggets.
Both KL and JS divergence performed marginally better than the uniform language model probabilistic scheme that we used in our initial experiments.
From inspection of the weighted list of terms, we observed that while low frequency relevant terms were boosted in strength, high frequency relevant terms still dominate the top of the weighted term list.
Only a handful of low frequency terms were weighted as strongly as topic keywords and combined with their low frequency, may have limited the impact of re-weighting such terms.
However we feel that despite this, Jensen-Shannon divergence does provide a small but measurable increase in the performance of our Human Interest Model.
1 JS divergence also has the property of being bounded, allowing the results to be treated as a probability if required.
However, the bounded property is not required here as we are only treating the divergence computed by JS divergence as term weights 5.2 Selecting Web Resources In one of our initial experiments, we observed that the quality of web resources included in the Interest Corpus may have a direct impact on the results we obtain.
We wanted to determine what impact the choice of web resources have on the performance of our Human Interest Model.
For this reason, we split our collection of web resources into four major groups listed here: N - News: Title and first paragraph of the top 50 most relevant articles found in NewsLibrary.
W - Wikipedia: Text from the most relevant article found in Wikipedia.
S - Snippets: Snippets extracted from the top 100 most relevant links after querying Google.
M - Miscellaneous sources: Combination of content (when available) from secondary sources including biography.com, s9.com, bartleby.com articles, Google definitions and WordNet definitions.
We conducted a gamut of runs on the TREC 2005 question set using all possible combinations of the above four groups of web resources to identify the best possible combination.
All runs were conducted on Human Interest Model using JS divergence as term weighting scheme.
The runs were sorted in descending F3-Score and the top 3 best performing runs for each entity class are listed in Table 2 together with earlier reported F3-scores from Figure 2 as a baseline reference.
A consistent trend can be observed for each entity class.
For PERSON and EVENT topics, NewsLibrary articles are the main source of interesting nuggets with Google snippets and miscellaneous articles offering additional supporting evidence.
This seem intuitive for events as newspapers predominantly focus on reporting breaking newsworthy events and are thus excellent sources of interesting nuggets.
We had expected Wikipedia rather than news articles to be a better source of interesting facts about people and were surprised to discover that news articles outperformed Wikipedia.
We believe that the reason is because the people selected as topics thus far have been celebrities or well known public figures.
Human readers are likely to be interested in news events that spotlight these personalities.
Conversely for ORGANIZATION and THING topics, the best source of interesting nuggets come from Wikipedia``s most relevant article on the topic with Google snippets again providing additional information for organizations.
With an oracle that can classify topics by entity class with 100% accuracy and by using the best web resources for each entity class as shown in Table 2, we can attain a F3-Score of 0.3158.
6.
UNIFYING INFORMATIVENESS WITH INTERESTINGNESS We have thus far been comparing the Human Interest Model against the Soft-Pattern model in order to understand the differences between interesting and informative nuggets.
However from the perspective of a human reader, both informative and interesting nuggets are useful and definitional.
Informative nuggets present a general overview of the topic while interesting nuggets give readers added depth and insight by providing novel and unique aspects about the topic.
We believe that a good definitional question answering system should provide the reader with a combined mixture of both nugget types as a definitional answer set.
Rank PERSON ORG THING EVENT Baseline Unigram Weighting Scheme, N+W+S+M 0.3279 0.3630 0.2551 0.2644 1 N+S+M W+S W+M N+M 0.3584 0.3709 0.2688 0.2905 2 N+S N+W+S W+S+M N+S+M 0.3469 0.3702 0.2665 0.2745 3 N+M N+W+S+M W+S N+S 0.3431 0.3680 0.2616 0.2690 Table 2: Top 3 runs using different web resources for each entity class We now have two very different experts at identifying definitions.
The Soft Pattern Bigram Model proposed by Cui et al. is an expert in identifying informative nuggets.
The Human Interest Model we have described in this paper on the other hand is an expert in finding interesting nuggets.
We had initially hoped to unify the two separate definitional question answering systems by applying an ensemble learning method [5] such as voting or boosting in order to attain a good mixture of informative and interesting nuggets in our answer set.
However, none of the ensemble learning methods we attempted could outperform our Human Interest Model.
The reason is that both systems are picking up very different sentences as definitional answers.
In essence, our two experts are disagreeing on which sentences are definitional.
In the top 10 sentences from both systems, only 4.4% of these sentences appeared in both answer sets.
The remaining answers were completely different.
Even when we examined the top 500 sentences generated by both systems, the agreement rate was still an extremely low 5.3%.
Yet, despite the low agreement rate between both systems, each individual system is still able to attain a relatively high F3 score.
There is a distinct possibility that each system may be selecting different sentences with different syntactic structures but actually have the same or similar semantic content.
This could result in both systems having the same nuggets marked as correct even though the source answer sentences are structurally different.
Unfortunately, we are unable to automatically verify this as the evaluation software we are using does not report correctly identified answer nuggets.
To verify if both systems are selecting the same answer nuggets, we randomly selected a subset of 10 topics from the TREC 2005 question set and manually identified correct answer nuggets (as defined by TREC accessors) from both systems.
When we compared the answer nuggets found by both system for this subset of topics, we found that the nugget agreement rate between both systems was 16.6%.
While the nugget agreement rate is higher than the sentence agreement rate, both systems are generally still picking up different answer nuggets.
We view this as further indication that definitions are indeed made up of a mixture of informative and interesting nuggets.
It is also indication that in general, interesting and informative nuggets are quite different in nature.
There are thus rational reasons and practical motivation in unifying answers from both the pattern based and corpus based approaches.
However, the differences between the two systems also cause issues when we attempt to combine both answer sets.
Currently, the best approach we found for combining both answer sets is to merge and re-rank both answer sets with boosting agreements.
We first normalize the top 1,000 ranked sentences from each system, to obtain the Normalized Human Interest Model score, him(s), and the Normalized Soft Pattern Bigram Model score, sp(s), for every unique sentence, s. For each sentence, the two separate scores for are then unified into a single score using Equation 5.
When only one system believes that the sentence is definitional, we simply retain that system``s normalized score as the unified score.
When both systems agree agree that the sentence is definitional, the sentence``s score is boosted by the degree of agreement between between both systems.
Score(s) = max(shim, ssp)1−min(shim,ssp) (5) In order to maintain a diverse set of answers as well as to ensure that similar sentences are not given similar ranking, we further re-rank our combined list of answers using Maximal Marginal Relevance or MMR [2].
Using the approach described here, we achieve a F3 score of 0.3081.
This score is equivalent to the initial Human Interest Model score of 0.3031 but fails to outperform the optimized Human Interest Model model.
7.
CONCLUSION This paper has presented a novel perspective for answering definitional questions through the identification of interesting nuggets.
Interesting nuggets are uncommon pieces of information about the topic that can evoke a human reader``s curiosity.
The notion of an average human reader is an important consideration in our approach.
This is very different from the lexico-syntactic pattern approach where the context of a human reader is not even considered when finding answers for definitional question answering.
Using this perspective, we have shown that using a combination of a carefully selected external corpus, matching against multiple centroids and taking into consideration rare but highly topic specific terms, we can build a definitional question answering module that is more focused on identifying nuggets that are of interest to human beings.
Experimental results has shown this approach can significantly outperform state-of-the-art definitional question answering systems.
We further showed that at least two different types of answer nuggets are required to form a more thorough set of definitional answers.
What seems to be a good set of definition answers is some general information that provides a quick informative overview mixed together with some novel or interesting aspects about the topic.
Thus we feel that a good definitional question answering system would need to pick up both informative and interesting nugget types in order to provide a complete definitional coverage on all important aspects of the topic.
While we have attempted to build such a system by combining our proposed Human Interest Model with Cui et al.``s Soft Pattern Bigram Model, the inherent differences between both types of nuggets seemingly caused by the low agreement rates between both models have made this a difficult task.
Indeed, this is natural as the two models have been designed to identify two very different types of definition answers using very different types of features.
As a result, we are currently only able to achieve a hybrid system that has the same level of performance as our proposed Human Interest Model.
We approached the problem of definitional question answering from a novel perspective, with the notion that interest factor plays a role in identifying definitional answers.
Although the methods we used are simple, they have been shown experimentally to be effective.
Our approach may also provide some insight into a few anomalies in past definitional question answering``s trials.
For instance, the top definitional system at the recent TREC 2006 evaluation was able to significantly outperform all other systems using relatively simple unigram probabilities extracted from Google snippets.
We suspect the main contributor to the system``s performance Entity Type Topics ORGANIZATION DePauw University, Merck & Co., Norwegian Cruise Lines (NCL), United Parcel Service (UPS), Little League Baseball, Cliffs Notes, American Legion, Sony Pictures Entertainment (SPE), Telefonica of Spain, Lions Club International, AMWAY, McDonald``s Corporation, Harley-Davidson, U.S. Naval Academy, OPEC, NATO, International Bureau of Universal Postal Union (UPU), Organization of Islamic Conference (OIC), PBGC PERSON Bing Crosby, George Foreman, Akira Kurosawa, Sani Abacha, Enrico Fermi, Arnold Palmer, Woody Guthrie, Sammy Sosa, Michael Weiss, Paul Newman, Jesse Ventura, Rose Crumb, Rachel Carson, Paul Revere, Vicente Fox, Rocky Marciano, Enrico Caruso, Pope Pius XII, Kim Jong Il THING F16, Bollywood, Viagra, Howdy Doody Show, Louvre Museum, meteorites, Virginia wine, Counting Crows, Boston Big Dig, Chunnel, Longwood Gardens, Camp David, kudzu, U.S. Medal of Honor, tsunami, genome, Food-for-Oil Agreement, Shiite, Kinmen Island EVENT Russian submarine Kursk sinks, Miss Universe 2000 crowned, Port Arthur Massacre, France wins World Cup in soccer, Plane clips cable wires in Italian resort, Kip Kinkel school shooting, Crash of EgyptAir Flight 990, Preakness 1998, first 2000 Bush-Gore presidential debate , 1998 indictment and trial of Susan McDougal, return of Hong Kong to Chinese sovereignty, 1998 Nagano Olympic Games, Super Bowl XXXIV, 1999 North American International Auto Show, 1980 Mount St. Helens eruption, 1998 Baseball World Series, Hindenburg disaster, Hurricane Mitch Table 3: TREC 2005 Topics Grouped by Entity Type is Google``s PageRank algorithm, which mainly consider the number of linkages, has an indirect effect of ranking web documents by the degree of human interest.
In our future work, we seek to further improve on the combined system by incorporating more evidence in support of correct definitional answers or to filter away obviously wrong answers.
8.
REFERENCES [1] S. Blair-Goldensohn, K. R. McKeown, and A. H. Schlaikjer.
A hybrid approach for qa track definitional questions.
In TREC ``03: Proceedings of the 12th Text REtrieval Conference, Gaithersburg, Maryland, 2003.
[2] J. G. Carbonell and J. Goldstein.
The use of MMR, diversity-based reranking for reordering documents and producing summaries.
In Research and Development in Information Retrieval, pages 335-336, 1998.
[3] Y. Chen, M. Zhou, and S. Wang.
Reranking answers for definitional qa using language modeling.
In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 1081-1088, Sydney, Australia, July 2006.
Association for Computational Linguistics.
[4] H. Cui, M.-Y.
Kan, and T.-S.
Chua.
Generic soft pattern models for definitional question answering.
In SIGIR ``05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 384-391, New York, NY, USA, 2005.
ACM Press.
[5] T. G. Dietterich.
Ensemble methods in machine learning.
Lecture Notes in Computer Science, 1857:1-15, 2000.
[6] S. Harabagiu, D. Moldovan, C. Clark, M. Bowden, A. Hickl, and P. Wang.
Employing two question answering systems at trec 2005.
In TREC ``05: Proceedings of the 14th Text REtrieval Conference, Gaithersburg, Maryland, 2005.
[7] M. Kaisser, S. Scheible, and B. Webber.
Experiments at the university of edinburgh for the trec 2006 qa track.
In TREC ``06 Notebook: Proceedings of the 14th Text REtrieval Conference, Gaithersburg, Maryland, 2006.
National Institute of Standards and Technology.
[8] J. Lin.
Divergence measures based on the shannon entropy.
IEEE Transactions on Information Theory, 37(1):145 - 151, Jan 1991.
[9] J. Lin, E. Abels, D. Demner-Fushman, D. W. Oard, P. Wu, and Y. Wu.
A menagerie of tracks at maryland: Hard, enterprise, qa, and genomics, oh my!
In TREC ``05: Proceedings of the 14th Text REtrieval Conference, Gaithersburg, Maryland, 2005.
[10] J. Lin and D. Demner-Fushman.
Automatically evaluating answers to definition questions.
In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 931-938, Vancouver, British Columbia, Canada, October 2005.
Association for Computational Linguistics.
[11] R. Sun, J. Jiang, Y. F. Tan, H. Cui, T.-S.
Chua, and M.-Y.
Kan..
Using syntactic and semantic relation analysis in question answering.
In TREC ``05: Proceedings of the 14th Text REtrieval Conference, Gaithersburg, Maryland, 2005.
[12] E. M. Voorhees.
Overview of the trec 2003 question answering track.
In Text REtrieval Conference 2003, Gaithersburg, Maryland, 2003.
National Institute of Standards and Technology.
[13] E. M. Voorhees.
Overview of the trec 2005 question answering track.
In TREC ``05: Proceedings of the 14th Text REtrieval Conference, Gaithersburg, Maryland, 2005.
National Institute of Standards and Technology.
[14] J. Xu, A. Licuanan, and R. Weischedel.
TREC 2003 QA at BBN: Answering definitional questions.
In TREC ``03: Proceedings of the 12th Text REtrieval Conference, Gaithersburg, Maryland, 2003.
[15] D. Zhang and W. S. Lee.
A language modeling approach to passage question answering.
In TREC ``03: Proceedings of the 12th Text REtrieval Conference, Gaithersburg, Maryland, 2003.

Interesting Nuggets and Their Impact on Definitional Question Answering
ABSTRACT
Current approaches to identifying definitional sentences in the context of Question Answering mainly involve the use of linguistic or syntactic patterns to identify informative nuggets.
This is insufficient as they do not address the novelty factor that a definitional nugget must also possess.
This paper proposes to address the deficiency by building a "Human Interest Model" from external knowledge.
It is hoped that such a model will allow the computation of human interest in the sentence with respect to the topic.
We compare and contrast our model with current definitional question answering models to show that interestingness plays an important factor in definitional question answering.
1.
DEFINITIONAL QUESTION ANSWERING
Definitional Question Answering was first introduced to the TExt Retrieval Conference Question Answering Track main task in 2003.
The Definition questions, also called Other questions in recent years, are defined as follows.
Given a question topic X, the task of a definitional QA system is akin to answering the question "What is X?"
or "Who is X?"
.
The definitional QA system is to search through a news corpus and return return a set of answers that best describes the question topic.
Each answer should be a unique topic-specific nugget that makes up one facet in the definition of the question topic.
1.1 The Two Aspects of Topic Nuggets
Officially, topic-specific answer nuggets or simply topic nuggets are described as "informative nuggets".
Each informative nugget is a sentence fragment that describe some factual information about the topic.
Depending on the topic type and domain, this can include topic properties, relationships the topic has with some closely related entity, or events that happened to the topic.
From observation of the answer set for definitional question answering from TREC 2003 to 2005, it seems that a significant number of topic nuggets cannot simply be described as informative nuggets.
Rather, these topic nuggets have a trivia-like quality associated with them.
Typically, these are out of the ordinary pieces of information about a topic that can pique a human reader's interest.
For this reason, we decided to define answer nuggets that can evoke human interest as "interesting nuggets".
In essence, interesting nuggets answer the questions "What is X famous for?"
, "What defines X?"
or "What is extraordinary about X?"
.
We now have two very different perspective as to what constitutes an answer to Definition questions.
An answer can be some important factual information about the topic or some novel and interesting aspect about the topic.
This duality of informativeness and interestingness can be clearly observed in the five vital answer nuggets for a TREC 2005 topic of "George Foreman".
Certain answer nuggets are more informative while other nuggets are more interesting in nature.
Informative Nuggets
- Was graduate of Job Corps.
- Became oldest world champion in boxing history.
Interesting Nuggets
- Has lent his name to line of food preparation products.
- Waved American flag after winning 1968 Olympics championship.
- Returned to boxing after 10 yr hiatus.
As an African-American professional heavyweight boxer, an average human reader would find the last three nuggets about George Foreman interesting because boxers do not usually lend their names to food preparation products, nor do boxers retire for 10 years before returning to the ring and become the world's oldest boxing champion.
Foreman's waving of the American flag at the Olympics is interesting because the innocent action caused some AfricanAmericans to accuse Foreman of being an Uncle Tom.
As seen here, interesting nuggets has some surprise factor or unique quality that makes them interesting to human readers.
1.2 Identifying Interesting Nuggets
Since the original official description for definitions comprise of
identifying informative nuggets, most research has focused entirely on identifying informative nuggets.
In this paper, we focus on exploring the properties of interesting nuggets and develop ways of identify such interesting nuggets.
A" Human Interest Model" definitional question answering system is developed with emphasis on identifying interesting nuggets in order to evaluate the impact of interesting nuggets on the performance of a definitional question answering system.
We further experimented with combining the Human Interest Model with a lexical pattern based definitional question answering system in order to capture both informative and interesting nuggets.
2.
RELATED WORK
There are currently two general methods for Definitional Question Answering.
The more common method uses a lexical patternbased approach was first proposed by Blair-Goldensohn et al. [1] and Xu et al. [14].
Both groups predominantly used patterns such as copulas and appositives, as well as manually crafted lexicosyntactic patterns to identify sentences that contain informative nuggets.
For example, Xu et al. used 40 manually defined "structured patterns" in their 2003 definitional question answering system.
Since then, in an attempt to capture a wider class of informational nuggets, many such systems of increasing complexity has been created.
A recent system by Harabagiu et al. [6] created a definitional question answering system that combines the use of 150 manually defined positive and negative patterns, named entity relations and specially crafted information extraction templates for 33 target domains.
Here, a musician template may contain lexical patterns that identify information such as the musician's musical style, songs sung by the musician and the band, if any, that the musician belongs to.
As one can imagine, this is a knowledge intensive approach that requires an expert linguist to manually define all possible lexical or syntactic patterns required to identify specific types of information.
This process requires a lot of manual labor, expertise and is not scalable.
This lead to the development of the soft-pattern approach by Cui et al. [4, 11].
Instead of manually encoding patterns, answers to previous definitional question answering evaluations were converted into generic patterns and a probabilistic model is trained to identify such patterns in sentences.
Given a potential answer sentence, the probabilistic model outputs a probability that indicates how likely the sentence matches one or more patterns that the model has seen in training.
Such lexicalosyntactic patterns approach have been shown to be adept at identifying factual informative nuggets such as a person's birthdate, or the name of a company's CEO.
However, these patterns are either globally applicable to all topics or to a specific set of entities such as musicians or organizations.
This is in direct contrast to interesting nuggets that are highly specific to individual topics and not to a set of entities.
For example, the interesting nuggets for George Foreman are specific only George Foreman and no other boxer or human being.
Topic specificity or topic relevance is thus an important criteria that helps identify interesting nuggets.
This leads to the exploration of the second relevance-based approach that has been used in definitional question answering.
Predominantly, this approach has been used as a backup method for identifying definitional sentences when the primary method of lexicalosyntactic patterns failed to find a sufficient number of informative nuggets [1].
A similar approach has also been used as a baseline system for TREC 2003 [14].
More recently, Chen et al. [3] adapted a bi-gram or bi-term language model for definitional Question Answering.
Generally, the relevance-based approach requires a "definitional corpus" that contain documents highly relevant to the topic.
The baseline system in TREC 2003 simply uses the topic words as its definitional corpus.
Blair-Goldensohn et al. [1] uses a machine learner to include in the definitonal corpus sentences that are likely to be definitional.
Chen et al. [3] collect snippets from Google to build its definitional corpus.
From the definitional corpus, a definitional centroid vector is built or a set of centroid words are selected.
This centroid vector or set of centroid words is taken to be highly indicative of the topic.
Systems can then use this centroid to identify definitional answers by using a variety of distance metrics to compare against sentences found in the set of retrieved documents for the topic.
BlairGoldensohn et al. [1] uses Cosine similarity to rank sentences by "centrality".
Chen et al. [3] builds a bigram language model using the 350 most frequently occurring google snippet terms, described in their paper as an ordered centroid, to estimate the probability that a sentence is similar to the ordered centroid.
As described here, the relevance-based approach is highly specific to individual topics due to its dependence on a topic specific definitional corpus.
However if individual sentences are viewed as a document, then relevance-based approaches essentially use the collected topic specific centroid words as a form of document retrieval with automated query expansion to identify strongly relevant sentences.
Thus such methods identify relevant sentences and not sentences containing definitional nuggets.
Yet, the TREC 2003 baseline system [14] outperformed all but one other system.
The bi-term language model [3] is able to report results that are highly competitive to state-of-the-art results using this retrieval-based approach.
At TREC 2006, a simple weighted sum of all terms model with terms weighted using solely Google snippets outperformed all other systems by a significant margin [7].
We believe that interesting nuggets often come in the form of trivia, novel or rare facts about the topic that tend to strongly cooccur with direct mention of topic keywords.
This may explain why relevance-based method can perform competitively in definitional question answering.
However, simply comparing against a single centroid vector or set of centroid words may have over emphasized topic relevance and has only identified interesting definitional nuggets in an indirect manner.
Still, relevance based retrieval methods can be used as a starting point in identifying interesting nuggets.
We will describe how we expand upon such methods to identify interesting nuggets in the next section.
3.
HUMAN INTEREST MODEL
3.1 Web Resources
3.2 Multiple Interesting Centroids
3.3 Implementation
4.
INITIAL EXPERIMENTS
4.1 Informativeness vs Interestingness
5.
REFINEMENTS
5.1 Weighting Interesting Terms
5.2 Selecting Web Resources
6.
UNIFYING INFORMATIVENESS WITH INTERESTINGNESS
7.
CONCLUSION
This paper has presented a novel perspective for answering definitional questions through the identification of interesting nuggets.
Interesting nuggets are uncommon pieces of information about the topic that can evoke a human reader's curiosity.
The notion of an" average human reader" is an important consideration in our approach.
This is very different from the lexico-syntactic pattern approach where the context of a human reader is not even considered when finding answers for definitional question answering.
Using this perspective, we have shown that using a combination of a carefully selected external corpus, matching against multiple centroids and taking into consideration rare but highly topic specific terms, we can build a definitional question answering module that is more focused on identifying nuggets that are of interest to human beings.
Experimental results has shown this approach can significantly outperform state-of-the-art definitional question answering systems.
We further showed that at least two different types of answer nuggets are required to form a more thorough set of definitional answers.
What seems to be a good set of definition answers is some general information that provides a quick informative overview mixed together with some novel or interesting aspects about the topic.
Thus we feel that a good definitional question answering system would need to pick up both informative and interesting nugget types in order to provide a complete definitional coverage on all important aspects of the topic.
While we have attempted to build such a system by combining our proposed Human Interest Model with Cui et al.'s Soft Pattern Bigram Model, the inherent differences between both types of nuggets seemingly caused by the low agreement rates between both models have made this a difficult task.
Indeed, this is natural as the two models have been designed to identify two very different types of definition answers using very different types of features.
As a result, we are currently only able to achieve a hybrid system that has the same level of performance as our proposed Human Interest Model.
We approached the problem of definitional question answering from a novel perspective, with the notion that interest factor plays a role in identifying definitional answers.
Although the methods we used are simple, they have been shown experimentally to be effective.
Our approach may also provide some insight into a few anomalies in past definitional question answering's trials.
For instance, the top definitional system at the recent TREC 2006 evaluation was able to significantly outperform all other systems using relatively simple unigram probabilities extracted from Google snippets.
We suspect the main contributor to the system's performance
Table 3: TREC 2005 Topics Grouped by Entity Type
is Google's PageRank algorithm, which mainly consider the number of linkages, has an indirect effect of ranking web documents by the degree of human interest.
In our future work, we seek to further improve on the combined system by incorporating more evidence in support of correct definitional answers or to filter away obviously wrong answers.

Interesting Nuggets and Their Impact on Definitional Question Answering
ABSTRACT
Current approaches to identifying definitional sentences in the context of Question Answering mainly involve the use of linguistic or syntactic patterns to identify informative nuggets.
This is insufficient as they do not address the novelty factor that a definitional nugget must also possess.
This paper proposes to address the deficiency by building a "Human Interest Model" from external knowledge.
It is hoped that such a model will allow the computation of human interest in the sentence with respect to the topic.
We compare and contrast our model with current definitional question answering models to show that interestingness plays an important factor in definitional question answering.
1.
DEFINITIONAL QUESTION ANSWERING
Definitional Question Answering was first introduced to the TExt Retrieval Conference Question Answering Track main task in 2003.
The Definition questions, also called Other questions in recent years, are defined as follows.
Given a question topic X, the task of a definitional QA system is akin to answering the question "What is X?"
.
The definitional QA system is to search through a news corpus and return return a set of answers that best describes the question topic.
Each answer should be a unique topic-specific nugget that makes up one facet in the definition of the question topic.
1.1 The Two Aspects of Topic Nuggets
Officially, topic-specific answer nuggets or simply topic nuggets are described as "informative nuggets".
Each informative nugget is a sentence fragment that describe some factual information about the topic.
From observation of the answer set for definitional question answering from TREC 2003 to 2005, it seems that a significant number of topic nuggets cannot simply be described as informative nuggets.
Rather, these topic nuggets have a trivia-like quality associated with them.
Typically, these are out of the ordinary pieces of information about a topic that can pique a human reader's interest.
For this reason, we decided to define answer nuggets that can evoke human interest as "interesting nuggets".
In essence, interesting nuggets answer the questions "What is X famous for?"
, "What defines X?"
.
We now have two very different perspective as to what constitutes an answer to Definition questions.
An answer can be some important factual information about the topic or some novel and interesting aspect about the topic.
This duality of informativeness and interestingness can be clearly observed in the five vital answer nuggets for a TREC 2005 topic of "George Foreman".
Certain answer nuggets are more informative while other nuggets are more interesting in nature.
Informative Nuggets
- Became oldest world champion in boxing history.
Interesting Nuggets
- Returned to boxing after 10 yr hiatus.
As seen here, interesting nuggets has some surprise factor or unique quality that makes them interesting to human readers.
1.2 Identifying Interesting Nuggets
Since the original official description for definitions comprise of
identifying informative nuggets, most research has focused entirely on identifying informative nuggets.
In this paper, we focus on exploring the properties of interesting nuggets and develop ways of identify such interesting nuggets.
A" Human Interest Model" definitional question answering system is developed with emphasis on identifying interesting nuggets in order to evaluate the impact of interesting nuggets on the performance of a definitional question answering system.
We further experimented with combining the Human Interest Model with a lexical pattern based definitional question answering system in order to capture both informative and interesting nuggets.
2.
RELATED WORK
There are currently two general methods for Definitional Question Answering.
The more common method uses a lexical patternbased approach was first proposed by Blair-Goldensohn et al. [1] and Xu et al. [14].
Both groups predominantly used patterns such as copulas and appositives, as well as manually crafted lexicosyntactic patterns to identify sentences that contain informative nuggets.
For example, Xu et al. used 40 manually defined "structured patterns" in their 2003 definitional question answering system.
Since then, in an attempt to capture a wider class of informational nuggets, many such systems of increasing complexity has been created.
A recent system by Harabagiu et al. [6] created a definitional question answering system that combines the use of 150 manually defined positive and negative patterns, named entity relations and specially crafted information extraction templates for 33 target domains.
As one can imagine, this is a knowledge intensive approach that requires an expert linguist to manually define all possible lexical or syntactic patterns required to identify specific types of information.
Instead of manually encoding patterns, answers to previous definitional question answering evaluations were converted into generic patterns and a probabilistic model is trained to identify such patterns in sentences.
Such lexicalosyntactic patterns approach have been shown to be adept at identifying factual informative nuggets such as a person's birthdate, or the name of a company's CEO.
However, these patterns are either globally applicable to all topics or to a specific set of entities such as musicians or organizations.
This is in direct contrast to interesting nuggets that are highly specific to individual topics and not to a set of entities.
For example, the interesting nuggets for George Foreman are specific only George Foreman and no other boxer or human being.
Topic specificity or topic relevance is thus an important criteria that helps identify interesting nuggets.
This leads to the exploration of the second relevance-based approach that has been used in definitional question answering.
Predominantly, this approach has been used as a backup method for identifying definitional sentences when the primary method of lexicalosyntactic patterns failed to find a sufficient number of informative nuggets [1].
A similar approach has also been used as a baseline system for TREC 2003 [14].
More recently, Chen et al. [3] adapted a bi-gram or bi-term language model for definitional Question Answering.
Generally, the relevance-based approach requires a "definitional corpus" that contain documents highly relevant to the topic.
The baseline system in TREC 2003 simply uses the topic words as its definitional corpus.
Blair-Goldensohn et al. [1] uses a machine learner to include in the definitonal corpus sentences that are likely to be definitional.
Chen et al. [3] collect snippets from Google to build its definitional corpus.
From the definitional corpus, a definitional centroid vector is built or a set of centroid words are selected.
This centroid vector or set of centroid words is taken to be highly indicative of the topic.
Systems can then use this centroid to identify definitional answers by using a variety of distance metrics to compare against sentences found in the set of retrieved documents for the topic.
BlairGoldensohn et al. [1] uses Cosine similarity to rank sentences by "centrality".
As described here, the relevance-based approach is highly specific to individual topics due to its dependence on a topic specific definitional corpus.
However if individual sentences are viewed as a document, then relevance-based approaches essentially use the collected topic specific centroid words as a form of document retrieval with automated query expansion to identify strongly relevant sentences.
Thus such methods identify relevant sentences and not sentences containing definitional nuggets.
Yet, the TREC 2003 baseline system [14] outperformed all but one other system.
The bi-term language model [3] is able to report results that are highly competitive to state-of-the-art results using this retrieval-based approach.
At TREC 2006, a simple weighted sum of all terms model with terms weighted using solely Google snippets outperformed all other systems by a significant margin [7].
We believe that interesting nuggets often come in the form of trivia, novel or rare facts about the topic that tend to strongly cooccur with direct mention of topic keywords.
This may explain why relevance-based method can perform competitively in definitional question answering.
However, simply comparing against a single centroid vector or set of centroid words may have over emphasized topic relevance and has only identified interesting definitional nuggets in an indirect manner.
Still, relevance based retrieval methods can be used as a starting point in identifying interesting nuggets.
We will describe how we expand upon such methods to identify interesting nuggets in the next section.
7.
CONCLUSION
This paper has presented a novel perspective for answering definitional questions through the identification of interesting nuggets.
Interesting nuggets are uncommon pieces of information about the topic that can evoke a human reader's curiosity.
The notion of an" average human reader" is an important consideration in our approach.
This is very different from the lexico-syntactic pattern approach where the context of a human reader is not even considered when finding answers for definitional question answering.
Using this perspective, we have shown that using a combination of a carefully selected external corpus, matching against multiple centroids and taking into consideration rare but highly topic specific terms, we can build a definitional question answering module that is more focused on identifying nuggets that are of interest to human beings.
Experimental results has shown this approach can significantly outperform state-of-the-art definitional question answering systems.
We further showed that at least two different types of answer nuggets are required to form a more thorough set of definitional answers.
What seems to be a good set of definition answers is some general information that provides a quick informative overview mixed together with some novel or interesting aspects about the topic.
Thus we feel that a good definitional question answering system would need to pick up both informative and interesting nugget types in order to provide a complete definitional coverage on all important aspects of the topic.
Indeed, this is natural as the two models have been designed to identify two very different types of definition answers using very different types of features.
As a result, we are currently only able to achieve a hybrid system that has the same level of performance as our proposed Human Interest Model.
We approached the problem of definitional question answering from a novel perspective, with the notion that interest factor plays a role in identifying definitional answers.
Although the methods we used are simple, they have been shown experimentally to be effective.
Our approach may also provide some insight into a few anomalies in past definitional question answering's trials.
For instance, the top definitional system at the recent TREC 2006 evaluation was able to significantly outperform all other systems using relatively simple unigram probabilities extracted from Google snippets.
We suspect the main contributor to the system's performance
Table 3: TREC 2005 Topics Grouped by Entity Type
In our future work, we seek to further improve on the combined system by incorporating more evidence in support of correct definitional answers or to filter away obviously wrong answers.

Interesting Nuggets and Their Impact on Definitional Question Answering
ABSTRACT
Current approaches to identifying definitional sentences in the context of Question Answering mainly involve the use of linguistic or syntactic patterns to identify informative nuggets.
This is insufficient as they do not address the novelty factor that a definitional nugget must also possess.
This paper proposes to address the deficiency by building a "Human Interest Model" from external knowledge.
It is hoped that such a model will allow the computation of human interest in the sentence with respect to the topic.
We compare and contrast our model with current definitional question answering models to show that interestingness plays an important factor in definitional question answering.
1.
DEFINITIONAL QUESTION ANSWERING
Definitional Question Answering was first introduced to the TExt Retrieval Conference Question Answering Track main task in 2003.
The Definition questions, also called Other questions in recent years, are defined as follows.
Given a question topic X, the task of a definitional QA system is akin to answering the question "What is X?"
or "Who is X?"
.
The definitional QA system is to search through a news corpus and return return a set of answers that best describes the question topic.
Each answer should be a unique topic-specific nugget that makes up one facet in the definition of the question topic.
1.1 The Two Aspects of Topic Nuggets
Officially, topic-specific answer nuggets or simply topic nuggets are described as "informative nuggets".
Each informative nugget is a sentence fragment that describe some factual information about the topic.
Depending on the topic type and domain, this can include topic properties, relationships the topic has with some closely related entity, or events that happened to the topic.
From observation of the answer set for definitional question answering from TREC 2003 to 2005, it seems that a significant number of topic nuggets cannot simply be described as informative nuggets.
Rather, these topic nuggets have a trivia-like quality associated with them.
Typically, these are out of the ordinary pieces of information about a topic that can pique a human reader's interest.
For this reason, we decided to define answer nuggets that can evoke human interest as "interesting nuggets".
In essence, interesting nuggets answer the questions "What is X famous for?"
, "What defines X?"
or "What is extraordinary about X?"
.
We now have two very different perspective as to what constitutes an answer to Definition questions.
An answer can be some important factual information about the topic or some novel and interesting aspect about the topic.
This duality of informativeness and interestingness can be clearly observed in the five vital answer nuggets for a TREC 2005 topic of "George Foreman".
Certain answer nuggets are more informative while other nuggets are more interesting in nature.
Informative Nuggets
- Was graduate of Job Corps.
- Became oldest world champion in boxing history.
Interesting Nuggets
- Has lent his name to line of food preparation products.
- Waved American flag after winning 1968 Olympics championship.
- Returned to boxing after 10 yr hiatus.
As an African-American professional heavyweight boxer, an average human reader would find the last three nuggets about George Foreman interesting because boxers do not usually lend their names to food preparation products, nor do boxers retire for 10 years before returning to the ring and become the world's oldest boxing champion.
Foreman's waving of the American flag at the Olympics is interesting because the innocent action caused some AfricanAmericans to accuse Foreman of being an Uncle Tom.
As seen here, interesting nuggets has some surprise factor or unique quality that makes them interesting to human readers.
1.2 Identifying Interesting Nuggets
Since the original official description for definitions comprise of
identifying informative nuggets, most research has focused entirely on identifying informative nuggets.
In this paper, we focus on exploring the properties of interesting nuggets and develop ways of identify such interesting nuggets.
A" Human Interest Model" definitional question answering system is developed with emphasis on identifying interesting nuggets in order to evaluate the impact of interesting nuggets on the performance of a definitional question answering system.
We further experimented with combining the Human Interest Model with a lexical pattern based definitional question answering system in order to capture both informative and interesting nuggets.
2.
RELATED WORK
There are currently two general methods for Definitional Question Answering.
The more common method uses a lexical patternbased approach was first proposed by Blair-Goldensohn et al. [1] and Xu et al. [14].
Both groups predominantly used patterns such as copulas and appositives, as well as manually crafted lexicosyntactic patterns to identify sentences that contain informative nuggets.
For example, Xu et al. used 40 manually defined "structured patterns" in their 2003 definitional question answering system.
Since then, in an attempt to capture a wider class of informational nuggets, many such systems of increasing complexity has been created.
A recent system by Harabagiu et al. [6] created a definitional question answering system that combines the use of 150 manually defined positive and negative patterns, named entity relations and specially crafted information extraction templates for 33 target domains.
Here, a musician template may contain lexical patterns that identify information such as the musician's musical style, songs sung by the musician and the band, if any, that the musician belongs to.
As one can imagine, this is a knowledge intensive approach that requires an expert linguist to manually define all possible lexical or syntactic patterns required to identify specific types of information.
This process requires a lot of manual labor, expertise and is not scalable.
This lead to the development of the soft-pattern approach by Cui et al. [4, 11].
Instead of manually encoding patterns, answers to previous definitional question answering evaluations were converted into generic patterns and a probabilistic model is trained to identify such patterns in sentences.
Given a potential answer sentence, the probabilistic model outputs a probability that indicates how likely the sentence matches one or more patterns that the model has seen in training.
Such lexicalosyntactic patterns approach have been shown to be adept at identifying factual informative nuggets such as a person's birthdate, or the name of a company's CEO.
However, these patterns are either globally applicable to all topics or to a specific set of entities such as musicians or organizations.
This is in direct contrast to interesting nuggets that are highly specific to individual topics and not to a set of entities.
For example, the interesting nuggets for George Foreman are specific only George Foreman and no other boxer or human being.
Topic specificity or topic relevance is thus an important criteria that helps identify interesting nuggets.
This leads to the exploration of the second relevance-based approach that has been used in definitional question answering.
Predominantly, this approach has been used as a backup method for identifying definitional sentences when the primary method of lexicalosyntactic patterns failed to find a sufficient number of informative nuggets [1].
A similar approach has also been used as a baseline system for TREC 2003 [14].
More recently, Chen et al. [3] adapted a bi-gram or bi-term language model for definitional Question Answering.
Generally, the relevance-based approach requires a "definitional corpus" that contain documents highly relevant to the topic.
The baseline system in TREC 2003 simply uses the topic words as its definitional corpus.
Blair-Goldensohn et al. [1] uses a machine learner to include in the definitonal corpus sentences that are likely to be definitional.
Chen et al. [3] collect snippets from Google to build its definitional corpus.
From the definitional corpus, a definitional centroid vector is built or a set of centroid words are selected.
This centroid vector or set of centroid words is taken to be highly indicative of the topic.
Systems can then use this centroid to identify definitional answers by using a variety of distance metrics to compare against sentences found in the set of retrieved documents for the topic.
BlairGoldensohn et al. [1] uses Cosine similarity to rank sentences by "centrality".
Chen et al. [3] builds a bigram language model using the 350 most frequently occurring google snippet terms, described in their paper as an ordered centroid, to estimate the probability that a sentence is similar to the ordered centroid.
As described here, the relevance-based approach is highly specific to individual topics due to its dependence on a topic specific definitional corpus.
However if individual sentences are viewed as a document, then relevance-based approaches essentially use the collected topic specific centroid words as a form of document retrieval with automated query expansion to identify strongly relevant sentences.
Thus such methods identify relevant sentences and not sentences containing definitional nuggets.
Yet, the TREC 2003 baseline system [14] outperformed all but one other system.
The bi-term language model [3] is able to report results that are highly competitive to state-of-the-art results using this retrieval-based approach.
At TREC 2006, a simple weighted sum of all terms model with terms weighted using solely Google snippets outperformed all other systems by a significant margin [7].
We believe that interesting nuggets often come in the form of trivia, novel or rare facts about the topic that tend to strongly cooccur with direct mention of topic keywords.
This may explain why relevance-based method can perform competitively in definitional question answering.
However, simply comparing against a single centroid vector or set of centroid words may have over emphasized topic relevance and has only identified interesting definitional nuggets in an indirect manner.
Still, relevance based retrieval methods can be used as a starting point in identifying interesting nuggets.
We will describe how we expand upon such methods to identify interesting nuggets in the next section.
3.
HUMAN INTEREST MODEL
Getting a computer system to identify sentences that a human reader would find interesting is a tall order.
However, there are many documents on the world wide web that are contain concise, human written summaries on just about any topic.
What's more, these documents are written explicitly for human beings and will contain information about the topic that most human readers would be interested in.
Assuming we can identify such relevant documents on the web, we can leverage them to assist in identifying definitional answers to such topics.
We can take the assumption that most sentences found within these web documents will contain interesting facets about the topic at hand.
This greatly simplifies the problem to that of finding within the AQUAINT corpus sentences similar to those found in web documents.
This approach has been successfully used in several factoid and list Question Answering systems [11] and we feel the use of such an approach for definitional or "Other" question answering is justified.
Identifying interesting nuggets requires computing machinery to understand world knowledge and human insight.
This is still a very challenging task and the use of human written documents dramatically simplifies the complexity of the task.
In this paper, we report on such an approach by experimenting with a simple word-level edit distance based weighted term comparison algorithm.
We use the edit distance algorithm to score the similarity of a pair of sentences, with one sentence coming from web resources and the other sentence selected from the AQUAINT corpus.
Through a series of experiments, we will show that even such a simple approach can be very effective at definitional question answering.
3.1 Web Resources
There exists on the internet articles on just about any topic a human can think of.
What's more, many such articles are centrally located on several prominent websites, making them an easily accessible source of world knowledge.
For our work on identifying interesting nuggets, we focused on finding short one or two page articles on the internet that are highly relevant to our desired topic.
Such articles are useful as they contain concise information about the topic.
More importantly, the articles are written by humans, for human readers and thus contain the critical human world knowledge that a computer system currently is unable to capture.
We leverage this world knowledge by collecting articles for each topic from the following external resources to build our "Interest Corpus" for each topic.
Wikipedia is a Web-based, free-content encyclopedia written collaboratively by volunteers.
This resource has been used by many Question Answering system as a source of knowledge about each topic.
We use a snapshot of Wikipedia taken in March 2006 and include the most relevant article in the Interest Corpus.
NewsLibrary is a searchable archive of news articles from over 100 different newspaper agencies.
For each topic, we download the 50 most relevant articles and include the title and first paragraph of each article in the Interest Corpus.
Google Snippets are retrieved by issuing the topic as a query to the Google search engine.
From the search results, we extracted the top 100 snippets.
While Google snippets are not articles, we find that they provide a wide coverage of authorative information about most topics.
Due to their comprehensive coverage of a wide variety of topics, the above resources form the bulk of our Interest Corpus.
We also extracted documents from other resources.
However, as these resources are more specific in nature, we do not always get any single relevant document.
These resources are listed below.
Biography.com is the website for the Biography television cable channel.
The channel's website contains searchable biographies on over 25,000 notable people.
If the topic is a person and we can find a relevant biography on the person, we include it it in our Interest Corpus.
Bartleby.com contains a searchable copy of several resources including the Columbia Encyclopedia, the World Factbook, and several English dictionaries.
s9.com is a biography dictionary on over 33,000 notable people.
Like Biography.com, we include the most relevant biography we can find in the Interest Corpus.
Google Definitions Google search engine offers a feature called "Definitions" that provides the definition for a query, if it has one.
We use this feature and extract whatever definitions the Google search engine has found for each topic into the Interest Corpus.
Figure 1: Human Interest Model Architecture.
WordNet WordNet is an well-known electronic semantic lexicon for the English language.
Besides grouping English words into sets of synonyms called synsets, it also provide a short definition on the meaning of words found in each synset.
We add this short definition, if there is one, into our Interest Corpus.
We have two major uses for this topic specific Interest Corpus, as a source of sentences containing interesting nuggets and as a unigram language model of topic terms, I.
3.2 Multiple Interesting Centroids
We have seen that interesting nuggets are highly specific to a topic.
Relevance-based approaches such as the bigram language model used by Chen et al. [3] are focused on identifying highly relevant sentences and pick up definitional answer nuggets as an indirect consequence.
We believe that the use of only a single collection of centroid words has over-emphasized topic relevance and choose instead to use multiple "centroids".
Since sentences in the Interest Corpus of articles we collected from the internet are likely to contain nuggets that are of interest to human readers, we can essentially use each sentence as "pseudocentroids".
Each sentence in the Interest Corpus essentially raises a different aspect of the topic for consideration as a sentence of interest to human readers.
By performing a pairwise sentence comparison between sentences in the Interest Corpus and candidate sentences retrieved from the AQUAINT corpus, we increase the number of sentence comparisons from O (n) to O (nm).
Here, n is the number of potential candidate sentences and m is the number of sentences in the Interest Corpus.
In return, we obtain a diverse ranked list of answers that are individually similar to various sentences found in the topic's Interest Corpus.
An answer can only be highly ranked if it is strongly similar to a sentence in the Interest Corpus, and is also strongly relevant to the topic.
3.3 Implementation
Figure 1 shows the system architecture for the proposed Human Interest-based definitional QA system.
The AQUAINT Retrieval module shown in Figure 1 reuses a document retrieval module of a current Factoid and List Question Answering system we have implemented.
Given a set of words describing the topic, the AQUAINT Retrieval module does query expansion using Google and searches an index of AQUAINT documents to retrieve the 800 most relevant documents for consideration.
The Web Retrieval module on the other hand, searches the online
resources described in Section 3.1 for "interesting" documents in order to populate the Interest Corpus.
The HIM Ranker, or Human Interest Model Ranking module, is the implementation of what is described in this paper.
The module first builds the unigram language model, I, from the collected web documents.
This language model will be used to weight the importance of terms within sentences.
Next, a sentence chunker is used to segment all 800 retrieved documents into individual sentences.
Each of these sentences can be a potential answer sentence that will be independently ranked by interestingness.
We rank sentences by interestingness using sentences from both the Interest Corpus of external documents as well as the unigram language model we built earlier which we use to weight terms.
A candidate sentence in our top 800 relevant AQUAINT documents is considered interesting if it is highly similar in content to a sentence found in our collection of external web-documents.
To achieve this, we perform a pairwise similarity comparison between a candidate sentence and sentences in our external documents using a weighted-term edit distance algorithm.
Term weights are used to adjust the relative importance of each unique term found in the Interest Corpus.
When both sentences share the same term, the similarity score is incremented by the two times the term's weight and every dissimilar term decrements the similarity score by the dissimilar term's weight.
We choose the highest achieved similarity score for a candidate sentence as the Human Interest Model score for the candidate sentence.
In this manner, every candidate sentence is ranked by interestingness.
Finally, to obtain the answer set, we select the top 12 highest ranked and non redundant sentences as definitional answers for the topic.
4.
INITIAL EXPERIMENTS
The Human Interest-based system described in the previous section is designed to identify only interesting nuggets and not informative nuggets.
Thus, it can be described as a handicapped system that only deals with half the problem in definitional question answering.
This is done in order to explore how interestingness plays a factor in definitional answers.
In order to compare and contrast the differences between informative and interesting nuggets, we also implemented the soft-pattern bigram model proposed by Cui et al. [4, 11].
In order to ensure comparable results, both systems are provided identical input data.
Since both system require the use of external resources, they are both provided the same web articles retrieved by our Web Retrieval module.
Both systems also rank the same same set of candidate sentences in the form of 800 most relevant documents as retrieved by our AQUAINT Retrieval module.
For the experiments, we used the TREC 2004 question set to tune any system parameters and use the TREC 2005 question sets to test the both systems.
Both systems are evaluated the results using the standard scoring methodology for TREC definitions.
TREC provides a list of vital and okay nuggets for each question topic.
Every question is scored on nugget recall (NR) and nugget precision (NP) and a single final score is computed using F-Measure (see equation 1) with β = 3 to emphasize nugget recall.
Here, NR is the number of vital nuggets returned divided by total number of vital nuggets while NP is computed using a minimum allowed character length function defined in [12].
The evaluation is automatically conducted using Pourpre v1 .0 c [10].
Table 1: Performance on TREC 2005 Question Set
Figure 2: Performance by entity types.
4.1 Informativeness vs Interestingness
Our first experiment compares the performance of solely identifying interesting nuggets against solely identifying informative nuggets.
We compare the results attained by the Human Interest Model that only identify interesting nuggets with the results of the syntactic pattern finding Soft-Pattern model as well as the result of the top performing definitional system in TREC 2005 [13].
Table 1 shows the F3 score the three systems for the TREC 2005 question set.
The Human Interest Model clearly outperform both soft pattern and the best TREC 2005 system with a F3 score of 0.303.
The result is also comparable with the result of a human manual run, which attained a F3 score of 0.299 on the same question set [9].
This result is confirmation that interesting nuggets does indeed play a significant role in picking up definitional answers, and may be more vital than using information finding lexical patterns.
In order to get a better perspective of how well the Human Interest Model performs for different types of topics, we manually divided the TREC 2005 topics into four broad categories of PERSON, ORGANIZATION, THING and EVENT as listed in Table 3.
These categories conform to TREC's general division of question topics into 4 main entity types [13].
The performance of Human Interest Model and Soft Pattern Bigram Model for each entity type can be seen in Figure 2.
Both systems exhibit consistent behavior across entity types, with the best performance coming from PERSON and ORGANIZATION topics and the worst performance from THING and EVENT topics.
This can mainly be attributed to our selection of web-based resources for the definitional corpus used by both system.
In general, it is harder to locate a single web article that describes an event or a general object.
However given the same set of web-based information, the Human Interest Model consistently outperforms the soft-pattern model for all four entity types.
This suggests that the Human Interest Model is better able to leverage the information found in web resources to identify definitional answers.
5.
REFINEMENTS
Encouraged by the initial experimental results, we explored two further optimization of the basic algorithm.
5.1 Weighting Interesting Terms
The word trivia refer to tidbits of unimportant or uncommon information.
As we have noted, interesting nuggets often has a trivialike quality that makes them of interest to human beings.
From this description of interesting nuggets and trivia, we hypothesize that interesting nuggets are likely to occur rarely in a text corpora.
There is a possibility that some low-frequency terms may actually be important in identifying interesting nuggets.
A standard unigram language model would not capture these low-frequency terms as important terms.
To explore this possibility, we experimented with three different term weighting schemes that can provide more weight to certain low-frequency terms.
The weighting schemes we considered include commonly used TFIDF, as well as information theoretic Kullback-Leiber divergence and Jensen-Shannon divergence [8].
TFIDF, or Term Frequency x Inverse Document Frequency, is a standard Information Retrieval weighting scheme that balances the importance of a term in a document and in a corpus.
For our experiments, we compute the weight of each term as tf x log (ntN), where tf is the term frequency, nt is the number of sentences in the Interest Corpus having the term and N is the total number of sentences in the Interest Corpus.
Kullback-Leibler Divergence (Equation 2) is also called KL Divergence or relative entropy, can be viewed as measuring the dissimilarity between two probability distributions.
Here, we treat the AQUAINT corpus as a unigram language model of general English [15], A, and the Interest Corpus as a unigram language model consisting of topic specific terms and general English terms, I. General English words are likely to have similar distributions in both language models I and A. Thus using KL Divergence as a term weighting scheme will cause strong weights to be given to topicspecific terms because their distribution in the Interest Corpus they occur significantly more often or less often than in general English.
In this way, high frequency centroid terms as well as low frequency rare but topic-specific terms are both identified and highly weighted using KL Divergence.
Due to the power law distribution of terms in natural language, there are only a small number of very frequent terms and a large number of rare terms in both I and A.
While the common terms in English consist of stop words, the common terms in the topic specific corpus, I, consist of both stop words and relevant topic words.
These high frequency topic specific words occur very much more frequently in I than in A.
As a result, we found that KL Divergence has a bias towards highly frequent topic terms as we are measuring direct dissimilarity against a model of general English where such topic terms are very rare.
For this reason, we explored another divergence measure as a possible term weighting scheme.
Jensen-Shannon Divergence or JS Divergence extends upon KL Divergence as seen in Equation 3.
As with KL Divergence, we also use JS divergence to measure the dissimilarity between our two language models, I and A.
Figure 3: Performance by various term weighting schemes on the Human Interest Model.
However, JS Divergence has additional properties1 of being symmetric and non-negative as seen in Equation 4.
The symmetric property gives a more balanced measure of dissimilarity and avoids the bias that KL divergence has.
We conducted another experiment, substituting the unigram languge model weighting scheme we used in the initial experiments with the three term weighting schemes described above.
As lower bound reference, we included a term weighting scheme consisting of a constant 1 for all terms.
Figure 3 show the result of applying the five different term weighting schemes on the Human Interest Model.
TFIDF performed the worst as we had anticipated.
The reason is that most terms only appear once within each sentence, resulting in a term frequency of 1 for most terms.
This causes the IDF component to be the main factor in scoring sentences.
As we are computing the Inverse Document Frequency for terms in the Interest Corpus collected from web resources, IDF heavily down-weights highly frequency topic terms and relevant terms.
This results in TFIDF favoring all low frequency terms over high frequency terms in the Interest Corpus.
Despite this, the TFIDF weighting scheme only scored a slight 0.0085 lower than our lower bound reference of constant weights.
We view this as a positive indication that low frequency terms can indeed be useful in finding interesting nuggets.
Both KL and JS divergence performed marginally better than the uniform language model probabilistic scheme that we used in our initial experiments.
From inspection of the weighted list of terms, we observed that while low frequency relevant terms were boosted in strength, high frequency relevant terms still dominate the top of the weighted term list.
Only a handful of low frequency terms were weighted as strongly as topic keywords and combined with their low frequency, may have limited the impact of re-weighting such terms.
However we feel that despite this, Jensen-Shannon divergence does provide a small but measurable increase in the performance of our Human Interest Model.
1JS divergence also has the property of being bounded, allowing the results to be treated as a probability if required.
However, the bounded property is not required here as we are only treating the divergence computed by JS divergence as term weights
5.2 Selecting Web Resources
In one of our initial experiments, we observed that the quality of web resources included in the Interest Corpus may have a direct impact on the results we obtain.
We wanted to determine what impact the choice of web resources have on the performance of our Human Interest Model.
For this reason, we split our collection of web resources into four major groups listed here: N - News: Title and first paragraph of the top 50 most relevant articles found in NewsLibrary.
W - Wikipedia: Text from the most relevant article found in Wikipedia.
S - Snippets: Snippets extracted from the top 100 most relevant links after querying Google.
M - Miscellaneous sources: Combination of content (when available) from secondary sources including biography.com, s9.com, bartleby.com articles, Google definitions and WordNet definitions.
We conducted a gamut of runs on the TREC 2005 question set using all possible combinations of the above four groups of web resources to identify the best possible combination.
All runs were conducted on Human Interest Model using JS divergence as term weighting scheme.
The runs were sorted in descending F3-Score and the top 3 best performing runs for each entity class are listed in Table 2 together with earlier reported F3-scores from Figure 2 as a baseline reference.
A consistent trend can be observed for each entity class.
For PERSON and EVENT topics, NewsLibrary articles are the main source of interesting nuggets with Google snippets and miscellaneous articles offering additional supporting evidence.
This seem intuitive for events as newspapers predominantly focus on reporting breaking newsworthy events and are thus excellent sources of interesting nuggets.
We had expected Wikipedia rather than news articles to be a better source of interesting facts about people and were surprised to discover that news articles outperformed Wikipedia.
We believe that the reason is because the people selected as topics thus far have been celebrities or well known public figures.
Human readers are likely to be interested in news events that spotlight these personalities.
Conversely for ORGANIZATION and THING topics, the best source of interesting nuggets come from Wikipedia's most relevant article on the topic with Google snippets again providing additional information for organizations.
With an oracle that can classify topics by entity class with 100% accuracy and by using the best web resources for each entity class as shown in Table 2, we can attain a F3-Score of 0.3158.
6.
UNIFYING INFORMATIVENESS WITH INTERESTINGNESS
We have thus far been comparing the Human Interest Model against the Soft-Pattern model in order to understand the differences between interesting and informative nuggets.
However from the perspective of a human reader, both informative and interesting nuggets are useful and definitional.
Informative nuggets present a general overview of the topic while interesting nuggets give readers added depth and insight by providing novel and unique aspects about the topic.
We believe that a good definitional question answering system should provide the reader with a combined mixture of both nugget types as a definitional answer set.
Table 2: Top 3 runs using different web resources for each entity class
We now have two very different "experts" at identifying definitions.
The Soft Pattern Bigram Model proposed by Cui et al. is an expert in identifying informative nuggets.
The Human Interest Model we have described in this paper on the other hand is an expert in finding interesting nuggets.
We had initially hoped to unify the two separate definitional question answering systems by applying an ensemble learning method [5] such as voting or boosting in order to attain a good mixture of informative and interesting nuggets in our answer set.
However, none of the ensemble learning methods we attempted could outperform our Human Interest Model.
The reason is that both systems are picking up very different sentences as definitional answers.
In essence, our two experts are disagreeing on which sentences are definitional.
In the top 10 sentences from both systems, only 4.4% of these sentences appeared in both answer sets.
The remaining answers were completely different.
Even when we examined the top 500 sentences generated by both systems, the agreement rate was still an extremely low 5.3%.
Yet, despite the low agreement rate between both systems, each individual system is still able to attain a relatively high F3 score.
There is a distinct possibility that each system may be selecting different sentences with different syntactic structures but actually have the same or similar semantic content.
This could result in both systems having the same nuggets marked as correct even though the source answer sentences are structurally different.
Unfortunately, we are unable to automatically verify this as the evaluation software we are using does not report correctly identified answer nuggets.
To verify if both systems are selecting the same answer nuggets, we randomly selected a subset of 10 topics from the TREC 2005 question set and manually identified correct answer nuggets (as defined by TREC accessors) from both systems.
When we compared the answer nuggets found by both system for this subset of topics, we found that the nugget agreement rate between both systems was 16.6%.
While the nugget agreement rate is higher than the sentence agreement rate, both systems are generally still picking up different answer nuggets.
We view this as further indication that definitions are indeed made up of a mixture of informative and interesting nuggets.
It is also indication that in general, interesting and informative nuggets are quite different in nature.
There are thus rational reasons and practical motivation in unifying answers from both the pattern based and corpus based approaches.
However, the differences between the two systems also cause issues when we attempt to combine both answer sets.
Currently, the best approach we found for combining both answer sets is to merge and re-rank both answer sets with boosting agreements.
We first normalize the top 1,000 ranked sentences from each system, to obtain the Normalized Human Interest Model score, him (s), and the Normalized Soft Pattern Bigram Model score,
sp (s), for every unique sentence, s. For each sentence, the two separate scores for are then unified into a single score using Equation 5.
When only one system believes that the sentence is definitional, we simply retain that system's normalized score as the unified score.
When both systems agree agree that the sentence is definitional, the sentence's score is boosted by the degree of agreement between between both systems.
In order to maintain a diverse set of answers as well as to ensure that similar sentences are not given similar ranking, we further re-rank our combined list of answers using Maximal Marginal Relevance or MMR [2].
Using the approach described here, we achieve a F3 score of 0.3081.
This score is equivalent to the initial Human Interest Model score of 0.3031 but fails to outperform the optimized Human Interest Model model.
7.
CONCLUSION
This paper has presented a novel perspective for answering definitional questions through the identification of interesting nuggets.
Interesting nuggets are uncommon pieces of information about the topic that can evoke a human reader's curiosity.
The notion of an" average human reader" is an important consideration in our approach.
This is very different from the lexico-syntactic pattern approach where the context of a human reader is not even considered when finding answers for definitional question answering.
Using this perspective, we have shown that using a combination of a carefully selected external corpus, matching against multiple centroids and taking into consideration rare but highly topic specific terms, we can build a definitional question answering module that is more focused on identifying nuggets that are of interest to human beings.
Experimental results has shown this approach can significantly outperform state-of-the-art definitional question answering systems.
We further showed that at least two different types of answer nuggets are required to form a more thorough set of definitional answers.
What seems to be a good set of definition answers is some general information that provides a quick informative overview mixed together with some novel or interesting aspects about the topic.
Thus we feel that a good definitional question answering system would need to pick up both informative and interesting nugget types in order to provide a complete definitional coverage on all important aspects of the topic.
While we have attempted to build such a system by combining our proposed Human Interest Model with Cui et al.'s Soft Pattern Bigram Model, the inherent differences between both types of nuggets seemingly caused by the low agreement rates between both models have made this a difficult task.
Indeed, this is natural as the two models have been designed to identify two very different types of definition answers using very different types of features.
As a result, we are currently only able to achieve a hybrid system that has the same level of performance as our proposed Human Interest Model.
We approached the problem of definitional question answering from a novel perspective, with the notion that interest factor plays a role in identifying definitional answers.
Although the methods we used are simple, they have been shown experimentally to be effective.
Our approach may also provide some insight into a few anomalies in past definitional question answering's trials.
For instance, the top definitional system at the recent TREC 2006 evaluation was able to significantly outperform all other systems using relatively simple unigram probabilities extracted from Google snippets.
We suspect the main contributor to the system's performance
Table 3: TREC 2005 Topics Grouped by Entity Type
is Google's PageRank algorithm, which mainly consider the number of linkages, has an indirect effect of ranking web documents by the degree of human interest.
In our future work, we seek to further improve on the combined system by incorporating more evidence in support of correct definitional answers or to filter away obviously wrong answers.