Word Substitution in Short Answer Extraction: A W ord N et-based Approach (2016)

Abstract: We describe the implementation of a short answer extraction system. It consists of a simple sentence selection front-end and a two phase approach to answer extraction from a sentence. In the first phase sentence classification is performed with a classifier trained with the passive aggressive algorithm utilizing the UIUC dataset and taxonomy and a feature set including word vectors. This phase outperforms the current best published results on that dataset. In the second phase, a sieve algorithm consisting of a series of increasingly general extraction rules is applied, using WordNet to find word types aligned with the UIUC classifications determined in the first phase. Some very preliminary performance metrics are presented.

## introduction short answer extraction refers to a set of information retrieval techniques that retrieve a short answer to a question from a sentence. for example, if we have the following question and answer sentence (1) q: who was the first president of the united states? a: george washington was the first president of the united states. we can use resnik's method @xcite to compute similarity. in particular we can use ted @xcite implementation @xcite , which gives the result of walk#n#4 amble#n#1 9.97400037941652 . word2vec @xcite using their 300-dimensional vectors trained on google news, also gives a relatively high similarity score for the two @xmath0.similarity('walk', 'amble') 0.525 ## is similarity the right measure? but what about if we have (4) "bob has an apple." (5) "who has a pear?" we find that this pair is even more similar than "walk" and "amble" > model.similarity('apple', 'pear') 0.645 and from @xcite concept #1: apple concept #2: pear apple pear apple#n#1 pear#n#1 10.15 and yet clearly 4 is not a valid answer to 5. one possibility is that synset subsumption as a measure of word substitution @xcite 1 2 may be the appropriate metric, rather than word similarity. ## question answering our approach starts with the user's question and the sentence that is most likely to contain the answer, which is selected with the bm25 algorithm @xcite . then we identify the incoming question as a particular question type according to the uiuc taxonomy foot_2 . to this taxonomy we have added the yes/no question type. then we pass the sentence and the question to a class written specifically to handle a particular uiuc question type. generally, all the base question types behave differently from one another. within a base question type, subtypes may be handled generically or with code specially targeted for that subtype. for this paper, we first discuss the approach to question classification, and then to answer extraction with a focus on the question subtypes that are amenable to a wordnet-based approach. ## question classification this section presents a question classifier with several novel semantic and syntactic features based on extraction of question foci. we use several sources of semantic information for representing features for each question focus. our model uses a simple margin-based online algorithm. we achieve state-of-the-art performance on both finegrained and coarse-grained question classification. as the focus of this paper is on wordnet, we leave many details to a future paper and primarily report the features used, the learning algorithm and results, without further justification ## answer extraction in this section we discuss techniques for short answer extraction once questions have been classified into a particular uiuc type. we employ a "sieve" approach, as in @xcite , that has seen some success in tasks like coreference resolution and is creating a bit of a renaissance in rule-based, as opposed to machine learning, approaches in nlp. we provide in this paper one example of how instead of taking an either/or approach, both methods can be combined into a high performance system. we focus below on the sieves that are specific to question types where we have been able to profitably employ wordnet for finding the right short answer. preliminary results have been positive employing this approach. we have two strategies that are used across the base question types: employing semantic role labels and recognizing appositives. the third test set is trec-8 @xcite . ## uiuc question types and synsets ## conclusion using a wordnet-based word replacement method appears to be better for question answering than using word similarity metrics. in preliminary tests 10 questions in a portion of our corpora are answerable with this approach just addressing two of the 22 entity subtypes with wordnet based matching. while more experimentation is needed, the results are intuitive and promising. the current approach should be validated and compared against other approaches on current data sets such as @xcite .