This work describes an answer ranking engine for non-factoid questions built using a large online community-generated question-answer collection ( Yahoo ! Answers ) . We show how such collections may be used to effectively set up large supervised learning experiments . Furthermore we investigate a wide range of feature types , some exploiting NLP processors , and demonstrate that using them in combination leads to considerable improvements in accuracy . The problem of Question Answering ( QA ) has received considerable attention in the past few years . Nevertheless , most of the work has focused on the task of factoid QA , where questions match short answers , usually in the form of named or numerical entities . Thanks to international evaluations organized by conferences such as the Text REtrieval Conference ( TREC)1 or the Cross Language Evaluation Forum ( CLEF ) Workshop2 , annotated corpora of questions and answers have become available for several languages , which has facilitated the development of robust machine learning models for the task . The situation is different once one moves beyond the task of factoid QA . Comparatively little research has focused on QA models for non-factoid questions such as causation , manner , or reason questions . Because virtually no training data is available for this problem , most automated systems train either on small hand-annotated corpora built in house ( Higashinaka and Isozaki , 2008 ) or on question-answer pairs harvested from Frequently Asked Questions ( FAQ ) lists or similar resources ( Soricut and Brill , 2006 ) . None of these situations is ideal : the cost of building the training corpus in the former setup is high ; in the latter scenario the data tends to be domain-specific , hence unsuitable for the learning of open-domain models . On the other hand , recent years have seen an explosion of user-generated content ( or social media ) . Of particular interest in our context are communitydriven question-answering sites , such as Yahoo ! Answers 3 , where users answer questions posed by other users and best answers are selected manually either by the asker or by all the participants in the thread . The data generated by these sites has significant advantages over other web resources : ( a ) it has a high growth rate and it is already abundant ; ( b ) it covers a large number of topics , hence it offers a better approximation of open-domain content ; and ( c ) it is available for many languages . Community QA sites , similar to FAQs , provide large number of questionanswer pairs . Nevertheless , this data has a significant drawback : it has high variance of quality , i.e. , answers range from very informative to completely irrelevant or even abusive . Table 1 shows some examples of both high and low quality content . In this paper we address the problem of answer ranking for non-factoid questions from social media content . Our research objectives focus on answering the following two questions : 1 . Is it possible to learn an answer ranking model for complex questions from such noisy data ? This is an interesting question because a positive answer indicates that a plethora of training data is readily available to QA researchers and system developers . 2 . Which features are most useful in this scenario ? Are similarity models as effective as models that learn question-to-answer transformations ? Does syntactic and semantic information help ? For generality , we focus only on textual features extracted from the answer text and we ignore all meta data information that is not generally available . Notice that we concentrate on one component of a possible social-media QA system . In addition to answer ranking , a complete system would have to search for similar questions already answered ( Jeon et al . , 2005 ) , and rank content quality using " social " features such as the authority of users ( Jeon et al . , 2006 ; Agichtein et al . , 2008 ) . This is not the focus of our work : here we investigate the problem of learning an answer ranking model capable of dealing with complex questions , using a large number of , possible noisy , question-answer pairs . By focusing exclusively on textual content we increase the portability of our approach to other collections where " social " features might not available , e.g. , Web search . The paper is organized as follows . We describe our approach , including all the features explored for answer modeling , in Section 2 . We introduce the corpus used in our empirical analysis in Section 3 . We detail our experiments and analyze the results in Section 4 . We overview related work in Section 5 and conclude the paper in Section 6 . In this work we described an answer ranking engine for non-factoid questions built using a large community-generated question-answer collection . On one hand , this study shows that we can effectively exploit large amounts of available Web data to do research on NLP for non-factoid QA systems , without any annotation or evaluation cost . This provides an excellent framework for large-scale experimentation with various models that otherwise might be hard to understand or evaluate . On the other hand , we expect the outcome of this process to help several applications , such as open-domain QA on the Web and retrieval from social media . For example , on the Web our ranking system could be combined with a passage retrieval system to form a QA system for complex questions . On social media , our system should be combined with a component that searches for similar questions already answered ; this output can possibly be filtered further by a content-quality module that explores " social " features such as the authority of users , etc . We show that the best ranking performance is obtained when several strategies are combined into a single model . We obtain the best results when similarity models are aggregated with features that model question-to-answer transformations , frequency and density of content , and correlation of QA pairs with external collections . While the features that model question-to-answer transformations provide most benefits , we show that the combination is crucial for improvement . Lastly , we show that syntactic dependency parsing and coarse semantic disambiguation yield a small , yet statistically significant performance increase on top of the traditional bag-of-words and n-gram representation . We obtain these results using only off-the-shelf NLP processors that were not adapted in any way for our task .

Challenge: Existing works on question answering focus on factoid tasks leaving non-factoid questions untouched because of a lack of training data which is expensive to obtain.
Approach: They propose an answer ranking engine for non-factoid questions using a large online community-generated question-answer collection which covers a large number of domains and languages.
Outcome: They show that the proposed model can exploit large Web data without additional cost and the best performing model is a combination of several strategies.