Original Content Is All You Need! an Empirical Study on Leveraging Answer Summary for W iki H ow QA Answer Selection Task (2022)

Abstract: Answer selection task requires finding appropriate answers to questions from informative but crowdsourced candidates. A key factor impeding its solution by current answer selection approaches is the redundancy and lengthiness issues of crowdsourced answers. Recently, Deng et al. (2020) constructed a new dataset, WikiHowQA, which contains a corresponding reference summary for each original lengthy answer. And their experiments show that leveraging the answer summaries helps to attend the essential information in original lengthy answers and improve the answer selection performance under certain circumstances. However, when given a question and a set of long candidate answers, human beings could effortlessly identify the correct answer without the aid of additional answer summaries since the original answers contain all the information volume that answer summaries contain. In addition, pretrained language models have been shown superior or comparable to human beings on many natural language processing tasks. Motivated by those, we design a series of neural models, either pretraining-based or non-pretraining-based, to check wether the additional answer summaries are helpful for ranking the relevancy degrees of question-answer pairs on WikiHowQA dataset. Extensive automated experiments and hand analysis show that the additional answer summaries are not useful for achieving the best performance.

## introduction answer selection task in community question answering (cqa) has been a popular research topic in both academy and industry due to its practical importance. in recent years, neural attention-based approaches for this task can be roughly categorized into two primary types. one type of them @xcite attempts to enhance the interactions of different granularity between question and candidate answer using the widely-adopted compare-aggregate framework @xcite . the another focuses on incorporating additional input information, such as user metadata information @xcite , the subject-body relationship of community questions @xcite , etc. however, real-life cqa datasets that contain open-domain and nonfactoid questions usually come along with long multi-sentence answer texts and noise. as a result, many previous neural answer selection approaches that were primarily designed to retrieve short answers fall short of expectations in such cases @xcite . recently, @xcite propose to leverage answer summaries to tackle the redundancy and lengthiness issues of original answers in long answer selection task. to this end, they created wik-ihowqa, the first large-scale open-domain cqa dataset that contains lengthy answers coupled with its summaries written by community users for nonfactoid questions starting with "how to". instead of relying on crowdsourcing, wikihowqa was generated based on the wikihow summarization dataset @xcite and the online wikihow knowledge base 1 . an example from the dataset is shown in however, though answer summaries are always much shorter and more concise than the answer texts being summarized, they present text information in an abridged form and do not include details. hence, the way that leverages the answer summaries to alleviate the answer redundancy and noise issues may also lead to the neglect of details question how to decorate a school locker ? answer do you want your locker to be calm and relaxing , to relieve you from those stressful classes ? or do you want your locker to be fun and exciting , colorful , or maybe you want it to show some of your rocker spirit. if you do n't want to express anything, pick a theme. a cool theme could be 70 's, giraffe 's, dogs ... (1112 words in total) summary think about the feeling you want to express. acquire all of the materials you will need to make your locker look amazing. hang up some pictures. put a sign on the door ... in this paper, we aim to conduct an in-depth and comprehensive analysis of this dataset and explore whether answer summaries could be helpful for "how-to" answer selection task. we demonstrate that, without the aid of answer summaries, simple, carefully designed lstm-based models and pretraining-based models could obtain high, stateof-the-art map score of 72.91% and 82.74% on the dataset respectively. we carry out a meticulous qualitative analysis on randomly-sampled instances to provide data on their difficulty and quality, and whether the utilization of answer summaries could improve human performance. we conclude that: (i) this answer selection task is relatively easy though it contains long multi-sentence answer texts. (ii) answer summaries do not convey additional information content and are not helpful for boosting both model and human performance. (iii) this dataset is noisy due to its method of data creation. ## the answer selection task the wikihowqa dataset introduced in @xcite is made from an online wiki-style commu-nity website -wikihow foot_1 . the questions contained in the dataset are all non-factoid and start with "how to". for a specific question, its accepted answers are considered as correct, and negative candidates were collected by retrieving the accepted answers to relevant questions. each answer written by community users details multiple steps of doing a procedural task for a specific how-to question. in addition, every candidate answer is associated with a short reference summary. to be specific, the answer selection task could be formally defined as follows: ## methods in this section, we describe two types of methods we implemented. the first class of methods directly models the relevancy degrees of question-answer pairs without using any answer summaries information. the second class of methods is partially inspired by @xcite . they jointly learn answer selection and answer summary generation so as to leverage short answer summary to aid in picking out long multi-sentence answer. different from them, we explicitly exploit the relevancy degrees of question-summary pair to aid in modeling the interaction between question and answer. for each class of methods, we from one side build our model based on bidirectional lstms so as to make a fair comparison with previous approaches and from the other we build our model based on a pretrained language model -albert @xcite to advance the state of the art. ## conclusion in this paper, we carefully study the recent wiki-howqa answer selection task. our models, either lstm-based or albert-based, outperform the previous state-of-the-art method by a large margin. more importantly, we do a careful hand-analysis of a small subset of the dataset. overall, we think the wikihowqa dataset is a valuable dataset, which provides a promising avenue for research on non-factoid, long answer selection task. nevertheless, we argue that: (i) this dataset is still noisy due to its method of data creation. (ii) for "how-to" answer selection task, the additional answer summaries can neither help to improve model performance nor can effect human annotation. (iii) the answer selection task is relatively easy though it contains long multi-sentence answer texts.