NUT - RC : Noisy User-generated Text-oriented Reading Comprehension (2020)

Abstract: Reading comprehension (RC) on social media such as Twitter is a critical and challenging task due to its noisy, informal, but informative nature. Most existing RC models are developed on formal datasets such as news articles and Wikipedia documents, which severely limit their performances when directly applied to the noisy and informal texts in social media. Moreover, these models only focus on a certain type of RC, extractive or generative, but ignore the integration of them. To well address these challenges, we come up with a noisy user-generated text-oriented RC model. In particular, we first introduce a set of text normalizers to transform the noisy and informal texts to the formal ones. Then, we integrate the extractive and the generative RC model by a multi-task learning mechanism and an answer selection module. Experimental results on TweetQA demonstrate that our NUT-RC model significantly outperforms the state-of-the-art social media-oriented RC models.

## introduction reading comprehension (rc), which aims to answer questions by comprehending the contexts of given passages, is a frontier topic in natural language processing research. recently, many rc models @xcite @xcite @xcite have been proposed and have achieved considerable successes. according to answer prediction methods, the rc models can be roughly divided into two major categories: extractive and generative. for an extractive rc model, the predicted answer is limited to be a consecutive span in the passage. for a generative rc model, it allows the answer to be free-text which can include novel words and phrases not appeared in passages. however, most of the existing rc models are developed for formal text, such as news articles @xcite and wikipedia @xcite , which severely limit their performances on noisy user-generated texts (nut). meanwhile, an increasing number of people are accustomed to getting real-time information via social media like twitter. recently, @xcite propose a large-scale dataset for question answering over social media texts, tweetqa, which is constructed in a crowd-sourcing way, and recommended annotators to write answers in their own language. tweet: @instyle: on kws cover: beautiful statement. thank u 4 opening this convo. its an important 1 that needs to be had.-kerry washington (@kerrywashington) february 5, 2015 q: what is kerry thankful for? a: instyle opening the conversation to address the above challenges, we come up with a noisy user-generated text-oriented reading comprehension model (nut-rc), which takes advantage of both the existing extractive and generative rc models for formal texts, and extends to the informal texts by text normalization. in particular, we first convert the noisy user-generated texts in social media to the formal texts via text normalization, which can benefit to improving the theoretical upper bound of performance of the rc models. second, in tweetqa, many answers are with the free-text format. for example, the answer in finally, an answer selection module is used to choose the more appropriate answer from the extractive rc model or the generative rc model. the experiments are carried out on the tweetqa dataset. the performances of our nut-rc model achieve to 76.1%, 72.1%, and 77.9% of bleu-1, meteor, rouge-l, respectively, yielding absolute improvements of 14.7% on bleu-1, 13.5% on meteor, and 13.8% on rouge-l over the baseline. note that the performance is also exceeding the human on both meteor and rouge-l metrics. our contributions in this paper are three-folds: (1) proposal of a novel noisy user-generated textoriented rc model which combines the extractive and the generative rc models by an answer selection model; (2) proposal of regarding the extractive rc as an auxiliary task to optimize the generative rc model by multi-task learning; and (3) empirical verification of the effectiveness of the model and achieving the state-of-the-art performance on tweetqa. ## related work reading comprehension reading comprehension (rc) aims to teach a machine answering questions by comprehending the context of given passages, which is one of the most important tasks in the nlp community automatically. the prior research on rc mainly focuses on either the cloze-style @xcite @xcite or multiple-choice @xcite . however, these models are difficult to be directly utilized for real application scenarios. recently, many large-scale rc datasets constructed by a crowdsourced way @xcite @xcite have been proposed and received widespread attentions @xcite . besides, to make rc models have the ability of conversation understanding like a human being, more challenging multi-round conversational rc datasets are proposed, such as quac @xcite and coqa @xcite . after transformer @xcite has been released, various pre-trained models @xcite @xcite have sprung up and achieved promising results on most of the rc datasets through purely fine-tuning. the parameters of our nut-rc model are also initialized by bert @xcite . according to the original formats of the predicted answers, the existing rc model can be roughly divided into two major categories: extractive and generative. in an extractive rc model, the answers are limited to be a span of the given passages. most of the mainstream rc models are the extractive rc model. in a generative rc model, the answers can be free-text and do not have to appear in the passages. normalizer informal type example splt mixed tokens #intheunlikelyevent (# in the unlikely event) expn abbreviation u (you), convo (conversation), addr (address) wdlk misspelling word preety (pretty), goverment (government) misc other sh*t (•), :-) (•) @xcite ; @xcite built a dependency parser for tweets based on the tweebank, which is the first dataset annotated with syntactic information on tweets. besides, the acl-ijcnlp workshop @xcite provides a shared task on noisy user-generated text processing, including twitter lexical normalization and named entity recognition, which stimulated many studies on this topic @xcite @xcite . recently, @xcite released the first large-scale rc dataset over social media data, tweetqa, which gathered tweets used in news articles and encouraged human annotators to write questions and answers upon these tweets in their language. to the best of our knowledge, there are rare reading comprehension models over social media, and the proposed nut-rc model is inspired by tweetqa. figure 1 illustrates the architecture of the noisy user-generated text-oriented reading comprehension model (nut-rc), which comprises six basic components: 1) text normalization to convert raw noisy and informal tweets to formal texts (subsection 3.1), 2) lexicon encoder to map the input into a sequence of input embedding vectors (subsection 3.2), 3) transformer with self-attention masks to map the input embeddings into contextual hidden representations (subsection 3.3), 4) extractive rc model identify the start and end positions of an answer in a tweet (subsection 3.4), 5) generative rc model with multi-task learning to generate an answer from the vocabulary conditioned on the question and the tweet (subsection 3.5), and 6) answer selection to choose the most appropriate answer from the extractive rc model and the generative rc model (subsection 3.6). ## conclusion in this paper, we come up with a noisy user-generated text-oriented rc model on social media texts. through text normalizers, the model transforms informal texts into well-formed texts. moreover, we train the generative rc model with the help of the extractive model by a multi-task learning mechanism. finally, we apply an answer selection module to choose the more appropriate answer from the generative and the extractive rc models. empirical results show that the proposed model significantly outperforms the state-of-the-art baselines on the tweetqa dataset. in future work, we plan to justify the proposed model on other free-form qa datasets. the source code of our nut-rc model is publicly available at https://github.com/whalefallzz/nut_rc .