Determining Question-Answer Plausibility in Crowdsourced Datasets Using Multi-Task Learning (2020)

Abstract: Datasets extracted from social networks and online forums are often prone to the pitfalls of natural language, namely the presence of unstructured and noisy data. In this work, we seek to enable the collection of high-quality question-answer datasets from social media by proposing a novel task for automated quality analysis and data cleaning: question-answer (QA) plausibility. Given a machine or user-generated question and a crowd-sourced response from a social media user, we determine if the question and response are valid; if so, we identify the answer within the free-form response. We design BERT-based models to perform the QA plausibility task, and we evaluate the ability of our models to generate a clean, usable question-answer dataset. Our highest-performing approach consists of a single-task model which determines the plausibility of the question, followed by a multi-task model which evaluates the plausibility of the response as well as extracts answers (Question Plausibility AUROC=0.75, Response Plausibility AUROC=0.78, Answer Extraction F1=0.665).

## introduction large, densely-labeled datasets are a critical requirement for the creation of effective supervised learning models. the pressing need for high quantities of labeled data has led many researchers to collect data from social media platforms and online forums @xcite @xcite . due to the presence of noise and the lack of structure that exist in these data sources, manual quality analysis (usually performed by paid crowdworkers) is necessary to extract structured labels, filter irrelevant examples, standardize language, and perform other preprocessing tasks before the data can be used. however, obtaining dataset annotations in this manner is a time-consuming and expensive process that is often prone to errors. in this work, we develop automated data cleaning and verification mechanisms for extracting high-quality data from social media platforms foot_0 . we specifically focus on the creation of questionanswer datasets, in which each data instance consists of a question about a topic and the corresponding answer. in order to filter noise and improve data quality, we propose the task of question-answer (qa) plausibility, which includes the following three steps: because we assume social media users generally answer questions in good faith (and are posed questions which they can answer), we can assume plausible answers are correct ones @xcite . necessarily, if this property were not satisfied, then any adequate solutions would require the very domain knowledge of interest. therefore, we look to apply this approach toward data with this property. in this study, we demonstrate an application of qa plausibility in the context of visual question answering (vqa), a well-studied problem in the field of computer vision @xcite . we assemble a large vqa dataset with images collected from an image-sharing social network, machinegenerated questions related to the content of the image, and responses from social media users. we then train a multitask bert-based model and evaluate the ability of the model to perform the three subtasks associated with qa plausibility. the methods presented in this work hold potential for reducing the need for manual quality analysis of crowdsourced data as well as enabling the use of questionanswer data from unstructured environments such as social media platforms. ## related work prior studies on the automated labeling task for datasets derived from social media typically focus on the generation of noisy labels; models trained on such datasets often rely on weak supervision to learn relevant patterns. however, approaches for noisy label generation, such as snorkel @xcite and curriculumnet @xcite , often use functions or other heuristics to generate labels. one such example is the sentiment140 dataset, which consists of 1.6 million tweets labeled with corresponding sentiments based on the emojis present in the tweet @xcite . in this case, the presence of just three category labels (positive, neutral, negative) simplifies the labeling task and reduces the effects of incorrect labels on trained models; however, this problem becomes increasingly more complex and difficult to automate as the number of annotation categories increases. previous researchers have studied question relevance by reasoning explicitly about the information available to answer the question. several vqa studies have explicitly extracted premises, or assumptions made by questions, to determine if the original question is relevant to the provided image @xcite . a number of machine comprehension models have been devised to determine the answerability of a question given a passage of text @xcite . in contrast, we are able to leverage the user's freeform response to determine if the original question was valid. our model is also tasked with supporting machine-generated questions, which may be unanswerable and lead to noisy user-generated responses. while the concept of answer plausibility in user responses has also been previously explored, existing approaches use hand-crafted rules and knowledge sources @xcite . by using a learned approach, we give our system the flexibility to adapt with the data and cover a wider variety of cases. ## dataset the dataset consists of questions and responses collected from an image-sharing social media platform. we utilize an automated question-generation bot in order to access public image posts, generate a question based on image features, and record data from users that replied to the question, as shown in figure 1 @xcite . because the question-generation bot was designed to maximize information gain, it generates questions across a wide variety of categories, including objects, attributes, spatial relationships, and activities (among others). for the sake of space, we refer readers to the original paper for more information on the method of question generation and diversity of the resulting questions asked. all users that contributed to the construction of this dataset were informed that they were participating in a research study, and irb approval was obtained for this work. for the privacy of our users, the dataset will not be released at this time. rather than focus on the specific dataset, we wish to instead present a general method for cleaning user-generated datasets and argue its generality even to tasks such as visualquestion-answering. it is important to note that since the questiongeneration process is automated, the question could be unrelated to the image due to bot errors; however, in such situations where the question is deemed to be implausible, the response may still be valid if it accurately refers to the content of the image. if the response is judged to be plausible, the amt crowdworker must then extract the answer span from the user's response. in order to capture the level of detail we required (while discouraging amt crowdworkers from simply copy/pasting the entire response), we set the maximum length of an answer span to be five words for the labeling step. however, the final model itself is not limited to answers of any particular length. for cost reasons, each example was labeled by only one annotator. while we could have averaged labels across annotators, we found that the majority of the labeling errors were due to misunderstandings of the non-standard task, meaning that errors were localized to particular annotators rather than randomly spread across examples. this issue was mitigated by adding a qualifying task and manually reviewing a subset of labels per worker for the final data collection. while one might expect images to be necessary (or at least helpful) for determining question and response plausibility, we found that human annotators were able to determine the validity of the inputs based solely on text without the need for the accompanying image. in our manual analysis of several hundred examples (approximately 5% of the dataset), we found that every example which required the image to label properly could be categorized as a "where" question. when the bot asked questions of the general form "where is the x" or "where was this taken," users assumed our bot had basic visual knowledge and was therefore asking a question not already answered by the image (such as "where is the dog now" or "what part of the world was this photo taken in"). this led to valid responses that did not pertain to image features and were therefore not helpful for training downstream models. our preprocessing steps and annotation procedure resulted in a total of 7200 question-response pairs with answer labels. we use a standard split of 80% of the dataset for training, 10% for validation, and 10% for testing. ## models and experiments model architecture: as shown in figure 3 , we utilized a modified bert model to perform the three sub-tasks associated with qa plausibility. the model accepts a concatenation of the machinegenerated question and user response as input, with the [cls] token inserted at the start of the sentence and the [sep] token inserted to separate the question and response. we evaluated performance on question and response plausibilities by computing accuracy and auc-roc scores. performance on the answer span extraction task was evaluated with f1 scores, which measure overlap between the predicted answer label and the true answer @xcite . ## results our results suggest that multi-task learning is most effective when the tasks are closely related, such as with response plausibility and answer extraction. since the bert architecture is extremely quick for both training and evaluation, we found that the increase in performance afforded by using a single-task model and multi-task model in series combined task question plausibility response plausibility answer extraction acc auroc acc auroc f1 question plausibility (qp) only 65.51% 0.7488 ---response plausibility (rp) only --64.62% 0.7674 -answer extraction only ----0.568 rp and answer extraction --70.13% 0.7870 0.665 qp, rp and answer extraction 63.90% 0.6803 60.91% 0.6881 0.6160 ## discussion deep learning studies are often hindered by lack of access to large datasets with accurate labels. in this paper, we introduced the question-answer plausibility task in an effort to automate the data cleaning process for question-answer datasets collected from social media. we then presented a multitask deep learning model based on bert, which accurately identified the plausibility of machinegenerated questions and user responses as well as extracted structured answer labels. although we specifically focused on the visual question answering problem in this paper, we expect that our results will be useful for other question-answer scenarios, such as in settings where questions are user-generated or images are not available. overall, our approach can help improve the deep learning workflow by processing and cleaning the noisy and unstructured natural language text available on social media platforms. ultimately, our work can enable the generation of large-scale, highquality datasets for artificial intelligence models.