Answer Quality Aware Aggregation for Extractive QA Crowdsourcing (2022)

Abstract: Quality control is essential for creating extractive question answering (EQA) datasets via crowdsourcing. Aggregation across answers, i.e. word spans within passages annotated, by different crowd workers is one major focus for ensuring its quality. However, crowd workers cannot reach a consensus on a considerable portion of questions. We introduce a simple yet effective answer aggregation method that takes into account the relations among the answer, question, and context passage. We evaluate answer quality from both the view of question answering model to determine how confident the QA model is about each answer and the view of the answer verification model to determine whether the answer is correct. Then we compute aggregation scores with each answer’s quality and its contextual embedding produced by pre-trained language models. The experiments on a large real crowdsourced EQA dataset show that our framework outperforms baselines by around 16% on precision and effectively conduct answer aggregation for extractive QA task.

## introduction extractive question answering (eqa) is a fundamental task in natural language processing @xcite . with access to large-scale datasets, deep neural models have achieved significant advances in the eqa task @xcite @xcite . creating large-scale high-quality datasets is one of the essential factors driving progress @xcite . currently, a prevalent method for creating eqa datasets is crowdsourcing @xcite @xcite @xcite thanks to its efficiency and scalability due to the availability of crowd workers. yet, answers collected from crowd workers often washington (cnn) --during the presidential campaign, then-candidate barack obama said that he hoped his administration wouldn't get [...]ssue. former republican speaker of the house newt gingrich called sotomayor a racist. conservative talk [...] a better conclusion than a white male who hasn't lived that life." one top gop senator said he wants more than an explanation. "i think she should apologize, but i don't believe any american wants a judge on the bench that's going to use empathy or their background to punish someon. "she's been called the equivalent of the head of the ku klux klan by rush limbaugh; [...] yor's appellate court decision against a mostly white group of firefighters who say they were discriminated against after a promotion test was thrown out, because critics said it discriminated against minority firefighters. but legal experts have said her full record on race isn't that controversial --in 96 race-related cases decided by sotomayor on the court of appeals, ... ## conclusion in this paper, we propose a novel answer annotation aggregation method for eqa crowdsourcing. we show that without any fine-tuning, our methods can achieve comparable performance with the trained qa and nli model using limited training data. we introduce a novel algorithm for combining the nli model, qa model and contextual text embedding for answer text sequence aggregation. the experiments on a real large-scale crowdsourced eqa dataset show the effectiveness and stability of the proposed method. the proposed methods outperform the baseline single metric method by around 16% absolute improvement on p@1 and 10% improvement on other ranking metrics. for future work, we will further explore methods incorporating crowd worker reliability and question answerability for better answer aggregation. we will also explore the applicability of our approaches to other tasks that deal with collecting extractive texts @xcite .