pwc-archive/papers-with-abstracts huggingface dataset:
CALOR-QUEST : generating a training corpus for Machine Reading Comprehension models from shallow semantic annotations
Authors: Frederic Bechet, Cindy Aloui, Delphine Charlet, Geraldine Damnati, Johannes Heinecke, Alexis Nasr, Frederic Herledan
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Machine Reading Comprehension, Natural Questions, Question Answering, Reading Comprehension

Machine reading comprehension is a task related to Question-Answering where questions are not generic in scope but are related to a particular document. Recently very large corpora (SQuAD, MS MARCO) containing triplets (document, question, answer) were made available to the scientific community to develop supervised methods based on deep neural networks with promising results. These methods need very large training corpus to be efficient, however such kind of data only exists for English and Chinese at the moment. The aim of this study is the development of such resources for other languages by proposing to generate in a semi-automatic way questions from the semantic Frame analysis of large corpora. The collect of natural questions is reduced to a validation/test set. We applied this method on the CALOR-Frame French corpus to develop the CALOR-QUEST resource presented in this paper.
Answer-Supervised Question Reformulation for Enhancing Conversational Machine Comprehension
Authors: Qian Li, Hui Su, Cheng Niu, Daling Wang, Zekang Li, Shi Feng, Yifei Zhang
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Reading Comprehension, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Sentence

In conversational machine comprehension, it has become one of the research hotspots integrating conversational history information through question reformulation for obtaining better answers. However, the existing question reformulation models are trained only using supervised question labels annotated by annotators without considering any feedback information from answers. In this paper, we propose a novel Answer-Supervised Question Reformulation (ASQR) model for enhancing conversational machine comprehension with reinforcement learning technology. ASQR utilizes a pointer-copy-based question reformulation model as an agent, takes an action to predict the next word, and observes a reward for the whole sentence state after generating the end-of-sequence token. The experimental results on QuAC dataset prove that our ASQR model is more effective in conversational machine comprehension. Moreover, pretraining is essential in reinforcement learning models, so we provide a high-quality annotated dataset for question reformulation by sampling a part of QuAC dataset.
Improving the Robustness of Deep Reading Comprehension Models by Leveraging Syntax Prior
Authors: Bowen Wu, Haoyang Huang, Zongsheng Wang, Qihang Feng, Jingsong Yu, Baoxun Wang
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Machine Reading Comprehension, Reading Comprehension, Sentence

Despite the remarkable progress on Machine Reading Comprehension (MRC) with the help of open-source datasets, recent studies indicate that most of the current MRC systems unfortunately suffer from weak robustness against adversarial samples. To address this issue, we attempt to take sentence syntax as the leverage in the answer predicting process which previously only takes account of phrase-level semantics. Furthermore, to better utilize the sentence syntax and improve the robustness, we propose a Syntactic Leveraging Network, which is designed to deal with adversarial samples by exploiting the syntactic elements of a question. The experiment results indicate that our method is promising for improving the generalization and robustness of MRC models against the influence of adversarial samples, with performance well-maintained.
Towards Answer-unaware Conversational Question Generation
Authors: Mao Nakanishi, Tetsunori Kobayashi, Yoshihiko Hayashi
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Question Generation, Question-Generation

Conversational question generation is a novel area of NLP research which has a range of potential applications. This paper is first to presents a framework for conversational question generation that is unaware of the corresponding answers. To properly generate a question coherent to the grounding text and the current conversation history, the proposed framework first locates the focus of a question in the text passage, and then identifies the question pattern that leads the sequential generation of the words in a question. The experiments using the CoQA dataset demonstrate that the quality of generated questions greatly improves if the question foci and the question patterns are correctly identified. In addition, it was shown that the question foci, even estimated with a reasonable accuracy, could contribute to the quality improvement. These results established that our research direction may be promising, but at the same time revealed that the identification of question patterns is a challenging issue, and it has to be largely refined to achieve a better quality in the end-to-end automatic question generation.
Cross-Task Knowledge Transfer for Query-Based Text Summarization
Authors: Elozino Egonmwan, Vittorio Castelli, Md. Arafat Sultan
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Machine Reading Comprehension, Machine Translation, Reading Comprehension, Sentence, Sentence Compression, Text Summarization, Transfer Learning, Translation

We demonstrate the viability of knowledge transfer between two related tasks: machine reading comprehension (MRC) and query-based text summarization. Using an MRC model trained on the SQuAD1.1 dataset as a core system component, we first build an extractive query-based summarizer. For better precision, this summarizer also compresses the output of the MRC model using a novel sentence compression technique. We further leverage pre-trained machine translation systems to abstract our extracted summaries. Our models achieve state-of-the-art results on the publicly available CNN/Daily Mail and Debatepedia datasets, and can serve as simple yet powerful baselines for future systems. We also hope that these results will encourage research on transfer learning from large MRC corpora to query-based summarization.
Book QA: Stories of Challenges and Opportunities
Authors: Stefanos Angelidis, Lea Frermann, Diego Marcheggiani, Roi Blanco, Llu{\'\i}s M{\`a}rquez
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Retrieval

We present a system for answering questions based on the full text of books (BookQA), which first selects book passages given a question at hand, and then uses a memory network to reason and predict an answer. To improve generalization, we pretrain our memory network using artificial questions generated from book sentences. We experiment with the recently published NarrativeQA corpus, on the subset of Who questions, which expect book characters as answers. We experimentally show that BERT-based retrieval and pretraining improve over baseline results significantly. At the same time, we confirm that NarrativeQA is a highly challenging data set, and that there is need for novel research in order to achieve high-precision BookQA results. We analyze some of the bottlenecks of the current approach, and we argue that more research is needed on text representation, retrieval of relevant passages, and reasoning, including commonsense knowledge.
Machine Comprehension Improves Domain-Specific Japanese Predicate-Argument Structure Analysis
Authors: Norio Takahashi, Tomohide Shibata, Daisuke Kawahara, Sadao Kurohashi
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Reading Comprehension

To improve the accuracy of predicate-argument structure (PAS) analysis, large-scale training data and knowledge for PAS analysis are indispensable. We focus on a specific domain, specifically Japanese blogs on driving, and construct two wide-coverage datasets as a form of QA using crowdsourcing: a PAS-QA dataset and a reading comprehension QA (RC-QA) dataset. We train a machine comprehension (MC) model based on these datasets to perform PAS analysis. Our experiments show that a stepwise training method is the most effective, which pre-trains an MC model based on the RC-QA dataset to acquire domain knowledge and then fine-tunes based on the PAS-QA dataset.
On Making Reading Comprehension More Comprehensive
Authors: Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, Sewon Min
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Machine Reading Comprehension, Question Answering, Reading Comprehension

Machine reading comprehension, the task of evaluating a machine{'}s ability to comprehend a passage of text, has seen a surge in popularity in recent years. There are many datasets that are targeted at reading comprehension, and many systems that perform as well as humans on some of these datasets. Despite all of this interest, there is no work that systematically defines what reading comprehension is. In this work, we justify a question answering approach to reading comprehension and describe the various kinds of questions one might use to more fully test a system{'}s comprehension of a passage, moving beyond questions that only probe local predicate-argument structures. The main pitfall of this approach is that questions can easily have surface cues or other biases that allow a model to shortcut the intended reasoning process. We discuss ways proposed in current literature to mitigate these shortcuts, and we conclude with recommendations for future dataset collection efforts.
Evaluating Question Answering Evaluation
Authors: Anthony Chen, Gabriel Stanovsky, Sameer Singh, Matt Gardner
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Answer Generation, Multiple-choice, Question Answering

As the complexity of question answering (QA) datasets evolve, moving away from restricted formats like span extraction and multiple-choice (MC) to free-form answer generation, it is imperative to understand how well current metrics perform in evaluating QA. This is especially important as existing metrics (BLEU, ROUGE, METEOR, and F1) are computed using n-gram similarity and have a number of well-known drawbacks. In this work, we study the suitability of existing metrics in QA. For generative QA, we show that while current metrics do well on existing datasets, converting multiple-choice datasets into free-response datasets is challenging for current metrics. We also look at span-based QA, where F1 is a reasonable metric. We show that F1 may not be suitable for all extractive QA tasks depending on the answer types. Our study suggests that while current metrics may be suitable for existing QA datasets, they limit the complexity of QA datasets that can be created. This is especially true in the context of free-form QA, where we would like our models to be able to generate more complex and abstractive answers, thus necessitating new metrics that go beyond n-gram based matching. As a step towards a better QA metric, we explore using BERTScore, a recently proposed metric for evaluating translation, for QA. We find that although it fails to provide stronger correlation with human judgements, future work focused on tailoring a BERT-based metric to QA evaluation may prove fruitful.
Bend but Don't Break? Multi-Challenge Stress Test for QA Models
Authors: Hemant Pugaliya, James Route, Kaixin Ma, Yixuan Geng, Eric Nyberg
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Question Answering

The field of question answering (QA) has seen rapid growth in new tasks and modeling approaches in recent years. Large scale datasets and focus on challenging linguistic phenomena have driven development in neural models, some of which have achieved parity with human performance in limited cases. However, an examination of state-of-the-art model output reveals that a gap remains in reasoning ability compared to a human, and performance tends to degrade when models are exposed to less-constrained tasks. We are interested in more clearly defining the strengths and limitations of leading models across diverse QA challenges, intending to help future researchers with identifying pathways to generalizable performance. We conduct extensive qualitative and quantitative analyses on the results of four models across four datasets and relate common errors to model capabilities. We also illustrate limitations in the datasets we examine and discuss a way forward for achieving generalizable models and datasets that broadly test QA capabilities.
Comprehensive Multi-Dataset Evaluation of Reading Comprehension
Authors: Dheeru Dua, Ananth Gottumukkala, Alon Talmor, Sameer Singh, Matt Gardner
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Entity Typing, Natural Language Understanding, Reading Comprehension, Representation Learning

Reading comprehension is one of the crucial tasks for furthering research in natural language understanding. A lot of diverse reading comprehension datasets have recently been introduced to study various phenomena in natural language, ranging from simple paraphrase matching and entity typing to entity tracking and understanding the implications of the context. Given the availability of many such datasets, comprehensive and reliable evaluation is tedious and time-consuming for researchers working on this problem. We present an evaluation server, ORB, that reports performance on seven diverse reading comprehension datasets, encouraging and facilitating testing a single model{'}s capability in understanding a wide variety of reading phenomena. The evaluation server places no restrictions on how models are trained, so it is a suitable test bed for exploring training paradigms and representation learning for general reading facility. As more suitable datasets are released, they will be added to the evaluation server. We also collect and include synthetic augmentations for these datasets, testing how well models can handle out-of-domain questions.
A Recurrent BERT-based Model for Question Generation
Authors: Ying-Hong Chan, Yao-Chung Fan
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Language Modeling, Language Modelling, model, Question Answering, Question Generation, Question-Generation, Text Generation

In this study, we investigate the employment of the pre-trained BERT language model to tackle question generation tasks. We introduce three neural architectures built on top of BERT for question generation tasks. The first one is a straightforward BERT employment, which reveals the defects of directly using BERT for text generation. Accordingly, we propose another two models by restructuring our BERT employment into a sequential manner for taking information from previous decoded results. Our models are trained and evaluated on the recent question-answering dataset SQuAD. Experiment results show that our best model yields state-of-the-art performance which advances the BLEU 4 score of the existing best models from 16.85 to 22.17.
Extractive NarrativeQA with Heuristic Pre-Training
Authors: Lea Frermann
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Natural Language Inference, Passage Retrieval, Question Answering, Retrieval, Unsupervised Pre-training

Although advances in neural architectures for NLP problems as well as unsupervised pre-training have led to substantial improvements on question answering and natural language inference, understanding of and reasoning over long texts still poses a substantial challenge. Here, we consider the task of question answering from full narratives (e.g., books or movie scripts), or their summaries, tackling the NarrativeQA challenge (NQA; Kocisky et al. (2018)). We introduce a heuristic extractive version of the data set, which allows us to approach the more feasible problem of answer extraction (rather than generation). We train systems for passage retrieval as well as answer span prediction using this data set. We use pre-trained BERT embeddings for injecting prior knowledge into our system. We show that our setup leads to state of the art performance on summary-level QA. On QA from full narratives, our model outperforms previous models on the METEOR metric. We analyze the relative contributions of pre-trained embeddings and the extractive training paradigm, and provide a detailed error analysis.
CLER: Cross-task Learning with Expert Representation to Generalize Reading and Understanding
Authors: Takumi Takahashi, Motoki Taniguchi, Tomoki Taniguchi, Tomoko Ohkuma
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Mixture-of-Experts, Multi-Task Learning, Reading Comprehension

This paper describes our model for the reading comprehension task of the MRQA shared task. We propose CLER, which stands for Cross-task Learning with Expert Representation for the generalization of reading and understanding. To generalize its capabilities, the proposed model is composed of three key ideas: multi-task learning, mixture of experts, and ensemble. In-domain datasets are used to train and validate our model, and other out-of-domain datasets are used to validate the generalization of our model{'}s performances. In a submission run result, the proposed model achieved an average F1 score of 66.1 {\%} in the out-of-domain setting, which is a 4.3 percentage point improvement over the official BERT baseline model.
Question Answering Using Hierarchical Attention on Top of BERT Features
Authors: Reham Osama, Nagwa El-Makky, Marwan Torki
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Question Answering

The model submitted works as follows. When supplied a question and a passage it makes use of the BERT embedding along with the hierarchical attention model which consists of 2 parts, the co-attention and the self-attention, to locate a continuous span of the passage that is the answer to the question.
Generalizing Question Answering System with Pre-trained Language Model Fine-tuning
Authors: Dan Su, Yan Xu, Genta Indra Winata, Peng Xu, Hyeondey Kim, Zihan Liu, Pascale Fung
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Language Modeling, Language Modelling, Multi-Task Learning, Question Answering, Reading Comprehension

With a large number of datasets being released and new techniques being proposed, Question answering (QA) systems have witnessed great breakthroughs in reading comprehension (RC)tasks. However, most existing methods focus on improving in-domain performance, leaving open the research question of how these mod-els and techniques can generalize to out-of-domain and unseen RC tasks. To enhance the generalization ability, we propose a multi-task learning framework that learns the shared representation across different tasks. Our model is built on top of a large pre-trained language model, such as XLNet, and then fine-tuned on multiple RC datasets. Experimental results show the effectiveness of our methods, with an average Exact Match score of 56.59 and an average F1 score of 68.98, which significantly improves the BERT-Large baseline by8.39 and 7.22, respectively
D-NET: A Pre-Training and Fine-Tuning Framework for Improving the Generalization of Machine Reading Comprehension
Authors: Hongyu Li, Xiyuan Zhang, Yibing Liu, Yiming Zhang, Quan Wang, Xiangyang Zhou, Jing Liu, Hua Wu, Haifeng Wang
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Machine Reading Comprehension, Multi-Task Learning, Question Answering, Reading Comprehension

In this paper, we introduce a simple system Baidu submitted for MRQA (Machine Reading for Question Answering) 2019 Shared Task that focused on generalization of machine reading comprehension (MRC) models. Our system is built on a framework of pretraining and fine-tuning, namely D-NET. The techniques of pre-trained language models and multi-task learning are explored to improve the generalization of MRC models and we conduct experiments to examine the effectiveness of these strategies. Our system is ranked at top 1 of all the participants in terms of averaged F1 score. Our codes and models will be released at PaddleNLP.
Dependency Tree Annotation with Mechanical Turk
Authors: Stephen Tratz
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Sentence

Crowdsourcing is frequently employed to quickly and inexpensively obtain valuable linguistic annotations but is rarely used for parsing, likely due to the perceived difficulty of the task and the limited training of the available workers. This paper presents what is, to the best of our knowledge, the first published use of Mechanical Turk (or similar platform) to crowdsource parse trees. We pay Turkers to construct unlabeled dependency trees for 500 English sentences using an interactive graphical dependency tree editor, collecting 10 annotations per sentence. Despite not requiring any training, several of the more prolific workers meet or exceed 90{\%} attachment agreement with the Penn Treebank (PTB) portion of our data, and, furthermore, for 72{\%} of these PTB sentences, at least one Turker produces a perfect parse. Thus, we find that, supported with a simple graphical interface, people with presumably no prior experience can achieve surprisingly high degrees of accuracy on this task. To facilitate research into aggregation techniques for complex crowdsourced annotations, we publicly release our annotated corpus.
Word Familiarity Rate Estimation Using a Bayesian Linear Mixed Model
Authors: Masayuki Asahara
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11

This paper presents research on word familiarity rate estimation using the {`}Word List by Semantic Principles{'}. We collected rating information on 96,557 words in the {`}Word List by Semantic Principles{'} via Yahoo! crowdsourcing. We asked 3,392 subject participants to use their introspection to rate the familiarity of words based on the five perspectives of {`}KNOW{'}, {`}WRITE{'}, {`}READ{'}, {`}SPEAK{'}, and {`}LISTEN{'}, and each word was rated by at least 16 subject participants. We used Bayesian linear mixed models to estimate the word familiarity rates. We also explored the ratings with the semantic labels used in the {`}Word List by Semantic Principles{'}.
Leveraging syntactic parsing to improve event annotation matching
Authors: Camiel Colruyt, Orph{\'e}e De Clercq, V{\'e}ronique Hoste
Date: 2019-11-01 00:00:00
Proceeding: WS 2019 11
Tasks: Event Extraction

Detecting event mentions is the first step in event extraction from text and annotating them is a notoriously difficult task. Evaluating annotator consistency is crucial when building datasets for mention detection. When event mentions are allowed to cover many tokens, annotators may disagree on their span, which means that overlapping annotations may then refer to the same event or to different events. This paper explores different fuzzy-matching functions which aim to resolve this ambiguity. The functions extract the sets of syntactic heads present in the annotations, use the Dice coefficient to measure the similarity between sets and return a judgment based on a given threshold. The functions are tested against the judgment of a human evaluator and a comparison is made between sets of tokens and sets of syntactic heads. The best-performing function is a head-based function that is found to agree with the human evaluator in 89{\%} of cases.