pwc-archive/papers-with-abstracts huggingface dataset:
Improving Personalized Dialogue Generation Models with Data-level Distillation and Diversification
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Dialogue Generation, Diversity

Personalized dialogue generation is a challenging task in which a persona-consistent response needs to be generated conditioning both persona texts and dialogue utterances, being more complex than conventional dialogues. Multiple persona texts and utterances exist in one sample and some of them can be distractors for generating. Thus even strong models have difficulty posing attention to suitable personas so generating persona-irrelevant responses. Besides, the limited data scale and diversity further affect the performance. Thus, we start from data and propose to boost the model in data-level distillation and diversification (D$^3$). We first distill the original training samples into simplified persona-consistent ones, lowering the difficulty by removing redundant information in personas and dialogue history. Next in the diversification, we increase both the amount and diversity of distilled data to ease its insufficiency. A model will be trained via curricula, first on easier augmented samples and then on harder original ones. Experiments on the PersonaChat benchmark dataset illustrate the superiority of our method when packed with two strong base dialogue models (Transformer and GPT2) on various automatic metrics and human evaluation.
GraphPrompt: Biomedical Entity Normalization Using Graph-based Prompt Templates
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9

Biomedical entity normalization unifies the language across biomedical experiments and studies, and further enables us to obtain a holistic view of life sciences. Current approaches mainly study the normalization of more standardized entities such as diseases and drugs, while disregarding the more ambiguous but crucial entities such as pathways, functions and cell types, hindering their real-world applications. To achieve biomedical entity normalization on these under-explored entities, we first introduce an expert-curated dataset OBO-syn encompassing 70 different types of entities and 2 million curated entity-synonym pairs. To utilize the unique graph structure in this dataset, we propose GraphPrompt, a prompt-based learning approach that creates prompt templates according to the graphs. GraphPrompt obtained 41.0% and 29.9% improvement on zero-shot and few-shot settings respectively, indicating the effectiveness of these graph-based prompt templates. We envision that our method GraphPrompt and OBO-syn dataset can be broadly applied to graph-based NLP tasks, and serve as the basis for analyzing diverse and accumulating biomedical data.
Pseudo-Error Generation for Grammatical Error Correction Based on Learner’s First Language
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Grammatical Error Correction, Translation

We propose to adapt grammatical error correction (GEC) systems to the learners' first language (L1) by generating artificial errors that reflect the L1 influence. To this end, we employ two simple approaches: fine-tuning a back-translation model on L1-annotated data; and controlling the output of a back-translation model and generating artificial errors that follow the L1-dependant error type distribution. We demonstrate that, despite the simplicity of the model and the paucity of the L1-annotated data, our methods succeed in adapting GEC models to some languages. We also show that generating L1-adapted artificial errors is orthogonal to the existing method that directly adapts the GEC model to each L1. Lastly, we present an analysis of the pseudo errors generated by our models and show that they approximately capture the L1-specific error patterns.
Learning Low-frequency Patterns with A Pre-trained Document-Grounded Conversation Model
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Decoder, Dialogue Generation

Owing to its perceived capability in recognizing the high-frequency patterns appeared in the large corpora, the Generative Pre-trained Transformer model (GPT-2) has demonstrated its remarkable performance in the document-grounded dialogue generation. Capturing low-frequency patterns, however, remains a challenging task. Here we consider a possible extension of the GPT-2 model with its improved capability of grasping the low-frequency patterns especially for task-specific dialogues. The extension consists of a semantic-oriented encoder and a GPT-2 decoder, the latter equipped with a knowledge-aware classification. The proposed encoder-decoder framework strengthens the GPT-2 in two task-specific aspects: One is in regard of a suitable way to select, on a semantic level, the crucial information of the dialogue context and the corresponding history knowledge from the documents; The other is in terms of the determination of the suitable time to generate a response with the knowledge from documents. With the enhanced capability to learn not only high-frequency and but also low-frequency patterns, the proposed extension is shown to outperform the state-of-the-art generative models.
DiS-ReX: A Multilingual Dataset for Distantly Supervised Relation Extraction
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Benchmarking, Relation, Relation Extraction

Our goal is to study the novel task of distant supervision for multilingual relation extraction (Multi DS-RE). Research in Multi DS-RE has remained limited due to the absence of a reliable benchmarking dataset. The only available dataset for this task, RELX-Distant  (Köksal and Özgür, 2020), displays several unrealistic characteristics, leading to a systematic overestimation of model performance. To alleviate these concerns, we release a new benchmark dataset for the task, named DiS-ReX. We also modify the widely-used bag attention models using an mBERT encoder and provide the first baseline results on the proposed task. We show that DiS-ReX serves as a more challenging dataset than RELX-Distant, leaving ample room for future research in this domain.
Denoising Large-Scale Image Captioning from Alt-text Data using Content Selection Models
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Caption Generation, Denoising, Image Captioning

Training large-scale image captioning (IC) models demands access to a rich and diverse set of training examples that are expensive to curate both in terms of time and man-power. Instead, alt-text based captions gathered from the web is a far cheaper alternative to scale with the downside of being noisy. Recent modeling approaches to IC often fall short in terms of performance in leveraging these noisy datasets in favor of clean annotations. We address this problem by breaking down the task into two simpler, more controllable tasks -- skeleton prediction and skeleton-based caption generation. Specifically, we show that sub-selecting content words as skeletons helps in generating improved and denoised captions when leveraging rich yet noisy alt-text--based uncurated datasets. We also show that the predicted English skeletons can further cross-lingually be leveraged to generate non-English captions, and present experimental results covering caption generation in French, Italian, German, Spanish and Hindi. We also show that skeleton-based prediction allows for better control of certain caption properties, such as length, content, and gender expression, providing a handle to perform human-in-the-loop interpretable semi-automatic corrections.
Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9

Although transformer-based Neural Language Models obtain impressive results on a wide variety of tasks, their generalization abilities are not well understood. They have been shown to perform strongly on subject-verb number agreement in a wide array of settings, suggesting that they learned to capture syntactic dependencies during their training even without explicit supervision. In this paper, we examine the extent to which BERT relies on lexical content to solve the number agreement (NA) task. To do so, we disrupt the lexical patterns found in naturally occurring stimuli in a novel fine-grained analysis of BERT's behavior. Our results on nonce sentences suggest that the model generalizes well for simple structures, but fails to perform lexically-independent syntactic generalization when as little as one attractor is present.
Commonsense Knowledge-Augmented Pretrained Language Models for Causal Reasoning Classification
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Classification, Commonsense Causal Reasoning, Language Modeling, Language Modelling

Commonsense knowledge can be leveraged for identifying causal relations in text. In this work, we convert triples in ATOMIC2020, a wide coverage commonsense reasoning knowledge graph, to natural language text and continually pretrain a BERT pretrained language model. We evaluate the resulting model on answering commonsense reasoning questions. Our results show that a continually pretrained language model augmented with commonsense reasoning knowledge outperforms our baseline on two commonsense causal reasoning benchmarks, COPA and BCOPA-CE, without additional improvement on the base model or using quality-enhanced data for fine-tuning.
Cross-Lingual Event Detection via Optimized Adversarial Training
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Event Detection

In this work, we focus on Cross-Lingual Event Detection (CLED) where a model is trained on data from a source language but its performance is evaluated on data from a second, target, language. Most recent works in this area have harnessed the language-invariant qualities displayed by pre-trained Multi-lingual Language Models (MLM). Their performance, however, reveals there is room for improvement as they mishandle delicate cross-lingual instances. We leverage the use of unlabeled data to train a Language Discriminator (LD) to discern between the source and target languages. The LD is trained in an adversarial manner so that our encoder learns to produce refined, language-invariant representations that lead to improved CLED performance. More importantly, we optimize the adversarial training by only presenting the LD with the most \textit{informative} samples. We base our intuition about \textit{what} makes a sample informative on two disparate metrics: sample similarity and event presence. Thus, we propose using Optimal Transport (OT) as a solution to naturally combine these two distinct information sources into the selection process. Extensive experiments on 8 different language pairs, using 4 languages from unrelated families, show the flexibility and effectiveness of our model that achieves new state-of-the-art results.
A Meta-framework for Spatiotemporal Quantity Extraction from Text
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9

News events are often associated with quantities (e.g., the number of COVID-19 patients or the number of arrests in a protest), and it is often important to extract their type, time, and location from unstructured text in order to analyze these quantity events. This paper thus formulates the NLP problem of spatiotemporal quantity extraction, and proposes the first meta-framework for solving it. This meta-framework contains a formalism that decomposes the problem into several information extraction tasks, a shareable crowdsourcing pipeline, and transformer-based baseline models. We demonstrate the meta-framework in three domains---the COVID-19 pandemic, Black Lives Matter protests, and 2020 California wildfires---to show that the formalism is general and extensible, the crowdsourcing pipeline facilitates fast and high-quality data annotation, and the baseline system can handle spatiotemporal quantity extraction well enough to be practically useful. All resources of this paper will be released for future research on this topic.
A Simple and Effective Model for Multi-Hop Question Generation
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Question Generation, Question-Generation, Sentence, Sentence Classification

Previous research on automated question generation has almost exclusively focused on generating factoid questions whose answers can be extracted from a single document. However, there is an increasing interest in developing systems that are capable of more complex multi-hop question generation (QG), where answering the question requires reasoning over multiple documents. In this work, we propose a simple and effective approach based on the transformer model for multi-hop QG. Our approach consists of specialized input representations, a supporting sentence classification objective, and training data weighting. Prior work on multi-hop QG considers the simplified setting of shorter documents and also advocates the use of entity-based graph structures as essential ingredients in model design. On the contrary, we showcase that our model can scale to the challenging setting of longer documents as input, does not rely on graph structures, and substantially outperforms the state-of-the-art approaches as measured by automated metrics and human evaluation.
Multilingual Generative Language Models for Zero-Shot Cross-Lingual Event Argument Extraction
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Cross-Lingual Transfer, Event Argument Extraction, Text Generation

We present a pioneering study on leveraging multilingual pre-trained generative language models for zero-shot cross-lingual event argument extraction (EAE) by formulating EAE as a language generation task. Compared to previous classification-based EAE models that build classifiers on top of pre-trained masked language models, our generative model effectively encodes the event structures and better captures the dependencies between arguments. To achieve cross-lingual transfer, we design language-agnostic templates to encode argument roles and train our models on source languages to "generate" arguments in the source languages to fill in the language-agnostic template. The trained model can then be directly applied to target languages to "generate" arguments in the target languages to fill in the template. Our experimental results demonstrate that the proposed model outperforms the current state-of-the-art results on zero-shot cross-lingual EAE. Comprehensive ablation study and error analysis are presented to better understand the advantages and the current limitations of using multilingual generative language models for cross-lingual transfer.
Digging Errors in NMT: Evaluating and Understanding Model Errors from Hypothesis Distribution
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Data Augmentation, Machine Translation, NMT, Translation

Sound evaluation of a neural machine translation (NMT) model is key to its understanding and improvement. Current evaluation of an NMT system is usually built upon a heuristic decoding algorithm (e.g., beam search) and an evaluation metric assessing similarity between the translation and golden reference (e.g., BLEU). However, this system-level evaluation framework is prone to its evaluation over only one best hypothesis and search errors brought by heuristic decoding algorithms. To better understand NMT models, we propose a novel evaluation protocol, which defines model errors with hypothesis distribution. In particular, we first propose an exact top-$k$ decoding algorithm, which finds top-ranked hypotheses in the whole hypothesis space and avoids search errors. Then, we evaluate NMT model errors with the distance between hypothesis distribution with the ideal distribution, aiming for a comprehensive interpretation. We apply our evaluation on various NMT benchmarks and model architectures to provide an in-depth understanding of how NMT models work. We show that the state-of-the-art Transformer models are facing serious ranking errors and do not even outperform the random chance level. We further provide several interesting findings over data-augmentation techniques, dropouts, and deep/wide models. 
Additionally, we analyze beam search's lucky biases and regularization terms. Interestingly, we find these lucky biases decrease when increasing model capacity.
SeaD: End-to-end Text-to-SQL Generation with Schema-aware Denoising
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Denoising, slot-filling, Slot Filling, Text to SQL, Text-To-SQL

On the WikiSQL benchmark, most methods tackle the challenge of text-to-SQL with predefined sketch slots and build sophisticated sub-tasks to fill these slots. Though achieving promising results, these methods suffer from over-complex model structure. In this paper, we present a simple yet effective approach that enables auto-regressive sequence-to-sequence model to robust text-to-SQL generation. Instead of formulating the task of text-to-SQL as slot-filling, we propose to train sequence-to-sequence model with Schema-aware Denoising (SeaD), which consists of two denoising objectives that train model to either recover input or predict output from two novel erosion and shuffle noises. These model-agnostic denoising objectives act as the auxiliary tasks for structural data modeling during sequence-to-sequence generation. In addition, we propose a clause-sensitive execution guided (EG) decoding strategy to overcome the limitation of EG decoding for generative model. The experiments show that the proposed method improves the performance of sequence-to-sequence model in both schema linking and grammar correctness and establishes new state-of-the-art on WikiSQL benchmark. Our work indicates that the capacity of sequence-to-sequence model for text-to-SQL may have been under-estimated and could be enhanced by specialized denoising task.
Modeling Future for Neural Machine Translation by Fusing Target Information
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Decoder, Machine Translation, NMT, Translation

Sequence-to-sequence Neural Machine Translation (NMT) models have achieved excellent performance. However, the NMT decoder only makes predictions based on the source and the target historical context, ignores the target future information completely, leading to a problem that NMT does not consider potential future information when making decisions. To alleviate this problem, we propose a simple and effective {\bf Fu}ture-fused {\bf NMT} model called \textsc{FuNMT}, which introduces a reverse decoder to explicitly model the target future information, then adopts an agreement mechanism to enable the forward decoder to learn this future information. Empirical studies on multiple benchmarks show that our proposed model significantly improves translation quality.
Human Schema Curation via Causal Association Rule Mining
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9

Event schemas are structured knowledge sources defining typical real-world scenarios (e.g., going to an airport). We present a framework for efficient human-in-the-loop construction of a schema library, based on a novel mechanism for schema induction and a well-crafted interface that allows non-experts to "program" complex event structures. Associated with this work we release a schema library: a machine readable resource of 232 detailed event schemas, each of which describe a distinct typical scenario in terms of its relevant sub-event structure (what happens in the scenario), participants (who plays a role in the scenario), fine-grained typing of each participant, and the implied relational constraints between them.  We freely release our schema library and custom annotation interface, SchemaBlocks.
A Few-Shot Semantic Parser for Wizard-of-Oz Dialogues with the Precise ThingTalk+ Representation
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9

Previous attempts to build effective semantic parsers for Wizard-of-Oz (WOZ) conversations suffer from the difficulty in acquiring a high-quality, manually annotated training set. Approaches based only on dialogue synthesis are insufficient as dialogues generated from state-machine based models are poor approximations of real-life conversations. Furthermore, previously proposed dialogue state representations are ambiguous and lack the precision necessary for building an effective agent.

This paper proposes a new dialogue representation and a sample-efficient methodology that can predict precise dialogue states in WOZ conversations. We propose a precise, complete, and executable dialogue representation called ThingTalk+, which captures all information an agent needs to respond properly. Our training strategy is sample-efficient: we combine (1) few-shot data sparsely sampling the full dialogue space and (2) synthesized data covering a subset space of dialogues generated by a succinct state-based dialogue model. The completeness of the ThingTalk+ language is demonstrated with a fully operational agent, which is also used in training data synthesis. 

We demonstrate the effectiveness of our methodology on MultiWOZ 3.0, a reannotation of the MultiWOZ 2.1 dataset in ThingTalk+. ThingTalk+ can represent 98% of the test turns, while the simulator can emulate 85% of the validation set. We train a contextual semantic parser using our strategy, and obtain 79% turn-by-turn exact match accuracy on the reannotated test set.
Multimodal Audio-textual Architecture for Robust Spoken Language Understanding
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), speech-recognition, Speech Recognition, Spoken Language Understanding

Tandem spoken language understanding (SLU) systems suffer from the so-called automatic speech recognition (ASR) error propagation problem. Additionally, as the ASR is not optimized to extract semantics, but solely the linguistic content, relevant semantic cues might be left out of its transcripts. In this work, we propose a multimodal language understanding (MLU) architecture to mitigate these problems. Our solution is based on two compact unidirectional long short-term memory (LSTM) models that encode speech and text information. A fusion layer is also used to fuse audio and text embeddings. Two fusion strategies are explored: a simple concatenation of these embeddings and a cross-modal attention mechanism that learns the contribution of each modality. The first approach showed to be the optimal solution to robustly extract semantic information from audio-textual data. We found that attention is less effective at testing time when the text modality is corrupted. Our model is evaluated on three SLU datasets and robustness is tested using ASR outputs from three off-the-shelf ASR engines. Results show that the proposed approach effectively mitigates the ASR error propagation problem for all datasets.
Combining (Second-Order) Graph-Based and Headed-Span-Based Projective Dependency Parsing
Authors: Anonymous
Date: 2021-09-17 00:00:00
Proceeding: ACL ARR September 2021 9
Tasks: ARC, Dependency Parsing

Graph-based methods are popular in dependency parsing for decades, which decompose the score of a dependency tree into scores of dependency arcs. Recently, (Yang and Tu, 2021) propose a headed-span-based method that decomposes the score of a dependency tree into scores of headed spans. In this paper, we combine the two types of methods by considering both arc scores and headed-span scores, designing three scoring methods and the corresponding dynamic programming algorithms for joint inference. Experiments show the effectiveness of our proposed methods.