pwc-archive/papers-with-abstracts huggingface dataset:
Self-Attention Guided Copy Mechanism for Abstractive Summarization
Authors: Song Xu, Haoran Li, Peng Yuan, Youzheng Wu, Xiaodong He, Bo-Wen Zhou
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Abstractive Text Summarization, Decoder

Copy module has been widely equipped in the recent abstractive summarization models, which facilitates the decoder to extract words from the source into the summary. Generally, the encoder-decoder attention is served as the copy distribution, while how to guarantee that important words in the source are copied remains a challenge. In this work, we propose a Transformer-based model to enhance the copy mechanism. Specifically, we identify the importance of each source word based on the degree centrality with a directed graph built by the self-attention layer in the Transformer. We use the centrality of each source word to guide the copy process explicitly. Experimental results show that the self-attention graph provides useful guidance for the copy distribution. Our proposed models significantly outperform the baseline methods on the CNN/Daily Mail dataset and the Gigaword dataset.
Fast and Accurate Non-Projective Dependency Tree Linearization
Authors: Xiang Yu, Simon Tannert, Ngoc Thang Vu, Jonas Kuhn
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Traveling Salesman Problem

We propose a graph-based method to tackle the dependency tree linearization task. We formulate the task as a Traveling Salesman Problem (TSP), and use a biaffine attention model to calculate the edge costs. We facilitate the decoding by solving the TSP for each subtree and combining the solution into a projective tree. We then design a transition system as post-processing, inspired by non-projective transition-based parsing, to obtain non-projective sentences. Our proposed method outperforms the state-of-the-art linearizer while being 10 times faster in training and decoding.
Boosting Neural Machine Translation with Similar Translations
Authors: Jitao Xu, Josep Crego, Jean Senellart
Date: 2020-07-01 00:00:00
Proceeding: AMTA 2022 9
Tasks: Data Augmentation, Machine Translation, Sentence, Translation

This paper explores data augmentation methods for training Neural Machine Translation to make use of similar translations, in a comparable way a human translator employs fuzzy matches. In particular, we show how we can simply present the neural model with information of both source and target sides of the fuzzy matches, we also extend the similarity to include semantically related translations retrieved using sentence distributed representations. We show that translations based on fuzzy matching provide the model with {``}copy{''} information while translations based on embedding similarities tend to extend the translation {``}context{''}. Results indicate that the effect from both similar sentences are adding up to further boost accuracy, combine naturally with model fine-tuning and are providing dynamic adaptation for unseen translation pairs. Tests on multiple data sets and domains show consistent accuracy improvements. To foster research around these techniques, we also release an Open-Source toolkit with efficient and flexible fuzzy-match implementation.
Language-aware Interlingua for Multilingual Neural Machine Translation
Authors: Changfeng Zhu, Heng Yu, Shanbo Cheng, Weihua Luo
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Decoder, Diversity, Machine Translation, NMT, Specificity, Translation

Multilingual neural machine translation (NMT) has led to impressive accuracy improvements in low-resource scenarios by sharing common linguistic information across languages. However, the traditional multilingual model fails to capture the diversity and specificity of different languages, resulting in inferior performance compared with individual models that are sufficiently trained. In this paper, we incorporate a language-aware interlingua into the Encoder-Decoder architecture. The interlingual network enables the model to learn a language-independent representation from the semantic spaces of different languages, while still allowing for language-specific specialization of a particular language-pair. Experiments show that our proposed method achieves remarkable improvements over state-of-the-art multilingual NMT baselines and produces comparable performance with strong individual models.
Parallel Sentence Mining by Constrained Decoding
Authors: Pin-zhen Chen, Nikolay Bogoychev, Kenneth Heafield, Faheem Kirefu
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Cross-Lingual Bitext Mining, Machine Translation, Parallel Corpus Mining, Sentence, Sentence Similarity

We present a novel method to extract parallel sentences from two monolingual corpora, using neural machine translation. Our method relies on translating sentences in one corpus, but constraining the decoding by a prefix tree built on the other corpus. We argue that a neural machine translation system by itself can be a sentence similarity scorer and it efficiently approximates pairwise comparison with a modified beam search. When benchmarked on the BUCC shared task, our method achieves results comparable to other submissions.
``You Sound Just Like Your Father'' Commercial Machine Translation Systems Include Stylistic Biases
Authors: Dirk Hovy, Federico Bianchi, Tommaso Fornaciari
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Machine Translation, Translation

The main goal of machine translation has been to convey the correct content. Stylistic considerations have been at best secondary. We show that as a consequence, the output of three commercial machine translation systems (Bing, DeepL, Google) make demographically diverse samples from five languages {``}sound{''} older and more male than the original. Our findings suggest that translation models reflect demographic bias in the training data. This opens up interesting new research avenues in machine translation to take stylistic considerations into account.
MMPE: A Multi-Modal Interface for Post-Editing Machine Translation
Authors: Nico Herbig, Tim D{\"u}wel, Santanu Pal, Kalliopi Meladaki, Mahsa Monshizadeh, Antonio Kr{\"u}ger, Josef van Genabith
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Machine Translation, Translation

Current advances in machine translation (MT) increase the need for translators to switch from traditional translation to post-editing (PE) of machine-translated text, a process that saves time and reduces errors. This affects the design of translation interfaces, as the task changes from mainly generating text to correcting errors within otherwise helpful translation proposals. Since this paradigm shift offers potential for modalities other than mouse and keyboard, we present MMPE, the first prototype to combine traditional input modes with pen, touch, and speech modalities for PE of MT. The results of an evaluation with professional translators suggest that pen and touch interaction are suitable for deletion and reordering tasks, while they are of limited use for longer insertions. On the other hand, speech and multi-modal combinations of select {\&} speech are considered suitable for replacements and insertions but offer less potential for deletion and reordering. Overall, participants were enthusiastic about the new modalities and saw them as good extensions to mouse {\&} keyboard, but not as a complete substitute.
Overestimation of Syntactic Representation in Neural Language Models
Authors: Jordan Kodner, Nitish Gupta
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6

With the advent of powerful neural language models over the last few years, research attention has increasingly focused on what aspects of language they represent that make them so successful. Several testing methodologies have been developed to probe models{'} syntactic representations. One popular method for determining a model{'}s ability to induce syntactic structure trains a model on strings generated according to a template then tests the model{'}s ability to distinguish such strings from superficially similar ones with different syntax. We illustrate a fundamental problem with this approach by reproducing positive results from a recent paper with two non-syntactic baseline language models: an n-gram model and an LSTM model trained on scrambled inputs.
You Don't Have Time to Read This: An Exploration of Document Reading Time Prediction
Authors: Orion Weller, Hildebr, Jordan t, Ilya Reznik, Christopher Challis, E. Shannon Tass, Quinn Snell, Kevin Seppi
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6

Predicting reading time has been a subject of much previous work, focusing on how different words affect human processing, measured by reading time. However, previous work has dealt with a limited number of participants as well as word level only predictions (i.e. predicting the time to read a single word). We seek to extend these works by examining whether or not document level predictions are effective, given additional information such as subject matter, font characteristics, and readability metrics. We perform a novel experiment to examine how different features of text contribute to the time it takes to read, distributing and collecting data from over a thousand participants. We then employ a large number of machine learning methods to predict a user{'}s reading time. We find that despite extensive research showing that word level reading time can be most effectively predicted by neural networks, larger scale text can be easily and most accurately predicted by one factor, the number of words.
Conversational Graph Grounded Policy Learning for Open-Domain Conversation Generation
Authors: Jun Xu, Haifeng Wang, Zheng-Yu Niu, Hua Wu, Wanxiang Che, Ting Liu
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Response Generation

To address the challenge of policy learning in open-domain multi-turn conversation, we propose to represent prior information about dialog transitions as a graph and learn a graph grounded dialog policy, aimed at fostering a more coherent and controllable dialog. To this end, we first construct a conversational graph (CG) from dialog corpora, in which there are vertices to represent {``}what to say{''} and {``}how to say{''}, and edges to represent natural transition between a message (the last utterance in a dialog context) and its response. We then present a novel CG grounded policy learning framework that conducts dialog flow planning by graph traversal, which learns to identify a what-vertex and a how-vertex from the CG at each turn to guide response generation. In this way, we effectively leverage the CG to facilitate policy learning as follows: (1) it enables more effective long-term reward design, (2) it provides high-quality candidate actions, and (3) it gives us more control over the policy. Results on two benchmark corpora demonstrate the effectiveness of this framework.
Recollection versus Imagination: Exploring Human Memory and Cognition via Neural Language Models
Authors: Maarten Sap, Eric Horvitz, Yejin Choi, Noah A. Smith, James Pennebaker
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6

We investigate the use of NLP as a measure of the cognitive processes involved in storytelling, contrasting imagination and recollection of events. To facilitate this, we collect and release Hippocorpus, a dataset of 7,000 stories about imagined and recalled events. We introduce a measure of narrative flow and use this to examine the narratives for imagined and recalled events. Additionally, we measure the differential recruitment of knowledge attributed to semantic memory versus episodic memory (Tulving, 1972) for imagined and recalled storytelling by comparing the frequency of descriptions of general commonsense events with more specific realis events. Our analyses show that imagined stories have a substantially more linear narrative flow, compared to recalled stories in which adjacent sentences are more disconnected. In addition, while recalled stories rely more on autobiographical events based on episodic memory, imagined stories express more commonsense knowledge based on semantic memory. Finally, our measures reveal the effect of narrativization of memories in stories (e.g., stories about frequently recalled memories flow more linearly; Bartlett, 1932). Our findings highlight the potential of using NLP tools to study the traces of human cognition in language.
Speakers enhance contextually confusable words
Authors: Eric Meinhardt, Eric Bakovic, Leon Bergen
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6

Recent work has found evidence that natural languages are shaped by pressures for efficient communication {---} e.g. the more contextually predictable a word is, the fewer speech sounds or syllables it has (Piantadosi et al. 2011). Research on the degree to which speech and language are shaped by pressures for effective communication {---} robustness in the face of noise and uncertainty {---} has been more equivocal. We develop a measure of contextual confusability during word recognition based on psychoacoustic data. Applying this measure to naturalistic speech corpora, we find evidence suggesting that speakers alter their productions to make contextually more confusable words easier to understand.
What determines the order of adjectives in English? Comparing efficiency-based theories using dependency treebanks
Authors: Richard Futrell, William Dyer, Greg Scontras
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6

We take up the scientific question of what determines the preferred order of adjectives in English, in phrases such as big blue box where multiple adjectives modify a following noun. We implement and test four quantitative theories, all of which are theoretically motivated in terms of efficiency in human language production and comprehension. The four theories we test are subjectivity (Scontras et al., 2017), information locality (Futrell, 2019), integration cost (Dyer, 2017), and information gain, which we introduce. We evaluate theories based on their ability to predict orders of unseen adjectives in hand-parsed and automatically-parsed dependency treebanks. We find that subjectivity, information locality, and information gain are all strong predictors, with some evidence for a two-factor account, where subjectivity and information gain reflect a factor involving semantics, and information locality reflects collocational preferences.
``None of the Above'': Measure Uncertainty in Dialog Response Retrieval
Authors: Yulan Feng, Shikib Mehri, Maxine Eskenazi, Tiancheng Zhao
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6
Tasks: General Classification, Retrieval

This paper discusses the importance of uncovering uncertainty in end-to-end dialog tasks and presents our experimental results on uncertainty classification on the processed Ubuntu Dialog Corpus. We show that instead of retraining models for this specific purpose, we can capture the original retrieval model{'}s underlying confidence concerning the best prediction using trivial additional computation.
Recursive Template-based Frame Generation for Task Oriented Dialog
Authors: Rashmi Gangadharaiah, Balakrishnan Narayanaswamy
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Decoder, Natural Language Understanding

The Natural Language Understanding (NLU) component in task oriented dialog systems processes a user{'}s request and converts it into structured information that can be consumed by downstream components such as the Dialog State Tracker (DST). This information is typically represented as a semantic frame that captures the intent and slot-labels provided by the user. We first show that such a shallow representation is insufficient for complex dialog scenarios, because it does not capture the recursive nature inherent in many domains. We propose a recursive, hierarchical frame-based representation and show how to learn it from data. We formulate the frame generation task as a template-based tree decoding task, where the decoder recursively generates a template and then fills slot values into the template. We extend local tree-based loss functions with terms that provide global supervision and show how to optimize them end-to-end. We achieve a small improvement on the widely used ATIS dataset and a much larger improvement on a more complex dataset we describe here.
GAN-BERT: Generative Adversarial Learning for Robust Text Classification with a Bunch of Labeled Examples
Authors: Danilo Croce, Giuseppe Castellucci, Roberto Basili
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6
Tasks: General Classification, Sentence, Sentence Classification, text-classification, Text Classification

Recent Transformer-based architectures, e.g., BERT, provide impressive results in many Natural Language Processing tasks. However, most of the adopted benchmarks are made of (sometimes hundreds of) thousands of examples. In many real scenarios, obtaining high- quality annotated data is expensive and time consuming; in contrast, unlabeled examples characterizing the target task can be, in general, easily collected. One promising method to enable semi-supervised learning has been proposed in image processing, based on Semi- Supervised Generative Adversarial Networks. In this paper, we propose GAN-BERT that ex- tends the fine-tuning of BERT-like architectures with unlabeled data in a generative adversarial setting. Experimental results show that the requirement for annotated examples can be drastically reduced (up to only 50-100 annotated examples), still obtaining good performances in several sentence classification tasks.
On Importance Sampling-Based Evaluation of Latent Language Models
Authors: Robert L. Logan IV, Matt Gardner, Sameer Singh
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6

Language models that use additional latent structures (e.g., syntax trees, coreference chains, knowledge graph links) provide several advantages over traditional language models. However, likelihood-based evaluation of these models is often intractable as it requires marginalizing over the latent space. Existing works avoid this issue by using importance sampling. Although this approach has asymptotic guarantees, analysis is rarely conducted on the effect of decisions such as sample size and choice of proposal distribution on the reported estimates. In this paper, we carry out this analysis for three models: RNNG, EntityNLM, and KGLM. In addition, we elucidate subtle differences in how importance sampling is applied in these works that can have substantial effects on the final estimates, as well as provide theoretical results which reinforce the validity of this technique.
Taxonomy Construction of Unseen Domains via Graph-based Cross-Domain Knowledge Transfer
Authors: Chao Shang, Sarthak Dash, Md. Faisal Mahbub Chowdhury, N Mihindukulasooriya, ana, Alfio Gliozzo
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Graph Neural Network, Transfer Learning

Extracting lexico-semantic relations as graph-structured taxonomies, also known as taxonomy construction, has been beneficial in a variety of NLP applications. Recently Graph Neural Network (GNN) has shown to be powerful in successfully tackling many tasks. However, there has been no attempt to exploit GNN to create taxonomies. In this paper, we propose Graph2Taxo, a GNN-based cross-domain transfer framework for the taxonomy construction task. Our main contribution is to learn the latent features of taxonomy construction from existing domains to guide the structure learning of an unseen domain. We also propose a novel method of directed acyclic graph (DAG) generation for taxonomy construction. Specifically, our proposed Graph2Taxo uses a noisy graph constructed from automatically extracted noisy hyponym hypernym candidate pairs, and a set of taxonomies for some known domains for training. The learned model is then used to generate taxonomy for a new unknown domain given a set of terms for that domain. Experiments on benchmark datasets from science and environment domains show that our approach attains significant improvements correspondingly over the state of the art.
To Pretrain or Not to Pretrain: Examining the Benefits of Pretrainng on Resource Rich Tasks
Authors: Sinong Wang, Madian Khabsa, Hao Ma
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Language Modeling, Language Modelling, text-classification, Text Classification

Pretraining NLP models with variants of Masked Language Model (MLM) objectives has recently led to a significant improvements on many tasks. This paper examines the benefits of pretrained models as a function of the number of training samples used in the downstream task. On several text classification tasks, we show that as the number of training examples grow into the millions, the accuracy gap between finetuning BERT-based model and training vanilla LSTM from scratch narrows to within 1{\%}. Our findings indicate that MLM-based models might reach a diminishing return point as the supervised data size increases significantly.
Investigating the effect of auxiliary objectives for the automated grading of learner English speech transcriptions
Authors: Hannah Craighead, Andrew Caines, Paula Buttery, Helen Yannakoudakis
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Automatic Speech Recognition, Automatic Speech Recognition (ASR), Language Identification, Language Modeling, Language Modelling, Multi-Task Learning, Native Language Identification, speech-recognition, Speech Recognition

We address the task of automatically grading the language proficiency of spontaneous speech based on textual features from automatic speech recognition transcripts. Motivated by recent advances in multi-task learning, we develop neural networks trained in a multi-task fashion that learn to predict the proficiency level of non-native English speakers by taking advantage of inductive transfer between the main task (grading) and auxiliary prediction tasks: morpho-syntactic labeling, language modeling, and native language identification (L1). We encode the transcriptions with both bi-directional recurrent neural networks and with bi-directional representations from transformers, compare against a feature-rich baseline, and analyse performance at different proficiency levels and with transcriptions of varying error rates. Our best performance comes from a transformer encoder with L1 prediction as an auxiliary task. We discuss areas for improvement and potential applications for text-only speech scoring.
Can We Predict New Facts with Open Knowledge Graph Embeddings? A Benchmark for Open Link Prediction
Authors: Samuel Broscheit, Kiril Gashteovski, Yanjie Wang, Rainer Gemulla
Date: 2020-07-01 00:00:00
Proceeding: ACL 2020 6
Tasks: Graph Embedding, Knowledge Graph Embedding, Knowledge Graph Embeddings, Link Prediction, Open Information Extraction, Open Knowledge Graph Embedding, Prediction, Relation

Open Information Extraction systems extract ({``}subject text{''}, {``}relation text{''}, {``}object text{''}) triples from raw text. Some triples are textual versions of facts, i.e., non-canonicalized mentions of entities and relations. In this paper, we investigate whether it is possible to infer new facts directly from the open knowledge graph without any canonicalization or any supervision from curated knowledge. For this purpose, we propose the open link prediction task,i.e., predicting test facts by completing ({``}subject text{''}, {``}relation text{''}, ?) questions. An evaluation in such a setup raises the question if a correct prediction is actually a new fact that was induced by reasoning over the open knowledge graph or if it can be trivially explained. For example, facts can appear in different paraphrased textual variants, which can lead to test leakage. To this end, we propose an evaluation protocol and a methodology for creating the open link prediction benchmark OlpBench. We performed experiments with a prototypical knowledge graph embedding model for openlink prediction. While the task is very challenging, our results suggests that it is possible to predict genuinely new facts, which can not be trivially explained.