pwc-archive/papers-with-abstracts huggingface dataset:
Stylistic Analysis of the French Presidential Speeches: Is Macron really different?
Authors: Dominique Labbé, Jacques Savoy
Date: 2021-05-06 00:00:00
Tasks: Sentence

Presidential speeches indicate the government's intentions and justifications supported by a dedicated style and rhetoric oscillating between explanation and controversy. Over a period of sixty years, can we observe stylistic variations by the different French presidents of the Fifth Republic (1958-2018)? Based on official transcripts of all their allocution, this paper illustrates the stylistic evolution and presents the underlying main trends. This study shows that de Gaulle's rhetoric is not mainly dedicated to his own person, or that the two terms of J. Chirac are not fully similar. According to several overall stylistic indicators, Macron's style does not appear as complex compared to his predecessors (F. Hollande or N. Sarkozy) but a more careful analysis clearly demonstrates his noticeable new style. Compared to the recent US presidents, the French ones present some similarities (e.g., similar mean sentence length) and dissimilarities (more I-words, less we-words). In this comparative analysis, Macron's style is also clearly distinctive from both the US and former French presidents. Opting for a more abstract discourse, less anchored in space, using less numbers, E. Macron tends to use long sentences. These various stylistic and rhetorical features could explain his being misunderstood by the French people and his recurrent low approval ratings.
The Authors Matter: Understanding and Mitigating Implicit Bias in Deep Text Classification
Authors: Haochen Liu, Wei Jin, Hamid Karimi, Zitao Liu, Jiliang Tang
Date: 2021-05-06 00:00:00
Proceeding: Findings (ACL) 2021 8
Tasks: Classification, Fairness, General Classification, text-classification, Text Classification

It is evident that deep text classification models trained on human data could be biased. In particular, they produce biased outcomes for texts that explicitly include identity terms of certain demographic groups. We refer to this type of bias as explicit bias, which has been extensively studied. However, deep text classification models can also produce biased outcomes for texts written by authors of certain demographic groups. We refer to such bias as implicit bias of which we still have a rather limited understanding. In this paper, we first demonstrate that implicit bias exists in different text classification tasks for different demographic groups. Then, we build a learning-based interpretation method to deepen our knowledge of implicit bias. Specifically, we verify that classifiers learn to make predictions based on language features that are related to the demographic attributes of the authors. Next, we propose a framework Debiased-TC to train deep text classifiers to make predictions on the right features and consequently mitigate implicit bias. We conduct extensive experiments on three real-world datasets. The results show that the text classification models trained under our proposed framework outperform traditional models significantly in terms of fairness, and also slightly in terms of classification performance.
Introducing Information Retrieval for Biomedical Informatics Students
Authors: Sanya B. Taneja, Richard D. Boyce, William T. Reynolds, Denis Newman-Griffis
Date: 2021-05-06 00:00:00
Proceeding: NAACL (TeachingNLP) 2021 6
Tasks: Information Retrieval, Retrieval

Introducing biomedical informatics (BMI) students to natural language processing (NLP) requires balancing technical depth with practical know-how to address application-focused needs. We developed a set of three activities introducing introductory BMI students to information retrieval with NLP, covering document representation strategies and language models from TF-IDF to BERT. These activities provide students with hands-on experience targeted towards common use cases, and introduce fundamental components of NLP workflows for a wide variety of applications.
What's in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus
Authors: Alexandra Sasha Luccioni, Joseph D. Viviano
Date: 2021-05-06 00:00:00

Whereas much of the success of the current generation of neural language models has been driven by increasingly large training corpora, relatively little research has been dedicated to analyzing these massive sources of textual data. In this exploratory analysis, we delve deeper into the Common Crawl, a colossal web corpus that is extensively used for training language models. We find that it contains a significant amount of undesirable content, including hate speech and sexually explicit content, even after filtering procedures. We discuss the potential impacts of this content on language models and conclude with future research directions and a more mindful approach to corpus collection and analysis.
Learning to Perturb Word Embeddings for Out-of-distribution QA
Authors: Seanie Lee, Minki Kang, Juho Lee, Sung Ju Hwang
Date: 2021-05-06 00:00:00
Proceeding: ACL 2021 5
Tasks: Data Augmentation, Domain Generalization, Question Answering

QA models based on pretrained language mod-els have achieved remarkable performance on various benchmark datasets.However, QA models do not generalize well to unseen data that falls outside the training distribution, due to distributional shifts.Data augmentation (DA) techniques which drop/replace words have shown to be effective in regularizing the model from overfitting to the training data.Yet, they may adversely affect the QA tasks since they incur semantic changes that may lead to wrong answers for the QA task. To tackle this problem, we propose a simple yet effective DA method based on a stochastic noise generator, which learns to perturb the word embedding of the input questions and context without changing their semantics. We validate the performance of the QA models trained with our word embedding perturbation on a single source dataset, on five different target domains.The results show that our method significantly outperforms the baselineDA methods. Notably, the model trained with ours outperforms the model trained with more than 240K artificially generated QA pairs.
Improving the Faithfulness of Attention-based Explanations with Task-specific Information for Text Classification
Authors: George Chrysostomou, Nikolaos Aletras
Date: 2021-05-06 00:00:00
Proceeding: ACL 2021 5
Tasks: text-classification, Text Classification

Neural network architectures in natural language processing often use attention mechanisms to produce probability distributions over input token representations. Attention has empirically been demonstrated to improve performance in various tasks, while its weights have been extensively used as explanations for model predictions. Recent studies (Jain and Wallace, 2019; Serrano and Smith, 2019; Wiegreffe and Pinter, 2019) have showed that it cannot generally be considered as a faithful explanation (Jacovi and Goldberg, 2020) across encoders and tasks. In this paper, we seek to improve the faithfulness of attention-based explanations for text classification. We achieve this by proposing a new family of Task-Scaling (TaSc) mechanisms that learn task-specific non-contextualised information to scale the original attention weights. Evaluation tests for explanation faithfulness, show that the three proposed variants of TaSc improve attention-based explanations across two attention mechanisms, five encoders and five text classification datasets without sacrificing predictive performance. Finally, we demonstrate that TaSc consistently provides more faithful attention-based explanations compared to three widely-used interpretability techniques.
Bird's Eye: Probing for Linguistic Graph Structures with a Simple Information-Theoretic Approach
Authors: Yifan Hou, Mrinmaya Sachan
Date: 2021-05-06 00:00:00
Proceeding: ACL 2021 5

NLP has a rich history of representing our prior understanding of language in the form of graphs. Recent work on analyzing contextualized text representations has focused on hand-designed probe models to understand how and to what extent do these representations encode a particular linguistic phenomenon. However, due to the inter-dependence of various phenomena and randomness of training probe models, detecting how these representations encode the rich information in these linguistic graphs remains a challenging problem. In this paper, we propose a new information-theoretic probe, Bird's Eye, which is a fairly simple probe method for detecting if and how these representations encode the information in these linguistic graphs. Instead of using classifier performance, our probe takes an information-theoretic view of probing and estimates the mutual information between the linguistic graph embedded in a continuous space and the contextualized word representations. Furthermore, we also propose an approach to use our probe to investigate localized linguistic information in the linguistic graphs using perturbation analysis. We call this probing setup Worm's Eye. Using these probes, we analyze BERT models on their ability to encode a syntactic and a semantic graph structure, and find that these models encode to some degree both syntactic as well as semantic information; albeit syntactic information to a greater extent.
TABBIE: Pretrained Representations of Tabular Data
Authors: Hiroshi Iida, Dung Thai, Varun Manjunatha, Mohit Iyyer
Date: 2021-05-06 00:00:00
Proceeding: NAACL 2021 4
Tasks: Cell Detection, Column Type Annotation, Representation Learning, Table annotation

Existing work on tabular representation learning jointly models tables and associated text using self-supervised objective functions derived from pretrained language models such as BERT. While this joint pretraining improves tasks involving paired tables and text (e.g., answering questions about tables), we show that it underperforms on tasks that operate over tables without any associated text (e.g., populating missing cells). We devise a simple pretraining objective (corrupt cell detection) that learns exclusively from tabular data and reaches the state-of-the-art on a suite of table based prediction tasks. Unlike competing approaches, our model (TABBIE) provides embeddings of all table substructures (cells, rows, and columns), and it also requires far less compute to train. A qualitative analysis of our model's learned cell, column, and row representations shows that it understands complex table semantics and numerical trends.
Assessing Dialogue Systems with Distribution Distances
Authors: Jiannan Xiang, Yahui Liu, Deng Cai, Huayang Li, Defu Lian, Lemao Liu
Date: 2021-05-06 00:00:00
Proceeding: Findings (ACL) 2021 8
Tasks: Dialogue Evaluation

An important aspect of developing dialogue systems is how to evaluate and compare the performance of different systems. Existing automatic evaluation metrics are based on turn-level quality evaluation and use average scores for system-level comparison. In this paper, we propose to measure the performance of a dialogue system by computing the distribution-wise distance between its generated conversations and real-world conversations. Specifically, two distribution-wise metrics, FBD and PRD, are developed and evaluated. Experiments on several dialogue corpora show that our proposed metrics correlate better with human judgments than existing metrics.
Capturing the diversity of multilingual societies
Authors: Thomas Louf, David Sanchez, Jose J. Ramasco
Date: 2021-05-06 00:00:00
Tasks: Diversity

Cultural diversity encoded within languages of the world is at risk, as many languages have become endangered in the last decades in a context of growing globalization. To preserve this diversity, it is first necessary to understand what drives language extinction, and which mechanisms might enable coexistence. Here, we study language shift mechanisms using theoretical and data-driven perspectives. A large-scale empirical analysis of multilingual societies using Twitter and census data yields a wide diversity of spatial patterns of language coexistence. It ranges from a mixing of language speakers to segregation with multilinguals on the boundaries of disjoint linguistic domains. To understand how these different states can emerge and, especially, become stable, we propose a model in which language coexistence is reached when learning the other language is facilitated and when bilinguals favor the use of the endangered language. Simulations carried out in a metapopulation framework highlight the importance of spatial interactions arising from people mobility to explain the stability of a mixed state or the presence of a boundary between two linguistic regions. Further, we find that the history of languages is critical to understand their present state.
Learn from Syntax: Improving Pair-wise Aspect and Opinion Terms Extractionwith Rich Syntactic Knowledge
Authors: Shengqiong Wu, Hao Fei, Yafeng Ren, Donghong Ji, Jingye Li
Date: 2021-05-06 00:00:00
Tasks: Boundary Detection, POS

In this paper, we propose to enhance the pair-wise aspect and opinion terms extraction (PAOTE) task by incorporating rich syntactic knowledge. We first build a syntax fusion encoder for encoding syntactic features, including a label-aware graph convolutional network (LAGCN) for modeling the dependency edges and labels, as well as the POS tags unifiedly, and a local-attention module encoding POS tags for better term boundary detection. During pairing, we then adopt Biaffine and Triaffine scoring for high-order aspect-opinion term pairing, in the meantime re-harnessing the syntax-enriched representations in LAGCN for syntactic-aware scoring. Experimental results on four benchmark datasets demonstrate that our model outperforms current state-of-the-art baselines, meanwhile yielding explainable predictions with syntactic knowledge.
Towards General Natural Language Understanding with Probabilistic Worldbuilding
Authors: Abulhair Saparov, Tom M. Mitchell
Date: 2021-05-06 00:00:00
Tasks: Natural Language Understanding, Question Answering, Semantic Parsing

We introduce the Probabilistic Worldbuilding Model (PWM), a new fully-symbolic Bayesian model of semantic parsing and reasoning, as a first step in a research program toward more domain- and task-general NLU and AI. Humans create internal mental models of their observations which greatly aid in their ability to understand and reason about a large variety of problems. In PWM, the meanings of sentences, acquired facts about the world, and intermediate steps in reasoning are all expressed in a human-readable formal language, with the design goal of interpretability. PWM is Bayesian, designed specifically to be able to generalize to new domains and new tasks. We derive and implement an inference algorithm that reads sentences by parsing and abducing updates to its latent world model that capture the semantics of those sentences, and evaluate it on two out-of-domain question-answering datasets: (1) ProofWriter and (2) a new dataset we call FictionalGeoQA, designed to be more representative of real language but still simple enough to focus on evaluating reasoning ability, while being robust against heuristics. Our method outperforms baselines on both, thereby demonstrating its value as a proof-of-concept.
A Unified Pre-training Framework for Conversational AI
Authors: Siqi Bao, Bingjin Chen, Huang He, Xin Tian, Han Zhou, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Yingzhan Lin
Date: 2021-05-06 00:00:00
Tasks: Chatbot, Interactive Evaluation of Dialog, Response Generation

In this work, we explore the application of PLATO-2 on various dialogue systems, including open-domain conversation, knowledge grounded dialogue, and task-oriented conversation. PLATO-2 is initially designed as an open-domain chatbot, trained via two-stage curriculum learning. In the first stage, a coarse-grained response generation model is learned to fit the simplified one-to-one mapping relationship. This model is applied to the task-oriented conversation, given that the semantic mappings tend to be deterministic in task completion. In the second stage, another fine-grained generation model and an evaluation model are further learned for diverse response generation and coherence estimation, respectively. With superior capability on capturing one-to-many mapping, such models are suitable for the open-domain conversation and knowledge grounded dialogue. For the comprehensive evaluation of PLATO-2, we have participated in multiple tasks of DSTC9, including interactive evaluation of open-domain conversation (Track3-task2), static evaluation of knowledge grounded dialogue (Track3-task1), and end-to-end task-oriented conversation (Track2-task1). PLATO-2 has obtained the 1st place in all three tasks, verifying its effectiveness as a unified framework for various dialogue systems.
Quantitative Evaluation of Alternative Translations in a Corpus of Highly Dissimilar Finnish Paraphrases
Authors: Li-Hsin Chang, Sampo Pyysalo, Jenna Kanerva, Filip Ginter
Date: 2021-05-06 00:00:00
Proceeding: MoTra (NoDaLiDa) 2021 5
Tasks: Translation

In this paper, we present a quantitative evaluation of differences between alternative translations in a large recently released Finnish paraphrase corpus focusing in particular on non-trivial variation in translation. We combine a series of automatic steps detecting systematic variation with manual analysis to reveal regularities and identify categories of translation differences. We find the paraphrase corpus to contain highly non-trivial translation variants difficult to recognize through automatic approaches.
XeroAlign: Zero-Shot Cross-lingual Transformer Alignment
Authors: Milan Gritta, Ignacio Iacobacci
Date: 2021-05-06 00:00:00
Proceeding: Findings (ACL) 2021 8
Tasks: Multilingual NLP, Natural Language Understanding, Sentence, Sentence Embeddings, text-classification, Text Classification, XLM-R

The introduction of pretrained cross-lingual language models brought decisive improvements to multilingual NLP tasks. However, the lack of labelled task data necessitates a variety of methods aiming to close the gap to high-resource languages. Zero-shot methods in particular, often use translated task data as a training signal to bridge the performance gap between the source and target language(s). We introduce XeroAlign, a simple method for task-specific alignment of cross-lingual pretrained transformers such as XLM-R. XeroAlign uses translated task data to encourage the model to generate similar sentence embeddings for different languages. The XeroAligned XLM-R, called XLM-RA, shows strong improvements over the baseline models to achieve state-of-the-art zero-shot results on three multilingual natural language understanding tasks. XLM-RA's text classification accuracy exceeds that of XLM-R trained with labelled data and performs on par with state-of-the-art models on a cross-lingual adversarial paraphrasing task.
Rethinking Search: Making Domain Experts out of Dilettantes
Authors: Donald Metzler, Yi Tay, Dara Bahri, Marc Najork
Date: 2021-05-05 00:00:00
Tasks: Information Retrieval, Question Answering, Retrieval

When experiencing an information need, users want to engage with a domain expert, but often turn to an information retrieval system, such as a search engine, instead. Classical information retrieval systems do not answer information needs directly, but instead provide references to (hopefully authoritative) answers. Successful question answering systems offer a limited corpus created on-demand by human experts, which is neither timely nor scalable. Pre-trained language models, by contrast, are capable of directly generating prose that may be responsive to an information need, but at present they are dilettantes rather than domain experts -- they do not have a true understanding of the world, they are prone to hallucinating, and crucially they are incapable of justifying their utterances by referring to supporting documents in the corpus they were trained over. This paper examines how ideas from classical information retrieval and pre-trained language models can be synthesized and evolved into systems that truly deliver on the promise of domain expert advice.
ADAM: A Sandbox for Implementing Language Learning
Authors: Ryan Gabbard, Deniz Beser, Jacob Lichtefeld, Joe Cecil, Mitch Marcus, Sarah Payne, Charles Yang, Marjorie Freedman
Date: 2021-05-05 00:00:00
Tasks: Language Acquisition

We present ADAM, a software system for designing and running child language learning experiments in Python. The system uses a virtual world to simulate a grounded language acquisition process in which the language learner utilizes cognitively plausible learning algorithms to form perceptual and linguistic representations of the observed world. The modular nature of ADAM makes it easy to design and test different language learning curricula as well as learning algorithms. In this report, we describe the architecture of the ADAM system in detail, and illustrate its components with examples. We provide our code.
Reconstructing shared dynamics with a deep neural network
Authors: Zsigmond Benkő, Zoltán Somogyvári
Date: 2021-05-05 00:00:00
Tasks: Time Series, Time Series Analysis

Determining hidden shared patterns behind dynamic phenomena can be a game-changer in multiple areas of research. Here we present the principles and show a method to identify hidden shared dynamics from time series by a two-module, feedforward neural network architecture: the Mapper-Coach network. We reconstruct unobserved, continuous latent variable input, the time series generated by a chaotic logistic map, from the observed values of two simultaneously forced chaotic logistic maps. The network has been trained to predict one of the observed time series based on its own past and conditioned on the other observed time series by error-back propagation. It was shown, that after this prediction have been learned successfully, the activity of the bottleneck neuron, connecting the mapper and the coach module, correlated strongly with the latent shared input variable. The method has the potential to reveal hidden components of dynamical systems, where experimental intervention is not possible.