pwc-archive/papers-with-abstracts huggingface dataset:
When a sentence does not introduce a discourse entity, Transformer-based models still often refer to it
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Negation, Sentence

Understanding longer narratives or participating in conversations requires tracking of discourse entities that have been mentioned. Indefinite noun phrases, such as 'a dog', frequently introduce discourse entities but this behavior is modulated by sentential operators such as negation. For example, 'a dog' in 'Arthur doesn't own a dog' does not introduce a discourse entity due to the presence of negation. In this work, we adapt the psycholinguistic assessment of language models paradigm to higher-level linguistic phenomena and introduce an English evaluation suite that targets the knowledge of the interactions between sentential operators and indefinite noun phrases. We use this evaluation suite for a fine-grained investigation of the entity tracking abilities of the Transformer-based models GPT-2 and GPT-3. We find that while the models are to a certain extent sensitive to the interactions we investigate, they are all challenged by the presence of multiple noun phrases and their behavior is not systematic, which suggests that even models at the scale of GPT-3 do not fully acquire basic entity tracking abilities.
Conceptualizing Treatment Leakage in Text-based Causal Inference
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Causal Inference

Causal inference methods that control for text-based confounders are becoming increasingly important in the social sciences and other disciplines where text is readily available. However, these methods rely on a critical assumption that there is no treatment leakage: that is, the text contains only information about the confounder and no information about treatment assignment (leading to post-treatment bias). However, this assumption may be unrealistic in real-world situations involving text, as human language is rich and flexible. We first define the leakage problem, discussing the identification and estimation challenges it raises. We also discuss the conditions under which leakage can be addressed by removing the treatment-related signal from the text in a pre-processing step we define as \emph{text distillation}. Then, using simulation, we investigate the mechanics of treatment leakage on estimates of the average treatment effect (ATE).
Visual content classifier for cultural heritage repositories
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

This work presents a novel approach for the automatic creation of an aligned image / text training set for the generation of descriptions of the visual content of artworks. To do this, we develop a classification tool based on a mix of heuristic rules and deep learning. This classifier is able to identify statements that describe visual art content, out of complex cultural heritage text that contains a mix of many other types of information on context, medium, author, etc. Our results are very promising when tested on texts from the Museo del Prado collections.
Zombies Eat Brains, You are Safe: A Knowledge Infusion based Multitasking System for Sarcasm Detection in Meme
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Emotion Recognition, Sarcasm Detection

Sarcasm detection is, in itself, a challenging task in the field of Natural Language Processing (NLP), and the task even becomes more complex when the target is a meme. In this paper, we first hypothesize that sarcasm detection is closely associated with emotions present in the meme. We propose a deep learning-based multitask model to perform these two tasks in parallel, where sarcasm detection is the primary, whereas emotion recognition is considered as an auxiliary task. Furthermore, we propose a novel knowledge infusion (KI) method to get a sentiment-aware knowledge representation on top of our multitasking model. This sentiment-aware knowledge representation is obtained from a pre-trained parent model and subsequently, this representation is used via a novel Gating Mechanism to train our downstream multitasking model. For training and evaluation purposes, we created a large-scale dataset consisting of 7,416 sample Hindi memes as there was no readily available dataset for building such multimodal systems. We collect the Hindi memes from various domains, such as politics, religious, racist, and sexist, and manually annotate each instance with three sarcasm categories, i.e., (i) Not Sarcastic, ii) Mildly Sarcastic or iii) Highly Sarcastic ) and 13 fine-grained emotion classes. We demonstrate the effectiveness of our proposed work through extensive experiments. The experimental results show that our proposed system achieves a 64.48% macro F1-score, outperforming all the baseline models. Finally, we note that our proposed system is model agnostic and can be used with any downstream model in practice. We will make the resources and codes available.
Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Do language models have beliefs about the world? Dennett (1995) famously argues that even thermostats have beliefs, on the view that a belief is simply an informational state decoupled from any motivational state. In this paper, we discuss approaches to detecting when models have beliefs about the world, updating model beliefs, and visualizing beliefs graphically. Our main contributions include: (1) new metrics for evaluating belief-updating methods focusing on the logical consistency of beliefs, (2) a training objective for Sequential, Local, and Generalizing updates (SLAG) that improves the performance of learned optimizers for updating beliefs, and (3) the introduction of the belief graph, a new form of interface with language models showing the interdependencies between model beliefs. Our experiments suggest that models possess belief-like qualities to only a limited extent, but update methods can both fix incorrect model beliefs and greatly improve their consistency. Although off-the-shelf optimizers are surprisingly strong belief-updating baselines, our learned optimizers can outperform them in more difficult settings than have been considered in past work.
Improving Data Augmentation in Low-resource Question Answering with Active Learning in Multiple Stages
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Active Learning, Answer Generation, Data Augmentation, Question-Answer-Generation, Question Answering

Neural approaches have become very popular in the domain of Question Answering, however they require a large amount of annotated data. Furthermore, they often yield very good performance but only in the domain they were trained on. In this work we propose a novel approach that combines data augmentation via question-answer generation and active learning to improve performance in low resource settings, where the target domain is vastly different from the source domain. Furthermore, we investigate data augmentation via generation for question answering in three different low-resource settings relevant in practice and how this can be improved: 1) No labels for the target domain, 2) static, labelled data for the target domain and 3) an Active Learning approach with labels for the target domain provided by an expert. In all settings we assume sufficient amount of labelled data from the source domain is available. We perform extensive experiments in each of the above conditions. Our findings show that our novel approach, which combines data augmentation with active learning, boosts performances in the low-resource, domain-specific setting, allowing for low-labelling-effort question answering systems in new, specialized domains. They further demonstrate how to best utilize data augmentation to boost performance in these settings.
On the data requirements of probing
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Diagnostic

As large and powerful neural language models are developed, researchers have been increasingly interested in developing diagnostic tools to probe them. There are many papers with conclusions of the form ``observation $X$ is found in model $Y$'', using their own datasets with varying sizes. Larger probing datasets bring more reliability, but are also expensive to collect. There is yet to be a quantitative method for estimating reasonable probing dataset sizes. We tackle this omission in the context of comparing two probing configurations: after we have collected a small dataset from a pilot study, how many additional data samples are sufficient to distinguish two different configurations? We present a novel method to estimate the required number of data samples in such experiments and, across several case studies, we verify that our estimations have sufficient statistical power. Our framework helps to systematically construct probing datasets to diagnose neural NLP models.
Improving Coherence of Language Model Generation with Latent Semantic State
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Language Modeling, Language Modelling

Sentences generated by neural language models (LMs) often suffer from coherence errors: they describe events and situations inconsistent with the state of the world described by preceding text. We show that coherence errors can arise at multiple stages of LM computation, and describe a procedure for distinguishing errors in inferring state from errors in generating sentences. In models with correctable errors of the first type, we show that targeted supervision can address them. We introduce two procedures for using explicit representations of world state as auxiliary supervision. These procedures efficiently improve LM coherence, in some cases providing the benefits of 1,000-9,000 training examples with only 500 state annotations.
Towards Understanding Large-Scale Discourse Structures in Pre-Trained and Fine-Tuned Language Models
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

In this paper, we extend the line of BERTology work by focusing on the important, yet less explored, alignment of pre-trained and fine-tuned PLMs with large-scale discourse structures. We propose a novel approach to infer discourse information for arbitrarily long documents. In our experiments, we find that the captured discourse information is local and general, even across a collection of fine-tuning tasks. We compare the inferred discourse trees with supervised, distantly supervised and simple baselines to explore the structural overlap, finding that constituency discourse trees align well with supervised models, however, contain complementary discourse information.Lastly, we individually explore self-attention matrices to analyze the information redundancy. We find that similar discourse information is consistently captured in the same heads.
Measuring Context-Dependent Syntactic Information Across Layers
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1

Probing studies have extensively explored where in neural language models linguistic information is located. While probing classifiers are a common instrument to approach such questions, it is less clear what evaluation metrics to choose, how to compare probes, and which baselines to use. We identify angles from which the question how linguistic information is structured within a model can be approached and propose two new setups that fill the gap of explicitly modelling local information gain compared to the previous layer.We apply the new setups, along with two from the literature, to probe models for a syntactic property that explicitly needs context to be retrieved: part-of-speech tags that are not the most common for a specific token. We test the hypothesis that more information is retrieved in deeper layers than for the most common tags, and find that while this is often true, the manifestation varies among metrics and models in different languages.
Massive-scale Decoding for Text Generation using Lattices
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Text Generation

Conditional neural text generation models generate high-quality outputs, but often concentrate around a mode when what we really want is a diverse set of options. We present a search algorithm to construct lattices encoding a massive number of generation options. First, we restructure decoding as a best-first search, which explores the space differently than beam search and improves efficiency by avoiding pruning paths. Second, we revisit the idea of hypothesis recombination: we can identify pairs of similar generation candidates during search and merge them as an approximation. On both summarization and MT, we show that our algorithm encodes thousands of diverse options that remain grammatical and high-quality into one lattice. This algorithm provides a foundation for building downstream generation applications on top of massive-scale diverse outputs.
Forecasting COVID-19 Caseloads Using Unsupervised Embedding Clusters of Social Media Posts
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Sentence, Time Series, Time Series Analysis, Time Series Forecasting

We present a novel approach incorporating transformer-based language models into infectious disease modelling. Text-derived features are quantified by tracking high-density clusters of sentence-level representations of Reddit posts within specific US states' COVID-19 subreddits. We benchmark these clustered embedding features against features extracted from other high-quality datasets. In a threshold-classification task, we show that they outperform all other feature types at predicting upward trend signals, a significant result for infectious disease modelling in areas where epidemiological data is unreliable. Subsequently, in a time-series forecasting task, we fully utilise the predictive power of the caseload and compare the relative strengths of using different supplementary datasets as covariate feature sets in a transformer-based time-series model.
Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Cross-Lingual Transfer, Zero-Shot Cross-Lingual Transfer

Adapter modules have recently been used for efficient fine-tuning and language specialization of massively multilingual Transformers (MMTs), improving downstream zero-shot cross-lingual transfer. In this work, we propose orthogonal language and task adapters (dubbed orthoadapters) for cross-lingual transfer. They are trained to encode language- and task-specific information that is complementary (i.e., orthogonal) to the knowledge already stored in the pretrained MMT parameters. Our zero-shot transfer experiments, involving three tasks and 10 diverse languages, 1) point to the usefulness of orthoadapters in cross-lingual transfer, especially for the most complex NLI task, but also 2) indicate that the optimal (ortho)adapter configuration highly depends on the task and the target language at hand. We hope that our work will motivate a wider investigation of usefulness of orthogonality constraints in language- and task-specific fine-tuning of pretrained transformers.
Bilingual Lexicon Induction for Low-Resource Languages using Graph Matching via Optimal Transport
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Bilingual Lexicon Induction, Graph Matching, Information Retrieval, Machine Translation, Retrieval, Translation

Bilingual lexicons form a critical component of various NLP applications, including unsupervised and semisupervised machine translation and crosslingual information retrieval. In this work, we improve bilingual lexicon induction performance across 32 diverse language pairs with a graph-matching method based on optimal transport. The method is especially strong with very low amounts of supervision.
Learning to Borrow– Relation Representation for Without-Mention Entity-Pairs for Knowledge Graph Completion
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Entity Embeddings, Graph Embedding, Knowledge Graph Completion, Knowledge Graph Embedding, Knowledge Graphs, Link Prediction, Relation, Sentence

Prior work on integrating text corpora with knowledge graphs (KGs) to improve Knowledge Graph Embedding (KGE) have obtained good performance for entities that co-occur in sentences in text corpora. Such sentences (textual mentions of entity-pairs) are represented as Lexicalised Dependency Paths (LDPs) between two entities. However, it is not possible to represent relations between entities that do not co-occur in a single sentence using LDPs. In this paper, we propose and evaluate several methods to address this problem, where we \emph{borrow} LDPs from the entity pairs that co-occur in sentences in the corpus (i.e. \emph{with mentions} entity pairs) to represent entity pairs that do \emph{not} co-occur in any sentence in the corpus (i.e. \emph{without mention} entity pairs). We propose a supervised borrowing method, \emph{SuperBorrow}, that learns to score the suitability of an LDP to represent a without-mentions entity pair using pre-trained entity embeddings and contextualised LDP representations. Experimental results show that SuperBorrow improves the link prediction performance of multiple widely-used prior KGE methods such as TransE, DistMult, ComplEx and RotatE.
Open Domain Response Generation Guided by Retrieved Conversations
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Keyword Extraction, Response Generation, Retrieval

Open domain response generation is the task of creating a response givena user query in any topics/domain. Limited by context and referenceinformation, responses generated by current systems are often "bland"or generic.  In this paper, we combine a response generation model witha retrieval system that searches for relevant utterances and responses,and extracts keywords from the retrieved results to guide the responsegeneration.  Our model uses a keyword extraction module to extract twotypes of keywords in an unsupervised fashion: (1) keywords in thequery not found in the retrieved utterances (DIFFKEY),and (2) overlapping keywords among the retrieved responses(SIMKEY).  Given these keywords, we use a two-stage transformer thatfirst decides where to insert the keywords in the response, and thengenerates the full response given the location of the keywords. Thekeyword extraction module and the two-stage transformer are connected ina single network, and so our system is trained end-to-end.Experimental results on Cornell Movie-Dialog corpus, Douban and Weibodemonstrate that our model outperforms state-of-the-art systems in termsof ROUGE, relevance scores and human evaluation.  Source code of ourmodel is available at: ANONYMISED.
Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Misinformation

Detecting out-of-context media, such as "miscaptioned" images on Twitter, is a relevant problem, especially in domains of high public significance. In this work we aim to develop defenses against such misinformation for the topics of Climate Change, COVID-19, and Military Vehicles. We first present a large-scale multimodal dataset with over 884k tweets relevant to these topics. Next, we propose a detection method, based on the state-of-the-art CLIP model, that leverages automatically generated hard image-text mismatches. While this approach works well on our automatically constructed out-of-context tweets, we aim to validate its usefulness on data representative of the real world. Thus, we test it on a set of human-generated fakes, created by mimicking in-the-wild misinformation. We achieve an 11% detection improvement in a high precision regime over a strong baseline. Finally, we share insights about our best model design and analyze the challenges of this emerging threat.
Hardness Masking via Auto-Regressive Language Model
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Language Modeling, Language Modelling, Large Language Model, model

Pre-trained masked language models have achieved tremendous success in natural language processing.    Most of these methods rely on recovering randomly masked tokens, which is in general not as good as when tokens are masked based on how well the model can predict. However, it is costly for a large-scale model to self-identify tokens that it still struggles to predict.  On the other hand, we observe that a smaller language model can often effectively find what a large model fails to learn. Inspired by this observation, we propose to leverage a compact bi-directional auto-regressive language model to dynamically discover tokens that a large language model has not learned well and guide its training via hardness masking. Comprehensive experiments demonstrate that our masking method can effectively boost the performance of pre-trained language models on general language understanding benchmarks.
Hybrid Semantic Type Representation for Zero-Shot Event Extraction
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Event Extraction, HTR, Vocal Bursts Type Prediction, Zero-shot Event Extraction

Event extraction is a significant task in natural language processing.   However, it is labor-intensive to get annotation when generalizing to new event types and ontologies. In this paper, we propose the HTR (Hybrid Type Representation) framework for zero-shot event extraction. We make a distinction of the abstraction level between events and roles, analyze role semantics, and propose a new representation approach, LRDB (label-related description-based), which is effective for both argument classification and collaboration with trigger extraction. We conduct extensive evaluation on ACE2005 dataset and achieve state-of-the-art.
Testing the Ability of Language Models to Interpret Figurative Language
Authors: Anonymous
Date: 2022-01-16 00:00:00
Proceeding: ACL ARR January 2022 1
Tasks: Open-Ended Question Answering

Figurative and metaphorical language are commonplace in discourse, and figurative expressions play an important role in communication and cognition. However, figurative language has been a relatively under-studied area in NLP, and it remains an open question to what extent modern language models can interpret nonliteral phrases. To address this question, we introduce Fig-QA, a Winograd-style nonliteral language understanding task consisting of correctly interpreting paired figurative phrases with divergent meanings. We evaluate the performance of several state-of-the-art language models on this task, and find that although language models achieve performance significantly over chance, they still fall  short of human performance, particularly in zero- or few-shot settings. This suggests that further work is needed to improve the nonliteral reasoning capabilities of language models.