pwc-archive/papers-with-abstracts huggingface dataset:
An Investigation of the (In)effectiveness of Counterfactually Augmented Data
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Diversity, Natural Language Understanding

While pretrained language models achieve excellent performance on natural language understanding benchmarks, they tend to rely on spurious correlations and generalize poorly to out-of-distribution (OOD) data. Recent work has explored using counterfactually-augmented data (CAD)---data generated by minimally perturbing examples to flip the ground-truth label---to identify robust features that are invariant under distribution shift. However, empirical results using CAD for OOD generalization have been mixed. To explain this discrepancy, we draw insights from a  linear Gaussian model and demonstrate the  pitfalls of CAD. Specifically, we show that (a) while CAD is effective at identifying robust features, it may prevent the model from learning unperturbed robust features; and (b) CAD may exacerbate existing spurious correlations in the data. On two crowdsourced CAD datasets, our results  show that the lack of perturbation diversity limits their effectiveness on OOD generalization, calling for innovative crowdsourcing procedures to elicit diverse perturbation of examples.
EncT5: Fine-tuning T5 Encoder for Discriminative Tasks
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR August 2021 8
Tasks: Decoder

Encoder-decoder transformer architectures have become popular recently with the advent of T5 models. While they demonstrate impressive performance on benchmarks such as GLUE (Wang et al., 2019), it is not clearly evident if the proposed encoder-decoder architecture is the most efficient for fine-tuning on downstream discriminative tasks. In this work, we study fine-tuning pre-trained encoderdecoder models such as T5. Particularly, we propose EncT5 as a way to efficiently finetune pre-trained encoder-decoder T5 models for classification and regression tasks by using only the encoder layers. Our experimental results show that EncT5 with less than half of the parameters of T5 performs similarly to T5 models on GLUE benchmark. We believe our proposed approach can be easily applied to any pre-trained encoder-decoder model.
Reinforce Attack: Adversarial Attack against BERT with Reinforcement Learning
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR August 2021 8
Tasks: Adversarial Attack, Adversarial Text, reinforcement-learning, Reinforcement Learning, Reinforcement Learning (RL), Semantic Similarity, Semantic Textual Similarity, Sentence

Adversarial  attacks  against  textual  data  has been drawing increasing attention in both the NLP and security domains. Current successful attack methods for text typically consist of two stages: word importance ranking and word replacement.  The first stage is usually achieved by masking each word in the sentence one at a time and obtaining the resulting output probability of the target model.  The second stage involves finding synonyms to replace “vulnerable” words by the order of ranking. In this paper, we first explore the effects of employing the model explanation tool LIME to generate word importance ranking, which has the advantage of taking the local information around the word into account to obtain word importance scores.  We then propose Reinforce Attack, a reinforcement learning (RL) based framework to  generate  adversarial  text.   Notably,  the  attack  process  is  controlled  by  a  reward  function rather than heuristics as in previous methods  to  encourage  higher  semantic  similarity and  lower  query  costs. Through  automatic and  human  evaluations,   we  show  that  our LIME + Reinforce Attack method achieves better or comparable attack success rate against other state-of-the-art attack frameworks, while the  generated  samples  preserve  significantly higher semantic similarity.
Adapting Multilingual Models for Code-Mixed Translation using Back-to-Back Translation
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR August 2021 8
Tasks: Data Augmentation, Machine Translation, Translation

In this paper, we explore the problem of translating code-mixed sentences to an equivalent monolingual form. The scarcity of gold standard code-mixed to pure language parallel data makes it difficult to train a translation model that can perform this task reliably. Prior work has addressed the paucity of parallel data with data augmentation techniques. Such techniques rely heavily on external resources, which make the systems difficult to train and scale effectively for multiple languages. We present a simple yet highly effective training scheme for adapting multilingual models to the task of code-mixed translation. Our method eliminates the dependence on external resources by creating synthetic data from a novel two-stage back-translation approach that we propose. We show substantial improvement in translation quality (measured through BLEU), beating existing prior work by up to +3.8 BLEU on code-mixed Hi$\rightarrow$En, Mr$\rightarrow$En, and Bn$\rightarrow$En tasks. On the LinCE Machine Translation leader board, we achieve the highest score for code-mixed Es$\rightarrow$En, beating existing best baseline by +6.5 BLEU, and our own stronger baseline by +1.1 BLEU.
CTRLsum: Towards Generic Controllable Text Summarization
Authors: Anonymous
Date: 2021-08-17 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Articles, Text Summarization

Current summarization systems yield generic summaries that are disconnected from users' preferences and expectations. To address this limitation, we present CTRLsum, a generic framework to control generated summaries through a set of keywords. During training keywords are extracted automatically without requiring additional human annotations. At test time CTRLsum features a control function to map control signal to keywords; through engineering the control function, the same trained model is able to be applied to control summaries on various dimensions, while neither affecting the model training process nor the pretrained models.  We additionally explore the combination of keywords and text prompts for more control tasks. Experiments demonstrate the effectiveness of CTRLsum on three domains of summarization datasets and five control tasks: (1) entity-centric and (2) length-controllable summarization, (3) contribution summarization on scientific papers, (4) invention purpose summarization on patent filings, and (5) question-guided summarization on news articles. Moreover, when used in a standard, unconstrained summarization setting, CTRLsum is comparable or better than the state-of-the-art systems.
ALLWAS: Active Learning on Language models in WASserstein space
Authors: Anonymous
Date: 2021-07-17 00:00:00
Proceeding: ACL ARR July 2021 7
Tasks: Active Learning, text-classification, Text Classification

Active learning has emerged as a standard paradigm in areas with scarcity of labeled training data, such as in the medical domain. Language models have emerged as the prevalent choice of several natural language tasks due to the performance boost offered by these models. However, in several domains, such as medicine, the scarcity of labeled training data is a common issue. Also, these models may not work well in cases where class imbalance is prevalent. Active learning may prove helpful in these cases to boost the performance with a limited label budget. To this end, we propose a novel method using sampling techniques based on submodular optimization and optimal transport for active Learning in language models, dubbed ALLWAS. We construct a sampling strategy based on submodular optimization of the designed objective in the gradient domain. Furthermore, to enable learning from few samples, we propose a novel strategy for sampling from the Wasserstein barycenters. Our empirical evaluations on standard benchmark datasets for text classification show that our methods perform significantly better (> 20% relative increase in some cases) than existing approaches for active learning on language models.
How to Do Human Evaluation: Best Practices for User Studies in NLP
Authors: Anonymous
Date: 2021-07-17 00:00:00
Proceeding: ACL ARR July 2021 7
Tasks: Explanation Generation, Machine Translation

Many research topics in natural language processing (NLP), such as explanation generation, dialog modeling or machine translation, require evaluation that goes beyond standard metrics like accuracy or F1 score toward a more human-centered approach.
Therefore, understanding how to design user studies becomes increasingly important.
However, few comprehensive resources exist on planning, conducting and evaluating user studies for NLP, making it hard to get started for researchers without prior experience in the field of human evaluation.
In this paper, we summarize the most important aspects of user studies and their design and evaluation, providing direct links to NLP tasks and NLP specific challenges where appropriate.
We (i) outline general study design, ethical considerations, and factors to consider for crowdsourcing, (ii) discuss the particularities of user studies in NLP and provide starting points to select questionnaires, experimental designs and evaluation methods that are tailored to the specific NLP tasks.
Additionally, we offer examples with accompanying statistical evaluation code in R throughout, to bridge the gap between theoretical guidelines and practical applications.
The Role of Context in Detecting Previously Fact-Checked Claims
Authors: Anonymous
Date: 2021-07-17 00:00:00
Proceeding: ACL ARR July 2021 7
Tasks: Fact Checking, Misinformation

Recent years have seen the proliferation of disinformation and misinformation online, thanks to the freedom of expression on the Internet and to the rise of social media. Two solutions were proposed to address the problem: (i) manual fact-checking, which is accurate and credible, but slow and non-scalable, and (ii) automatic fact-checking, which is fast and scalable, but lacks explainability and credibility. With the accumulation of enough manually fact-checked claims, a middle-ground approach has emerged: checking whether a given claim has previously been fact-checked. This can be made automatically, and thus fast, while also offering credibility and explainability, thanks to the human fact-checking and explanations in the associated fact-checking article. This is a relatively new and understudied research direction, and here we focus on claims made in a political debate, where context really matters. Thus, we study the impact of modeling the context of the claim: both on the source side, i.e., in the debate, as well as on the target side, i.e., in the fact-checking explanation document. We do this by modeling the local context, the global context, as well as by means of co-reference resolution, and reasoning over the target text using Transformer-XH. The experimental results show that each of these represents a valuable information source, but that modeling the source-side context is more important, and can yield 10+ points of absolute improvement.
A Vector-Based Approach to Few-Shot Veracity Classification for Automated Fact-Checking
Authors: Anonymous
Date: 2021-07-17 00:00:00
Proceeding: ACL ARR July 2021 7
Tasks: Classification, Fact Checking, Language Modeling, Language Modelling, Sentence, Veracity Classification

As progress on automated fact-checking continues to be called, veracity classification has gained more attention. It is the task of predicting the veracity of a given claim by comparing it with retrieved pieces of evidence. One of the challenges for this task is to obtain manual annotations for large datasets, especially when it comes to new domains for which labelled data is unavailable in the first instance. In this paper, we describe a vector-based approach that achieves significant performance improvement on veracity classification in few-shot settings. Performance is compared with two competitive baselines: (1) fine-tuning BERT / RoBERTa, and (2) the state-of-the-art few-shot veracity classification approach leveraging language model perplexity with thresholds. Our approach first utilises sentence-BERT to get sentence vectors of claims and evidences. We then create a relation vector for each claim-evidences pairs, by applying absolute operation on their vector offsets. Experiments show significant improvements over the baselines.
A Survey on Geocoding: Algorithms and Datasets for Toponym Resolution
Authors: Anonymous
Date: 2021-07-17 00:00:00
Proceeding: ACL ARR May 2021 5
Tasks: Survey, Toponym Resolution

Geocoding, the task of converting unstructured text to structured spatial data, has recently seen progress thanks to a variety of new datasets, evaluation metrics, and machine-learning algorithms. We provide a survey to review, organize and analyze recent work on geocoding (also known as toponym resolution) where the text is matched to geospatial coordinates and/or ontologies. We summarize the findings of this research and suggest some promising directions for future work.
Using Language Models on Low-end Hardware
Authors: Anonymous
Date: 2021-07-17 00:00:00
Proceeding: ACL ARR July 2021 7
Tasks: Classification, Language Modeling, Language Modelling, Multi-Label Classification, MUlTI-LABEL-ClASSIFICATION, text-classification, Text Classification

This paper evaluates the viability of using fixed language models for training text classification networks on low-end hardware. We combine language models with a CNN architecture and put together a comprehensive benchmark with 8 datasets covering single and multi-label classification of topic, sentiment, and genre. Our observations are distilled into a list of trade-offs, concluding that there are scenarios, where not fine-tuning a language model yields competitive effectiveness at faster training, requiring only a quarter of the memory compared to fine-tuning.
Rethinking Offensive Text Detection as a Multi-Hop Reasoning Problem
Authors: Anonymous
Date: 2021-07-17 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Text Detection

We introduce the task of implicit offensive language detection in dialogues, where a statement may have either an offensive or unoffensive interpretation, depending on the listener and context. We argue that inference is crucial for understanding this broader set of offensive utterances, and create a dataset featuring chains of reasoning to describe how an offensive interpretation may be reached. Experiments show that state-of-the-art methods of offense classification perform poorly on this task, achieving less than 0.12 average accuracy. We explore the use of pre-trained entailment models % to score links as part of a multi-hop approach to the problem, showing improved accuracy in most situations.  We discuss the feasibility of our approach and the types of external knowledge necessary to support it.
AlephBERT: Pre-training and End-to-End Language Models Evaluation from Sub-Word to Sentence Level
Authors: Anonymous
Date: 2021-07-17 00:00:00
Proceeding: ACL ARR July 2021 7
Tasks: Sentence

Large Pre-trained Language Models (PLMs) have become ubiquitous in the development of language understanding technology and lie at the heart of many artificial intelligence advances. While advances reported for English using PLMs are unprecedented, reported  advances using PLMs in Hebrew are few and far between. The problem is twofold. First, Hebrew resources for training large language models are not at the same order of  magnitude as their English counterparts.  Second, there are no accepted tasks and  benchmarks to evaluate the progress of Hebrew PLMs on, and in particular, evaluation on sub-word (morphological) tasks. We aim to remedy both aspects. We present AlephBERT, a large PLM for Modern Hebrew, trained on larger vocabulary and a larger dataset  than any Hebrew PLM before. Moreover, we introduce a novel language-agnostic architecture that extracts all of the sub-word  morphological segments encoded in contextualized word embedding vectors. Utilizing this new morphological component we offer a new PLM evaluation pipeline of multiple Hebrew tasks and benchmarks, that cover word-level, sub-word level and sentence level tasks. With AlephBERT we achieve state-of-the-art results compared against contemporary baselines. We make our AlephBERT model and evaluation pipeline publicly available, providing a single point of entry for evaluating and comparing Hebrew PLMs.
Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation
Authors: Anonymous
Date: 2021-07-17 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Machine Translation, Sentence, Synthetic Data Generation, Translation

The widespread online communication in a modern multilingual world has provided opportunities to blend more than one language (aka. code-mixed language) in a single utterance. This has resulted a formidable challenge for the computational models due to the scarcity of annotated data and presence of noise. A potential solution to mitigate the data scarcity problem in low-resource setup is to leverage existing data in resource-rich language through translation. In this paper, we tackle the problem of code-mixed (Hinglish and Bengalish) to English machine translation. First, we synthetically develop HINMIX, a parallel corpus of Hinglish to English,  with ~5M sentence pairs. Subsequently, we propose JAMT, a robust perturbation based joint-training model that learns to handle noise in the real-world code-mixed text by parameter sharing across clean and noisy words. Further, we show the adaptability of JAMT in a zero-shot setup for Bengalish to English translation. Our evaluation and comprehensive analyses qualitatively and quantitatively demonstrate the superiority of JAMT over state-of-the-art code-mixed and robust translation methods.
EIDER: Evidence-enhanced Document-level Relation Extraction
Authors: Anonymous
Date: 2021-07-17 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Document-level Relation Extraction, Relation, Relation Extraction, Sentence

Document-level relation extraction (DocRE) aims at extracting the semantic relations among entity pairs in a document. In DocRE, a subset of the sentences in a document, called the evidence sentences, might be sufficient for predicting the relation between a specific entity pair. To make better use of the evidence sentences, in this paper, we propose a three-stage evidence-enhanced DocRE framework called Eider consisting of joint relation and evidence extraction, evidence-centered relation extraction (RE), and fusion of extraction results. We first jointly train an RE model with a simple and memory-efficient evidence extraction model. Then, we construct pseudo documents based on the extracted evidence sentences and run the RE model again. Finally, we fuse the extraction results of the first two stages using a blending layer and make a final prediction. Extensive experiments show that our proposed framework achieves state-of-the-art performance on the DocRED dataset, outperforming the second-best method by 1.37/1.26 Ign F1/F1. In particular, Eider-RoBERTa$_\text{large}$ significantly improves the performance on entity pairs requiring co-reference and multi-hop reasoning by 1.98/2.08 F1, respectively, which cover around 75\% of the cross-sentence samples.
Predicting Visual Futures with Image Captioning and Pre-Trained Language Models
Authors: Anonymous
Date: 2021-06-16 00:00:00
Proceeding: ACL ARR Jun 2021 6
Tasks: Image Captioning

The task of visual forecasting deals with predicting future events from a sequence of input images. Purely pixel-based approaches find this challenging due to the presence of abstract concepts and temporal events at different timescales. In this paper, we present an approach that combines image captioning with pre-trained language models to predict visual futures. By leveraging language as an intermediate medium, our model is able to perform more effective temporal reasoning on two different tasks -- visual story cloze and action forecasting. Despite making the final predictions using only the generated captions, our approach outperforms state-of-the-art systems by $4\%$ and $6\%$ respectively on the two tasks. We find that our model consistently picks images/actions that are semantically relevant to the given image sequence instead of simply relying on visual similarity.
Restoring Hebrew Diacritics Without a Dictionary
Authors: Anonymous
Date: 2021-06-16 00:00:00
Proceeding: ACL ARR Jun 2021 6

We demonstrate that it is feasible to diacritize Hebrew script without any human-curated resources other than plain diacritized text. We present NAKDIMON, a two-layer character level LSTM, that performs on par with much more complicated curation-dependent 
 systems, across a diverse array of modern Hebrew sources.
A Recipe For Arbitrary Text Style Transfer with Large Language Models
Authors: Anonymous
Date: 2021-06-16 00:00:00
Proceeding: ACL ARR October 2021 10
Tasks: Sentence, Sentence ReWriting, Style Transfer, Text Style Transfer, Zero-Shot Learning

In this paper, we leverage large language models (LLMs) to perform zero-shot text style transfer. We present a prompting method that we call augmented zero-shot learning, which frames style transfer as a sentence rewriting task and requires only a natural language instruction, without model fine-tuning or exemplars in the target style. Augmented zero-shot learning is simple and demonstrates promising results not just on standard style transfer tasks such as sentiment, but also on arbitrary transformations such as 'make this melodramatic' or 'insert a metaphor.'
So Different Yet So Alike! Constrained Unsupervised Text Style Transfer
Authors: Anonymous
Date: 2021-06-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Attribute, Generative Adversarial Network, Style Transfer, Text Style Transfer, Unsupervised Text Style Transfer

Transferring text from one domain to the other has seen tremendous progress in the recent past. However, these methods do not aim to explicitly maintain constraints such as similar text length, descriptiveness between the source and the translated text. To this end, we introduce two complementary cooperative losses to the generative adversarial network family. Here, both the generator and the critic reduce the contrastive and/or the classification loss aiming to satisfy the constraints. These losses allow lexical, syntactic, and domain-specific consistencies to persist across domains.  We demonstrate the effectiveness of our method over multiple benchmark datasets, both with single and multi-attribute transfers.  The complimentary cooperative losses also improve text quality across datasets as judged by current, automated generation and human evaluation metrics.
Augmented Neural Story Generation with Commonsense Inference
Authors: Anonymous
Date: 2021-06-16 00:00:00
Proceeding: ACL ARR Jun 2021 6
Tasks: Language Modeling, Language Modelling, Story Generation

Transformer-based language model approaches to automated story generation currently provide state-of-the-art results. However, they still suffer from plot incoherence when generating narratives over time, and critically lack basic commonsense reasoning. Furthermore, existing methods generally focus only on single-character stories, or fail to track characters at all. To improve the coherence of generated narratives and to expand the scope of character-centric narrative generation, we introduce Commonsense-inference Augmented neural StoryTelling (CAST), a framework for introducing commonsense reasoning into the generation process while modeling the interaction between multiple characters. We find that our CAST method produces significantly more coherent and on-topic two-character stories, outperforming baselines in dimensions including plot plausibility and staying on topic. We also show how the CAST method can be used to further train language models that generate more coherent stories and reduce computation cost.