pwc-archive/papers-with-abstracts huggingface dataset:
A Simple Unsupervised Approach for Coreference Resolution using Rule-based Weak Supervision
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: coreference-resolution, Coreference Resolution

Labeled data for the task of Coreference Resolution is a scarce resource, requiring significant human effort. While state-of-the-art coreference models rely on such data, we propose an approach that leverages an end-to-end neural model in settings where labeled data is unavailable. Specifically, using weak supervision, we transfer the linguistic knowledge encoded by Stanford’s rule-based coreference system to the end-to-end model, which jointly learns rich, contextualized span representations and coreference chains. Our experiments on the English OntoNotes corpus demonstrate that our approach effectively benefits from the noisy coreference supervision, producing an improvement over Stanford’s rule-based system (+3.7 F$_1$) and outperforming the previous best unsupervised model (+0.9 F$_1$). Additionally, we validate the efficacy of our method on two other datasets: PreCo and Litbank (+2.5 and +4 F$_1$ on Stanford's system, respectively).
Are We NER Yet? Measuring the Impact of ASR Errors on Named Entity Recognition in Spontaneous Conversation Transcripts
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER

Transcriptions of spontaneous human conversations present a significant obstacle for traditional NER models trained on prescriptive written language. The lack of grammatical structure of spoken utterances, combined with word errors introduced by the ASR, makes downstream NLP tasks challenging. In this paper, we examine the impact of ASR errors on the ability of NER models to recover entity mentions from transcripts of spontaneous human conversations in English. We experimentally compare several commercial ASR systems paired with state-of-the-art NER models. We use both publicly available benchmark datasets (Switchboard Named Entity Corpus, SWNE), and the proprietary, real-life dataset of gold (human-transcribed) phone conversation transcripts. To measure the performance of NER models on ASR transcripts, we introduce a new method of token alignment between transcripts. Our findings unequivocally show that NER models trained on the written language struggle when processing transcripts of spontaneous human conversations. The presence of ASR errors only exacerbates the problem.
An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Knowledge Graphs

Large knowledge graphs have been shown to benefit zero-shot evaluation of downstream tasks, through continual pre-training of language models. Yet, little is known about how to optimally learn from this knowledge, and what is the impact of the resulting models on different task partitions. This paper studies the effect of model architectures, loss functions, and knowledge subsets on the generalization of zero-shot models across task partitions. Our experiments show that data size, model size, model architecture, and loss function all play an important role in the accuracy and generalizability of the models. Most of the improvement occurs on questions with short answers and dissimilar answer candidates, which corresponds to the characteristics of the data used for pre-training. These findings inform future work that uses self-supervision with large knowledge graphs in order to create generalizable commonsense reasoning agents.
Aligning Annotation with User Perceptions for the Evaluation of Voice-Based Intelligent Personal Digital Assistants
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Evaluating voice-based Intelligent Personal Digital Assistants (or IPDAs) like Alexa, Google Assistant, and Siri is a challenge, as they offer increasingly complex interactions and are found in ever-increasing contexts. The problem is compounded by users expecting more sophisticated, human-like behavior of IPDAs. Building an automated evaluation mechanism requires high quality ground truth data on user-IPDA interaction success, which incorporates these diverse and changing user expectations. Ignoring them may lead to divergence between user perceptions and what IPDAs are designed to optimize. To address this problem, we describe Response Quality (RQ) annotation and introduce response categories that incorporate the user’s perspective. To validate these categories, we present an in-lab study — the first on a large-scope conversational system that covers hundreds of domains — that tests the hypothesis that users judge human-like responses from IPDAs (like admitted fallibility and relevance) more favorably than non-human-like ones. We find support for this hypothesis, confirming that (1) Generic Fails are highly rated by older users in particular, (2) Irrelevant Responses are rated as poorly as Generic Fails and (3) disambiguation responses are highly rated (with caveats). We then baseline RQ ratings against user ratings to improve RQ’s alignment with user perceptions.
From the Detection of Toxic Spans in Online Discussions to the Analysis of Toxic-to-Civil Transfer
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Toxic Spans Detection

We study the task of toxic spans detection, which concerns the detection of the spans that make a text toxic, when detecting such spans is possible. We introduce a dataset for this task, ToxicSpans, which we release publicly. By experimenting with several methods, we show that sequence labeling models perform best, but methods that add generic rationale extraction mechanisms on top of classifiers trained to predict if a post is toxic or not are also surprisingly promising. Finally, we use ToxicSpans and systems trained on it, to provide further analysis of state-of-the-art toxic to non-toxic transfer systems, as well as human performance on that latter task. Our work highlights challenges in finer toxicity detection and mitigation.
Compact Token Representations with Contextual Quantization for Efficient Document Re-ranking
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Quantization, Re-Ranking

Transformer based re-ranking models can achieve  high search relevance through context-aware soft matching of query tokens with document tokens. To alleviate runtime complexity of such inference, previous work has adopted a late interaction architecture
with pre-computed contextual token representations at the cost of a large online storage. This paper proposes contextual quantization of token embeddings by decoupling document-specific and document-independent ranking contributions during codebook-based compression. This allows effective online decompression and embedding composition for better search relevance. This paper presents  an evaluation of the above compact token representation model in terms of relevance and space efficiency.
Perturbations in the Wild: Leveraging Human-Written Text Perturbations for Realistic Adversarial Attack and Defense
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Adversarial Attack

We proposes a novel algorithm, ANTHRO,  that inductively extracts over 600K human-written text perturbations in the wild and leverages them for realistic adversarial attack. Unlike existing character-based attacks which often deductively hypothesize a set of manipulation strategies, our work is grounded on actual observations from real-world texts. We find that adversarial texts generated by ANTHRO achieve the best trade-off between (1) attack success rate, (2) semantic preservation of the original text, and (3) stealthiness--i.e. indistinguishable from human writings hence harder to be flagged as suspicious. Specifically, our attacks accomplished around 83% and 91% attack success rates on BERT and RoBERTa, respectively. Moreover, it outperformed the TextBugger baseline with an increase of 50% and 40% in terms of semantic preservation and stealthiness when evaluated by both layperson and professional human workers. ANTHRO can further enhance a BERT classifier's performance in understanding different variations of human-written toxic texts via adversarial training when compared to the Perspective API. All source code will be released.
Repetition Facilitates Processing: The Processing Advantage of Construction Repetition in Dialogue
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Language Modeling, Language Modelling

Repetitions occur frequently in dialogue. This study focuses on the repetition of lexicalised constructions—i.e., recurring multi-word units—in English open domain spoken dialogues. We hypothesise that construction repetition is an efficient communication strategy that reduces processing effort, and make three predictions based on this hypothesis. Our three predictions are confirmed: repetitions facilitate the processing of constructions and of their linguistic context; facilitating effects are higher when repetitions accumulate, and lower when repetitions are less locally distributed. We measure reduction in processing effort using two surprisal-based measures and estimate surprisal with an adaptive neural language model. Our findings suggest that human-like patterns of repetitions can be learned implicitly by utterance generation models equipped with psycholinguistically motivated surprisal-based objectives and adaptation mechanisms.
Fine-Grained Controllable Text Generation Using Non-Residual Prompting
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Decoder, Text Generation

The introduction of immensely large Causal Language Models (CLMs) has rejuvenated the interest in open-ended text generation. However, controlling the generative process for these Transformer-based models is at large an unsolved problem. Earlier work has explored either plug-and-play decoding strategies, or more powerful but blunt approaches such as prompting. There hence currently exists a trade-off between fine-grained control, and the capability for more expressive high-level instructions. To alleviate this trade-off, we propose an encoder-decoder architecture that enables intermediate text prompts at arbitrary time steps. We propose a resource-efficient method for converting a pre-trained CLM into this architecture, and demonstrate its potential on various experiments, including the novel task of contextualized word inclusion. Our method provides strong results on multiple experimental settings, proving itself to be both expressive and versatile.
Transductive Learning for Abstractive News Summarization
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Articles, News Summarization, Transductive Learning

Pre-trained and fine-tuned news summarizers are expected to generalize to news articles unseen in the fine-tuning (training) phase. However, these articles often contain specifics, such as events and people, a summarizer could not learn about in training. This applies to scenarios such as when a news publisher trains a summarizer on dated news and wants to summarize incoming recent news. In this work, we explore the first application of transductive learning to summarization where we further fine-tune models on test set’s input. Specifically, we construct references for learning from article salient sentences and condition on the randomly masked articles. We show that this approach is also beneficial in the fine-tuning phase when extractive references are jointly predicted with abstractive ones in the training set. In general, extractive references are inexpensive to produce as they are automatically created without human effort. We show that our approach yields state-of-the-art results on CNN/DM and NYT datasets, for instance, more than 1 ROUGE-L points improvement on the former. Moreover, we show the benefits of transduction from dated to more recent CNN news. Finally, through human and automatic evaluation, we demonstrate improvements in summary abstractiveness and coherence.
Training Dynamics for Text Summarization Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Hallucination, News Summarization, Text Summarization

Pre-trained language models (e.g. BART) have shown impressive results when fine-tuned on large summarization datasets. However, little is understood about this fine-tuning process, including what knowledge is retained from pre-training models or how content selection and generation strategies are learnt across iterations. In this work, we analyze the training dynamics for generation models, focusing on news summarization. Across different datasets (CNN/DM, XSum, MediaSum) and model behaviors (content selection, abstractiveness, hallucination), we study what the model learns at different stages of its fine-tuning process. We find that properties such as copy behavior and content selection are learnt earlier in the training process and these observations are robust across domains. On the other hand, factual errors, such as hallucination of unsupported facts, are learnt in the later stages, and this behavior is more varied across domains. Based on these observations, we demonstrate two techniques for modifying training: first, disregarding high-loss tokens that are challenging to learn and second, disregarding low-loss tokens that are learnt very quickly. We show that these simple modifications can help achieve different goals, such as improving factuality or improving abstractiveness.
SHIELD: Defending Textual Neural Networks against Black-Box Adversarial Attacks with Stochastic Multi-Expert Patcher
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Even though several methods have proposed to defend textual neural network (NN) models against black-box adversarial attacks, they often defend against a specific text perturbation strategy and/or require re-training the models from scratch. This leads to a lack of generalization in practice and redundant computation. In particular, the state-of-the-art transformer models (e.g., BERT, RoBERTa) require great time and computation resources. By borrowing an idea from software engineering, in order to address these limitations, we propose a novel algorithm, SHIELD, which modifies and re-trains only the last layer of a textual NN, and thus it "patches" and "transforms" the NN into a stochastic weighted ensemble of multi-expert prediction heads. Considering that most of current black-box attacks rely on iterative search mechanisms to optimize their adversarial perturbations, SHIELD confuses the attackers by automatically utilizing different weighted ensembles of predictors depending on the input. In other words, SHIELD breaks a fundamental assumption of the attack, which is a victim NN model remains constant during an attack. By conducting comprehensive experiments, we demonstrate that all of CNN, RNN, BERT, and RoBERTa-based textual NNs, once patched by SHIELD, exhibit a relative enhancement of 15%--70% in accuracy on average against 14 different black-box attacks, outperforming 6 defensive baselines across 3 public datasets. All codes are to be released.
Sentence Selection Strategies for Distilling Word Embeddings from BERT
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Sentence, Word Embeddings

Many applications crucially rely on the avail-ability of high-quality word vectors. To learn such representations, several strategies based on language models have been proposed in recent years. While effective, these methods typically rely on a large number of contextualised vectors for each word, which makes them impractical. In this paper, we investigate whether similar results can be obtained when only a few contextualised  representations  of  each  word can be used.  To this end, we analyze a range of strategies for selecting the most informative sentences.  Our results show that with a careful selection strategy, high-quality word vectors can be learned from as few as 5 to 10 sentences.
Improving GPT-3 after deployment with a dynamic memory of feedback
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Large LMs such as GPT-3 while powerful, are not immune to mistakes, but are prohibitively costly to retrain. One failure mode is misinterpreting a user's instruction (e.g., GPT-3 interpreting "What word is similar to `good'?" to mean a homonym, while the user intended a synonym). Our goal is to allow users to correct such errors directly through interaction -- without retraining. Our approach is to pair GPT-3 with a growing memory of cases where the model misunderstood the user's intent and was provided with feedback, clarifying the instruction. Given a new query, our memory-enhanced GPT-3 uses feedback from similar, prior queries to enrich the prompt. Through simple proof-of-concept experiments, we demonstrate how a user can interactively teach a deployed GPT-3, doubling its accuracy on basic lexical tasks (e.g., generate a synonym) where users query in different, novel (often misunderstood) ways. In such scenarios, memory helps avoid  repeating similar past mistakes. Our simple idea is a first step towards strengthening deployed models, potentially broadening their utility.
Image Retrieval from Contextual Descriptions
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Image Retrieval, Retrieval

The ability to integrate context, including perceptual and temporal cues, plays a pivotal role in grounding the meaning of a linguistic utterance. In order to measure to what extent current vision-and-language models master this ability, we devise a new multimodal challenge, Image Retrieval from Contextual Descriptions (ImageCoDe). In particular, models are tasked with retrieving the correct image from a set of 10 minimally contrastive candidates based on a contextual description.
As such, each description contains only the details that help distinguish between images.
Because of this, descriptions tend to be complex in terms of syntax and discourse and require drawing pragmatic inferences. 
Images are sourced from both static pictures and video frames.
We benchmark several state-of-the-art models, including both cross-encoders such as ViLBERT and bi-encoders such as CLIP, on ImageCoDe.
Our results reveal that these models dramatically lag behind human performance: the best variant achieves an accuracy of 20.9 on video frames and 59.4 on static pictures, compared with 90.8 in humans.
Furthermore, we experiment with new model variants that are better equipped to incorporate visual and temporal context into their representations, which achieve modest gains. 
Our hope is that ImageCoDE will foster progress in grounded language understanding by encouraging models to focus on fine-grained visual differences.
Good Examples Make A Faster Learner: Simple Demonstration-based Learning for Low-resource NER
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Domain Adaptation, Few-Shot Text Classification, In-Context Learning, named-entity-recognition, Named Entity Recognition, Named Entity Recognition (NER), NER, Sentence, text-classification, Text Classification

Recent advances in prompt-based learning have shown strong results on few-shot text classification by using cloze-style templates.
Similar attempts have been made on named entity recognition (NER) which manually design templates to predict entity types for every text span in a sentence. However, such methods may suffer from error propagation induced by entity span detection, high cost due to enumeration of all possible text spans, and omission of inter-dependencies among token labels in a sentence. Here we present a simple demonstration-based learning method for NER, which lets the input be prefaced by task demonstrations for in-context learning. We perform a systematic study on demonstration strategy regarding what to include (entity examples, with or without surrounding context), how to select the examples, and what templates to use. Results on in-domain learning and domain adaptation show that the model's performance in low-resource settings can be largely improved with a suitable demonstration strategy (e.g., a 4-17% improvement on 25 train instances). We also find that good demonstration can save many labeled examples and consistency in demonstration contributes to better performance.
Incremental Topic Modeling for Scientific Trend Topics Extraction
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Articles

Caused by the exponential growth of scientific research, the number of scientific publications and reports, one of the most urgent and challenging tasks now is the early detection of trending topics. In this paper, we investigate recent topic modeling approaches to accurately extract trending topics at an early stage. The incremental training technique is suggested so that the model can operate on data in real-time. For validation, we propose a novel dataset that contains a collection of early-stage articles and a set of key collocations for each trend. The proposed metric estimates the delay in days when determining the trend, and the developed matching method suffices to calculate it automatically. The conducted experiments demonstrate that the topic model with regularization, namely ARTM, is superior to the base PLSA model. Apart from that, the best ARTM-based model is able to extract most of the labeled trends during the first year of their evolution.
The Sensitivity of Annotator Bias to Task Definitions
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11
Tasks: Sensitivity

NLP models are biased by the data they are trained on, including how it is annotated, but NLP research increasingly examines the em social biases of models, often in the light of their training data. This paper is first to examine to what extent social bias is sensitive to how data is annotated. We do so by collecting annotations of arguments in the same documents following four different guidelines and from four different demographic annotator backgrounds. We show that annotations exhibit widely different levels of group disparity depending on which guidelines annotators follow. The differences are not explained by task complexity, but rather by characteristics of these groups, as previously identified by sociological studies.
Exploring Meaning Encoded in Random Character Sequences with Character-Aware Language Models
Authors: Anonymous
Date: 2021-11-16 00:00:00
Proceeding: ACL ARR November 2021 11

Natural language processing models learn word representations based on the distributional hypothesis, which asserts that word context (e.g., co-occurrence) correlates with semantic meaning. We propose that n-grams composed of random character sequences, or garble, provide a novel context for studying word meaning both within and beyond extant language. In particular, randomly-generated character n-grams lack semantic meaning but contain primitive information based on the distribution of characters they contain. By studying the embeddings of a large corpus of garble, extant language, and pseudowords using CharacterBERT, we identify an axis in the model's high-dimensional embedding space that separates these classes of n-grams. Furthermore, we show that this axis relates to structure within extant language, including word part of speech, morphology, and concreteness. Thus, in contrast to studies that are mainly limited to extant language, our work reveals that semantic meaning and primitive information are intrinsically linked.