pwc-archive/papers-with-abstracts huggingface dataset:
Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Fair
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12

More capable language models increasingly saturate existing task benchmarks, in some cases outperforming humans, leaving little headroom with which to measure further progress. Adversarial dataset creation has been proposed as a strategy to construct more challenging datasets, and two common approaches are: (1) filtering out easy examples and (2) model-in-the-loop data collection. In this work, we study the impact of applying each approach to create more challenging evaluation datasets. We adapt the AFLite algorithm to filter evaluation data, and run experiments against 18 different adversary models. We find that AFLite indeed selects more challenging examples, lowering the performance of evaluated models more as stronger adversary models are used. However, the resulting ranking of models can also be unstable and highly sensitive to the choice of adversary model used. Moreover, AFLite oversamples examples with low annotator agreement, meaning that model comparisons hinge on the most contentiously labeled examples. Smaller-scale experiments on the adversarially collected datasets ANLI and AdversarialQA show similar findings, broadly lowering performance with stronger adversaries while disproportionately affecting the adversary model.
SHAP-Based Explanation Methods: A Review for NLP Interpretability
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12

Model explanations are crucial for the transparent, safe, and trustworthy deployment of machine learning models. The \emph{SHapley Additive exPlanations} (SHAP) framework is considered by many to be a gold standard for local explanations thanks to its solid theoretical background and general applicability. In the years following its publication, several variants appeared in the literature---presenting adaptations in the core assumptions and target applications. In this work, we review all relevant SHAP-based interpretability approaches available to date and provide instructive examples as well as recommendations regarding their applicability to NLP use cases.
AutoGraphex: Zero-shot Biomedical Definition Generation with Automatic Prompting
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: Descriptive, Language Modeling, Language Modelling, Text Generation, Zero-Shot Learning

Describing terminologies with definition texts is an important step towards understanding the scientific literature, especially for domains with limited labeled terminologies. Previous works have sought to design supervised neural text generation models to solve the biomedical terminology generation task, but most of them failed to define never-before-seen terminologies in newly emerging research fields. Here, we tackle this challenge by introducing a zero-shot definition generation model based on prompting, a recent approach for eliciting knowledge from pre-trained language models, with automatically generated prompts. Furthermore, we enhanced the biomedical terminology dataset by adding descriptive texts to each biomedical subdiscipline, thus enabling zero-shot learning scenarios. Our model outperformed existing supervised baseline and the baseline pre-trained language model that employs manually crafted prompts by up to 52 and 6 BLEU score, respectively.
Flat and Nested Negation and Uncertainty Detection with PubMed BERT
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: Multi-class Classification, Negation, Negation Detection

Negation and uncertainty detection is an oft-studied challenge in biomedical NLP. Annotation style for the task has not been standardized and as such, the existing datasets not only vary in domain but require various algorithmic designs due to their structural differences. We present a new negation detection dataset in two versions from clinical publications. We further developed two BERT-based models to evaluate on each dataset version. Both models treat the task as a token-level multi-class classification task, one of which is capable of assigning more than one label per token in the case of recursive nesting. Our models achieve F1 scores of 76% and 72% on the development and test sets, respectively.
Towards Faithful Personalized Response Selection in Retrieval Based Dialog Systems
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: Retrieval

Personalized response selection systems are generally grounded on persona. However, the angle of emotion influencing response selection is not explored. Also, faithfulness to the conversation context of these systems plunges when a contradictory or an off-topic response is selected. This paper makes an attempt to address these issues by proposing a suite of fusion strategies that capture the interaction between persona, emotion, and entailment information of the utterances. A concept-flow encoder is designed which capture the relevant concept knowledge both in context and responses. Ablation studies were done on Persona-Chat dataset show that incorporating emotion, entailment improves the accuracy of response selection. We combine our fusion strategies and concept-flow encoding to train a BERT based model which outperforms the previous methods by margins larger than 1.9% on original personas and 1.7% on revised personas in terms of hits@1 (top-1 accuracy), achieving a new state-of-the-art performance on the Persona-Chat dataset.
Hate Speech and Counter Speech Detection: Context Does Matter
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12

Hate speech is plaguing the cyberspace along with user-generated content. Adding counter speech has become an effective way to combat hate speech online. Existing datasets and models target either (a) hate speech or (b) hate and counter speech but disregard the context. This paper investigates the role of context in the annotation and detection of online hate and counter speech, where context is defined as the preceding comment in a conversation thread. We created a context-aware dataset for a 3-way classification task on Reddit comments: hate speech, counter speech, or neutral. Our analyses indicate that context is critical to identify hate and counter speech: human judgments change for most comments depending on whether we show annotators the context. A linguistic analysis draws insights into the language people use to express hate and counter speech. Experimental results show that neural networks obtain significantly better results if context is taken into account. We also present qualitative error analyses shedding light into (a) when and why context is beneficial and (b) the remaining errors made by our best model when context is taken into account.
Time Waits for No One! Analysis and Challenges of Temporal Misalignment
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12

When an NLP model is trained on text data from one time period and tested or deployed on data from another, the resulting temporal misalignment can degrade end-task performance. In this work, we establish a suite of eight diverse tasks across different domains (social media, science papers, news, and reviews) and periods of time (spanning five years or more) to quantify the effects of temporal misalignment. Our study is focused on the ubiquitous setting where a pretrained model is optionally adapted through continued domain-specific pretraining, followed by task-specific finetuning. We establish a suite of tasks across multiple domains to study temporal misalignment in modern NLP systems. We find stronger effects of temporal misalignment on task performance than have been previously reported. We also find that, while temporal adaptation through continued pretraining can help, these gains are small compared to task-specific finetuning on data from the target time period. Our findings motivate continued research to improve temporal robustness of NLP models.
Knowledge Distillation Improves Stability in Retranslation-based Simultaneous Translation
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: Knowledge Distillation, Translation

In simultaneous translation, the \emph{retranslation} approach has the advantage of requiring no modifications to the inference engine. However in order to reduce the undesirable instability (flicker) in the output, previous work has resorted to increasing the latency through masking, and introducing specialised inference, losing the simplicity of the approach. In this paper, we argue that the flicker is caused by both non-monotonicity of the training data, and by non-determinism of the resulting model. Both of these can be addressed using knowledge distillation. We evaluate our approach using simultaneously interpreted test sets for English-German and English-Czech and demonstrate that the distilled models have an improved flicker-latency tradeoff, with quality similar to the original.
Creation and evaluation of timelines for longitudinal user posts
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12

There is increasing interest to work with user generated content in social media and especially textual posts over time. Currently there is no consistent way of segmenting user posts into timelines in a meaningful way that can improve the quality and cost of manual annotation. Here we propose a set of methods for segmenting longitudinal user posts into timelines that are likely to contain interesting moments of change in a user's behaviour based on the content they have shared online and their online activity. We also propose a framework for evaluating the timelines returned in terms of containing candidate moments of change in close proximity to manually annotated timelines dense in such moments of change. Finally, we present a discussion of the linguistic content of highly ranked timelines.
PInKS: Preconditioned Commonsense Inference with Weak Supervision
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: Informativeness

Reasoning with preconditions such as "glass can be used for drinking water unless the glass is shattered" remains an open problem for language models. The main challenge lies in the scarcity of preconditions data and the model's lack of support for such reasoning. We present PInKS, Preconditioned Commonsense Inference with Weak Supervision, an improved model for reasoning with preconditions through minimum supervision. We show, both empirically and theoretically, that PInKS improves the results across the benchmarks on reasoning with the preconditions of commonsense knowledge~(up to 0.4 macro-f1 scores). We further investigate the robustness of our method through PAC-Bayesian informativeness analysis, recall measures, and ablation study.
Contrastive Learning for Fair Representations
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: Attribute, Classification, Contrastive Learning, Multi-class Classification

Trained classification models can unintentionally lead to biased representations and predictions, which can reinforce societal preconceptions and stereotypes. Existing debiasing methods for classification models, such as adversarial training, are often expensive to train and fragile to optimise. Here, we propose a method for mitigating bias in classifier training by incorporating contrastive learning, in which instances sharing the same class label are encouraged to have similar representations, while instances sharing a protected attribute are forced further apart. In such a way our method learns representations that capture the task label in focused regions, while ensuring the protected attribute has diverse spread, and thus has limited impact on prediction and thereby results in fairer models. Extensive experimental results on three tasks show that: our method achieves fairer representations larger bias reduction than competitive baselines; it does so without sacrificing main task performance; and it generalizes across modalities and binary- and multi-class classification tasks, being conceptually simple and agnostic to network architecture, and incurring minimal additional compute cost.
Speeding Up Entmax
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: Machine Translation, Text Generation, Translation

Softmax is the de facto standard for  normalizing logits in modern neural networks for language processing. However, by producing a  dense probability distribution each token in the vocabulary has a nonzero chance of being selected at each generation step, leading to a variety of reported problems in text generation. $\alpha$-entmax of Peters et al. (2019) solves this problem, but is unfortunately slower than softmax. In this paper, we propose an alternative to $\alpha$-entmax, which keeps its virtuous characteristics, but is as fast as optimized softmax and achieves on par or better performance in machine translation task.
Embedding Convolutions for Short Text Extreme Classification with Millions of Labels
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: text-classification, Text Classification

In this paper, we propose a convolutional architecture InceptionXML which is light-weight, yet powerful, and robust to the inherent lack of word-order in short-text queries in search and recommendation tasks. We demonstrate the efficacy of applying convolutions by recasting the operation along the embedding dimension instead of the word dimension as done in conventional usage of CNNs for text classification. Towards scaling our model to problems with millions of labels, we also propose InceptionXML+ framework. This addresses the shortcomings of the dynamic hard-negative mining framework in the recently proposed LightXML by improving the alignment between the label-shortlister and extreme classifier. InceptionXML+ is not only smaller than state-of-the-art deep extreme classifier, Astec, in terms of model size but also significantly outperforms it on popular benchmark datasets. For reproducibility, the code is made available as part of this submission.
WARM: A Weakly (+Semi) Supervised Math Word Problem Solver
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: Math

Solving math word problems (MWPs) is an important and challenging problem in natural language processing. Existing approaches to solve MWPs require full supervision in the form of intermediate equations. However, labeling every MWP with its corresponding equations is a time-consuming and expensive task. In order to address this challenge of equation annotation, we propose a weakly supervised model for solving MWPs by requiring only the final answer as supervision. We approach this problem by first learning to generate the equation using the problem description and the final answer, which we subsequently use to train a supervised MWP solver. We propose and compare various weakly supervised techniques to learn to generate equations directly from the problem description and answer. Through extensive experiments, we demonstrate that without using equations for supervision, our approach achieves accuracy gains of 4.5% and 32% over the state-of-the-art weakly supervised approach (Hong et al., 2021), on the standard Math23K (Wang et al., 2017) and AllArith (Roy and Roth, 2017) datasets respectively. Additionally, we curate and release new datasets of roughly 10k MWPs each in English and in Hindi (a low-resource language). These datasets are suitable for training weakly supervised models. We also present an extension of WARM to semi-supervised learning and present further improvements on results, along with insights.
Negative Sample is Negative in Its Own Way: Tailoring Negative Sentences for Image-Text Retrieval
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: Image-text Retrieval, Retrieval, Sentence, Text Retrieval, Triplet

Matching model is essential for Image-Text Retrieval framework. Existing research usually train the model with a triplet loss and explore various strategy to retrieve hard negative sentences in the dataset. We argue that current retrieval-based negative sample construction approach is limited in the scale of the dataset thus fail to identify negative sample of high difficulty for every image. We propose our TAiloring neGative Sentences with Discrimination and Correction (TAGS-DC) to generate synthetic sentences automatically as negative samples. TAGS-DC is composed of masking and refilling to generate synthetic negative sentences with higher difficulty. To keep the difficulty during training, we mutually improve the retrieval and generation through parameter sharing. To further utilize fine-grained semantic of mismatch in the negative sentence, we propose two auxiliary tasks, namely word discrimination and word correction to improve the training. In experiments, we verify the effectiveness of our model on MS-COCO and Flickr30K compared with current state-of-the-art models and demonstrates its robustness and faithfulness in the further analysis.
Balancing out Bias: Achieving Fairness Through Balanced Training
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: Fairness

Bias in natural language processing manifests as disparities in error rates across author demographics, typically disadvantaging minority groups. Although dataset balancing has been shown to be effective in mitigating bias, existing approaches do not directly account for correlations between author demographics and linguistic variables. To achieve Equal Opportunity fairness, this paper introduces a simple but highly effective objective for countering bias using balanced training. We extend the method in the form of a gated model, which incorporates protected attributes as input, and show that it is effective at reducing bias in predictions through demographic input perturbation, outperforming all other bias mitigation techniques when combined with balanced training.
BnPC: A Corpus for Paraphrase Detection in Bangla
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: Paraphrase Identification, Sentence

In this paper, we present the first benchmark dataset for paraphrase detection in Bangla language.  Despite being the sixth most spoken language in the world, paraphrase identification in the Bangla language is barely explored. Our dataset contains 8,787 human-annotated sentence pairs collected from a total of 23 newspaper outlets' headlines on four categories. We explore different linguistic features and pre-trained language models to benchmark the dataset. We perform a human evaluation experiment to obtain a better understanding of the task's constraints, which reveals intriguing insights. We make our dataset and code publicly available.
ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: nlg evaluation, Text Generation

Automatic evaluations for natural language generation conventionally rely on token-level or embedding-level comparisons with the text references. This is different from human evaluation manners, in which people also form pictures of the text contents in their minds during reading. In this work, we propose ImaginE, an imagination-based automatic evaluation metric for natural language generation. With the help of CLIP and DALL-E, two cross-modal models pre-trained on large-scale image-text pairs, we automatically generate an image as the embodied imagination for the text snippet, and compute the imagination similarity using contextual embeddings. Experiments spanning several text generation tasks demonstrate that adding imagination with our ImaginE displays great potential in introducing multi-modal information into NLG evaluation, and improves existing automatic metrics’ correlations with human similarity judgments in many circumstances.
A Survey on Multimodal Disinformation Detection
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: Misinformation, Survey

Recent years have witnessed the proliferation of offensive content online such as fake news, propaganda, misinformation, and disinformation. While initially this was mostly about textual content, over time images and videos gained popularity, as they are much easier to consume, attract more attention, and spread further than simple text. As a result, researchers started leveraging different modalities and combinations thereof to combat online multimodal offensive content. In this study,  we offer a survey that carefully studies the state-of-the-art on multimodal disinformation detection covering various combinations of modalities: text, images, speech, video, social media network structure, and temporal information. Moreover, while some studies focused on factuality, others investigated how harmful the content is. While these two components in the definition of disinformation -- (i) factuality, and (ii) harmfulness, are equally important, they are typically studied in isolation. Thus, we argue for the need to tackle disinformation detection by taking into account multiple modalities as well as both factuality and harmfulness, in the same framework. Finally, we discuss current challenges and future research directions.
Literature-Augmented Clinical Outcome Prediction
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: Prediction

We present BEEP (Biomedical Evidence-Enhanced Predictions), a novel approach for clinical outcome prediction that retrieves patient-specific medical literature and incorporates it into predictive models. Based on each individual patient's clinical notes, we train language models (LMs) to find relevant papers and fuse them with information from notes to predict outcomes such as in-hospital mortality. We develop methods to retrieve literature based on noisy, information-dense patient notes, and to augment existing outcome prediction models with retrieved papers in a manner that maximizes predictive accuracy. Our approach boosts predictive performance on three important clinical tasks in comparison to strong recent LM baselines, increasing F1 by up to 5 points and precision@Top-K by a large margin of over 25%.
Multi-Vector Models with Textual Guidance for Fine-Grained Scientific Document Similarity
Authors: Anonymous
Date: 2021-12-17 00:00:00
Proceeding: ACL ARR December 2022 12
Tasks: Sentence

We present a new scientific document similarity model based on matching fine-grained aspects of texts. To train our model, we exploit a naturally-occurring source of supervision: sentences in the full-text of papers that cite multiple papers together (co-citations). Such co-citations not only reflect close paper relatedness, but also provide textual descriptions of how the co-cited papers are related. This novel form of textual supervision is used for learning to match aspects across papers. We develop multi-vector representations where vectors correspond to sentence-level aspects of documents, and present two methods for aspect matching: (1) A fast method that only matches single aspects, and (2) a method that makes sparse multiple matches with an Optimal Transport mechanism that computes an Earth Mover's Distance between aspects. Our approach improves performance on document similarity tasks in four datasets. Further, our fast single-match method achieves competitive results, paving the way for applying fine-grained similarity to large scientific corpora.