NLM s: Augmenting Negation in Language Models (2023)

Abstract: Negation is the fundamental component in a natural language that reverses the semantic meaning of a sentence. It plays an extremely important role across a wide range of applications, yet they are underrepresented in pre-trained language models (LMs), resulting often in wrong inferences. In this work, we try to improve the underlying understanding of the negation in the pre-trained LMs. To augment negation understanding, we propose a language model objective with a weighted cross-entropy loss and elastic weight consolidation regularization. We reduce the mean top 1 error rate for BERT-base to 1.1%, BERT-large to 0.78%, RoBERTA-base to 3.74%, RoBERTA-large to 0.01% on the negated LAMA dataset. It minimizes the BERT error rate by a margin of 8% and also outperform the existing negation models. We also provide empirical evidences that negated augmented models outperform the classical models on original as well as negation benchmarks on natural language inference tasks.

## introduction negation plays a pivotal role in many natural language understanding tasks, such as sentiment analysis, question answering, natural language inference (nli) tasks, etc. while large language models such as bert have pushed state-of-the-art and are being used widely across various domains, it fails dramatically to understand negation. negation relates an expression e to another expression with a meaning that is in some way contrary to the meaning of the original sentence e. negation is ubiquitous in english language text and comprises approximately 25% sentences with negation clues or genre in some form @xcite . despite this fact, understanding of negation is under-represented in the language models. the current language models (lms) cannot distinguish between negated and non-negated forms of masking tasks @xcite . for example, when asked to predict the @xcite token in the sentences: the capital of south korea is @xcite . and the capital of south korea is not @xcite .; bert often generates the same answer as "seoul". the exercise gives evidence that lms do not appropriately model the distribution of negation sentences. lms when fine-tuned on nli tasks, current pre-trained models tend to misclassify the examples which contain negations as contradictions when the true label is neutral or entailment. in this work, we address the challenge of augmenting negation understanding of language models and alleviating the model bias. to enhance negation understanding, we adopt a data-driven approach by leveraging a large dataset extracted from wikipedia. the dataset serves as the foundation for generating a diverse set of negated sentences. our approach incorporates dependency parsing and tense morphological structure to systematically construct negated counterparts of the original sentences. by explicitly introducing negation into the data, we aim to improve the model's ability to handle negated statements effectively. the next step involves training the language models using a paired set of affirmative-affirmative and affirmative-negative sentence pairs. to effectively capture the nuances of negation, we employ a weighted loss function and an elastic weight regularizer during the training process. this combination enables the models to focus on learning from both affirmative and negative instances, thereby reducing any inherent model bias towards affirmative statements. to evaluate the effectiveness of our models, we conduct experiments on standard datasets, specifically focusing on knowledge completion and nli tasks. the work presented here addresses the shortcomings of negation understanding of lms and makes the following contributions: ## related work language models such as bert @xcite and roberta @xcite have achieved remarkable results in various natural language understanding tasks such as question answering, sentiment analysis, named entity recognition @xcite , etc. their ability to capture contextual information, leverage large scale pre-training and the incorporation of novel training techniques have contributed to their impressive performance. these models also learn broader range of factual and common-sense knowledge @xcite . despite these abilities and performance, @xcite shows that these models fall short on understanding the negated factual sentences. various works investigate the linguistic knowledge captured in these lms. the authors @xcite investigate the grammatical knowledge of the lms and conclude that these models have near human performance on simple sentences and fail to make fine-grained grammatical distinctions. authors @xcite propose a dataset for evaluating the knowledge of understanding grammatically correct as well as incorrect sentences. the olmpics work investigates bert and roberta model on the reasoning tasks which require comparison, conjunctions and composition. other studies include diagnosing syntactic heuristics in nli tasks @xcite , inner workings of these models on (negative) polarity items @xcite and discovering of nlp pipelines in bert @xcite . some recent work studies negation understanding in the area of negation detection @xcite , negation scope detections @xcite @xcite , attention analysis in negation scope (zhao and bethard, 2020) and focus detection @xcite . authors @xcite study linguistic phenomenons such as antonyms, negation, and spelling mismatch and find that these models rely on syntactic clues to make the nle inferences. hossain et al. (2020) proposes a benchmark for natural language inference tasks in a form of a text-hypothesis pair in which negation plays a critical role. also, the authors find that the current state-of-the-art transformers struggle to understand negation which often results in wrong inferences. @xcite shows the utilities of explicit negative examples to improve the syntactic abilities of models for the nli tasks. the studies @xcite show that these state-of-the-art transformer models struggle to understand the negation and there exists the need to inject negation understanding in these models. these works investigate the negation understanding of lms. however, only a few works try to fix this negation bias in lms. @xcite augments negation understanding in bert by modelling objectives with an unlikelihood based on negated generic sentences. the proposed method successfully injects knowledge into bert-base, however, fails in bert-large model. ## dataset in order to effectively augment negation, we need a dataset for affirmative and negative sentences compatible with lm's input requirements. however, a publicly available dataset with such characteristics is not available. an alternative approach is undertaken to extract the wiki-raw-text corpus and negate the sentences. ## model in this section, we describe the training sample generation and the model details. ## conclusion negation is ubiquitous in a language and plays a crucial aspect in language understanding. we introduce a negation aware technique that addresses the negation bias of the existing lms. we modified the training data by augmenting it with explicitly negation data and trained the model with ewc regularization with weighted loss. by incorporating, explicit negation cues during training, the proposed approach enables the model to better capture the subtle nuances of negation, thus resulting in improved contextual understanding and more accurate language completion. the results demonstrate that nlms significantly improve the lms performance in understanding and representing negated contexts, leading to more accurate predictions in knowledge completion and nli tasks. in the future, we plan to augment negation in cross-lingual and generative models. augmenting negation in these models involves several challenges such as different syntactic and lexical cues, inherent difficulty of aligning and training paradigms.