Developmental Negation Processing in Transformer Language Models (2022)

Abstract: Reasoning using negation is known to be difficult for transformer-based language models. While previous studies have used the tools of psycholinguistics to probe a transformer’s ability to reason over negation, none have focused on the types of negation studied in developmental psychology. We explore how well transformers can process such categories of negation, by framing the problem as a natural language inference (NLI) task. We curate a set of diagnostic questions for our target categories from popular NLI datasets and evaluate how well a suite of models reason over them. We find that models perform consistently better only on certain categories, suggesting clear distinctions in how they are processed.

## introduction negation is an important construct in language for reasoning over the truth of propositions (heinemann, 2015), garnering interest from philosophy @xcite , psycholinguistics @xcite , and natural language processing (nlp) @xcite . while transformer language models (tlms) @xcite have achieved impressive performance across many nlp tasks, a great deal of recent work has found that they do not process negation well, and often make predictions that would be trivially false in the eyes of a human @xcite @xcite . in developmental psychology, there has likewise been a great deal of interest in how a child's ability to comprehend negation emerges in the early years of life @xcite @xcite . unlike in nlp, which typically treats negation as representing a single monolithic competency, this research has long understood that there are many kinds of negation used in everyday interactions @xcite . this ranges from using negation to express a child's rejection of something to clarifying a child's knowledge. these "developmental" categories of negation do not emerge simultaneously; children tend to start using certain kinds before others @xcite . given that these categories represent some of the earliest uses of negation among humans, understanding how well tlms can master them is important for building more human-like models of language processing. understanding how well models perform on different categories will indicate whether they have mastery of some forms of negation, while also helping to identify failure points. another interesting question is whether the proficiency of tlms on these categories is at all related to competencies in human children (e.g., is the category which models consistently perform the best on the same that children most frequently employ?). however, to our knowledge, no prior work in nlp has focused on how well models perform on the forms of negation of interest to developmental psychology. in this short paper, we investigate how well a suite of tlms can process developmental negation, foot_1 by framing the problem as a natural language inference (nli) task. we develop a rulebased parser to extract problems from existing nli datasets, and evaluate our models on each category, in order to determine (i) whether certain categories are more solvable by our models than others, and (ii) what relationships exist among the categories. we find that models can consistently achieve stronger performance only on certain categories, and that training on combinations or sequences of these categories does not substantially improve a model's downstream performance. ## related work negation is known to be frequently used in everyday conversation. while this includes its logical form, we primarily focus on negation's psycholinguistic forms, especially those that have been studied in the context of developmental psychology. negation emerges early in child development, with 'no' sometimes being a child's first word @xcite , and even infants appear to understand forms of negation @xcite . preschool children use at least three different kinds of negation @xcite , but possibly as many as nine @xcite . as noted by @xcite , one of the first categories children use is rejection, where a child rejects an object or activity. this is later followed by existence, where a child might express the lack of an object, and later still denial, which a child uses to deny the truth of a claim. larger scale studies of child-directed speech have found that truth-functional kinds of negation tend to emerge later @xcite , but individual children do vary in their specific order of acquisition @xcite . it is unknown whether this ordering reflects any deeper dependencies among the different categories, or whether the ordering is reflected in how artificial language models (lms) learn negation. in nlp, methods from psycholinguistics have been used to probe the reasoning capabilities of lms. results from some studies have indicated that tlms are not human-like in their processing of negation @xcite . a similar line of work has used the nli task to probe a model's ability to process negation and found that tlms will often alter their predictions when negation is inserted or removed, even when the negation does not alter the entailment relationship @xcite . as argued by @xcite , part of the challenge of modeling purely logical negation is that a predicate often occurs in very similar contexts regardless of whether it is being negated. they argue that we should view negation as being a "graded similarity function", and show that distributional models can predict human plausibility judgments quite well, even in the presence of negation. these works show that it is unclear how well distributional models, especially tlms, are actually processing negation. we contribute to this literature from a new perspective, by studying how well models can reason over forms of negation common in developmental psychology. ## the developmental negation corpus we use the nli task to study the negation reasoning capabilities of our models. nli problems consist of two sentences: a premise (p) and hypothesis (h), and solving such a problem involves assessing whether p textually entails h. the generic structure of the nli task makes it suitable for studying a variety of underlying reasoning skills, including negation. we specifically use the snli @xcite and mnli @xcite datasets. to automatically identify questions that contain a specific kind of negation, we rely on the work by @xcite which studied how frequently different kinds of developmental negation occur in child-directed speech, using the data from the childes corpus (macwhinney, 2014). to do this, they created a simple rule-based parser to automatically tag each sentence in childes with the type of negation it contained (if any). we reimplement their parser, in some cases tweaking the rules slightly to better suit the structure of the nli task. for each example across all the splits of both datasets, we first obtain a dependency parse of both p and h using the diaparser package @xcite , and check if either contains an explicit negation marker ("no", "not", or "n't"). if one span contains negation, we check if the syntactic structure obeys the rules of any of our categories. if the span falls into a category, we mark it as belonging to that category. we use these questions as the diagnostic set for our experiments, splitting out 1/3 of the questions in each category as a diagnostic test set, and leaving the remainder as a diagnostic train set (and we will refer to them as such). we place the remaining nli questions containing no negation in a separate n li train set, giving us about 730,000 examples we use to finetune our models on the nli task. we split out 9,000 questions from this train set at random to use as a n li dev set, bal- category premise hypothesis p o yeah you probably don't have the right temperatures... you probably have ideal temperatures... ex this analysis pooled estimates... the analysis proves that there is no link... l not orders, no. it is not orders. p r two people are sitting against a building near shopping carts. run that way but don't run into the... i his manner was unfortunate, i observed thoughtfully. i could not pick out what kind of manner he... ep yeah i don't know why i know why r i lowered my voice... i didn't want to be overheard. possession (p o) we require that the lemma of the root be have, has, or had, and that the root is directly modified by both the negation and the verb do. existence (ex) we require that there occur in the text and precede the negative marker and that the negative marker directly modifies a noun phrase, determiner, or an adverb. ## experiments using the curated dataset, we performed a series of exploratory experiments to help us understand how well tlms process each of the negation categories. we use bert @xcite , and roberta @xcite , two popular transformer lms that have demonstrated impressive results on a variety of language understanding tasks. we also examine miniberta @xcite and babyberta @xcite , which are both based on the roberta architecture but were pretrained on a much smaller number of tokens (10 million and 5 million respectively), which is more realistic to the amount of language a child is exposed to in the first few years of life. we use the huggingface implementation of all models @xcite , and use both the base and large version of bert and roberta, which differ only in the number of trainable parameters. experiment 1: we began by investigating whether tlms would master certain negation categories sooner than others over the course of training. we train our models on n li train for 10 epochs, using a learning rate of 1e -5, a weight decay of 0.01, a batch size of 16, and a maximum sequence length 175. 3 we selected these hyperparameters to be similar to those which were previously reported to yield strong results when training on nli datasets @xcite . we additionally evaluated the models on n li dev , and found that they all achieved a matthews correlation of at least 0.6 @xcite , and thus concluded that these hyperparameters were suitable. for every end of epoch checkpoint across all models, we obtained evaluation results on each diagnostic test set. importantly, the models are not finetuned on any negated nli questions for this experiment, meaning that all knowledge of negation comes from pre-training. results are shown in figure 1 . we see that the categories have similar rankings in terms of accuracy. for example, l and p o are among the top two best-performing categories, while r is generally one of the worst-performing ones, indicating clear distinctions in how lms process the categories. babyberta, unlike other models, also shows stronger similarities to how children acquire negation. for instance, while r is thought to be one of the first categories children acquire, babyberta is the only model where r is one of the highest-ranking categories in terms of accuracy. experiment 2: one might expect that children develop a more abstract understanding of negation as they are exposed to different categories. this was suggested by @xcite who argued that more abstract forms of negation develop from less abstract ones, suggesting that mastering one form of negation can lead to positive transfer on others. in experiment 2, we examined how much positive 3 we set the maximum sequence length for babyberta to 128, which is the longest that the model supports. transfer could be obtained from training on one of the negation categories, and then testing on the others. we adopt a similar methodology to pruksachatkun et al. (2020), who explored the conditions that affect intermediate task transfer learning. using the models trained in experiment 1, we further finetune these models for 25 epochs on each diagnostic train set separately. we then evaluate the finetuned models on each diagnostic test set, which allows us to examine all possible pairwise interactions among categories. figure 2 shows the results for all combinations of diagnostic categories for training and testing. surprisingly, we find that positive transfer generally only occurs when a model is trained on the same category it is being tested on. training on a different category has little to no effect on the target category. babyberta is again an exception, as we do see positive transfer for most pairs, suggesting the model is generalizing across categories experiment 3: building on experiment 2, we examined how the performance of our models is affected when trained on all diagnostic categories in sequence. assuming that no positive transfer exists among the categories, we would expect to see a model's performance on a particular category improve only after it has been trained on that same category, and even training on multiple other categories should not substantially improve performance on the target. using the models from experiment 1, we finetune each model for 10 epochs on every diagnostic train set, using the sequence of categories shown in the x-axis of figure 3 . additionally, we under-sample all diagnostic train sets to have the same number of questions as p r, so that all categories contribute the same amount of data. figure 3 shows the results. for some categories, such as l and p r, we see the expected trend. the largest accuracy gain for these categories occurs whenever the model is trained on the same category it is being tested on, and performance drops slightly after being trained on others. however, for categories such as r, the best performance gain is not always after being trained on the same category. we sometimes see the model continue to improve on r after being trained on r, and in some cases, training on r causes performance on r to decrease. ## discussion and conclusion in this paper, we have explored how well transformers process categories of developmental negation. we find that performance rankings across categories are generally consistent, but that the categories seem to test for orthogonal skills in the majority of lms. in babyberta, we see significant similarities with the order of negation acquisition in children. two of the best performing categories are r and l, while two of the worst are ex and figure 3 : results from experiment 3. the x-axis shows the sequence of categories on which all models were trained, while the y-axis shows the accuracy obtained after being trained on a category. p r, which aligns quite well to the order observed by @xcite . it thus seems that tlms do at least partially reflect the order of negation acquisition observed in children, although more experiments would be needed to understand the extent of this correlation. that we found category rankings to generally be consistent across lms may have interesting implications, and understanding why lms struggle with certain categories may help to improve the ability of lms to process negation. future work can build on these experiments in several ways. in experiments 2 and 3, we modeled interactions among the negation categories in either a pairwise or sequential fashion, which is unlikely to reflect how children are exposed to negation. more experiments, mixing all of the categories at once in various proportions, might yield a more realistic model of cognitive development. our approach also requires that each category fits into a specific structure, which limits the amount of examples that can be extracted. future work will need to expand our ruleset to include more variations in the negated utterances covered. finally, while we primarily focus on finetuning, pre-training is likely to impact the proficiency of our models on the categories as well. future work should precisely control the prevalence of each category in the pretraining corpus, to observe what effect this has on downstream performance.