Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence (2022)

Abstract: The logical negation property (LNP), which implies generating different predictions for semantically opposite inputs (p is true iff ¬p is false), is an important property that a trustworthy language model must satisfy. However, much recent evidence shows that large-size pre-trained language models (PLMs) do not satisfy this property. In this paper, we perform experiments using probing tasks to assess PLMs’ LNP understanding. Unlike previous studies that only examined negation expressions, we expand the boundary of the investigation to lexical semantics. Through experiments, we observe that PLMs violate the LNP frequently. To alleviate the issue, we propose a novel intermediate training task, named meaning-matching, designed to directly learn a meaning text correspondence, instead of relying on the distributional hypothesis. Through multiple experiments, we find that the task enables PLMs to learn lexical semantic information. Also, through fine-tuning experiments on 7 GLUE tasks, we confirm that it is a safe intermediate task that guarantees a similar or better performance of downstream tasks. Finally, we observe that our proposed approach outperforms our previous counterparts despite its time and resource efficiency.

## introduction contemporary large-size plms, such as bert @xcite , electra @xcite , and gpt-2 and -3 @xcite , have shown excellent results in many downstream tasks, even performing better than humans in the glue @xcite and super-glue @xcite ) benchmark datasets. however, their reliability is recently being challenged. many studies have conducted various probing tasks and observed that plms exhibit faulty behaviours, such as insensitiveness to sentence ordering (pham et al., 2021; gupta et al., 2021; sinha 1 https://github.com/mj-jang/beyond-distributional et al., 2021b), incomprehension on number-related representations @xcite @xcite , and lack of semantic content understanding @xcite . these issues raise concerns about plms' stability and reliability, precluding them from applications in practice, especially in risk-sensitive areas. another critical problem of plms is their inaccurate behaviour on negation, which is a principal property in many language understanding tasks. for tasks where the lnp holds (p is true iff ¬p is false; see @xcite , plms should make different answers for the original and negated inputs. however, several studies observed that plms violate this property. in masked knowledge retrieval tasks, plms frequently generate incorrect answers for negated input queries @xcite . in other studies, plms show a poor generalisation ability on negated natural language inference (nli) datasets @xcite . although the aforementioned studies produced promising analysis results, they limited the scope of the lnp only to adding negation expressions (e.g., "no" and "not"). however, other perturbations that generate the opposite meaning also can be applied to the property. therefore, a consideration of such perturbation methods is necessary to fully assess whether plms satisfy the lnp. also, remedies to alleviate the problem have not been studied much yet. @xcite recently employed data augmentation and unlikelihood training @xcite to prevent models from generating unwanted words, given the augmented negated data during masked language modelling (mlm). however, this approach has several downsides. first, like previous works, @xcite only considered negation expressions. second, the data augmentation method is contingent on many additional linguistic compo-nents, which causes the dependency of a model's performance on certain modules and precludes applying the method to other languages where such resources are unavailable. finally, the model should be pre-trained from scratch with the unlikelihood objective, which consumes considerable time and resources. in this paper, we expand the boundary of the lnp to lexical semantics, i.e., synonyms and antonyms, and ascertain that plms are prone to violate the lnp. next, we propose a remedy, called intermediate-training on meaning-matching (im foot_0 ), which hardly employs additional linguistic components. we hypothesise that a leading cause lies in the mlm training objective, which assumes the distributional hypothesis for learning the meaning of the text @xcite . instead, we design a model that directly learns the correspondence between words and their semantic contents. through experiments, we verify that our approach improves the model's comprehension of the lnp, while showing a stable performance on multiple downstream tasks. our main contributions are as follows: (i) we extend the investigation of the lnp from negation to lexical semantics (section 2), (ii) we reveal that plms are prone to violate the lnp (section 3), (iii) we propose a novel remedy, named im 2 , which is decoupled from the distributional hypothesis but learns meaning-text correspondence instead (section 4), (iv) through experiments, we ascertain that the proposed approach improves the understanding of negation and lexical semantic information (sections 5.1 and 5.2), and (v) we verify that meaningmatching is a stable and safe intermediate task that produces a similar or better performance in multiple downstream tasks (sections 5.3 and 5.4). ## probing tasks for investigating the logical negation property we design three probing tasks to evaluate whether plms satisfy the lnp: masked knowledge retrieval on negated queries (mkr-nq), masked word retrieval (mwr), and synonym/antonym recognition (sar). brief illustrations of each task are in figure 1 . ## plms lack information of negation and lexical semantics we select the following plms for the experiments: bidirectional encoder representations from transformers (bert)-base/large @xcite , roberta-base/large @xcite , and albert-base/large @xcite . these plms are pre-trained with the mlm training objective. we added the electra-small/base/large models @xcite for the sar task, but it is not used for the mkr-nq and mwr experiments, as the discriminator of the electra models are trained with the replaced token prediction (rtp) training objective and have no mlm classifier. no additional training is required for the mkr-nq and mwr tasks. for the sar task, we fine-tune each plm for 10 epochs and apply the early stopping technique. we use the adamw optimiser @xcite for training with a learning rate of 5e -6 and a batch size of 32. ## intermediate training on meaning matching task: im 2 ## experiments and results we conduct the same probing tasks after the intermediate training on the meaning-matching task. foot_2 ## related work plms are at the core of many success stories in natural language processing (nlp). however, it remains unclear to what extent plms understand the syntactic and semantic properties of the human language. a series of probing tasks have been conducted on plms and have found them lacking or falling short on some language properties. among the many findings of these probing tasks, plms have been found to be insensitive to the order of sentences when generating representations @xcite @xcite , struggle to comprehend number-related representations @xcite @xcite , and display a lack of semantic content understanding @xcite . in addition to the above faulty behaviours, ettinger (2020) and @xcite show that plms fail to comprehend negation, which is an important property of language in many natural language understanding (nlu) tasks. ettinger (2020) check the ability of plms to understand the meaning of negation in given contexts. in their work, they check whether models are sensitive in their completions of sentences that either include negation or not. under normal circumstances, the completions are expected to vary in truth depending on the presence or absence of negation in given sentences. their results show that plms are insensitive to the impacts of negations when completing sentences. kassner and schütze (2020) construct the negated lama dataset by inserting negation elements (e.g., "not") in the lama cloze questions @xcite . they use negated and original question pairs to query plms and establish that models are equally prone to make the same predictions for both the original and negated questions. in a well-informed setting, it is expected that plms should make different predictions for the original and negated questions. this shows that plms struggle to comprehend negation. in light of the highlighted faulty behaviours of plms, especially their struggle to comprehend negation, @xcite propose a remedy to alleviate the problem. in their remedy, they augment the language modelling objective with an unlikelihood objective @xcite based on negated sentences from the training corpus. they use a syntactic augmentation method to generate negated sentences. in this method, the dependency parse of the sentences, pos tags, and morphological information of each word are taken as input, and the negation of sentences is done using sets of dependency tree regular expression patterns, such as semgrex @xcite . during training, they replace objects in negated sentences with [mask] tokens and use unlikelihood training to make the masked-out tokens unlikely under the plm distribution. to ensure that negated sentences are factually false, they use the corresponding positive sentences as context for the unlikelihood prediction task. previous studies (e.g., @xcite ) have mostly limited the scope of the logical negation property only to the negation expressions (e.g., "no" and "not"). however, the core spirit of this property is the opposite meaning, which is not only limited to the negation. @xcite consider negating sentences using dependency tree regular expression patterns. this widens the scope of negation, as it is not only limited to the negation expressions "no" and "not". however, their approach relies on other components, such as semgrex, and dependency and pos parsers, which could impact the quality of the data, hence impact the models' performance. in this work, we consider other perturbation methods to generate the opposite-meaning sentences to investigate whether plms satisfy the logical negation property, and we propose a remedy, called intermediate-training on meaning-matching (im 2 ), which hardly employs additional linguistic components. ## summary and outlook in this work, we investigated plms' lnp. compared to previous works that only examine negation expressions, we expanded the boundary of lnp to lexical semantics. we confirmed that plms are likely to violate lnp through extensive experiments. we hypothesise that the distributional hypothesis is an insufficient basis for understanding the semantic meaning of texts. to alleviate the issue, we proposed a novel intermediate task: meaning-matching. via experiments, we verified that meaningmatching is a stable intermediate task that substantially improves plms' understanding of negation and lexical semantic information while guaranteeing a positive transfer with multiple downstream tasks. also, our approach produces a better performance on the negated nli datasets compared to the unlikelihood training-based method, which leverages much more time and resources. our work suggests that it is time to move beyond the distributional hypothesis to develop logically consistent and stable language models.