BERT Knows P unta C ana is not just beautiful, it’s gorgeous: Ranking Scalar Adjectives with Contextualised Representations (2020)

Abstract: Adjectives like pretty, beautiful and gorgeous describe positive properties of the nouns they modify but with different intensity. These differences are important for natural language understanding and reasoning. We propose a novel BERT-based approach to intensity detection for scalar adjectives. We model intensity by vectors directly derived from contextualised representations and show they can successfully rank scalar adjectives. We evaluate our models both intrinsically, on gold standard datasets, and on an Indirect Question Answering task. Our results demonstrate that BERT encodes rich knowledge about the semantics of scalar adjectives, and is able to provide better quality intensity rankings than static embeddings and previous models with access to dedicated resources.

## introduction scalar adjectives describe a property of a noun at different degrees of intensity. identifying the scalar relationship that exists between their meaning (for example, the increasing intensity between pretty, beautiful and gorgeous) is useful for text understanding, for both humans and automatic systems. it can serve to define the sentiment and subjectivity of a text, perform inference and textual entailment @xcite , build question answering and recommendation systems @xcite , and assist language learners in distinguishing between semantically similar words @xcite . we investigate the knowledge that the pretrained bert model @xcite encodes about the intensity expressed on an adjective scale. given that this property is acquired by humans during language learning, we expect a language model (lm) exposed to massive amounts of text data during training to have also acquired some figure 1 : full scale of adjectives describing positive and negative sentiment at different degrees from the so-cal dataset @xcite . our work falls in the neural network interpretation paradigm which explores the knowledge about language encoded in the representations of deep learning models @xcite @xcite @xcite . the bulk of this interpretation work addresses structural aspects of language such as syntax, word order, or number agreement @xcite @xcite ; shallow semantic phenomena closely related to syntax such as semantic role labelling and coreference @xcite ; or the symbolic reasoning potential of language model representations @xcite . our work makes a contribution towards the study of the knowledge pre-trained lms encode about word meaning, generally overlooked until now in interpretation work. we evaluate the representations generated by bert against gold standard adjective intensity estimates @xcite @xcite and apply them directly to a question answering task @xcite . our results show that bert clearly encodes the intensity variation between adjectives on scales describing different properties. our proposed method can be easily applied to new datasets and languages where scalar adjective resources are not available. 1 2 related work the analysis of scalar adjective relationships in the literature has often been decomposed into two steps: grouping related adjectives together and ranking adjectives in the same group according to intensity. the first step can be performed by distributional clustering approaches @xcite which can also address adjectival polysemy. hot, for example, can be on the temperature scale (a @xmath0, the attractiveness (a @xmath1, depending on the attribute it modifies. other works @xcite @xcite directly address the second step, ranking groups of semantically related adjectives from lexicographic resources (e.g., wordnet) @xcite . this ranking is the focus of this work. we show that bert contextualised representations encode rich information about adjective intensity, and can provide high quality rankings of adjectives in a scale. adjective ranking has been traditionally performed using pattern-based approaches which extract lexical or syntactic patterns indicative of an intensity relationship from large corpora @xcite @xcite . for example, the patterns "x, but not y" and "not just x but y" provide evidence that x is an adjective less intense than y. another common approach is lexicon-based and draws upon a resource that maps adjectives to scores encoding sentiment po-larity (positive or negative) and intensity. such resources can be manually created, like the so-cal lexicon @xcite , or automatically compiled by mining adjective orderings from star-valued product reviews where people's comments have associated ratings @xcite @xcite . @xcite combine knowledge from lexico-syntactic patterns and the so-cal lexicon with paraphrases in the paraphrase database (ppdb) @xcite . our approach is novel in that it does not need specified patterns or access to lexicographic resources. it, instead, relies on the knowledge about intensity encoded in scalar adjectives' contextualised representations. our best performing method is inspired by work on gender bias which relies on simple vector arithmetic to uncover gender-related stereotypes. a gender direction is determined (for example, by comparing the embeddings of she and he, or woman and man) and the projection of the vector of a potentially biased word on this direction is then calculated @xcite . we extend this method to scalar adjectives and bert representations. kim and de marneffe (2013) also consider vector distance in the semantic space to encode scalar relationships between adjectives. they specifically examine a small set of word pairs, and observe that the middle point in space between the word2vec @xcite embeddings of two antonyms (e.g., furious and happy) falls close to the embedding of a mid-ranked word in their scale (e.g., unhappy). their experiments rely on antonym pairs extracted from wordnet. we show that contextualised representations are a better fit for this task than static embeddings, encoding rich information about adjectives' meaning and intensity. ## data we experiment with three scalar adjective datasets. demelo (de melo and bansal, 2013). 2 adjective sets were extracted from wordnet 'dumbbell' structures @xcite . the sets represent full-scales (e.g., from horrible to awesome) and are partitioned into half-scales (from horrible to bad, and from good to awesome) based on pattern-based evidence in the google n-grams corpus @xcite . the dataset contains 87 half-scales with 548 adjective pairs, manually annotated for intensity relations (<, >, @xmath2. crowd @xcite . 3 the dataset consists of a set of adjective scales with high coverage of the ppdb vocabulary. it was constructed by a threestep process: crowd workers were first asked to determine whether pairs of adjectives describe the same attribute (e.g., temperature) and should, therefore, belong to the same scale. sets of samescale adjectives were then refined over multiple rounds. finally, workers ranked the adjectives in each set by intensity. the final dataset includes 330 adjective pairs along 79 half-scales. wilkinson @xcite . 4 this dataset was generated through crowdsourcing. crowd workers were presented with small seed sets (e.g., huge, small, microscopic) and were asked to propose similar adjectives, resulting in twelve adjective sets. sets were automatically cleaned for consistency, and then annotated for intensity by the crowd workers. the original dataset contains full scales. we use its division in 21 half-scales (with 61 adjective pairs) proposed by @xcite . in the rest of the paper, we use the term "scale" to refer to the half-scales contained in these datasets. ## bert contextualised representations 4.1 sentence collection to explore the knowledge bert has about relationships in an adjective scale s, we generate a contextualised representation for each @xmath3. since such cases are rare in running text, we construct two sentence sets that satisfy this condition using the ukwac corpus @xcite 5 and the flickr 30k dataset @xcite . 6 for every @xmath4, a dataset from section 3, and for each @xmath5, we collect 1,000 instances (sentences) from each corpus. 7 we substitute each instance i of @xmath6, with each @xmath7, creating |s| -1 new sentences. 8 for example, for an instance of thick from the scale [@xmath8, we generate two new sentences where thick is substituted by each of the other adjectives in the same context. ## indirect question answering we conduct an additional evaluation in order to assess how useful diffvec adjective rankings can be in a real application. as in @xcite , we address indirect question answering (qa) @xcite . the task consists in interpreting indirect answers to yes/no questions involving scalar adjectives. these do not straightforwardly convey a yes or no answer, but the intended reply can be inferred. for example, if someone is asked "was it a good ad?" and replies "it was a great ad", the answer is yes. this makes indirect qa a good fit for scalar adjective ranking evaluation since it allows to directly assess a model's capability to detect the difference in intensity and direction (positive or negative) in an adjective pair. we use the de @xcite dataset for evaluation, which consists of 125 qa pairs manually annotated with their implied answers (yes or no). we adopt a decision procedure similar to the one proposed by @xcite . we compute the bert embeddings of the adjective in the question (a q ) and the adjective in the answer (a a ). if a a (e.g., great) has the same or higher intensity than a q (e.g., good) the prediction is yes; otherwise, the prediction is no. if the answer contains a negation, we switch yes to no, and no to yes. in previous work, indirect qa evaluation was performed on 123 or 125 examples, depending on whether cases labelled as "uncertain" were included @xcite @xcite . we report all available results from previous work, and our scores on the 123 yes/no examples as in the most recent work by @xcite . we report results using diffvec with the adjustment for ties, where two adjectives are considered to be of the same intensity if they are similarly close to the absolute value of dif f @xmath9.01, we count them as a tie. we compare our method to previous results, to freq and sense, and to a baseline predicting always the majority label (yes). results of this evaluation are given in ## discussion our initial exploration of the knowledge encoded in bert representations about scalar adjectives using bertsim (section 5.1) showed they can successfully rank them by intensity. then our diffvec method (sections 5.2 and 5.3) outperformed bert-sim, providing even better ranking predictions with as few resources as a single adjective pair. this difference can be due to the composition of the vectors in the two cases. the a ext representation in bertsim contains information about the meaning of the extreme adjective alongside its intensity, while the ---→ dv ec vector is a cleaner representation of intensity: the subtraction of ---→ a mild from --→ a ext removes the common core meaning expressed by their scale (e.g., beauty, temperature, size). consequently, ---→ dv ec is a pure and general representation of intensity which can successfully serve to rank adjectives from any scale, as shown by our results. the diffvec method can estimate adjectives' relative intensity on the fly, and performs better than the bertsim model which needs a reference point to propose a ranking. it does not use any external knowledge source -a requirement in previous approaches -and one of its highest performing variations (diffvec-1 (+)) makes best quality predictions with a single adjective pair example. our assumption concerning the need for the sentences used for extracting bert representations to be a good semantic fit for adjectives in a scale, has not been confirmed by our evaluation. precisely, differences between our methods when relying on carefully vs randomly selected sentences are minor. this might be due to several reasons: one is that although bert representations are contextualised, they also encode knowledge about the meaning and intensity of words acquired through pre-training, independent of the new context of use. another possible explanation is that due to the skewed distribution of word senses @xcite , a high proportion of our randomly selected sentences might contain instances of the adjectives in their most frequent sense. if this is also the meaning of the corresponding scale, then the sentences are a good fit. the diffvec-1 (+) method, which uses a vector derived from a single positive pair, yields consistently better results than diffvec-1 (-) which relies on a single negative pair. to better understand this difference in performance, we examine the composition of demelo and crowd, specifically whether there is an imbalance in terms of polarity as reflected in the frequency of positive vs negative adjectives in the two datasets. we check the polarity of the adjectives in two sentiment lexicons: so-cal @xcite and @xcite . the two lexicons cover a portion of the adjectives in demelo and crowd: 68% and 79%, respectively. the demelo dataset is well-balanced in terms of positive and negative adjectives: 51% and 49% of the covered adjectives fall in each category. in crowd, we observe a slight skew towards positive: 61% vs 39%. according to this analysis, the difference in performance between the two methods could only partially be explained by an imbalance in terms of polarity. we perform an additional analysis based on the google ngram frequency of the positive and negative words that were used for deriving diffvec. the adjectives good (276m) and awesome (10m) are more frequent than bad (65m) and horrible (4m). in fact, we find that the 1,000 most frequent positive words in so-cal and afinn are, on average, much more frequent (18m) than the 1,000 most frequent negative words (8m). word frequency has a direct impact on word representations, since having access to sparse information about a word's usages does not allow the model to acquire rich information about its linguistic properties as in the case of frequent words. the high frequency of good and awesome results in better quality representations than the ones obtained for their antonyms, and could explain to some extent the improved performance of diffvec-1 (+) compared to diffvec-1 (-) with bert embeddings. however, this analysis does not explain the difference in the performance of diffvec (+) and (-) between bert and word2vec. this would require a better understanding of how words with different polarity (antonyms) are represented in bert's space compared to word2vec, and how negation affects their representations. we leave these explorations for future work. regarding the performance of different bert layers, we observe that knowledge relevant for scalar adjective ranking is situated in the last layers of the transformer network. figure 3 shows how the performance of diffvec-1 (+) changes across different bert layers: model predictions improve after layer 3, and performance peaks in one of the last four layers. this is in accordance with the findings of @xcite that semantic information is mainly located in the upper layers of the model, but is more spread across the network than syntactic information which is contained in a few middle layers. ## conclusion we have shown that bert representations encode rich information about the intensity of scalar adjectives which can be efficiently used for their ranking. although our method is simple and resource-light, solely relying on an intensity vector which can be derived from as few as a single example, it clearly outperforms previous work on the scalar adjective ranking and indirect question answering tasks. our performance analysis across bert layers highlights that the lexical semantic knowledge needed for these tasks is mostly located in the higher layers of the bert model. in future work, we plan to extend our methodology to new languages, and experiment with multilingual and language specific bert models. to create scalar adjective resources in new languages, we could either translate the english datasets or mine adjective scales from starred product reviews as in de @xcite . our intention is also to address adjective ranking in full scales (instead of half-scales) and evaluate the capability of contextualised representations to detect polarity. we use these patterns to detect sentences where adjective substitution should not take place, as described in section 4.2 of the paper. we remove these sentences from our ukwac and flickr datasets. 17