Scalar Adjective Identification and Multilingual Ranking (2021)

Abstract: The intensity relationship that holds between scalar adjectives (e.g., nice < great < wonderful) is highly relevant for natural language inference and common-sense reasoning. Previous research on scalar adjective ranking has focused on English, mainly due to the availability of datasets for evaluation. We introduce a new multilingual dataset in order to promote research on scalar adjectives in new languages. We perform a series of experiments and set performance baselines on this dataset, using monolingual and multilingual contextual language models. Additionally, we introduce a new binary classification task for English scalar adjective identification which examines the models’ ability to distinguish scalar from relational adjectives. We probe contextualised representations and report baseline results for future comparison on this task.

## introduction scalar adjectives relate the entities they modify to specific positions on the evoked scale (e.g., good-ness, temperature, size): a wonderful view is nicer than a good view, and one would probably prefer a delicious to a tasty meal. but not all adjectives express intensity or degree. relational adjectives are derived from nouns (e.g., @xmath0, @xmath1, have no antonyms and serve to classify nouns (e.g., a wooden table, a chemical substance) @xcite . the distinction between scalar and relational adjectives is an important one. identifying adjectives that express intensity can serve to assess the emotional tone of a given text, as opposed to words that mostly contribute to its descriptive content. additionally, estimating the intensity of a scalar adjective is useful for textual entailment (wonderful |= good but good |= wonderful), product review analysis and recommendation systems, emotional chatbots and question answering @xcite . work on scalar adjectives has until now evolved around pre-compiled datasets (de @xcite @xcite . reliance on external resources has also restricted research to english, and has led to the prevalence of pattern-based and lexicon-based approaches. recently, garí soler and apidianaki (2020) showed that bert representations @xcite encode intensity relationships between english scalar adjectives, paving the way for applying contextualised representations to intensity detection in other languages. 1 in our work, we explicitly address the scalar adjective identification task, overlooked until now due to the focus on pre-compiled resources. we furthermore propose to extend scalar adjective ranking to new languages. we make available two new benchmark datasets for scalar adjective identification and multilingual ranking: (a) scal-rel, a balanced dataset of relational and scalar adjectives which can serve to probe model representations for scalar adjective identification; and (b) multi-scale, a scalar adjective dataset in french, spanish and greek. in order to test contextual models on these two tasks, the adjectives need to be seen in sentential context. we thus provide, alongside the datasets, sets of sentences that can be used to extract contextualised representations in order to promote model comparability. we conduct experiments and report results obtained with simple baselines and state-of-the-art monolingual and multilingual models on these new benchmarks, opening up avenues for research on sentiment analysis and emotion detection in different languages. foot_1 2 the datasets ## scalar adjective identification for each english adjective in the scal-rel dataset, we generate a representation from the available ten sentences (cf. section 2.2) using the bert-base-uncased model (with wp and wp-1). we experiment with a simple logistic regression classifier that uses the averaged representation for an adjective (adj-rep) as input and predicts whether it is scalar or relational. we also apply the diffvec-1 (+) method to this task and measure how intense an adjective is by calculating its cosine with ---→ dv ec. the absolute value of the cosine indicates how clearly an adjective encodes the notion of intensity. in figure 1 , we show two scalar adjective vectors with negative and positive cosine similarity to ---→ dv ec, and another vector that is perpendicular to ---→ dv ec, i.e. describing a relational adjective for which the notion of intensity does not apply. 10 we train a logistic regression model to find a cosine threshold separating scalar from relational adjectives (dv-1 (+)). finally, we also use as a feature the cosine similarity of the adjective representation to the vector of "good", which we consider as a prototypical scalar adjective (proto-sim). the best bert layer is selected based on the accuracy obtained on the development set. we report accuracy on the test set. the baseline classifiers only use frequency (freq) and polysemy (sense) as features. we use these baselines on scal-rel because the wordnet pertainyms included in the dataset are rarer than the scalar adjectives. the intuition behind the sense baseline explained in section 3.1 also applies here. results on this task are given in ## conclusion we propose a new multilingual benchmark for scalar adjective ranking, and set performance baselines on it using monolingual and multilingual contextual language model representations. our results show that adjective intensity information is present in the contextualised representations in the studied languages. we also propose a new classification task and a dataset that can serve as a benchmark to estimate the models' capability to identify scalar adjectives when relevant datasets are not available. we make our datasets and sentence contexts available to promote future research on scalar adjectives detection and analysis in different languages. bryan wilkinson and tim oates. 2016. a gold standard for scalar adjectives. in proceedings of the tenth international conference on language resources and evaluation (lrec'16), pages 2669-2675, portorož, slovenia. european language resources association (elra). george kingsley zipf. 1945. the meaning-frequency relationship of words. journal of general psychology, 33(2):251-256.