Manual Clustering and Spatial Arrangement of Verbs for Multilingual Evaluation and Typology Analysis (2020)

Abstract: We present the first evaluation of the applicability of a spatial arrangement method (SpAM) to a typologically diverse language sample, and its potential to produce semantic evaluation resources to support multilingual NLP, with a focus on verb semantics. We demonstrate SpAM’s utility in allowing for quick bottom-up creation of large-scale evaluation datasets that balance cross-lingual alignment with language specificity. Starting from a shared sample of 825 English verbs, translated into Chinese, Japanese, Finnish, Polish, and Italian, we apply a two-phase annotation process which produces (i) semantic verb classes and (ii) fine-grained similarity scores for nearly 130 thousand verb pairs. We use the two types of verb data to (a) examine cross-lingual similarities and variation, and (b) evaluate the capacity of static and contextualised representation models to accurately reflect verb semantics, contrasting the performance of large language specific pretraining models with their multilingual equivalent on semantic clustering and lexical similarity, across different domains of verb meaning. We release the data from both phases as a large-scale multilingual resource, comprising 85 verb classes and nearly 130k pairwise similarity scores, offering a wealth of possibilities for further evaluation and research on multilingual verb semantics.

## introduction many recent efforts in semantic modeling have focused on unsupervised pretraining to extend the benefits offered by recently proposed text encoders @xcite to new languages and domains. in these approaches, general language representations are learned from large volumes of unlabeled text, and subsequently leveraged in downstream systems by means of fine-tuning on a given supervised task. the release of large multilingual pretrained encoders @xcite boosted the state of the art on a range of multilingual tasks @xcite @xcite @xcite @xcite . in parallel, the number of language-specific pretrained architectures available has also been steadily growing, with the advantage of being more attuned to the properties of the language in question @xcite . the ease of incorporating these powerful encoders into downstream task pipelines has made them widely popular. however, there is a disproportionate shortage of resources allowing for probing of the learned representations in most languages. the aim of this work is to address this deficit by releasing a multilingual resource targeting verb semantics in a typologically diverse selection of languages where no such datasets have hitherto been available. the motivation behind the specific focus on verbs is twofold: (i) the importance of accurate and nuanced representation of verb meaning in light of their pivotal role in sentence structure and the still subpar verbal reasoning ability of sota models @xcite , and (ii) the scarcity of verb data in evaluation datasets currently available. to this end, we employ a recently proposed two-phase data collection method @xcite combining semantic clustering (phase 1) and finer-grained spatial arrangements of words based on their similarity (phase 2), and evaluate its cross-lingual applicability. using cross-lingual mappings, ## background and design motivation word similarity has been widely used as a go-to intrinsic evaluation task, in which rankings of similarity scores computed between word embeddings produced by representation models are compared against ranked human similarity judgments. the dataset design involving sets of word pairs and their associated rating on a discrete scale has been particularly common, due to its reliance on non-expert native speaker judgments, quicker and cheaper to obtain than the large expert-curated lexical-semantic or semanticsyntactic resources such as wordnet @xcite or verbnet @xcite . in english, examples include wordsim-353 @xcite , men @xcite and simlex-999 @xcite . analogous datasets have been created in other languages, either through translation from an existing english dataset (e.g., from simlex: german, italian, and russian @xcite , hebrew and croatian @xcite and polish @xcite ), or from a new set of concept pairs (e.g., turkish @xcite , mandarin chinese @xcite , japanese @xcite ). while these datasets are dominated by nouns (e.g., simlex includes 222 verb pairs), verb-oriented datasets are harder to come by. in english, these include datasets of @xcite (130 verb pairs), @xcite (143 verb pairs), @xcite (3,500 verb pairs). a recent multilingual word similarity dataset, multi-simlex @xcite , extends coverage of verb semantic similarity to 469 verb pairs in 12 languages, including mandarin chinese, finnish, and polish. another recently introduced large-scale english verb resource of @xcite (hereafter spa-verb) comprises verb classes and unmatched coverage of nearly 30k verb similarity scores. in this work, we demonstrate that their large-scale dataset creation methodology based on spatial arrangement (spam) can be extended to other and typologically diverse languages such as mandarin chinese, japanese, finnish, polish, and italian. for each language, we create a dataset comprising 16-18 verb classes with similarity scores between all class members, resulting in over 20k verb pair similarity scores within each language ( we start from the english spa-verb sample translated into five target languages and apply the two-phase annotation method combining semantic clustering and spatial arrangements based on semantic similarity proposed in @xcite . the method adapts a spam approach previously used in cognitive science and psychology in behavioural studies of visual similarity between concrete objects @xcite @xcite to lexical stimuli. in phase 1, a large word sample is divided into a number of broad categories of similar and related items. each of these classes is then used as input in phase 2, where the related class members are arranged in a 2d space based on their semantic similarity, with similar words placed closer together. each item placement simultaneously communicates figure 1 : consecutive phase 2 trials on a class of polish emotion verbs. in the first trial (1-2), the whole class is displayed around the arena and word labels are placed one by one based on the similarity of their meaning. words put closer together in the first trial (2) are subsampled for the subsequent trial (3), and arranged again in a less crowded space (annotators are asked to use the entire space available in each trial and the relative inter-item distances, not the absolute on-screen distances, represent the dissimilarities). the main advantages of the spatial arrangement method lie in its intuitiveness, rooted in psychology @xcite @xcite , and flexibility, due to the reliance on fluid item placements simultaneously expressing multi-way similarity judgments, rather than discrete numerical scores. by repeatedly considering subsets of items, the users reflect on relative differences in meaning between different configurations of words, which decreases bias from placement error, order of presentation and judgment context. the two-phase design offers a practical advantage for porting the method to other languages. the approach starts from a verb sample, rather than a set of word pairs, which allows for easy translation into the target language, avoiding many of the complications encountered in translation of pairs, including cases where both words in the source language pair translate into the same word (e.g., cup -@xmath0, or several pairs in the source language translate into identical target pairs (e.g., easy -hard, easy -@xmath1. foot_0 ## data collection and analysis we sampled languages from 5 different language families to ensure typological diversity: sino-tibetan (mandarin chinese zh), japonic (japanese ja), uralic (finnish fi), slavic (polish pl) and romance (italian it). following translation from english (en), the two data collection phases were set up on an online platform (meadows-research.com) as two separate studies for each language. recruitment was carried out on a crowd-sourcing website, prolific.co. participants were native speakers of the target language with at least undergraduate education level and at least a 90% approval rating. each phase featured a short qualification task testing the participants' understanding of the guidelines. ## evaluation evaluation is focused on two types of representation architectures: static word embeddings @xcite and more recently proposed large pretrained encoders @xcite . we compare their ability to capture word-level semantics across languages and domains of verb meaning. we also contrast the performance of language-specific bert models with their massively multilingual counterpart @xcite , and examine the impact of computing word-level representations in context, rather than by feeding items to a pretrained model in isolation. representation models. we evaluate fasttext (ft) as a representative non-contextualised word embedding model with proven representation capabilities on diverse nlp tasks @xcite and coverage of 157 languages. for multi-word expressions, we compute their representations by averaging the vectors of their constituent words. we contrast the performance of ft vectors with the omnipresent state-of-the-art bert model @xcite . we derive word-level bert representations of words and multi-word expressions in two different ways: (a) in isolation and (b) in context. in method (a), we follow the steps of @xcite 1 ). we report scores for (m-)bert embeddings computed in isolation (iso) and in context (ctx) (see §4). the h hidden representations for each of the subword tokens constituting the item, and finally (3) taking the average of these subword representations to obtain the final d-dimensional representation (@xmath2. again, following @xcite , we average over all layers (12 with bert-base). in (b), we encode word meaning in context of other words using external corpora foot_3 in the following way. first, we randomly sample n sentences containing each item in the corpus, then, we compute the item's representation in each of n sentential contexts (averaging over constituent subword representations and hidden layers as in steps ( 2 )-(3) above), and finally average over the n sentential representations to obtain the final representation for each item. 5 we evaluate the uncased multilingual bert model (m-bert) @xcite , pretrained on monolingual wikipedia corpora of 102 languages, as well as language-specific pretrained bert encoders released for zh, ja (bert-base with and without whole word masking (+wwm)), pl, fi, and it (bert-base and bert-base-xxl trained on a larger italian corpus), available in the transformers repository @xcite ). 4 ## conclusion and future work we presented the first large-scale multilingual evaluation resource, constructed via spatial arrangement and targeting verb semantics in chinese, japanese, polish, finnish, and italian. it includes semantic classes and fine-grained pairwise lexical similarity scores, which we release with this paper. the dual nature and vast coverage of the dataset enables evaluation of representation models on two tasks, semantic clustering and word similarity, and focused probing analyses on specific semantic domains, revealing aspects of verbal meaning which elude models' representation capacity. the low overall model performance indicates that estimating similarity between a large number of semantically proximate concepts linked by fine-grained relations is a challenging task. in future work, we will use the spatial arrangement data for in-depth analyses of cross-lingual typological variation and model evaluation on fine-grained semantic clusters from phase 2 to explore the potential for (semi-)automatic creation of verb classes and semantic resources in languages where those are still lacking. we will also evaluate cross-lingual representation learning algorithms on mapped cross-lingual verb similarity datasets for all language pairs created in this project.