S em E val-2020 Task 2: Predicting Multilingual and Cross-Lingual (Graded) Lexical Entailment (2020)

Abstract: Lexical entailment (LE) is a fundamental asymmetric lexico-semantic relation, supporting the hierarchies in lexical resources (e.g., WordNet, ConceptNet) and applications like natural language inference and taxonomy induction. Multilingual and cross-lingual NLP applications warrant models for LE detection that go beyond language boundaries. As part of SemEval 2020, we carried out a shared task (Task 2) on multilingual and cross-lingual LE. The shared task spans three dimensions: (1) monolingual vs. cross-lingual LE, (2) binary vs. graded LE, and (3) a set of 6 diverse languages (and 15 corresponding language pairs). We offered two different evaluation tracks: (a) Dist: for unsupervised, fully distributional models that capture LE solely on the basis of unannotated corpora, and (b) Any: for externally informed models, allowed to leverage any resources, including lexico-semantic networks (e.g., WordNet or BabelNet). In the Any track, we recieved runs that push state-of-the-art across all languages and language pairs, for both binary LE detection and graded LE prediction.

## introduction lexical entailment (le; hyponymy-hypernymy or is-a relation) is a core asymmetric lexico-semantic relation @xcite and a crucial building block of lexico-semantic networks and knowledge bases such as wordnet @xcite , babelnet @xcite or conceptnet @xcite . the ability to reason about concept-level entailment supports a plethora of tasks such as taxonomy induction @xcite @xcite , natural language inference @xcite @xcite , and text generation @xcite or metaphor detection @xcite . binary and graded lexical entailment. for this task, we follow the definition of lexical entailment as thoroughly discussed in @xcite , namely as a taxonomical asymmetric hyponymy-hypernymy or is-a relation. although commonly treated as a binary relation ("is x a type of y?"), cognitive theories of concept (proto)typicality and category vagueness @xcite suggest that le is rather a graded relation: humans can perceive the degree to which the le relation holds between concepts ("to which degree is x a type of y?"). 1 the graded nature of the le relation has been empirically validated in human annotations crowdsourced for the hyperlex dataset @xcite . its creation catalyzed research on models for predicting graded le @xcite @xcite @xcite . multilingual and cross-lingual lexical entailment. despite its potential for a variety of crosslingual and multilingual applications such as multilingual taxonomy construction, machine translation, and multilingual natural language inference @xcite @xcite @xcite , le detection, especially its graded variant, has so far been predominantly studied in monolingual settings @xcite @xcite @xcite @xcite with most models and evaluations, unsurprisingly, focusing on english. existing work on multilingual and cross-lingual le @xcite @xcite has been rather limited and focused dominantly on major and mutually similar languages and binary le detection. shared task. aiming to catalyze the development of models for predicting le, we organized the shared task described in this paper. our shared task had a broad scope aiming to cover reasoning over lexical entailment from multiple perspectives. namely, the subtasks covered both monolingual and cross-lingual setups as well as both binary le detection and graded le prediction (i.e., prediction of a degree to which le holds between concepts). our shared task encompassed a set of 6 etymologically and typologically diverse languages (and 15 corresponding language pairs): english (en), german (de), italian (it), croatian (hr), turkish (tr), and albanian (sq). we offered two different evaluation tracks. in the distributional (dist) track we allowed only for fully distributional systems, capturing le only on the basis of unannotated corpora. in contrast, the any track invited systems that exploit any kind of additional external resources, including lexico-semantic networks. overall, we did not observe any empirically confirmed strong systems in the dist track, further corroborating the findings from prior work that building le-oriented vectors distributionally is more difficult than for some other relations such as broader semantic relatedness @xcite . however, several runs submitted to the any track pushed the state of the art both in binary le detection and graded le prediction, for most of the languages and language pairs in our evaluation. ## data we started from the le datasets we previously created and published @xcite , covering four languages (en, de, it, hr) and extended those datasets to two new languages (tr, sq). for completeness, we describe the details of the annotation process and the creation of final multilingual and cross-lingual datasets for the shared task. starting point: graded le in english. hyperlex @xcite comprises 2,616 english (en) word pairs (2,163 noun pairs and 453 verb pairs) annotated for the graded le relation. unlike in symmetric similarity datasets @xcite @xcite , word order in each pair (x, y ) is important: this means that pairs (x, y ) and (y, x) can obtain drastically different graded le ratings. the word pairs were first sampled from wordnet to represent a spectrum of different word relations (e.g., hyponymy-hypernymy, meronymy, co-hyponymy, synonymy, antonymy, no relation). the ratings in the [0, 6] interval were then collected through crowdsourcing by posing the graded le "to what degree is x a type of y?" question to human subjects, with each pair rated by at least 10 raters: the score of 6 indicates a perfect le relation between the concepts x and y (in that order), and 0 indicates absence of the le relation. the final score was averaged across individual ratings. the final en hyperlex dataset reveals that gradience effects are indeed present in human annotations: it contains word pairs with ratings distributed across the entire [0, 6] rating interval. ## participating systems we now describe in more detail the approaches adopted by the three teams who submitted their system description papers. 3 team bmeaut @xcite . the bmeaut method for le detection and prediction is a rule-based approach that exploits wiktionary definitions @xcite and relies on dependency parsing and semantic graphs. in the first step, the authors apply the dict to 4lang tool @xcite on wiktionary definitions of concepts (which can be both unigrams and multi-word expressions, i.e., phrases) in order to induce the directed graphs conforming to the 4lang formalism @xcite . 4lang graphs are directed graphs with concepts as nodes and three types of edges: edges of type 0 denote attribution (cat 0 -→ four-legged), lexical entailment (cat 0 -→ mammal), or unary predication (cat 0 -→ meow); edges of type 1 and 2 denote relations between the predicate and its subject and object, respectively (e.g., cat 1 ←catch 2 -→ mouse). 4 kovács et al. (2020) first extract definitions from wiktionary using language-specific templates. each definition is then transformed into a 4lang graph with the help of a language-specific universal dependencies @xcite parser. let (x, y) be the candidate word pair from one of our test sets. @xcite then start from the 4lang graph of x and include all concepts to which x has an outgoing edge of type 0. they then iteratively expand this initial 4lang graph of x by adding in each iteration all concepts for which they have an incoming edge of type 0 from any of the nodes already in the graph. if, via this extension, they get to include y into the graph, they conclude (in the binary setup) that x indeed entails y, i.e., that (x, y) is a positive le instance. somewhat expectedly, this rule-based approach has a very high precision, but relatively low recall. this is why, in the next step, the authors augment their 4lang graph containing extractions from wiktionary definitions with the hypernymy pairs from wordnet (for en and it they use language-specific wordnets, for de they translate german terms to english and use english wordnet). the reliance on time-consuming rule-based design of language-specific wiktionary extractors and language-specific le detection rules on 4lang graphs prevented the authors from submitting predictions for lower-resource languages (hr, sq, tr). also, the authors did not submit any graded le predictions nor cross-lingual predictions. building reliable scores on a continuous scale has been proven difficult for semantic similarity @xcite and distributional similarity measures have been shown to perform better. we presume something similar to be the case with le and the proposed 4lang graphs -it is inherently difficult to create a reliable le score based on paths and distances in a symbolic representation that is a (directed) graph. team ualberta @xcite . the approach of ualberta for cross-lingual binary le detection combines sentence-level translations (i.e., parallel corpora), distributional word vectors (i.e., word embeddings) and multilingual lexical resources. their base method, dubbed bitext, mines candidates for cross-lingual le from parallel corpora -they simply run the fastalign @xcite word alignment algorithm and assume that the le relation holds between all aligned pairs of words. as clarified by the authors, this will, in most cases, extract cross-lingual synonyms, which, strictly speaking, do satisfy the le relation; also, in some cases, the alignments will be established between close (e.g., first order) hyponymy-hypernymy pairs -in this case, however, the bitext alignment of words alone does not suggest the direction of the le relation. the authors simply declare any pair of words from our cross-lingual datasets to stand in the le relation if they find this pair in the list of generated word alignments. the second run, dubbed vectors, extends bitext by exploiting similarities between distributional embeddings of words. to paraphrase the authors, the intuition behind vectors is that mutually semantically similar concepts tend to entail the same set of other concepts. let (x, y) be a cross-lingual word alignment generated with bitext. let @xmath0, . . . , x n } be the set of terms which are semantically most similar to x in one language and @xmath1, . . . , y m } be the set of terms which are semantically most similar to y in the other language. note that this does not require a bilingual word embedding space, merely two monolingual word embedding spaces: sets x and y are obtained by thresholding monolingual word embedding similarities (the threshold value is tuned on the development portions of our le sets). finally, all possible pairs (x i , y j ) ∈ x × y are considered to stand in the le relation. finally, in the third run, the authors couple the bitext-based fasttext aligner with the babelalign algorithm, which aligns concepts across languages based on babelnet @xcite , a massively multilingual lexico-semantic network. team shikeblcu @xcite . the approach of @xcite extends the wellestablished line of work based on specializing (i.e., fine-tuning) distributional word vectors for lexical relations, be it symmetric semantic similarity @xcite @xcite @xcite or the asymmetric le relation @xcite @xcite , using constraints from external lexico-semantic resources like wordnet for supervision. at the core of the approach is the lexical-entailment attract repel (lear) @xcite , a retrofitting model that specializes distributional vectors of words from external constraints (synonyms, antonyms, and le pairs). @xcite first le-specialize with lear monolingual word embedding spaces of each language independently using language-specific constraints collected from conceptnet @xcite . the number of constraints they obtain for other languages is, however, significantly smaller than for en, with especially few constraints obtained for lower-resource languages -hr, tr, and sq. because of this, they add additional constraints in target languages by translating en constraints to target languages via google translate. a similar approach of automatic constraint translation has already been proven very effective in the context of symmetric similarity-based specialization of embedding spaces for low-resource languages @xcite . this way, @xcite obtain an le-specialized embedding space for each language. following that, in the second step, they learn a linear projection mapping between the le-specialized monolingual spaces with the vecmap tool for inducing bilingual word embedding spaces @xcite . recent comparative evaluations @xcite ) rendered vecmap as one of the most robust algorithms for inducing cross-lingual embedding spaces. the word translations obtained with google translate when translating en constraints are also forwarded to vecmap as supervision for inducing bilingual embedding spaces. ## official evaluation we now report the official results of our evaluation. we first describe the baselines (section 5.1) and then show the performances for all submitted runs (section 5.2). foot_3 ## conclusion as a fundamental asymmetric lexico-semantic relation, lexical entailment (le) supports construction of concept hierarchies and downstream applications that require reasoning and inference. in multilingual and cross-lingual applications we need models that can detect le across languages. this is why we carried out a semeval shared task (task 2) on predicting le, spanning (1) monolingual vs. cross-lingual le detection, (2) binary le detection vs. graded le prediction, and (3) a set of 6 diverse languages (and 15 language pairs). in the track in which we allowed for any external resources to be used (any track), we received submissions that substantially push the state of the art across all languages and language pairs, for both binary le detection and graded le prediction. we hope that these methodological advances, instigated by the semeval task and the constructed datasets, will inform and inspire further work in fields such as multilingual taxonomy induction and language inference.