BMEAUT at S em E val-2020 Task 2: Lexical Entailment with Semantic Graphs (2020)

Abstract: In this paper we present a novel rule-based, language independent method for determining lexical entailment relations using semantic representations built from Wiktionary definitions. Combined with a simple WordNet-based method our system achieves top scores on the English and Italian datasets of the Semeval-2020 task “Predicting Multilingual and Cross-lingual (graded) Lexical Entailment” (Glavaš et al., 2020). A detailed error analysis of our output uncovers future di- rections for improving both the semantic parsing method and the inference process on semantic graphs.

## introduction we present a rule-based, multilingual system for detecting monolingual binary entailment between pairs of words using wiktionary definitions, dependency parsing, and semantic graphs. we define entailment over pairs of semantic graphs built from dictionary definitions and achieve near-perfect precision on the semeval-2020 task "predicting multilingual and cross-lingual (graded) lexical entailment" @xcite , where we participate in the any track of the monolingual task that allows for the use of external lexico-semantic resources. our system improves the performance of strong wordnet-based baselines on three languages, achieving top results on english and italian and second-best on german. our pipeline can be easily extended to support any language given a monolingual dictionary and a universal dependency (ud) parser. a detailed error analysis shows multiple directions for further improvement, most notably the refinement of the mechanism responsible for recursively extending semantic graphs based on the definition of its nodes. section 2 briefly describes the lexical entailment task, the 4lang semantic representation @xcite , and the dict to 4lang tool @xcite for generating graphs from dictionary definitions @xcite . section 3 outlines the architecture of our current system and presents our method for detecting entailment over pairs of 4lang graphs. our results on the shared task and a detailed error analysis is presented in section 4. our system is available for download under an mit license from github under https://github.com/adaamko/wikt2def/tree/semeval . ## evaluation we participated in the any track of the semeval-2020 task "predicting multilingual and cross-lingual (graded) lexical entailment", a detailed description of which is available in @xcite . we did not experiment with detecting cross-lingual or graded entailment, our systems produce binary output for pairs of words of the same language only, which we submitted to both the binary and graded subtasks of the monolingual task. we implemented wiktionary extractor modules for three languages: english, german, and italian. for each of these we measured on the development set not only the performance of our best-performing hybrid system but also that of the stand-alone wordnet and 4lang-based systems. figures are shown in the most common case of our method failing to detect entailment between two concepts based on their definition graphs is when the recursive extension of the premise graph contains most of the semantic content of the hypothesis without actually making the connection with the right concept. an example is the entailment pair @xmath0. the graph built from the definition of the premise (an edible plant, lactuca sativa and its close relatives, having a head of green and/or purple leaves.), then extended using the definition of edible (can be eaten without harm) and finally with that of eat (to ingest) will still fail to contain the concept food. such mismatches highlight the need for reducing all such semantic representations to a small common set of defining concepts, a step which could then be performed for both premise and hypothesis words. our future plans include implementing such a reduction along the premise hypothesis premise definition graph chart a data chart (graphical representation of data) intended to illustrate the relationship between a set (or sets) of numbers saturn planet sechster und zweitgrößter planet unseres sonnensystem 'sixth and second-largest planet of our solar system' test esame esame per verificare qualcosa 'exam to check something' ## conclusion we presented a system of entailment detection that relies on a considerable amount of manual work for its data sources: both wikitionary and wordnet were crafted by many years of human labor, and the ud parser trainsets are generally hand-corrected silver or hand-parsed gold sets. but the adaptive layer between these two, consisting mostly in trivial scripts that convert the formats, and a simple rule-based parser to extract the pivot representations from ud parses, is relatively thin, and quite easy to extend to further languages. perhaps the most important takeaway from our work is that the classical resources of computational linguistics are exactly the kind of structures a system needs to learn. as the old miners' adage goes, gold is where you find it. knowledge, in distilled and higly leverageable form, is in the dictionaries. even if our ultimate goal is, as it should be, to extract the knowledge from raw data, experimentation with hybrid systems is warranted by the fact that symbolic systems, and so far only these, can be meaningfully debugged on the kind of relatively small but well-crafted datasets our shared tasks provide. the hungarian scientific research found (otka), contract number 120145. recski was partly supported by brise-vienna (uia04-081), a european union urban innovative actions project.