SHIKEBLCU at S em E val-2020 Task 2: An External Knowledge-enhanced Matrix for Multilingual and Cross-Lingual Lexical Entailment (2020)

Abstract: Lexical entailment recognition plays an important role in tasks like Question Answering and Machine Translation. As important branches of lexical entailment, predicting multilingual and cross-lingual lexical entailment (LE) are two subtasks of SemEval2020 Task2. In previous monolingual LE studies, researchers leverage external linguistic constraints to transform word embeddings for LE relation. In our system, we expand the number of external constraints in multiple languages to obtain more specialised multilingual word embeddings. For the cross-lingual subtask, we apply a bilingual word embeddings mapping method in the model. The mapping method takes specialised embeddings as inputs and is able to retain the embeddings’ LE features after operations. Our results for multilingual subtask are about 20% and 10% higher than the baseline in graded and binary prediction respectively.

## introduction lexical entailment (le) refers to the hyponymy-hypernymy relation, also known as type-of, or is-a, which is a fundamental asymmetric lexical relation @xcite . it is a basic requirement for tasks like question answering (qa) and recognizing textual entailment (rte). and more general reasoning over cross-lingual and multilingual le relationships can improve language understanding in multilingual contexts @xcite . cross-lingual le recognition is crucial to tasks such as recognizing cross-lingual textual entailment @xcite and machine translation @xcite . predicting binary and graded scores for multilingual and cross-lingual lexical entailment is the task of semeval 2020 task 2 @xcite . there are two subtasks. subtask a is to predict binary or graded le on a monolingual pair of words, e.g., (building, construction) and the subtask is in six multiple languages (i.e., english, german, italian, croatian, turkish, albanian). subtask b, predicting cross-lingual le, gives a pair of words in two different languages with prefix, e.g., (en dinosaur, de kreatur 1 ). there are 15 cross-lingual pairs as languages above combine with each other, i.e., de-x, en-x, it-x, hr-x and sq-x sets 2 . each subtask includes both binary and graded prediction for a given pair of words. binary prediction is to determine whether there is a le relation between two concepts, while graded lexical entailment (gr-le) measures the strength of le relation on a continuous 0-6 scale @xcite . for instance, (apple, fruit) gains a score of 6, while the pair (apple, flower) gets 1.2. this is because apple is more like a kind of fruit instead of flower. since le is an asymmetric relation, the score of (fruit, apple) is not the same as (apple, fruit). researches in monolingual gr-le mainly focus on training le-specialiased word embeddings based on external linguistic constraints @xcite @xcite . the constraints, namely external knowledge, are some synonymy (pretty, beautiful), antonymy (nice, bad) and lexical entailment word pairs (sandwich, food). these are useful for le relation and are extracted from lexical resources (such as wordnet). the size of external constraints decides the size of word embeddings get trained, thus for each language, the amount of external knowledge needs to be large enough. for cross-lingual gr-le prediction, previous works transfer the space from the source language to the target language @xcite . the basis of the methods is to obtain unified bilingual word embeddings, which can also be trained by mapping two languages space into a shared one based a bilingual dictionary @xcite . after getting bilingual embeddings, operations in cross-lingual models are similar with the monolingual one. in our work, we apply massive external constraints to train specialised word embeddings for monolingual le. and we introduce a bilingual word embedding mapping method on cross-lingual subtask. the inputs of the mapping method are the le-specialised word embeddings which are outputs of subtask a. several experiments are conducted to prove the effect of external constraints. our contributions are as follows: ## system description the components for our system are shown in figure 1 . we propose a system that focuses on transforming word embeddings for le relation. for monolingual le subtask, the inputs are word pairs and external lexical constraints in the same language. we employ lear @xcite to train specialised input vectors based on external constraints. the method was used for english gr-le. here we use the model for all six languages with more external constraints added. after transforming the input embeddings, the outputs are monolingual le-specialised word embeddings (section 2.1). next, to solve cross-lingual subtask, we treat the outputs of subtask a as inputs for a bilingual mapping method to map two different vector spaces into a shared one (section 2.2). the final step of both subtasks is to score the entailment relation using the trained word embeddings (section 2.3). 2.1 subtask a: monolingual le training pairs) closer, while the repel (rep) objective pushes antonymy pairs (repel pairs) away from each other. meanwhile, the model adjusts vector norms so that the higher-level concepts have larger norms and lower-level concepts have smaller norms in euclidean space. the set of k word pairs for which the att or rep score is to be computed is denoted by the most important objective of the method is an asymmetric distance-based objective which aims to rearrange norms of vectors. this is to obtain specialised vectors reflecting the asymmetry of the le relation. we adopt the best-performing asymmetric objective from @xcite : figure 2 shows the whole transformation of word embeddings. first, the model adjusts the vectors direction according to att objective or rep objective. this step captures the symmetric similarity of word pairs. and next step is to rearrange vector norms according to le(b l ), so that norms reveal the concepts' level. the final transformed word embeddings are saved for the following task. ## conclusion we use lear model to fine-tune the input word vector spaces, and expand the number of external constraints used in the model to obtain more global word embeddings. the results demonstrate that with more knowledge added into the model, especially le constraints, the relation among vectors will be more significant and beneficial to the monolingual le relation prediction. and we apply above specialised word embeddings and the mapping method in cross-lingual word embeddings models to predict cross-lingual le. for future work, we think bi-dictionary and proper polysemy process should improve the performance. besides, we will consider applying external constraints in both languages into the cross-lingual model since they work well in monolingual le prediction.