The C og AL ex Shared Task on Monolingual and Multilingual Identification of Semantic Relations (2020)

Abstract: The shared task of the CogALex-VI workshop focuses on the monolingual and multilingual identification of semantic relations. We provided training and validation data for the following languages: English, German and Chinese. Given a word pair, systems had to be trained to identify which relation holds between them, with possible choices being synonymy, antonymy, hypernymy and no relation at all. Two test sets were released for evaluating the participating systems. One containing pairs for each of the training languages (systems were evaluated in a monolingual fashion) and the other proposing a surprise language to test the crosslingual transfer capabilities of the systems. Among the submitted systems, top performance was achieved by a transformer-based model in both the monolingual and in the multilingual setting, for all the tested languages, proving the potentials of this recently-introduced neural architecture. The shared task description and the results are available at https://sites.google.com/site/cogalexvisharedtask/ .

## introduction determining whether two words are related and what kind of relations holds between them is an important task in natural language processing, and it has inspired a lot of research for more than one decade @xcite . discovering relations between words is essential also for the creation of linguistic resources, such as ontologies and thesauri @xcite , and this is especially true for specialized domains. research on semantic relations benefited from the success of distributional semantic models @xcite , since they allow to easily generate semantic representations for words from text, in the form of semantic vectors. however, the semantic similarity measured by vector models is an underspecified relation, and it is not easy to tell, given two similar words, in which way they are similar @xcite @xcite . in the previous edition of the cogalex workshop, co-located with coling 2016 in osaka, the organizers set up a shared task dedicated to the corpus-based identification of semantic relations for english @xcite . for the first time, systems were being evaluated in a shared task on the classification of multiple relations at once and, not surprisingly, the task proved to be challenging for computational models. for this new edition of the workshop, we have decided to launch a new version of the same shared task, adding more languages to the evaluation and encouraging the participants to evaluate their system also in a multilingual setting. among the three teams that submitted their systems, the top performance was achieved by a roberta-based system, xlm-r, in all the four languages, and both in the monolingual and in the multilingual setting. ## related work the earlier methods for identifying semantic relations were based on patterns. patterns are generally very precise for identifying relations such as hypernymy-hyponymy @xcite and meronymy @xcite , or even multiple relations at once @xcite , but their limit is that the two related words have to occur together in a corpus, and thus their recall is limited @xcite . distributional models, which do not suffer from such limitations, became then the first choice for the nlp research on semantic relations. in a first phase, researchers focused on the similarity metric, proposing alternatives to cosine that can be more efficient in setting apart a specific semantic relation from the others, e.g. hypernymy @xcite @xcite , synonymy @xcite or antonymy @xcite , or looked for specific differences in their distributional contexts @xcite . in parallel, the first large datasets for evaluating the identification of semantic relations were being released, including relations such as hypernymy, cohyponymy and antonymy @xcite @xcite @xcite . in a second phase, following the increasing popularity of publicly-available frameworks for training word embeddings such as word2vec @xcite and glove @xcite , the focus quickly shifted on the usage of these vectors as features for supervised classifiers. some of these methods train classifiers directly on pairs of vectors @xcite , while others compute dsms-based metrics first and then use them as features @xcite . late attempts to conciliate similarity metrics and word embeddings brought to proposals such as apsyn @xcite . some of the more recent contributions proposed even more sophisticated classification approaches. @xcite aim at integrating word embeddings with information coming from lexical patterns, which proved to be extremely accurate for detecting relations. other researchers introduced modifications to the structure of the vector spaces with the goal of identifying a specific type of semantic relation, for example by modifying the objective function of the word2vec training to inject external knowledge from a lexical resource (e.g. wordnet) @xcite , or by adding an extra postprocessing step that projects the word vectors into a new space, expressly specialized for modeling the target relation @xcite or even more refined techniques of vector space specialisation (e.g. adversarial specialisation) @xcite . however, these contributions mostly tried to address one relation at a time, with rare attempts of tackling the problem in a multiclass setting. the shared task organized in coincidence with cogalex 2016 @xcite was one of the few exceptions, and the low results achieved by most systems (the top f-score being 0.44) showed the difficulty of distinguishing between multiple relations at once. for this reason, we have decided to propose a similar challenge, yet including another factor of complexity: multilingualism. considering the recent approaches that have been introduced for semantic relations in multilingual @xcite , crosslingual @xcite and meta learning @xcite ) settings, we provided datasets in multiple languages (english, german, chinese and italian) and encouraged the participants to train their systems for both monolingual and multilingual evaluation. ## shared task the cogalex-vi shared task was organized as a friendly competition: participants had access to both training and testing datasets, which were respectively released on august 1 and september 1, 2020. the scores of the participating systems were evaluated with the official scripts, and each team had to submit a short paper containing the system description. among the three participants that submitted their systems, one only submitted results for the english data. ## evaluation for the evaluation, participants had to submit their predictions. the output files were expected to contain exactly the same pairs, in the same order, and using the same annotation labels of the gold standard. given the gold standard and the system output, our script calculates precision, recall and f1 score. the weighted performance average across the classes was calculated ignoring the random pairs, as our focus is on the system's capability of detecting actual semantic relations. to this end, only syn, ant and hyp were averaged in the final score. the overall ranking was based on such a weighted average. ## conclusions in this paper, we have described the cogalex-vi shared task, which focused on monolingual and multilingual identification of semantic relations. three teams have submitted their system. all of them have addressed subtask 1 (i.e. identifying relations in languages for which a training set was released: english, german and chinese), while only two teams addressed subtask 2 (i.e. identifying relations in a surprise language: italian). all the submitted systems utilized neural networks, but with different level of complexity. the evaluation shows that transformer-based approaches obtain better performance than other neural methods. these approaches, in particular, are also behind the recent developments in natural language processing, showing incredible capabilities in a large set of domains and applications. probably because of such capabilities, traditional nlp tasks, such as the identification of semantic relations, are gradually losing traction in the community. this shared task meant to show how such core nlp tasks remain a big challenge, which would require more attention by the community to possibly generate even more powerful and robust semantic representations.