C og AL ex- VI Shared Task: Transrelation - A Robust Multilingual Language Model for Multilingual Relation Identification (2020)

Abstract: We describe our submission to the CogALex-VI shared task on the identification of multilingual paradigmatic relations building on XLM-RoBERTa (XLM-R), a robustly optimized and multilingual BERT model. In spite of several experiments with data augmentation, data addition and ensemble methods with a Siamese Triple Net, Translrelation, the XLM-R model with a linear classifier adapted to this specific task, performed best in testing and achieved the best results in the final evaluation of the shared task, even for a previously unseen language.

## introduction determining whether a semantic relation exists between words and which type of relation it represents is a central challenge in numerous nlp tasks, such as extracting terminological concept systems and paraphrase generation. adding a multilingual dimension renders this task at the same time more relevant and more challenging. recent approaches rely on aligned vector spaces for individual languages @xcite or meta-learning approaches @xcite for hypernymy detection and a siamese triple net for antonymy-synonymy distinction inherent in word embeddings @xcite . however, in general a distinction of paradigmatic relations with word embeddings is difficult (im walde, 2020). in a multilingual scenario, frequently lexical resources are utilized to reinforce the model's transfer learning abilities @xcite . given relatively small training datasets and a necessity to support a previously unknown language, we decided to rely on a multilingual pretrained language model. the cogalex-vi shared task focuses on the identification of semantic relations of the types synonymy (e.g. chap and man), antonymy (e.g. big and small), hypernymy (e.g. screech and noise), or random (e.g. ink and closure) between a given word pair. random indicates that the word pair is unrelated. the shared task provided two subtasks. for the first subtask, participating teams were allowed to design monolingual systems being provided training and validation data for the languages mandarin chinese, german, and english. for the second subtask, participating teams were expected to design a single multilingual system that can correctly classify semantic relations in all three languages as well as a previously unknown surprise language, which turned out to be italian. additional resources were permitted with the exclusion of anything related to wordnet @xcite or conceptnet @xcite . our initial intention was to target the second subtask with a multilingual system relying on the state-ofthe-art multilingual model xlm-roberta (xlm-r) @xcite adapted to the task at hand utilizing a linear layer and cogalex-vi training datasets, a model we call transrelation that we provided within the text to terminological concept system (text2tcs) 1 project. to support the model's ability to distinguish relations we experimented with data augmentation, data addition and ensemble methods, joining transrelation 2 with a model trained on a siamese triple net. finally, the adapted xml-r model outperformed all other experiments as well as all other submitted models to cogalex-vi on both tasks. ## results and analysis fig. 1 shows the normalized confusion matrix based on the joined results on all four test sets. besides confusing meaningful relations with random, which can be explained by the fact that random is the majority class, the highest confusion exists between hypernyms and synonyms. for chinese, for instnace, 19 hyp/syn labeled test examples were confused. from these examples, in 11 pairs some characters in one sequence are present in the other, such as 海水-水(sea water -water) (label: hyp) and 船-船舶(ship/boat -ship) (label: syn). this also occurred in four syn/ant labeled examples, e.g. 無 線-有線(wireless -wired) (gold: ant). for the remainder of wrongly classified syn/ant examples, our model frequently selected random, e.g. 私人-公立(private individual -public) (gold: ant). the learning curve shown in fig. 2 plots the achieved weighted f1 score in relation to the number of samples in the training set. for each training set size we trained four models and reported the highest observed score. the model greatly benefits from additional training samples when the training set size is below 8,000. however, the usefulness of adding more data diminishes quickly as the learning curve seems to plateau towards the end. this was confirmed when we tried to add additional training data to data provided by cogalex-vi observing the wordnet/conceptnet exclusion. ## discussion in additional experiments we trained a siamese triplet net @xcite to learn metaembeddings that contrast synonyms and antonyms, which we also tried for hypernym and synonym distinction. however, an ensemble method combining this model and xlm-r performed worse than xlm-r on its own. due to our model's strong performance in chinese we also experimented with data augmentation by machine translating the training and validation sets from chinese to the other languages. the model's performance on these translated datasets was, however, considerably worse than solely on the original untranslated datasets. additionally, performance of both models trained for individual languages or consecutively one language after another lagged considerably behind our final model. given the vast differences in model performance on the different languages, we briefly analyzed the data quality. in the confusion matrix in fig. 1 it becomes evident that our model tended to confuse hypernyms and synonyms a well as random and antonyms. a brief check on the german data where the model performed worse showed that some word pairs labeled as hypernyms might be understood as synonyms by human classifiers, e.g. fett (fat) -dick (plump), unruhig (anxious/restless) -erregt (excited/aroused), and radikal (radical) -drastisch (radical/extreme) could instead be labeled as synonyms. additional training data not related to wordnet or conceptnet we experimented with (e.g. @xcite ) had similar issues and data addition did not improve performance of both the tested models. so on the one hand we attribute this confusion problem of our model to word pairs that might easily be confused by human users. on the other hand, the number of training examples was rather low and data augmentation/addition with high-quality data might have considerably improved performance. depending on the fact that the semantics of these examples change with context, we believe that providing words in context could be one way to alleviate this misclassification problem. one curious example underlining this issue was the result we got for the surprise language italian not seen during training, where farfalla (butterfly) and coccinella (ladybug) are labeled as antonyms, while our system labeled the pair as a synonym. since both can be used to lovingly refer to a young female person in italian, the result of our system could be regarded as correct if the words are understood in this sense. further such examples can be found in great number in the training, validation and test datasets. curiously, performance on mandarin chinese did not seem to be impacted as heavily by this problem, which might be due to the fact that the training datasets were compiled from a different source of different quality. ## conclusion in this paper, we present our system transrelation for the cogalex vi shared task on multilingual relation identification called transrelation. we experimented with data addition, data augmentation and ensemble methods joining pretrained transformer-based models with a siamese triple net. the final system is based on the multilingual pretrained language model xlm-r, which turned out to be the winning system and delivered a strong performance on all four languages, including one previously unknown and unseen additional language. in the future, it would be interesting to apply ideas from curriculum learning @xcite or meta-learning, as already done for simpler models in the case of hypernymy detection @xcite to improve the learning process of our model. this would especially apply to similar scenarios of few available training datasets. furthermore, it would be interesting to evaluate the model's performance on different lexico-semantic relations as well as languages from different language families, e.g. slavic.