C og AL ex- VI Shared Task: Bidirectional Transformer based Identification of Semantic Relations (2020)

Abstract: This paper presents a bidirectional transformer based approach for recognising semantic relationships between a pair of words as proposed by CogALex VI shared task in 2020. The system presented here works by employing BERT embeddings of the words and passing the same over tuned neural network to produce a learning model for the pair of words and their relationships. Afterwards the very same model is used for the relationship between unknown words from the test set. CogALex VI provided Subtask 1 as the identification of relationship of three specific categories amongst English pair of words and the presented system opts to work on that. The resulted relationships of the unknown words are analysed here which shows a balanced performance in overall characteristics with some scope for improvement.

## introduction predicting the relationship between two words in terms of semantics has become a quintessential problem to be solved in the present day nlp world and reflect great impacts on the theoretical psycholinguistic modeling of the mental lexicon as well. the field of nlp finds many useful applications through tackling this direction, such as thesaurus generation @xcite , ontology learning @xcite , paraphrase generation and identification @xcite , question answering and recognizing textual entailment @xcite , as well as drawing inferences @xcite . many nlp applications make use of handcrafted resources such as wordnet @xcite . as a matter of fact, wordnet came from a similar direction with a substantial manual effort. creating such resources is expensive and time consuming; thus efforts of this sort do not cover the variety of languages equally. practically, coverage of a wide range of languages through such manual initiatives also far from completion. many organizations and institutes, who are interested in creating knowledge bases on the field of their practice have attempted to classify such word pairs to shape taxonomies @xcite . lately distributional or corpus based approaches came into popularity for investigating the semantic linkage between words; this approach utilizes the usage and appearance of the words in the corpus. these methods have been able to reflect potential in pattern-recognition-based exploration for word to word semantic mapping through distributional parameters. exploring and connecting semantic relationships are quite difficult and variety of approaches have been tried yet. cognitive aspects of the lexicon vi (cogalex vi) foot_1 has arranged a shared task in 2020; it was looking to explore different efforts to figure out paradigmatic semantic relations, specifically synonymy, antonymy and hypernymy. in the field of nlp these type of relations are notoriously difficult to be distinguished between word pairs given a distribution. to tackle this problem, we employ a deep learning framework to develop training models and test their performances through semantic link prediction between unknown word set. in this paper, we demonstrate our bidirectional transformer based approach to classify whether a given pair of words are semantically connected with one of the three relationships just mentioned above or some other random ones: as asked by this shared task. in the following section 2 we describe the related work, section 3 provides the system description; following that section 4 describes the experimental set up, while section 5 describes the results and section 6 lays out the conclusion and future direction. ## system description the system first reads the training data and preprocess it to a structure consumable further in the process flow. we use bert @xcite embeddings to represent each words. bert adopts the transformer architecture to learn embeddings for words. since each term can have one or more words, we use an lstm layer to track the context of the terms. each of the two terms is first sent into a bert embedding layer providing the embeddings of the two terms. afterword each embedding is individually sent into an lstm layer. the output of the two lstm layers are concatenated and sent to a convolutional layer. the output of the convolutional layer is then flattened and sent to dense layer and further into a softmax output layer of the model as final phase of training. system produced model trained on training data is used for predicting on test data. cross entropy loss has been chosen as the loss function for this solution and a softmax output layer is chosen since multiple semantic relationship classes have to be learnt and predicted. figure 1 shows the system architecture deployed for our experimentation. ## conclusion the current bert based system demonstrates here a prominent approach for recognizing semantical classes between word pairs. this approach has been applied to the shared task proposed by cogalex vi 2 on their subtask 1 for english data to identify synonymy, antonymy, hypernymy or random semantic relationship amongst provided words. these relationships are paradigmatic ones and quite hard for discovering them together. the approach employed here through bidirectional transformer based deep learning network bert handles this task quite well and produce reasonable precision and recall for each category. looking at the difference between the evaluation parameters observed amongst the relationship classes on testing it seems like a thorough investigation should be guided further for better outcomes. to improve the outcomes, introducing variability in deep learning models as well as tuning inside of the model could well have potential. some specific example based error analysis showed there is ample scope of improvement in the identification of closely related word relationships as well relationship prediction for rarely used words. the exploration of the multiclass semantic relationship mappings could be further interesting if the classes for the experimentation are closely connected ones and many in numbers. different distributional training data should be exploited further to train model for evaluating the efficacy of such models on predicting such semantic relationships.