TLDR at S em E val-2022 Task 1: Using Transformers to Learn Dictionaries and Representations (2022)

Abstract: We propose a pair of deep learning models, which employ unsupervised pretraining, attention mechanisms and contrastive learning for representation learning from dictionary definitions, and definition modeling from such representations. Our systems, the Transformers for Learning Dictionaries and Representations (TLDR), were submitted to the SemEval 2022 Task 1: Comparing Dictionaries and Word Embeddings (CODWOE), where they officially ranked first on the definition modeling subtask, and achieved competitive performance on the reverse dictionary subtask. In this paper we describe our methodology and analyse our system design hypotheses.

## introduction dictionaries are some of the linguistically richest resources available for a language, in addition to being extremely clean and unbiased in comparison to most naturally occurring language data, which is noisy and shows domain specific bias based on its source. thus, there has been considerable interest towards using nlp models to harness this knowledge, especially for low-resource languages. broadly there are two sets of approaches towards the same -the first is to use dictionaries for representation learning and using these representations for transfer learning in other tasks, such as in the work by @xcite where they use an lstm based auto-encoder to learn rich representations from dictionary definitions such that the definitions can also be generated back from the representations. @xcite also used dictionary definitions to build sets of 'strong' and 'weak' pairs of words to get improved word representations with greater interpretability by moving words which show a stronger semantic-relatedness closer together in the embedding space. the second approach has been to move in the opposite direction, such as in the work by @xcite where the authors try to map contextualized word representations to their dictionary definitions in an effort towards word-sense disambiguation. this has been further explored by @xcite , where they use an rnn based language model to generate definitions for representations. recent work by @xcite improves upon this by leveraging pre-trained encoderdecoder models like bart @xcite in order to generate the definitions of words. these pre-trained language models, significantly outperform the rnn based models. the ability to generate definitions for representations makes these contextual representations of words explainable. nlp has advanced leaps and bounds within the past decade, with a major push coming from the advent and utilization of transfer learning via representation learning techniques such as word2vec @xcite and elmo @xcite . the more recent methods to employ transfer learning use large pretrained language models, such as bert @xcite and xlm @xcite . these models are jointly used with unsupervised training objectives such as mlm and causal-lm, to transform natural language into information rich meaning representations which are then used for many different downstream tasks. we aim to replicate the same in our experiments to give a better prior for the model, before it learns to generate desired representations. one of the characteristics common to all the language models mentioned above is that they are all based on transformers. transformers use the multiheaded attention and self-attention mechanisms to learn extremely effective language representations. transformers also scale well since unlike their predecessors, the rnns, they process information in parallel and thus are much faster. for semeval 2022's task 1, comparing dictio-naries and word embeddings @xcite , the participants were asked to design systems for the following two subtasks; 1. subtask 1: reconstruct sgns @xcite , character and electra @xcite embeddings from their dictionary glosses. the subtasks are called the reverse dictionary and definition modeling subtasks respectively. the first subtask is evaluated on the mean squared error (mse), cosine similarity and cosine ranking between the generated representations and the gold representations. the second subtask is evaluated on mover score, sense level bleu score (s-bleu) and lemma level bleu score (l-bleu). in this system description paper we detail our model architectures, training, evaluation and testing methodologies, and try to analyse our hypotheses and their impact on the final scores. for the reverse dictionary subtask we designed a simple bert-like model, pretrained it on the mlm objective, and finetuned it for the subtask on a combination of cosine embedding loss and mse loss, alongside negative sampling of the dataset to add a contrastive loss to the overall objective function. for the definition modeling subtask we designed a model based on transformer decoders for natural language generation, with masked self-attention over the inputs in addition to multi-head attention over the representations from the three embeddings spaces. we submitted our systems for evaluation over english data and our systems demonstrated very good performance in the contest itself, with the definition model system outperforming all other submissions and taking first place, and the the reverse dictionary system achieving the fifth, sixth and seventh place on the character, electra and sgns targets respectively. lastly, we also show some post-contest improvements on the reverse dictionary system. the code for our experiments has been open sourced and is available on github. 1 2 subtask 1: reverse dictionary ## conclusion in this paper, we explored a variety of approaches towards dictionary definitions and embeddings generation, especially under the constraint of being unable to use external monolingual data. we have analyzed the effectiveness of each of these methods, performing ablation studies showing the impact that various objective functions like mse, cosine embedding loss and contrastive loss have in reconstructing representations from text, and coming up with an attention mechanism utilizing all the provided representations to generate definitions to produce exceptional results. as part of future work, we plan to explore the performance of our models in multilingual settings. we would also use test the performance of these models against pre-trained language models like bart, bert etc. to gauge the impact language model pre-trainig since we did not have enough monolingual data in the tasks to train them. finally, we plan to experiment with a joint training mechanism where instead of training on each subtask independently, the models can inform and improve each other by collaboratively learning both the subtasks.