RIGA at S em E val-2022 Task 1: Scaling Recurrent Neural Networks for CODWOE Dictionary Modeling (2022)

Abstract: Described are our two entries “emukans” and “guntis” for the definition modeling track of CODWOE SemEval-2022 Task 1. Our approach is based on careful scaling of a GRU recurrent neural network, which exhibits double descent of errors, corresponding to significant improvements also per human judgement. Our results are in the middle of the ranking table per official automatic metrics.

## introduction the definition modeling track of semeval-2022 task 1: codwoe -comparing dictionaries and word embeddings @xcite challenged participants to generate dictionary glosses from individual word embedding vectors. this paper describes two codwoe submissions, "emukans" and "guntis", where the first focuses on the automatic codwoe scores, but the second attempts to gauge the relationships between scaling laws, the automated metrics, and human evaluation. our submissions achieved competitive results (see figure 3 ) on the moverscore official metric -scoring 1st for french, 2nd for spanish, and 3rd for russian. our approach was to apply classical recurrent networks, such as long short-term memory (lstm) @xcite and gated recurrent units (gru) @xcite , to definition modeling and investigate how model scaling impacts performance. the scaling effect is well investigated for transformers, but not so much for rnns. recently, the main focus in deep learning has skewed from searching for new model architectures to investigating how various factors impact the training process and overall system performance @xcite @xcite . the main factors are: the amount of data, the amount of compute, and the size of the model (parameter count). in the competition the data amount is fixed and no use of external data is permitted, thus we investigate how scaling model size and training time impacts training progress and model performance for recurrent models. in our experiments we did observe deep double descent effects: epoch-wise double descent with respect to both cross-entropy loss and prediction accuracy on a validation data set, more pronounced with increasing model size. we also investigated the automatic metrics used for evaluating submissions and their correlation with human evaluation, focusing primarily on the moverscore metric @xcite . moverscore does correlate with human evaluation, but not necessarily very strong, at least for this dataset. we find that the double descent effect seen with respect to prediction accuracy can also be observed for moverscore. ## background the codwoe shared task invites participants to compare two types of semantic descriptions: dictionary glosses and word embedding representations. the task consists of 2 subtracks: definition modeling and reverse dictionary. in definition modeling participants have to generate glosses from word embedding representations. the reverse dictionary task is the inverse: reconstruct a word embedding from the corresponding gloss. considering results achieved by the baseline models provided by the organisers, we decided to participate only in the definition modeling track, as it seems the more challenging task, with more room for potential improvement. for the definition modeling track, inputs are 256dimensional embedding vectors and outputs are plain text. data is provided for 5 languages: english, french, spanish, italian and russian. every language is scored separately. we submitted for all 5 languages. the provided word embedding vec-tors are of 3 types: char, sgns, and electra. ## system overview for the definition modeling task, we used classical recurrent networks, experimenting with both lstm and gru architectures. we added an initial fully connected input layer to scale a given word embedding vector to higher dimensions. we used the adam optimizer @xcite in the training process. the learning rate was set in the @xmath0, 3e-5, 1e-4]. a linear learning rate decay schedule with warm-up over 0.01 was used. no preprocessing was applied to training data. the code is available on github 1 the very first step is creating a tokenizer and building its vocabulary. we use sentencepiece tokenization @xcite , trained on the training dataset only. we carried out experiments across a range of vocabulary sizes. we used a classical approach and a decoder only part of standard seq2seq (sequence to sequence) recurrent neural network models without attention. the gru/lstm state vector is initialized from the given defmod embedding vector. in our case, we use a single word embedding vector type. for the first time step we pass a @xmath1. the model outputs a single predicted token and a new state vector. to avoid exploding gradients, the outputs of the model are normalized. the token selected by the model is appended to the generated gloss, and is also used as input to the model for the next time step. this process is repeated until reaching the iteration limit. at every time step, the model can make mistakes. if the initial part of the input sequence fed to a seq2seq model is bad, most likely the subsequent output sequence will also be wrong. to mitigate this accumulation of errors and speed up the training process, we use the teacher forcing technique @xcite . with teacher forcing, the model is trained by supplying input tokens from the target sequence of the dataset and using the network's one-step-ahead predictions to do multi-step sampling. we also tried a more advanced teacher forcing technique: scheduled sampling @xcite , where input sequence tokens are given ground-truth values only with some probability. unfortunately, scheduled sampling did not give good results -the loss plot was very noisy. it is likely that the codwoe definition modeling task itself is a very hard task with too much variability relative to the amount of provided training data; scheduled sampling might be better suited for language model fine-tuning when the model weights are pretrained on a large corpus and already correlate fairly well with natural language syntax and semantics. after each training epoch, the model is evaluated on a validation dataset using the same cross-entropy loss function as used for training. we also use an accuracy metric for evaluating model performance, as it correlates with perplexity and human judgement for large language models. the accuracy is calculated by dividing the count of correctly predicted tokens (under teacher-forcing) by the number of total tokens. for "emukans" submissions, model training is stopped using early stopping @xcite based on the accuracy score for the validation data, while "guntis" submissions were intentionally trained long past the overfitting point to observe scaling and double descent effects. ## experimental setup for our experiments, we have 5 tesla v100 16gb gpus provided by our institute. during the competition, our focus was on exploring different training effects and model tuning. most of the experiments were focused on primary factors of "scaling laws": model size and the amount of compute (training epochs). for simplicity and consistency of presentation, in most of the following tables and figures (all except for figure 3 ) we report experimental performance evaluated against a trial dataset provided by the codwoe organizers, which consists of only 200 glosses. apart from the automatic metrics, our focus was on (informal) manual evaluation of generated glosses. ## results our team "emukans" and "guntis" placed in the middle of the final ranking table. however, if we inspect the scores in figure 3 , we see that our solution (a green line) does outperform others in some metrics for some languages (i.e., top score for french, 2nd for spanish, 3rd for russian). analysing our submission results, we noticed that moverscore can give even negative scores and is quite variable from one example to another. the score is generally very low if the generated gloss length differs substantially (either too long or too word ground-truth epoch predicted moverscore scraggy lean or thin, scrawny. 5 a slightly used to slightly. 0.11885 193 adorned with one or more gauntlets 0.06659 315 ase, slender, thin . 0.12597 coal a glowing or charred piece of coal, wood, or other solid fuel. 5 slightly; to slightly. 0.00863 193 a blust or furnished vehicle . 0.17182 315 supply with energy, especially of a person's size. 0.10929 beautiful pleasant; clear. 5 having been (a person); to suggest or despons. 0.11287 193 sufficient attention or thought, especially concerning the avoidance of harm. 0.03470 315 suitable or proper; extraordinary; epic. 0.19444 thirsty craving something. 5 having been used to suggest or slightly. 0.04727 193 causing by a sensation of alcohol or narcotics. -0.02330 315 causing by anger or excitement. 0.05691 analysing the available data, we see that many of the glosses are relatively short: up to 20 tokens (but there are also very long examples). we conjecture that one strategy for increasing moverscore might be to simply limit all generated glosses to 20 tokens or less. ## conclusion in this competition, we tried a classical recurrent neural network approach for the codwoe definition modeling task, and obtained positive results. several topics require deeper investigation. a good metric for automatically measuring how semantically close are two sentences is still an unsolved problem. moverscore is still too far from human judgement. taking into account even the best scores for the definition modeling task, the task is still in very early stages, and models that are trained only on the provided data cannot generate any practically useful outputs. this could possibly be addressed with much larger training datasets, or by allowing the use of external data (or of large pretrained language models). in general, it seems that a word semantic could not be represented using a single vector. the task requires more context to capture the semantics. maybe the task could be changed to generating a gloss for a set of synonyms or semantically close words.