S emeval-2022 Task 1: CODWOE – Comparing Dictionaries and Word Embeddings (2022)

Abstract: Word embeddings have advanced the state of the art in NLP across numerous tasks. Understanding the contents of dense neural representations is of utmost interest to the computational semantics community. We propose to focus on relating these opaque word vectors with human-readable definitions, as found in dictionaries This problem naturally divides into two subtasks: converting definitions into embeddings, and converting embeddings into definitions. This task was conducted in a multilingual setting, using comparable sets of embeddings trained homogeneously.

## introduction word embeddings are a success story in nlp. they have been equated to distributional semantics models @xcite , a theory of semantics which relates the meaning of words to their distribution in context @xcite . recently introduced contextualized word embeddings (e.g. @xcite have set a new state of the art on a wide variety of tasks. for this reason, they have attracted much research interest. do they depict consistent semantic spaces and are they theoretically valid @xcite ? what limitations are to be expected in these models @xcite )? can they scale up in performance @xcite ? word embeddings are dense vector representations of meaning which are not easily intelligible to a human observer. many techniques have been employed to make embedding spaces more interpretable. a promising approach consists in converting these opaque vectors into human readable definitions, as one could find in a dictionary: accurately translating a dense, opaque vector representation into an equivalent human-readable piece of text would allow us to peer into the black box of modern neural network architectures. this avenue of research, known as definition modeling, was pioneered by @xcite . one may however question whether the task is at all feasible: there is no guarantee that the information content of a dictionary definition is similar to that which is described by real-valued vectors inferred from word distributions. the semeval shared task on comparing dictionaries and word embeddings (codwoe) sets out to study whether embeddings and dictionaries encode similar information. we present the task and relevant state of the art in section 2. we describe the data collected and presented to participants in section 3. in section 4, we discuss the metrics used to rank participant submissions. our baseline model is presented in section 5. we list results from participants' submissions in section 6 and provide a more in-depth discussion in section 7. ## what we are fishing for what is in a word embedding? are word embeddings semantic descriptions, in the same sense that dictionary definitions are? if so, embeddings and definitions must be translatable into one another. the codwoe shared task was set up to test this. the shared task participants investigated whether a word vector-e.g. ⃗ cod-contains the same information as the corresponding dictionary definitionviz. "any of various bottom-dwelling fishes (family gadidae, the cod family) that usually occur in cold marine waters and often have barbels and three dorsal fins." 1 we decompose this research problem into two tracks: the first corresponds to the vector-tosequence task of definition modeling, the second to the sequence-to-vector reverse dictionary task. the task of definition modeling consists in using the vector representation of ⃗ cod to produce the associated gloss, "any of various bottom-dwelling fishes (family gadidae, the cod family) that usually occur in cold marine waters and often have barbels and three dorsal fins". the reverse dictionary task is the mathematical inverse: reconstruct an embedding ⃗ cod from the corresponding gloss. these two tracks display a number of interesting characteristics. these tasks are obviously useful for explainable ai, since they involve converting human-readable data into machine-readable data and back. they also have a theoretical significance: both glosses and word embeddings are representations of meaning, and therefore involve the conversion of distinct non-formal semantic representations. from a practical point of view, the ability to infer word-embeddings from dictionary resources, or dictionaries from large un-annotated corpora, would prove a boon for many under-resourced languages. ## what's in the nets: data used the definition modeling and reverse dictionary tasks both require a parallel dataset, where dictionary definitions are aligned with corresponding word embeddings. the task is held in a multilingual setting. we provide data in english, french, russian, italian and spanish. we selected these languages to facilitate the collection of comparable data: all these languages possess comparable large scale resources, including online dictionaries as well as corpora that can be used to train comparable embeddings. our datasets are made available online at https://codwoe.atilf.fr/ . the aim of both tracks of codwoe is to compare the semantic contents of definitions and embeddings. as a consequence, we ask participants to refrain from using external data such as pretrained with examples without en 0 806297 es 0 132583 fr 431793 573313 it 16127 86959 ru 122282 485208 ## testing the waters: baseline architectures we implement simple neural network architecture baselines to lower the barrier to entry to this shared task. they are based on the transformer architecture of @xcite and designed to be as simple as possible. our code is publicly available at https://github.com/ timotheemickus/codwoe. we illustrate our reverse dictionary baseline architecture in figure 3a . it consists in feeding the input gloss 〈 ⃗ bos, ⃗ w 1 , . . . , ⃗ w n , ⃗ eos〉 into a simple transformer encoder, and then summing all the hidden representations to produce the prediction p i . in practice, the summed hidden states are passed into a small non-linear feed-forward module to derive the prediction: our definition modeling baseline is presented in figure 3b . it consists in a simple transformer encoder, where earlier time-step representations are prevented from attending to later time-step representations. to provide information about the definiendum to the model, we use the definiendum embedding ⃗ d i as the input for the first time-step instead of a start-of-sequence token. we train the models with teacher-forcing: i.e., during training we ignore the definientia p 1 i , . . . , p n i that the model produces; instead we feed it the target w 1 , . . . , w m attested in the training set at each time-step. during inference, we feed the model with its own prediction. this creates a train-test mismatch, which we alleviate by using a beam-search. we stop generation when all beams have produced an end-ofsequence token. for both tracks, we train one model for each distinct pair of language and embedding architecture. we start by re-tokenizing the datasets using sentence piece with a vocabulary size of 15000. this is done in order to mitigate the effects of different scores attained by participants are shown in tables 3 and 4. in in total, we received 159 valid submissions from 15 different users; out of which 11 teams produced a submission paper. 9 of these teams tackled the definition modeling, and 10 addressed the reverse dictionary track. competition rankings are established by ranking each submission received, selecting for each participant the best performance on all metrics, and finally taking the average best rank. some participants' submissions were faulty and could not be processed by the evaluation website scoring program. five submissions specifically focus on the reverse dictionary task. @xcite compare the performances of mlpbased to lstm-based networks for reverse dictionary. @xcite study pretraining objectives for the reverse dictionary track. @xcite pay specific attention to how the not-so-satisfactory quality of the spanish dataset impacts results on spanish reverse dictionary. cerniavski and stymne (2022, uppsala) study whether foreign language entries can improve the performance of the english reverse dictionary baseline model. @xcite introduce multiple technical tweaks for reverse dictionary, such as a dynamic weight averaging loss, languagespecific tags and residual cutting. the last four submissions addressed both tracks. @xcite propose to project embeddings and definitions on a shared representational space. korenčić and grubišić (2022, irb-nlp) take inspiration from @xcite to address definition modeling, and experiment with pooling strategies over transformer embeddings for the reverse dictionary track. @xcite focus on comparing the effects of adding lstm and bilstm layers on top of a transformer model, as well as zero-shot cross-lingual generalization. @xcite propose two transformer-based architectures for the two tracks, leveraging contrastive learning and unsupervised pretraining. looking at tables 3 and 4 , we see that the metrics we chose in section 4 are not always aligned. on the definition modeling track ( ## a deeper dive into our results when looking at the competition results, two trends emerge. first, the baseline architectures from section 5 remain quite competitive with solutions proposed by participants. second, scores are generally unsatisfactory, especially in the definition modeling track: we do not see a clear divide between char embeddings and distributional semantic representations. the nlg metrics are, in absolute terms, low compared to modern nlp standards and results reported elsewhere on other definition modeling benchmarks. as for the reverse dictionary track, we see that across all submissions, at least a third of the test set is closer (in terms of cosine distance) to the production than the intended target. participants have suggested multiple reasons for these hardships. in particular, @xcite highlight that the automated data compilation in dbnary @xcite is of an unsatisfactory quality. similar remarks can be made with respect to the embeddings, which are trained on rather small corpora. other submissions such as @xcite , @xcite , @xcite highlight the limited applicability of mainstream nlg metrics, as we ourselves have discussed in section 4. 12 one last remark is the limited size of our dataset, discussed by the edinburgh and riga teams. all these remarks suggest avenues for future research: in particular, the release of the full dataset should alleviate some of the concerns with respect to dataset size. the mmg team also suggest some concrete preprocessing steps to handle some of the issues they identify in the proposed definitions. in terms of solutions explored, we can stress that teams have adopted a variety of strategies and architectures: systems used transformer, rnn and cnn components, often leveraging or exploring multilingualism @xcite @xcite , bl.research), multitasking, or multiple training objectives @xcite @xcite @xcite . multi-task training tends to yield varied yet competitive results for our data. no preponderant architecture emerges from the system descriptions; we note that multiple submissions based their work on other contextualized embedding architectures, trained from scratch on the codwoe dataset @xcite . the comprehensive review of architectures by team 1cadamy suggests nonetheless that transformers might be less suited to this shared task than recurrent models. ## conclusions and future perspectives the codwoe shared task was constructed so that participants' submissions would be likely to have linguistic significance. yet, it is not trivial to tease apart the various factors that lead to the overall low results we observed. while the inadequacy of mainstream nlg metrics and the limitations of the dataset certainly play a role, they do not resolve the fundamental issue that we wished to investigate with codwoe. whether word embeddings and dictionaries contain the same information is not a solved research problem. this has two immediate consequences: firstly, one can question the use of definition modeling as an evaluation tool for embeddings, as suggested by the seminal work of @xcite . the codwoe shared task results indicate that the metrics currently used in the field are rife with caveats; in the controlled setup we have proposed here, participants rarely, if ever, found that character-based embeddings starkly contrasted with distributional semantic representations. second, one can question whether definition modeling and reverse dictionary are fit for building lexical resources for under-tooled languages: the crosslingual route proposed by @xcite seems more practical than training models from scratch, even with relatively large datasets. our embeddings were trained on corpora comparable in size to the 1b words benchmark @xcite : while modern text corpora are now several orders of magnitude larger, this dataset remained a landmark for several years. our definitions were selected from dbnary @xcite , which focuses the largest wiktionary projects. overall, the codwoe shared task has been a success: we were able to show that the task at hand was far from trivial and we drew significant interest towards the issues addressed in the definition modeling and reverse dictionary literature. in future work, we plan to investigate better ways to perform nlg evaluation for the definition modeling task (in particular relying on human annotations) and we plan to focus on existing embeddings trained from very large corpora.