Self- S tr AE at S em E val-2024 Task 1: Making Self-Structuring A uto E ncoders Learn More With Less (2024)

Abstract: We present two simple improvements to the Self-Structuring AutoEncoder (Self-StrAE). Firstly, we show that including reconstruction to the vocabulary as an auxiliary objective improves representation quality. Secondly, we demonstrate that increasing the number of independent channels leads to significant improvements in embedding quality, while simultaneously reducing the number of parameters. Surprisingly, we demonstrate that this trend can be followed to the extreme, even to point of reducing the total number of non-embedding parameters to seven. Our system can be pre-trained from scratch with as little as 10M tokens of input data, and proves effective across English, Spanish and Afrikaans.

## introduction natural language is generally understood to be compositional. to understand a sentence, all you need to know are the meanings of the words and how they fit together. the mode of combination is generally conceived as an explicitly structured hierarchical process which can be described through, for example, a parse tree. recent work by @xcite presents the self-strae (self-structuring autoencoder), a model which learns embeddings such that they define their own hierarchical structure and extend to multiple levels (i.e. from the subword to the sentence level and beyond). the strengths of this model lie in its parameter and data efficiency achieved through the inductive bias towards hierarchy. learning embeddings such that they meaningfully represent semantics is crucial for many modern nlp applications. for example, retrieval augmented generation @xcite ) is predicated on the fact that the correct contexts for a given query can be determined. the semantic relation between a query and a context is encompassed by the notion of semantic relatedness. they are not equivalent to one another (i.e. paraphrases), but are close in meaning in a broader, more contextual sense. the focus of task one of this year's semeval @xcite is capturing this notion of semantic relatedness, with a particular focus on african and asian languages generally characterised by a lack of nlp resources. in this work, we investigate whether self-strae can learn embeddings which capture semantic relatedness, when trained from scratch on moderately sized pre-training corpora. we turn to the competition in order to examine whether the model can even compare with dedicated str systems. in order to determine whether self-strae can provide an alternative approach in low resource settings where systems that rely on large pre-trained transformers @xcite may not have sufficient scale to prove effective. we show that with two simple changes, self-strae's performance can be substantially improved. moreover, we demonstrate that the the resulting system is not limited to english, but can work equally well (if not better) for both spanish and afrikaans foot_0 . ## conclusion we show that two simple changes can make self-strae significantly more performant: adding a discrete reconstruction objective and increasing the number of independent channels. the latter also has the added benefit of reducing the number of parameters in the model, and surprisingly means that simpler is better. more broadly, we believe these findings demonstrate the potential of an inductive biases towards explicit structure. self-strae, at present, is a very simple model. the only thing it really has going for it is the inductive bias which tasks embeddings with organising themselves hierarchically. while the gap between self-strae and sota systems still remains, the fact that it is able to perform at all demonstrates the promise. moreover, the fact that the two simple changes demonstrated in this paper can lead to such improvements indicates that the full potential of the inductive bias has yet to be reached, and it is likely that further refinements can lead to even more substantial benefits. finally, because this model does not require significant scale to optimise pursuing further improvements may provide substantial benefits for low resource languages where pre-training data is scarce.