Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance? (2023)

Abstract: Understanding how and what pre-trained language models (PLMs) learn about language is an open challenge in natural language processing. Previous work has focused on identifying whether they capture semantic and syntactic information, and how the data or the pre-training objective affects their performance. However, to the best of our knowledge, no previous work has specifically examined how information loss in input token characters affects the performance of PLMs. In this study, we address this gap by pre-training language models using small subsets of characters from individual tokens. Surprisingly, we find that pre-training even under extreme settings, i.e. using only one character of each token, the performance retention in standard NLU benchmarks and probing tasks compared to full-token models is high. For instance, a model pre-trained only on single first characters from tokens achieves performance retention of approximately 90% and 77% of the full-token model in SuperGLUE and GLUE tasks, respectively.

## introduction the use of pre-trained language models (plms), such as bert @xcite , t5 @xcite , gpt-3 @xcite and many more, has led to significant advances in natural language processing (nlp) tasks, with many achieving stateof-the-art results. however, the exact mechanisms by which these models learn foot_1 and represent language are still not fully understood. the pre-training data evidently plays an important role in the downstream performance of figure 1 : humans are able to understand natural language text consisting of words with altered letters (e.g., part bold, or entiely msig letters). can language models do the same? plms @xcite . previous studies have mainly focused on exploring how its size affects the performance @xcite @xcite , demonstrating the benefits of using more data together with a larger number of parameters. more recent work has focused on studying the relationship between plms performance and the quality and relevance of the pretraining data @xcite @xcite @xcite , showing that high-quality data leads to improved performance. interestingly, contrary to these findings, other studies showed that language models can still achieve comparable performance even when they are pretrained using randomly selected character n-grams @xcite or pre-trained using synthetic tasks @xcite . however, no previous studies have so far explored the amount of information that plms actually need from individual tokens in the pre-training data. motivated by this, we pre-train language models by introducing partial information loss in the token sequence. this is achieved by using subsets of characters to represent each token in the pretraining data, i.e. keeping only one, two or three characters at a time (figure 1 ). to the best of our knowledge, this study is the first to explore how informative subsets of tokens are compared to using full tokens in language models pre-training. specifically, we draw inspiration from studies in cognitive science and psycholinguistics that have shown that humans possess remarkable flexibility in processing partial word information, such as comprehending text by only reading the first or last characters of each word @xcite @xcite . this intriguing ability raises the question of whether computational models can exhibit similar behavior, bringing forth novel avenues for nlp research. in this paper, we explore the parallels between human reading strategies and language models when presented with partial word information, aiming to uncover common principles and differences underlying both systems. our main contributions are as follows: probing has shown that plms encode linguistic information, such as syntactic and semantic, in their representations @xcite @xcite @xcite @xcite @xcite @xcite . pre-training objectives another line of work has focused on exploring how pre-training objectives affect the performance of plms. @xcite demonstrated that plms pre-trained on masked language modeling (mlm) perform well in downstream tasks due to their ability to learn semantic dependencies between tokens. however, many different pre-training objectives are effective in downstream tasks @xcite @xcite . in addition, studies examined how different types of pre-training tasks help plms learn, showing that even objectives that are not intuitive for humans, such as tokens grouped in arbitrary random classes, can still help plms achieve good performance on downstream tasks and acquire linguistic information @xcite . furthermore, a recent study by @xcite has shown that the performance of plms on downstream tasks is significantly influenced by the pre-training objective complexity. pre-training data closer to our work are studies on investigating the learning process of plms by pre-training them on synthetic data. @xcite found that plms pre-trained on sentences with shuffled word order still perform well on downstream tasks, likely due to their ability to capture higher-order co-occurrence statistics. @xcite used a corpus of randomly selected character n-grams to pre-train models on synthetic summarization tasks. they found that the performance of these models was nearly equivalent to models pre-trained on real data. @xcite demonstrated that pre-training on artificial languages that mimic the structural properties of natural language can still result in knowledge transferable to natural language. even when pre-trained on non-linguistic data such as music or code, language models can still perform well on natural language downstream tasks @xcite . similarly, hashing semantically unrelated tokens together in the same embedding results in negligible performance loss @xcite . unlike previous work, we explore how much information loss affects the performance of plms inspired by human reading comprehension @xcite @xcite @xcite . ## models various studies in cognitive science have attempted to measure the ability of humans to process words and understand sentences when retaining only some of the characters and their relative position in each word while others are masked, i.e. orthographic priming @xcite @xcite . inspired by this, we aim to explore how much information plms need from individual tokens by pre-training using only a small subset of characters in each token. we experiment using three different input token subsets (i.e. one, two and three characters) and three relative positions of characters in the token (i.e. first, middle and last). we also experiment with retaining only consonant or vowel characters from each input token. ## analysis 6.1 pre-training loss curves figure 2 shows the pre-training loss curves for the single character models illustrating the progression and convergence of the loss function throughout the pre-training process. we observe that both the f and m models have very close pre-training losses and that the model l has the lowest pre-training loss. overall, we note that the three models have stable pre-training which indicates that the models are able to learn using limited input information. the pre-training loss curves for the other pre-trained models can be found in appendix c. ## discussion our results reveal that certain subsets of input tokens are more informative than others. for instance, we found that plms perform better when pre-trained using the first and last characters instead of the first or last two characters of input tokens. this appears to align with research in cognitive science and psycholinguistics which suggests that the first and last letters of a word are more crucial for human reading comprehension than the interior letters and that the positions of letters are not given equal weight in the cognitive representation of a word @xcite . previous studies have also suggested that the first and last letters of a word hold more importance because they pertain to the way the mind organizes and retrieves lexical information @xcite @xcite . other studies have suggested that the first and last letters of a word are more easily identified as they are less impacted by external interference or the presence of surrounding letters compared to inner letters @xcite @xcite . surprisingly, our results also highlight that plms are still able to learn from data that may be incomprehensible to humans. our models perform well on natural language understanding tasks or appear to 'learn' semantic and syntactic information (i.e. probing) by using only one or two characters from each token. this demonstrates the ability of plms to find patterns in seemingly random or limited data. additionally, our findings also suggest that these models may have the capability to process language in ways that are fundamentally different from human cognition. ## conclusion in this work, we explored how the pre-training of language models using different subsets of characters from the input tokens affect their performance on downstream tasks. to our surprise, we found that even using just one character from each input token achieves performance way above chance. our probing results further revealed that specific character positions within tokens affect how the models capture linguistic knowledge.