Character-Level C hinese Backpack Language Models (2023)

Abstract: The Backpack is a Transformer alternative shown to improve interpretability in English language modeling by decomposing predictions into a weighted sum of token sense components. However, Backpacks’ reliance on token-defined meaning raises questions as to their potential for languages other than English, a language for which subword tokenization provides a reasonable approximation for lexical items. In this work, we train, evaluate, interpret, and control Backpack language models in character-tokenized Chinese, in which words are often composed of many characters. We find that our (134M parameter) Chinese Backpack language model performs comparably to a (104M parameter) Transformer, and learns rich character-level meanings that log-additively compose to form word meanings. In SimLex-style lexical semantic evaluations, simple averages of Backpack character senses outperform input embeddings from a Transformer. We find that complex multi-character meanings are often formed by using the same per-character sense weights consistently across context. Exploring interpretability-through control, we show that we can localize a source of gender bias in our Backpacks to specific character senses and intervene to reduce the bias.

## introduction language modeling is a crucial task in natural language processing, where the goal is to compute the probability of the next word in a sequence given the preceding words. recently, large language models based on the transformer architecture @xcite have achieved remarkable success in various nlp applications, including text generation @xcite @xcite , machine translation @xcite , and question-answering @xcite @xcite . however, transformers are notoriously hard to interpret and control. their non-linear contextualization functions imply that intervening on their internal activations can have unpredictable consequences. the recently proposed backpack architecture @xcite tackles the interpretability problem by decomposing its predictions as a sum of non-contextual vectors, which then provide an interface for interpretability. intuitively, it combines the expressivity of transformers with some of the interpretability and control benefits of log-linear models. it was shown to have similar language modeling capacity to transformers on english, and performed comparably on perplexity and lam-bada @xcite tests, at a tax of 1.4x more parameters. the effectiveness of the backpack architecture in languages with different morphological structures than english remains uncertain due to challenges in interpreting and controlling individual tokens without stable explicit semantics. in chinese, most vocabulary consists of compound words with mul-tiple characters. however, these characters often have implicit meanings @xcite , making it challenging to infer the meaning of these words based solely on the individual meanings of their constituent characters. additionally, some characters represent the pronunciation of foreign words and lack semantic associations, which requires characters to learn more complex semantic connections within limited sense vectors. the english-based backpack model is trained on often complete words with more explicit meanings, making it uncertain whether backpacks will perform well in character-level chinese. in this paper, we trained the first non-english (and first character-based language) backpack language model and evaluate its performance and learned lexical semantics on character level. 1 we trained several backpack and transformer baseline models and evaluated them on perplexity and word prediction accuracy tasks. our experiments show that our pretrained 134m backpack language model with 16 sense vectors, which uses character-based tokenization, performs comparably to a 104m transformer model. to understand the backpack's success, we first study how it composes word meaning from noncontextual token senses. we hypothesize word meaning is formed because tokens of a multicharacter word receive similar weighting in the backpack's sum across all contexts the word appears in. we find that indeed the proportion of these composed characters on each sense vector changes by no more than 20% in over 90% of cases. moreover, we achieve better word representation under three chinese corpora by simply averaging the sense vectors of composed characters compared to the character embeddings of the pretrained transformer model. additionally, we propose and evaluate character-level interventions to mitigate gender bias and control how word meaning is composed from character meaning , which demonstrate promising results for generating controllable text in character-based chinese backpack models. these experiments show that our chinese backpack model learns the implicit semantics of characters, making it possible to control the emphasis or weakening of certain characteristics of a word during generation tasks. ## experiment training backpack lms to compare the performance of our models against the baseline models in general language modeling evaluations, we first pretrained our 134m "backpack-small" and 27m "backpack-micro" chinese backpack language models and the baseline 104m "gpt2-small" and 18m "gpt2-micro" gpt2 models on large chinese corpus. these sizes are set so the transformer used in the backpack's weight computation is the same size as the corresponding gpt2-like transformer model. ## conclusion in this paper, we presented implementing, pretraining, and evaluating a character-based chinese backpack language model. we conducted extensive experiments on sense vector visualizations, word representations, lexical relationships, and idiom compositions and explored two approaches to character-level interventions. our results demonstrate the potential of backpack lm in language modeling tasks for character-based languages, the interpretability of the sense vectors on the character and word level, and the potential of character-level interventions across various contexts.