Roles of Scaling and Instruction Tuning in Language Perception: Model vs. Human Attention (2023)

Abstract: Recent large language models (LLMs) have revealed strong abilities to understand natural language. Since most of them share the same basic structure, i.e. the transformer block, possible contributors to their success in the training process are scaling and instruction tuning. However, how these factors affect the models’ language perception is unclear. This work compares the self-attention of several existing LLMs (LLaMA, Alpaca and Vicuna) in different sizes (7B, 13B, 30B, 65B), together with eye saccade, an aspect of human reading attention, to assess the effect of scaling and instruction tuning on language perception. Results show that scaling enhances the human resemblance and improves the effective attention by reducing the trivial pattern reliance, while instruction tuning does not. However, instruction tuning significantly enhances the models’ sensitivity to instructions. We also find that current LLMs are consistently closer to non-native than native speakers in attention, suggesting a sub-optimal language perception of all models. Our code and data used in the analysis is available on GitHub.

## introduction large language models (llms), e.g., gpt-4, chat-gpt and vicuna, have shown nearly human-level understanding of text inputs, indicating better human-like language perception compared with their predecessors such as bert @xcite and gpt-2 @xcite . however, the mechanism behind such improvement is largely unknown. one way to interpret such mechanism is comparing model computation processes to human data. for example, prior work compared language models (mainly gpt-2) with human neuroimaging data during language comprehension, suggesting that models with higher human resemblance also perform better in nlp tasks. (hasson * corresponding author @xcite . however, given recent breakthroughs in llms , it remains to be tested whether the newest llms still align with human perception data. based on the training pipeline of current llms, there are two possible sources for the potential change: the scaled model/data size and the aligning process after pretraining, such as instruction tuning @xcite @xcite . this paper aims to provide a further understanding of llms' mechanism by investigating the roles of these two factors in language understanding. in this paper, we analyze self-attention as a channel to the language perception ablility of llms, because it is the key mechanism of transformer language models, and it naturally resembles the human attention mechanism in language perceiving @xcite . the attention patterns under investigation includes those from trending open-sourced llms with different sizes, i.e., 7b, 13b, 30b, 65b, and different training stages, i.e., pretrained (llama) and instruction-tuned (alpaca and vicuna) models. given these models are all trained mainly with english data, we also compare the attention patterns of llms with human saccade (a form of eye-movement) data, including both native speakers (l1) and non-native learners (l2) of english @xcite to see if the models correlates more with l1 subjects. our analysis is three-fold. first, we compare the general attention distributions of llms to measure the impact of these two varying factors on the model ( §3.1). second, we perform linear regressions (lrs) between the model self-attention and human saccade matrices to see how human-like they are, and how the resemblance is affected by the two factors ( §3.2). third, we analyze the relation between the given attention and some trivial patterns, revealing the difference between different levels of language perception abilities ( §3.3). the main findings are: ## related work self-attention analysis is a common way to interpret transformer models. several studies has shown that it is correlated to linguistic information, such as syntactic or semantic structures @xcite @xcite . there are also some attempts and debates to explain the models' prediction with attention @xcite @xcite . different from them, we use self-attention patterns as a measurement of the language perception abilities of llms. also, some new ways to interpret model self-attention have been proposed, such as attention flow (abnar and zuidema, 2020), vector norm @xcite , alti @xcite , value zeroing @xcite , and logit analysis @xcite . these methods can raise the interpretability of model attention w.r.t. linguistic structural information, but are not tested with human eye-tracking data. also, some of them suffer from huge computational cost. instead, we choose to use the raw attention scores, as it is produced in the models' original computational processes, and they already correlate the human data, demonstrating their interpretability. instruction tuned llms achieve better performances in performing tasks @xcite @xcite . it seems that the llms are better at understanding human instructions. however, there is little discussion on how the instruction tuning process affects the language perception. eye-movement in the human reading process has drawn attention in the computational linguistic field. it is proposed that scanpaths, i.e. tracks of eye-movement, reveals characteristics of the parsing processes and relates to working memory during reading (von der malsburg and vasishth, 2013), and the irregularity of scanpaths is affected by sentence structures and age @xcite . such work provides theoretical ground for our analysis on saccade data. joint study of language models and human eye-movement is also receiving growing interest. prior studies have shown the correlation between language processing of human and artificial neural networks @xcite , especially pretrained language models such as gpt-2 @xcite @xcite . especially, oh and schuler (2023a,b) explored the fit of model surprisal to human reading times, noting the effect of models' sequencial memorization and dataset size. @xcite studied the correlation between relative word importance derived from models and humans, and @xcite extended that research to a multilingual setting, as well as analysis on model attention. some studies also use human attention as supervision to assist specific tasks @xcite , or use sequencial models to predict eye-tracking data @xcite . different from them, we use human attention as references for the language perception abilities. ## methods to show the effect of scaling and instruction tuning on different models, we compare the self-attention scores of different llms given the same input, by viewing model attention as probability distributions and calculating the general attention divergence based on jensen-shannon divergence. to analyze model self-attention, we take the human saccade as a reference, and design a human resemblance metric based on linear regression scores. in addition, we select several trivial, contextfree attention patterns, and design a trivial pattern reliance metric to help demonstrate the difference between models and human subject groups. ## results figure 2 : mean j-s divergence between 7b and 13b model attention in each layer quarter, measured on noninstruction sentences. ## conclusion this research evaluates the effects of scaling and instruction tuning on llms' attention. we find that scaling effectively changes the general attention distribution, enhances the human resemblance, and reduces the trivial pattern reliance, while instruction tuning does the opposite, but also increases the models' sensitivity to instructions. furthermore, we find current open-sourced llms closer to non-native english speakers in language perception, which contains more trivial attention patterns. to the best of our knowledge, we are the first to analyze the effect of instruction tuning on the human resemblance of model attention. we hope this can inspire future work on analyzing llms.