C og BERT : Cognition-Guided Pre-trained Language Models (2022)

Abstract: We study the problem of integrating cognitive language processing signals (e.g., eye-tracking or EEG data) into pre-trained language models like BERT. Existing methods typically fine-tune pre-trained models on cognitive data, ignoring the semantic gap between the texts and cognitive signals. To fill the gap, we propose CogBERT, a framework that can induce fine-grained cognitive features from cognitive data and incorporate cognitive features into BERT by adaptively adjusting the weight of cognitive features for different NLP tasks. Extensive experiments show that: (1) Cognition-guided pre-trained models can consistently perform better than basic pre-trained models on ten NLP tasks. (2) Different cognitive features contribute differently to different NLP tasks. Based on this observation, we give a fine-grained explanation of why cognitive data is helpful for NLP. (3) Different transformer layers of pre-trained models should encode different cognitive features, with word-level cognitive features at the bottom and semantic-level cognitive features at the top. (4) Attention visualization demonstrates that CogBERT aligns with human gaze patterns and improves its natural language comprehension ability.

## introduction pre-trained models such as bert @xcite , gpt @xcite and roberta @xcite have brought promising improvements to natural language processing (nlp) tasks, such as event prediction @xcite and sentiment analysis @xcite . on the other hand, from a language processing perspective, cognitive neuroscience studies the biological and cognitive processes underlying language processing in human brains. researchers specifically designed pre-trained models to capture how the brain represents language meaning 1 our code will be released upon publication. the isis has done so many terrible things in syria and caused chaos in the mid east the isis has done so many terrible things in syria and caused chaos in the mid east the isis has done so many terrible things in syria and caused chaos in the mid east @xcite . prior work mainly incorporates cognitive signals by explicitly fine-tuning the pre-trained model to predict language-induced brain recordings. however, this line of work cannot give a finegrained analysis and interpretation of why cognitive data is helpful for nlp. this is of great importance to guide future cognition-inspired nlp studies on what kind of cognitive features should be induced from cognitive data and how these cognitive features contribute to nlp tasks. for example, figure 1 shows eye-tracking data from native speakers of english, where figure 1 (a ) illustrates the number of fixations during the normal reading of humans. figure 1 (b ) and (c) show the number of fixations given the nlp task of sentiment classification (sc) and named entity recognition (ner), respectively. we can observe that human attention is different for the same sentence, given different nlp tasks. in particular, for the sc task, people pay more attention to emotion words, such as "terrible" and "chaos". while for the ner task, people tend to focus on named entity words, such as "isis" and "syria". unfortunately, prior studies cannot give such fine-grained analysis by simply fine-tuning pre-trained models on cognitive data. to facilitate this, we propose cogbert, a cognition-guided pre-trained model. specifically, we focus on the effects of using eye-tracking data, which provides gaze information from native speakers by tracking eye movements and measuring fixation duration. instead of directly fine-tuning bert name independent strands index word position ✓ lower 1 word length ✓ lower 2 ner word ✓ lower 3 content word ✓ lower 4 noun phrase ✓ upper 5 emotion word ✓ upper 6 active/passive aux ✓ upper 7 poss/prep mod coordinating conj prep/adj comp ✓ upper 8 prep obj on cognitive data, we first extract psycholinguistic features based on cognition theory @xcite . then we filter out statistically insignificant features in the eye-tracking data (which means that the human attention of words with these features is not significantly higher/lower than the average attention of words). subsequently, we incorporate these cognition-validated features into bert by fine-tuning on different nlp tasks. in the fine-tuning process, we would learn different weights for each type of feature according to different nlp tasks. our results show that cogbert can perform better than baseline systems on three benchmark datasets across ten nlp tasks. we give a detailed quantitative analysis of the contributions of different cognitive features to different nlp tasks. case studies show that cogbert does learn patterns of human reading behavior compared to bert. our findings can provide more insights into cognitionenhanced nlp. ## cogbert cogbert works in a two-stage setting. the first stage is to induce cognitive features from texts and assign weights for these features guided by human reading signals. in the second stage, we integrate cognitive features into bert and learn taskspecific feature weights for different nlp tasks. ## experiments to show the effectiveness of cogbert, we compare it with baselines on three benchmark datasets across ten nlp tasks and an eye-tracking prediction task. ## related work prior neuroscience studies have demonstrated that cognitive data is associated with language comprehension activity in human brains @xcite , showing longer duration links to a greater processing load in different language units (words, clauses, texts). these studies established the theoretical grounding for cognition-guided nlp. in cognitively motivated nlp, researchers investigate the impact of cognitive language processing signals on nlp tasks, especially focusing on improving neural networks by utilizing cognitive data. early researches mainly used lstm or cnn to incorporate cognitive signals @xcite @xcite @xcite . as pre-trained models have shown their great power on various nlp tasks, a line of work focuses on exploring cognitive-data-enhanced pre-trained models @xcite @xcite , mainly by fine-tuning pretrained models on cognitive data. however, these methods cannot give a finegrained analysis of how cognitive data contributes to different nlp tasks. in contrast, cogbert is inspired by the theory in psycholinguistics and encodes cognitive features induced from eye-tracking data into pre-trained models in a layer-wise manner with carefully designed architecture, enabling us to perform a fine-grained analysis of how cognitive data contributes to different nlp tasks and further improved the performance of pre-trained models comparing to previous simply fine-tuned ones. ## conclusion we present cogbert, a framework that can effectively incorporate cognitive signals into pre-trained models. experimental results show that cogbert achieves new sota results on three nlp benchmark datasets. analyses suggest that cogbert can adaptively learn task-specific cognitive feature weights to give fine-grained explanations of how cognitive data works on nlp tasks. this work provides a new paradigm in learning cognitionenhanced pre-trained models, and extensive elaborated experiments can guide future researches. ratio as 0.15, and the dimension of the fully connected layer before softmax activation as 2000. the learning rate is initialized as 5e-4 for the adamw optimizer, and we use the linear learning rate schedule with the warm-up ratio of 0.1. for bert and roberta, we use the pre-trained version of different models released by hugging-face. 6 for glue benchmark. we use adamw optimizer with learning rate from [1e-5, 3e-5, 5e-5] on glue benchmark. the training epochs are from @xcite @xcite . we also use the linear learning rate schedule with a warm-up ratio of 0.1. the batch size is 32 for all tasks. the kernel size for both 1dcnn and 2dcnn is 3. the 2dcnn module does not have bias, and the padding strategy is the "same". for conll-2000 chunk and conll-2003 ner tasks, we use adamw optimizer with learning rate from [1e-6, 3e-6, 5e-6]. the training epochs are from @xcite . we also use the linear learning rate schedule with a warm-up ratio of 0.1. the batch size is 32 with a gradient accumulation step of 4 to avoid memory issues. for the eye-tracking prediction task, we follow @xcite and replace the bert model with our model. the training setting is precisely the same as the original task setting released on the page. 7 the results are adamw optimizer with a learning rate of 5e-5 and a weight decay of 0.01. we employ a linear learning rate decay schedule over the total number of training steps. we clip all gradients exceeding the maximal value of 1. we train the models for 100 epochs, with early stopping after seven epochs without improving the validation accuracy.