C og A lign: Learning to Align Textual Neural Representations to Cognitive Language Processing Signals (2021)

Abstract: Most previous studies integrate cognitive language processing signals (e.g., eye-tracking or EEG data) into neural models of natural language processing (NLP) just by directly concatenating word embeddings with cognitive features, ignoring the gap between the two modalities (i.e., textual vs. cognitive) and noise in cognitive features. In this paper, we propose a CogAlign approach to these issues, which learns to align textual neural representations to cognitive features. In CogAlign, we use a shared encoder equipped with a modality discriminator to alternatively encode textual and cognitive inputs to capture their differences and commonalities. Additionally, a text-aware attention mechanism is proposed to detect task-related information and to avoid using noise in cognitive features. Experimental results on three NLP tasks, namely named entity recognition, sentiment analysis and relation extraction, show that CogAlign achieves significant improvements with multiple cognitive features over state-of-the-art models on public datasets. Moreover, our model is able to transfer cognitive information to other datasets that do not have any cognitive processing signals.

## introduction cognitive neuroscience, from a perspective of language processing, studies the biological and cognitive processes and aspects that underlie the mental language processing procedures in human brains while natural language processing (nlp) teaches machines to read, analyze, translate and generate human language sequences @xcite . the commonality of language processing shared by these two areas forms the base of * corresponding author cognitively-inspired nlp, which uses cognitive language processing signals generated by human brains to enhance or probe neural models in solving a variety of nlp tasks, such as sentiment analysis @xcite , named entity recognition (ner) @xcite , dependency parsing @xcite , relation extraction @xcite , etc. in spite of the success of cognitively-inspired nlp in some tasks, there are some issues in the use of cognitive features in nlp. first, for the integration of cognitive processing signals into neural models of nlp tasks, most previous studies have just directly concatenated word embeddings with cognitive features from eye-tracking or eeg, ignoring the huge differences between these two types of representations. word embeddings are usually learned as static or contextualized representations of words in large-scale spoken or written texts generated by humans. in contrast, cognitive language processing signals are collected by specific medical equipments, which record the activity of human brains during the cognitive process of language processing. these cognitive processing signals are usually assumed to represent psycholinguistic information @xcite or cognitive load @xcite . intuitively, information in these two types of features (i.e., word embeddings and cognitive features) is not directly comparable to each other. as a result, directly concatenating them could be not optimal for neural models to solve nlp tasks. the second issue with the incorporation of cognitive processing signals into neural models of nlp is that not all information in cognitive processing signals is useful for nlp. the recorded signals contain information covering a wide variety of cognitive processes, particularly for eeg @xcite . for different tasks, we may need to detect elements in the recorded signals, which are closely related to specific nlp tasks, and neglect features that are noisy to the tasks. in order to address the two issues, we propose cogalign, a multi-task neural network that learns to align neural representations of texts to cognitive processing signals, for several nlp tasks. as shown in figure 1 , instead of simply concatenating cognitive features with word embeddings, we use two private encoders to separately encode cognitive processing signals and word embeddings. the two encoders will learn task-specific representations for cognitive and textual inputs in two disentangled spaces. to align the representations of neural network with cognitive processing signals, we further introduce an additional encoder that is shared by both data sources. we alternatively feed cognitive and textual inputs into the shared encoder and force it to minimize an adversarial loss of the discriminator stacked over the shared encoder. the discriminator is task-agnostic so that it can focus on learning both differences and deep commonalities between neural representations of cognitive and textual features in the shared encoder. we want the shared encoder to be able to transfer knowledge of cognitive language processing signals to other datasets even if cognitive processing signals are not available for those datasets. therefore, cogalign does not require cognitive processing signals as inputs during inference. partially inspired by the attentive pooling network @xcite , we propose a text-aware attention mechanism to further align textual inputs and cognitive processing signals at the word level. the attention network learns a compatibility matrix of textual inputs to cognitive processing signals. the learned text-aware representations of cognitive processing signals also help the model to detect task-related information and to avoid using other noisy information contained in cognitive processing signals. in a nutshell, our contributions are listed as follows: eeg for nlp. electroencephalography (eeg) measures potentials fluctuations caused by the activity of neurons in cerebral cortex. the exploration of eeg data in nlp tasks is relatively limited. chen et al. (2012) improve the performance of automatic speech recognition (asr) by using eeg signals to classify the speaker's mental state. hollenstein et al. (2019a) incorporate eeg signals into nlp tasks, including ner, relation extraction and sentiment analysis. additionally, muttenthaler et al. (2020) leverage eeg features to regularize attention on relation extraction. adversarial learning. the concept of adversarial training originates from the generative adversarial nets (gan) @xcite in computer vision. since then, it has been also applied in nlp @xcite . recently, a great variety of studies attempt to introduce adversarial training into multi-task learning in nlp tasks, such as chinese ner @xcite , crowdsourcing learning @xcite , cross-lingual transfer learning @xcite , just name a few. different from these studies, we use adversarial learning to deeply align cognitive modality to textual modality at the sentence level. ## cogalign cogalign is a general framework for incorporating cognitive processing signals into various nlp tasks. the target task can be specified at the predictor layer with corresponding task-specific neural network. cogalign focuses on aligning cognitive processing signals to textual features at the word and encoder level. the text-aware attention aims at learning task-related useful cognitive information (thus filtering out noises) while the shared encoder and discriminator collectively learns to align representations of cognitive processing signals to those of textual inputs in a unified semantic space. the matched neural representations can be transferred to another datasets of the target task even though cognitive processing signals is not present. the neural architecture of cogalign is visualized in figure 1 . we will elaborate the components of model in the following subsections. ## experiments we conducted experiments on three nlp tasks, namely ner, sentiment analysis and relation extraction with two types of cognitive processing signals (eye-tracking and eeg) to validate the effectiveness of the proposed cogalign. ## analysis 6.1 text-aware attention analysis in addition to denoising the cognitive processing signals, the text-aware attention mechanism also obtains the task-specific features. to have a clear view of the role that the text-aware attention mechanism plays in cogalign, we randomly choose samples and visualize the average attention weights over each signal in figure 3 . for eye-tracking, signals reflecting the late syntactic processing, such as 'nfix' (number of fixation), 'tfd' (total fixation duration), play an important role in the three tasks. these results are consistent with findings in cognitive neuroscience. in cognitive neuroscience, researchers have shown that readers tend to gaze at nouns repeatedly @xcite (related to the eye-tracking signal nfix, the number of fixations) and there is a dependency relationship between regression features and sentence syntactic structures @xcite . in other nlp tasks that infused eye-tracking features, the late gaze features have also proved to be more important than early gaze features, such as multiword expression extraction @xcite . moreover, from the additional eye-tracking used in ner, we can find that the cognitive features from the neighboring words are helpful to identify entity, such as 'w-2 fp' (w-2 fixation probability), 'w+1 fp' (w+1 fixation probability). since a single eeg signal has no practical meaning, we only visualize the attention weights over eeg signals used in the ner task. obviously, attentions to 't1' (theta1) and 'a2' (alpha2) are stronger than other signals, suggesting that low frequency electric activities in the brain are obvious when we recognize an entity. ## conclusions in this paper, we have presented cogalign, a framework that can effectively fuse cognitive processing signals into neural models of various nlp tasks by learning to align the textual and cognitive modality at both word and sentence level. experiments demonstrate that cogalign achieves new state-ofthe-art results on three nlp tasks on the zuco dataset. analyses suggest that the text-aware attention in cogalign can learn task-related cognitive processing signals by attention weights while the modality discriminator with adversarial learning forces cogalign to learn cognitive and textual representations in the unified space. further experiments exhibit that cogalign is able to transfer cognitive information from zuco to other datasets without cognitive processing signals.