Enhancing EEG -to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG -Text Masked Autoencoder (2024)

Abstract: Reconstructing natural language from non-invasive electroencephalography (EEG) holds great promise as a language decoding technology for brain-computer interfaces (BCIs). However, EEG-based language decoding is still in its nascent stages, facing several technical issues such as: 1) Absence of a hybrid strategy that can effectively integrate cross-modality (between EEG and text) self-learning with intra-modality self-reconstruction of EEG features or textual sequences; 2) Under-utilization of large language models (LLMs) to enhance EEG-based language decoding. To address above issues, we propose the Contrastive EEG-Text Masked Autoencoder (CET-MAE), a novel model that orchestrates compound self-supervised learning across and within EEG and text through a dedicated multi-stream encoder. Furthermore, we develop a framework called E2T-PTR (EEG-to-Text decoding using Pretrained Transferable Representations), which leverages pre-trained modules alongside the EEG stream from CET-MAE and further enables an LLM (specifically BART) to decode text from EEG sequences. Comprehensive experiments conducted on the popular text-evoked EEG database, ZuCo, demonstrate the superiority of E2T-PTR, which outperforms the baseline framework in ROUGE-1 F1 and BLEU-4 scores by 8.34% and 32.21%, respectively. Our proposed pre-trained EEG-Text model shows the potential to improve downstream tasks involving EEG and text. This opens up promising avenues for its application in inner speech BCI paradigms, meriting further investigation.

## introduction decoding natural language from non-invasive brain recordings with electroencephalography (eeg) is an emerging topic that holds promising benefits inspired by the above prevailing ssl strategies, we propose a novel pre-trained model to align eeg and text, contrastive eeg-text masked autoencoder (cet-mae), as shown in figure2(a). cet-mae integrates contrastive learning and masked signal modeling through a dedicated multi-stream encoder. it effectively learns pre-trained representations of eeg and text by balancing the latent embeddings represented by self-reconstruction and the semantic-level aligned embeddings of text tokens and text-evoked eeg features. in terms of masked signal modeling, cet-mae implements a high mask ratio (specifically, 75%) on both eeg and text data, presenting a meaningful challenge for the model to handle an increased amount of missing information during the reconstruction phase. this setting not only enhances the model's understanding of individual modality but also facilitates cross-modal interactions and support. furthermore, to make the most of llms' capability in language understanding and generation as well as to fully use pre-trained representations learned by cet-mae, we introduce a new eeg-to-text decoding framework, eeg-to-text using pre-trained transferable representations (e2t-ptr). e2t-ptr utilizes pre-trained modules alongside the eeg stream from cet-mae and further adopts the bart @xcite to decode language from eeg sequences. by transferring the pre-trained representations from cet-mae, e2t-ptr significantly enhances eeg-to-text decoding, surpassing both the baseline and state-of-the-art (sota) methods. our main contributions are summarised below: ## conclusion this study contributes to the development of eegbased language decoding by introducing an effective eeg-text pre-trained model, cet-mae, and a highly capable and llm-empowered eeg-to-text decoding framework, e2t-ptr. cet-mae uses a multi-stream architecture to incorporate both intraand cross-modality ssl within one unified system: 1) intra-modality streams explore representative embeddings that reflect the intrinsic characteristics of eeg or text sequences, leveraging masked modeling with a mask ratio of up to 75%; 2) intermodality stream provides dual-modal representations to enhance intra-modality reconstruction and constrains the encoder to maximize semantic consistency between text and its corresponding eeg sequences. e2t-ptr transfers pre-trained eeg representations and leverages bart's capabilities for text generation from these consistent and representative features. extensive experiments on the latest text-evoked eeg dataset, zuco, demonstrate the superiority of this work in both qualitative and quantitative assessments. the proposed cet-mae model shows great potential for enhancing eegbased language decoding tasks and could be utilized for other inner speech bci datasets.