M ap G uide: A Simple yet Effective Method to Reconstruct Continuous Language from Brain Activities (2024)

Abstract: Decoding continuous language from brain activity is a formidable yet promising field of research. It is particularly significant for aiding people with speech disabilities to communicate through brain signals. This field addresses the complex task of mapping brain signals to text. The previous best attempt reverse-engineered this process in an indirect way: it began by learning to encode brain activity from text and then guided text generation by aligning with predicted brain responses. In contrast, we propose a simple yet effective method that guides text reconstruction by directly comparing them with the predicted text embeddings mapped from brain activities. Comprehensive experiments reveal that our method significantly outperforms the current state-of-the-art model, showing average improvements of 77% and 54% on BLEU and METEOR scores. We further validate the proposed modules through detailed ablation studies and case analyses and highlight a critical correlation: the more precisely we map brain activities to text embeddings, the better the text reconstruction results. Such insight can simplify the task of reconstructing language from brain activities for future work, emphasizing the importance of improving brain-to-text-embedding mapping techniques.

## introduction decoding continuous language text from brain activity stands as a groundbreaking endeavor at the nexus of neuroscience, linguistics, and artificial intelligence. such an advancement promises to revolutionize communication, offering a new voice to those with speech impairments @xcite . beyond enhancing communication, this research offers profound insights into the brain's language processing, paving the way for interfaces that integrate thought and * corresponding author speech effortlessly @xcite @xcite . while trials using invasive technologies like ecog have shown promise @xcite , the broad application of these methods is hampered by the limited public availability of invasive data and the complexities associated with neurosurgery. decoding continuous language from noninvasive brain recordings, which are more accessible, remains a formidable challenge. this difficulty mainly stems from the intricate and dynamic relationship between language and the neural responses it elicits, further complicated by the inherently noisy nature of non-invasive neuroimaging. the previous best attempt to tackle this issue first encoded brain activity from text with a linear model and then used this to guide text generation by aligning it with predicted brain responses @xcite . however, whether such an indirect method is optimal for the decoding task and whether a linear model is adequate for continuous text generation are questionable. although this method has shown some improvement over random-level performance, the advancements are marginal. addressing the complex challenge of decoding continuous language from brain activities, we introduce mapguide, a simple yet effective two-stage framework. the first stage learns to map brain activity to text embeddings with a transformer-based mapper. we improve the mapper's resilience to neural noise by employing a random mask method for data augmentation and contrastive learning. in the second stage, a pre-trained text generator is guided by text embeddings predicted with the mapper to produce text that closely aligns with the embeddings. mapguide's integration of these two stages offers a more direct and effective solution for translating neural signals into a coherent text. experiments show that the proposed method achieves a new state-of-the-art (sota) result in reconstructing continuous language from fmri-based brain recordings, significantly higher than the previous best attempt as measured by four different types of metrics. our investigation further reveals an interesting contrast in compatibility patterns between frameworks: while previous encoding-based frameworks excel with linear models in linking brain activity and language, the decoding-based framework demonstrates superior performance when paired with non-linear models, underscoring a pivotal shift in approach for optimal results. we also find a clear link between the accuracy of mapping brain activities to text embeddings and improved text reconstruction performance. this insight simplifies the task of reconstructing language from brain activities, emphasizing the importance of refining the brain-to-text embedding mapping process. ## results in this section, we will first compare the metrics of text reconstruction of mapguide against the previous sota. we will then conduct a detailed ablation study to evaluate the efficacy of mapguide's modules and discuss the influence of hyper-parameter setting. we will lastly do a correlation analysis of the experimental results. ## conclusion in this paper, we propose mapguide, a simple yet effective double-stage framework for reconstructing continuous language from brain activities. in the first stage, we learn a mapper that decodes text embeddings from brain activities with contrastive learning. in the second stage, the mapper is applied to supervise a text generation model. mapguide exceeds the previous sota by a large margin on all evaluation metrics. through comprehensive ablation studies and in-depth case analyses, we further substantiate the efficacy of mapguide's modules. our research further reveals a direct correlation between the precision of mapping brain activities to text embeddings and the subsequent improvements in text reconstruction performance. this insight can be informative in streamlining the intricate process of language reconstruction from brain activities. by enhancing the mapping from brain activities to text embeddings, we can significantly simplify and improve the task of language reconstruction.