Decoding Part-of-Speech from Human EEG Signals (2022)

Abstract: This work explores techniques to predict Part-of-Speech (PoS) tags from neural signals measured at millisecond resolution with electroencephalography (EEG) during text reading. We first show that information about word length, frequency and word class is encoded by the brain at different post-stimulus latencies. We then demonstrate that pre-training on averaged EEG data and data augmentation techniques boost PoS decoding accuracy for single EEG trials. Finally, applying optimised temporally-resolved decoding techniques we show that Transformers substantially outperform linear-SVMs on PoS tagging of unigram and bigram data.

## introduction electro-/magnetoencephalography (eeg/meg), which measures neural activity at millisecond resolution, is a key neuroscientific method to assess how neural representations unfold dynamically in language processing. early event related potential (erp) studies that rely on averaging eeg activity across multiple trials have shown that eeg signal magnitude and topography depend on word length, frequency and open vs. closed class. word length effects arose in eeg at about 150 ms, frequency effects at 200 ms and word class effects from 400-700 ms @xcite @xcite @xcite @xcite . recent studies were able to predict these and other (e.g. semantic) aspects based on single trial multi-channel eeg/meg activity @xcite @xcite . importantly, the aim of cognitive neuroscience studies is to dissociate when (i.e. latency) and where (i.e. brain region) specific linguistic information is explicitly encoded in neural activity. neuroscience studies therefore typically use linear decoders and try to disentangle neural activity for linguistic and non-linguistic dimensions that covary in natural language statistics (e.g. word class vs. length). by contrast, engineering applications mainly aim to maximise performance accuracy, utilising all available information and more powerful nonlinear classifiers. intriguingly, recent studies have shown that adding human eye tracking data @xcite or morphosyntactic information extracted from human functional magnetic resonance imaging (fmri) signals during sentence reading, can substantially improve pos induction @xcite ). yet, morphosynactic information obtained from fmri is limited, because fmri measures only the slow changes in blood oxygenation, peaking 5-6 s after stimulus onset, rather than the rapid neural activity during language processing. contributions. this interdisciplinary paper decodes pos tags from eeg signals with linear svms and transformers, pursuing several aims relevant for neuroscience and/or engineering. neuroscience-focused section 3 uses linear svms to define the distinct neural representations of word length, frequency and class based on a new eeg data set, in which a single subject reads an extensive syntactically annotated corpus. to dissociate these linguistic and non-linguistic aspects, typically correlated in natural language statistics, section 3 matched the stimulus distributions for each classification task with respect to the confounding dimensions of no interest. consistent with previous reports, we show that word length, frequency and class can be decoded at different post-stimulus latencies based on single trial and trial-averaged data. this replication part serves to validate the new eeg data set. methods-focused section 4 moves beyond open vs. closed word class decoding that were the focus of previous meg/eeg studies and decodes 6 pos tags from eeg activity with linear svms and transformers. we show that pretraining on trial-averaged data with subsequent fine-tuning on single-trial data, alongside data augmentation, boosts pos decoding accuracy from single-trial eeg data selectively for transformers (but not for linear svms). engineering-focused section 5 finally uses linear svms and transformers together with pretraining and augmentation techniques from section 4 to assess how pos information about unigrams and bigrams becomes progressively available in eeg activity across post-stimulus time. comparing eeg decoding from sliding and incremental time windows suggests that transformers outperform linear svms particularly when information needs being integrated across longer time windows. our results raise the possibility of combining postagging based on eeg decoding with corpora and dependency tree annotation to obtain more reliable morphosyntactic information for low-resource languages. ## general methods our experiments used a new corpus annotated with eeg data, previously acquired at the university of birmingham following ethical approval and participant's informed consent. the eeg annotated corpus is available 1 under a public license (cc by-sa 4.0). ## decoding word class, length, frequency for comparison with previous research @xcite , we decoded word length, frequency and class with linear svms in a temporally-resolved fashion from 0 to 700 ms poststimulus eeg, recorded during sentence reading. data set. we decoded word class from eeg via binary classification between open class words (i.e. noun, verb, adj, and propn) vs. closed class words (i.e. det, adp, aux, pron, sconj and cconj). likewise, for decoding word frequency and length, words were assigned to two classes based on the median values in the data set (i.e. zipf-@xmath0.91 = high else low; word @xmath1, else short). eeg decoding analyses were performed for single-trials, averages across 3 and 10 trials. to dissociate the distinct contributions word length, frequency and class that are highly correlated in natural language statistics, we decoded one variable by controlling for the other two variables. for instance, when decoding open vs. closed class words, we selected a subset of trials such that the joint distributions over the three confounding variables of word frequency (discretised to the nearest 0.25), length (number of characters) and sentence position (i.e. sentence initial, mid, end) were equated for the categories of open and closed class words. to minimise confounds arising from the preceding word in the sentence, we balanced the test set with respect to the open/closed class status of the previous word. similarly, we controlled the decoding of word frequency for word length, and the analysis of word length for word frequency, and both analyses for open/closed class and sentence position. each analysis across training, dev and test sets. methods. to temporally resolve how the brain encodes word length, frequency and class, we trained and tested linear svms on eeg signals separately for sliding windows of 64 ms (i.e. 16 time points) that shift in increments of 4 ms (i.e. one time sample). figure 2 shows the mean accuracy values (averaged across 10 seed points) from the test set (centred on the last bin of each time window @xcite ) @xmath2. the classification responses for the test set from the model that performed best on the dev set were entered into a two-sided binomial test, separately for each time window. solid lines in figure 2 above the decoding accuracy time courses indicate time points that were significant at (@xmath3.05) false discovery rate (fdr) corrected for multiple comparisons @xcite across time (i.e. 160 tests). results. figure 2 (top rows of a, b, c) shows butterfly plots for the effects of word length, frequency and class across 64 electrodes. our linear svm decoding analysis replicates the temporal cascade of word length, frequency and class effects previously reported for eeg responses averaged across a large number of trials. the word length effect arises early at about 100 ms, previously associated with visual word processing in occipitotemporal cortices @xcite @xcite . word frequency influenced neural processing later from 200 ms onwards with a slight left-hemispheric predominance @xcite . the word class effect emerged in early and late time windows with the effect at about 550 ms in line with the wellknown p600 as an erp indicator for syntactic processing @xcite @xcite . word length and frequency effects were stronger than the word class effect; see @xcite . as expected, decoding accuracy increased when eeg signals were averaged across trials. thus, carefully controlling each comparison of interest (e.g. word class) for the effects of no interest (e.g. word length and frequency) enabled us to dissociate word length, frequency and class effects, despite their high correlation in natural language, thereby validating our new annotated eeg data set and analysis procedure. ## temporally-resolved pos decoding sections 3 and 4 were driven by the neuroscience goal of dissociating neural representations associated with pos from confounding factors such as word length or frequency, which are typically correlated with pos in natural language statistics. to control for these confounds sections 3 and 4 generated data sets, in which e.g. pos classes were equated with respect to word length. by contrast, section 5 pursues the engineering goal of maximising pos decoding accuracy. here, correlations between word frequency, length and pos class are no longer considered a confound but a useful source of information. capitalising on the optimised 3-1 training scheme with 250k data augmentation from section 4, section 5 will assess whether pos information about unigrams and bigrams can be decoded from eeg signals (without any confound controls). for both unigrams and bigrams, we will first investigate how pos information becomes available dynamically across post-stimulus time by training svms and transformers on eeg signals from 64 ms sliding time windows. second, we will assess how svms and transformers integrate pos information across post-stimulus time by training them on eeg signals from incremental time windows. methods.we implemented the 3-1 pretraining with 250k data augmentation (section 4). for the sliding window analysis, we trained and tested svms and transformers on eeg signals from 64 ms windows (i.e. 16 time points) that shifted in increments of 16 ms from 0 ms to 700 ms (i.e. resulting in a sequence of 41 decoding accuracies). for the incremental window analysis, we successively increased the initial [0 16] ms time window (i.e. 4 samples) by 4 additional sampling points resulting in a temporal sequence of 44 decoding accuracies. we computed decoding accuracies (mean across seeds, ± 68% ci) from the test set. across time windows we compared the decoding accuracies on the test set of the best dev set svm and transformer using the wilcoxon signed rank-test (reported at @xmath4.05, fdr-corrected for multiple comparisons across time i.e. 41 resp. 44 tests). results. in the sliding window analysis, the decoding accuracies of svms and transformers show two prominent peaks at 200 ms and 400 ms suggesting that pos decoding relies on several aspects of information encoded in the eeg. based on our confound-controlled analysis (section 3) the first peak reflects word length and frequency information, while the second peak is more closely related to semantic and syntactic aspects of the word. the incremental window analysis showed an accuracy benefit of 4.5% for the transformer starting in the very first [0 16] ms time window. this difference in performance between the two models further widened, reaching a maximum advantage of 11.6% around 360 ms. transformers thus benefit from integrating information about word frequency, length and class that arise at different post-stimulus latencies. moreover, because pos classes of subsequent words are correlated in natural language, the transformer may also benefit via self-attention from information about the next word that is presented and progressively encoded in eeg activity about 240 ms after the current (i.e. to be decoded) word. statistical testing confirmed that the transformer significantly outperformed the svm for all sliding and incremental windows (wilcoxon-signed rank test, fdr-corrected at @xmath5.05). ## related work the confound-controlled analysis dissociated word length, frequency and class effects in eeg. this replication of earlier erp @xcite and meg decoding work @xcite validates a new eeg data set for an extensive morphosyntactically gold annotated corpus; c.f. @xcite . transformers successfully decoded 6 pos tags from single trial eeg with data augmentation and 3-1 pretraining (≈ 40% accuracy), raising the possibility to boost pos induction with eeg-decoded pos tags. while we acknowledge that our results are limited to eeg data from a single subject, given the spatial smoothness of eeg scalp topographies, we envision pretraining on eeg obtained from different participants. further, because human brains generate similar neural signatures for word classes across different languages (c.f. yudes (2016); @xcite ; @xcite ), pretraining pos-eeg decoders on large morphosyntactically annotated eeg datasets for english followed by fine-tuning on a smaller annotated eeg data set for a low-resource language may enable successful generalisation to eeg obtained from reading non-annotated texts in this low-resource language. pos-induction jointly based on annotated texts and eeg signals could thus be transformative for corpus generation of low-resource languages. ## conclusion combining neural signals measured at millisecond resolution with eeg and a linguistically annotated corpus, this work shows -to the best of our knowledge -the first time that unigram and bigram pos tags can be decoded successfully from single-trial eeg data. temporally-resolved eeg decoding unraveled how information about linguistic and nonlinguistic aspects evolved dynamically across time. unsurprisingly, transformers with self-attention mechanisms outperformed svms across all experiments. in particular, they benefited from integrating information across time, data augmentation and pretraining methods. our work paves the way for future applications that incorporate human brain signals in traditional nlp methods.