From Language to Language-ish: How Brain-Like is an LSTM ’s Representation of Nonsensical Language Stimuli? (2020)

Abstract: The representations generated by many models of language (word embeddings, recurrent neural networks and transformers) correlate to brain activity recorded while people read. However, these decoding results are usually based on the brain’s reaction to syntactically and semantically sound language stimuli. In this study, we asked: how does an LSTM (long short term memory) language model, trained (by and large) on semantically and syntactically intact language, represent a language sample with degraded semantic or syntactic information? Does the LSTM representation still resemble the brain’s reaction? We found that, even for some kinds of nonsensical language, there is a statistically significant relationship between the brain’s activity and the representations of an LSTM. This indicates that, at least in some instances, LSTMs and the human brain handle nonsensical data similarly.

## introduction when people read or listen to language, brain imaging studies have shown us that the brain's activity correlates to lstm (long short term memory) state representations for the same text @xcite . in those studies (and others like them) the stimuli used to test for this correlation was based on language with no errors. 1 this implies that during the processing of within-distribution data (i.e. well-formed sentences/stories), lstms and the human brain show similar representational patterns. but what happens when language is out-of-distribution (e.g. nonsensical sentences or pseudo-words)? can we expect 1 nonsensical language is often used when measuring event related potentials. here we speak of decoding studies only. here we use brain imaging data (electroencephalography, eeg) collected in three conditions, sentence: well-formed grammatical sentences, jabberwocky: pseudo-word sentences that preserved word order, morphosyntax, and sentential prosody without lexical or compositional semantics, and word-list: the words of the sentence condition in a pseudo-random order without sentence prosody, syntax, or compositional meaning. we ran a character-level lstm model on the stimuli, and trained a decoding model to predict the lstm's internal representations from eeg signals. using data from the sentence condition, we corroborated previous results and showed that lstm representations are correlated with brain activity for withindistribution language. but, when it came to nonsensical language stimuli, it was unclear if lstm representations would still correlate to brain activity. our original hypothesis was that lstm representations for out-of-distribution language would no longer correlate to brain activity. however, we found that our decoding model worked quite well even when all content words of the stimuli were pseudo-words (jabberwocky). to summarize, we show that: ## results we were interested in comparing the representations generated by an lstm to that of the human brain, in response to both within-and out-ofdistribution language. our sentence stimuli, which represent within-distribution language, contain semantic and syntactic information. we used two kinds of out-of-distribution stimuli: jabberwocky, which was designed to have syntactic information only, and word-list, which has only semantic information. we attempted to learn a mapping from eeg to lstm representations (to test if the lstm and brain handle the stimuli similarly). to begin, we examined the difference in the semantic and syntactic information encoded by each of the lstm representations. then, we developed analyses to test for a similarity in the representation of semantic and syntactic information across the experimental conditions. we investigated using the following questions: ## discussion considering the results as a whole, several points become clear. there is a relationship between the semantic and/or syntactic information as represented by the brain and by lstm representations, at least for the sentence and jabberwocky conditions. the probing results are quite consistent with the results of analyses 1-3: lstm1 has a strong signal for syntax, lstm2 has syntax and semantics, and lstm3 has some syntax and/or semantic signal, but the signal is weaker than for lstm1-2. lstm1 shows only minor changes in performance in analysis 2 and 3. so the syntactic information encoded in this layer is fairly consistent for stimuli from both the sentence and jabberwocky conditions, and it correlates well to either eeg data source. there is likely not much semantic information to leverage here, as the performance of models trained on sentence eeg change by only a small amount in analysis 2 and 3. in analysis 2 we saw similar drops in lstm2 performance for both sentence and jabberwocky conditions. the drop in performance using the sentence eeg could be attributed to the lack of semantic information in the jabberwocky lstm representations. however, we see a similar size drop in performance for the jabberwocky condition, which implies that there is a mismatch even in the syntactic information available in lstm2 for the two conditions. in analysis 3, when we swap the test data, the pattern learned to predict lstm2 in the sentence condition (leveraging semantics and syntax) is not as effective when tested on jabberwocky data. the performance of lstm3 is harder to explain, possibly because it has weaker semantic/syntactic signal (as evidenced by the probing tasks). there is a small performance hit when training on sentence eeg data in analysis 2, but a very large drop in analysis 3. this pattern could result if lstm3's representations of syntax are similar for sentence and jabberwocky stimuli, but the brain showed differing representations for the syntactic information in the two conditions. then, it is possible that only the sentence eeg would correlate to the syntactic information in lstm3. we wondered if there could be another explanation for our ability to decode in the jabberwocky condition. one possibility is that the eeg and lstm layers contain a correlate of the position in a sentence (1st word, 2nd word, etc.), and our models are using that information to decode (7/8 2 vs. 2 tests will use words at different positions). to test for this possibility, we trained a classifier to predict the ordering of two random words selected from a sentence, as suggested by @xcite . the input to the classifier is the lstm representation of the two words at their positions in a sentence, and the output is a binary decision for which of the two words appears sooner in the sentence. a model trained using our lstm and the sentence stimuli produced 80% accuracy on this task. thus, we cannot say unequivocally that our results are not due in some part to positional information. however, our probing results are consistent with there being semantic/syntactic information in the representations, and those results are very consistent with the decoding analysis. this is strong evidence that our results are not entirely due to positional information. we wondered also if the lexical semantics of the jabberwocky stimuli could be leaking into the lstm vectors, perhaps because the pseudo-words were repaired in the convolution step of the lstm. note, however, that lexical semantics are entirely intact in the word-list condition, but the lstm representations are of no use in that condition. morphosyntax and syntax are maintained in the jabberwocky condition, which appears to be enough to drive the correlation between lstm representations and eeg recordings. the lstm may be picking up on bi-and tri-gram signals related to morphosyntax cueing syntactic structure @xcite , but more work is needed to rule out alternative explanations. recall that the sentence and jabberwocky stimuli share some orthographic/phonological information. could our jabberwocky results, and the results of analysis 2 (swap g(s)), be due only to the eeg encoding phonological or orthographic information? if our models were able to leverage such information, we would expect to see comparable decoding results in analysis 1 and the wordlist condition, where the stimuli are perfect orthographic matches to the eeg. however, that analysis did not produce significantly above-chance accuracy. furthermore, if the information leveraged in analysis 2 was at the character-level, we would expect to see significantly above-chance accuracy in the character embedding or convolutional layers. however, it is not until the first lstm layer (where contextual information is first incorporated) that any decoding model performs significantly above chance in any condition. this is evidence that the information being leveraged is not simply phonological or orthographic. our stimuli are composed of two conjoined sentences. how much composition have dutch listeners done by the time when they get to the conjunction word "en?" how does the processing differ between the first vs the second of the conjoined sentences? previous work on the brain's processing of syntactic structures and coordinate clauses proposed an "active structure maintenance model", where neural activity increases as a function of syntactic complexity @xcite ; @xcite . they found that neural activity in certain left-hemispheric regions indeed increased when more constituents had to be integrated, for both sentences and jabberwocky stimuli. it may be that the second coordinate constituent in our stimuli sentences elicit stronger neural activity than the first, but more analysis would be required to verify this. the first example of mapping brain responses onto corpus-derived representations appeared in @xcite . this study encoded word meaning into vectors of word co-occurrence features. the authors showed that a trained linear regression model could predict fmri activation in response to single concrete noun stimuli. from there, decoding models were shown to work with dependency-parse-based representations @xcite and with concept-relation-features extracted from topic models @xcite . @xcite demonstrated that decoding models can learn the pattern of the brain's response to abstract concepts/nouns. some of the first examples of decoding language in context were from @xcite and @xcite . the first models used a combination of (non-contextual) corpus-derived, acoustically-derived and/or hand-coded representations. several groups then began to experiment with encoding models based on contextual language representations, like those in recurrent neural network (rnn) language models @xcite @xcite . these models showed that vectors incorporating contextual information could be decoded from brain imaging data, and contextual models actually outperformed non-contextual word vectors. we confirmed those findings here. though there are fewer decoding models trained on eeg, there are a few recent examples. @xcite showed that the operations performed by an rnn-grammar trained to parse sentences correlated to eeg collected while people listened to a story. @xcite found connections between bi-lstm representations and the erps (event related potentials) more classically used to study language in the brain. our work adds to the new body of work showing that eeg can be a powerful data source in this space. ## conclusion and future work in this study, we explored the correlation of a character-level lstm with the brain's response for two kinds of out-of-distribution language. the jabberwocky condition used pseudo-word translations of the sentence stimuli (ablate semantics, preserve syntax). the word-list stimuli was a pseudorandom re-ordering of the words in each of the sentence stimuli (ablate syntax, preserve seman-tics). we ran a character-based lstm to create contextual embeddings for the stimuli of each condition. our linear-regression decoding models were trained to predict the various lstm representations from the eeg signals. our results showed that the lstm layers of this character-based lstm do in fact correlate with eeg signals in both the sentence and jabberwocky conditions, but not in the word-list condition. by training models with various alterations to the data, we were able to determine which lstm representations carry semantic and syntactic information. we verified those results using a probing task on our dutch lstm, as well as an identical model trained on english. there are multiple avenues for future work. for example, dutch has a fairly transparent phonemegrapheme correspondence; would our results still hold for a language with deeper orthography? we were surprised to find that some lstm representations resembled the jabberwocky eeg signals. are there other examples of out-of-distribution language where this relationship holds? and, perhaps more interestingly, where it does not hold? finding ways in which the brain's representations differ from an lstm could help us to build models closer to the true nature of human language processing.