Modeling Bilingual Sentence Processing: Evaluating RNN and Transformer Architectures for Cross-Language Structural Priming (2024)

Abstract: This study evaluates the performance of Recurrent Neural Network (RNN) and Transformer models in replicating cross-language structural priming, a key indicator of abstract grammatical representations in human language processing. Focusing on Chinese-English priming, which involves two typologically distinct languages, we examine how these models handle the robust phenomenon of structural priming, where exposure to a particular sentence structure increases the likelihood of selecting a similar structure subsequently. Our findings indicate that transformers outperform RNNs in generating primed sentence structures, with accuracy rates that exceed 25.84% to 33. 33%. This challenges the conventional belief that human sentence processing primarily involves recurrent and immediate processing and suggests a role for cue-based retrieval mechanisms. This work contributes to our understanding of how computational models may reflect human cognitive processes across diverse language families.

## introduction structural priming refers to the phenomenon where encountering a specific syntactic structure boosts the probability of generating or understanding sentences with a comparable structure @xcite . it serves as a valuable method for exploring the capabilities of language models and probing their internal states and their potential relation to human sentence processing. studies show that recurrent neural networks (rnn), particularly gated recurrent unit models (gru), have been pivotal in modeling human sentence processing, including structural priming @xcite . meanwhile, transformers also demonstrate structural priming ability similar to that of humans @xcite . this suggests the representations learned by the models may capture not only sequential structure but also some degree of hierarchical syntactic information. that said, to our knowledge, no study has compared these models' ability to syntactically prime across two typologically distant languages. in the current study, we address this gap by comparing the models' ability to prime syntactically across two languages from vastly different families. consider a case where a human participant reads a passive chinese (c) sentence and is then asked to describe a separate picture in english (e) (see figure 1 ). here, the passive sentence c influences the structure of the target sentence e, leading the participant to use passive voice in their description. our study explores structural priming in translation models, highlighting their ability to generate syntactically diverse english outputs from chinese inputs. a key contribution is a set of insights into syntactic representation across typologically distinct languages in computation models. we demonstrate that transformers outperform rnns in generating primed sentence structures, challenging the belief that human sentence processing relies mainly on recurrent and immediate processing. the next section reviews work on crosslinguistic priming. section 3 introduces our study, exploring insights into syntactic representation across typologically distinct languages in computational models. section 4 introduces a newly designed test set to evaluate our models. section 5 details the implementation and training of two distinct models. section 6 discusses the design of our experimental setup, followed by a comprehensive analysis and interpretation of our results. ## related work this section focuses on work related to crosslinguistic priming, as exemplified in figure 1 . prior experiments induce cross-linguistic structural priming by instructing bilingual participants to use two languages: presenting primes in one language and eliciting targets in another. these studies show that specific sentence structures in one language can influence the use of similar structures in the other language @xcite . computational modeling studies have shown that rnns exhibit structural priming effects akin to those observed in human bilinguals @xcite . these models process sequential information through recurrence, a feature thought to resemble human cognitive processing. the emergence of such priming effects in language models suggests that they develop implicit syntactic representations that resemble those employed by human language systems @xcite . however, the transformer model, which uses self-attention mechanisms instead of recurrence, challenges this notion. the transformer's ability to directly access past input information, regardless of temporal distance, offers a fundamentally different approach from rnns. the effectiveness of transformers in various nlp tasks makes us wonder if they can emulate rnns in modeling crosslanguage structural priming. the current study is inspired by two prior studies. @xcite compare transformer and rnn models' ability to account for measures of monolingual (english) human reading effort. they show that transformers outperform rnns in explaining self-paced reading times and neural activity during english sentence reading, challenging the widely held idea that human sentence processing relies on recurrent and immediate processing. their study is monolingual and english-centric. frank (2021) investigates crosslanguage structural priming, finding that rnns trained on english-dutch sentences account for garden-path effects and are sensitive to structural priming, within and between languages. recent studies on structural priming in neural language models have shown significant progress, with researchers quantifying this phenomenon using various methods across different languages. prasad et al. (2019) demonstrate that lstm language models can hierarchically organize syntactic representations in a manner that reflects abstract sentence properties. @xcite show that transformer models exhibit structural priming, suggesting these models capture both sequential and hierarchical syntactic information. @xcite provide evidence that large multilingual language models possess abstract grammatical representations that influence text generation similarly across different languages. together, these findings underscore the capacity of neural models to develop and apply structural abstractions, contributing to a deeper understanding of language processing in ai. ## the current study our study examines structural priming in translation models, demonstrating their capability to generate syntactically diverse english outputs from chinese inputs. this approach offers insights into syntactic representations across typologically distinct languages in computational models. to compare rnns and transformers in their ability to model cross-language structural priming, we adopt a new approach. while frank (2021) trains models on comprehension, where a longer response time indicates greater difficulty in understanding a new sentence and thus a weaker priming effect, the current study focuses on production. here, the structure of each generated sentence is compared with that of the input sentence to assess the presence of a priming effect. as shown in figure 2 , chinese has equivalents for structures that are passive (e.g., many trees were planted by them) and active (e.g., they planted many trees). it also includes structures for prepositional objects (e.g., the cowboy gave the book to the sailor) and double objects (e.g., the cowboy gave the sailor the book). in our study, the input sentence is in chinese and system output is an english version of the sentence. bleu scores are calculated between the system-generated english sentence and both a "correct" english sentence that shares the structure with the chinese inputand an "incorrect" sentence. we then calculate the difference between the two bleu scores, as depicted in figure 3 . another novel aspect of our study is the selection of two languages from vastly divergent language families, challenging the models to develop abstract representations for distinct structures. ## data preparation we select and process a chinese-english corpus which contains 5.2 million chinese-english parallel sentence pairs @xcite . 1 we employ a dataloader 2 to facilitate batch processing, transforming text into token ids suitable for model interpretation. we then use the helsinki-nlp tokenizer (tiedemann and thottin-1 the source can be found at https://drive.google.com/ file/d/1ex8ee5ywbxcaohbo8fh4e2j3b9c2btvq/ view?@xmath0 in sequence-to-sequence models, assigning a value of -100 to padding tokens ensures they are excluded from loss calculations. this setup is crucial for effective model training, enabling precise adjustment of model parameters based on the tokenized input and target sequences. proper data formatting through this preprocessing step facilitates optimal training outcomes. we also design a test dataset, initially sampling five sentences for each of the four sentence structures (active voice, passive voice, prepositional object, and double object) from the crosslanguage structural priming corpus @xcite . to augment the data, we employ a llm, chatgpt 3.5 @xcite , by providing a one-shot learning prompt, we expand each set to 30 sentences, resulting in a total of 120 sentences for our test dataset: generate 30 sentences with the following structure: the cowboy gave the book to the sailor. replace all the words while keeping the sentence structure the same. in our test set, each chinese sentence is paired with a correct and an incorrect english sentence. subsequently, a bilingual annotator proficient in both mandarin and english carefully reviews the sentence outputs generated by the llm, ensuring that each triplet comprises translation equivalents. the review also confirms that only the 'correct' answer maintains syntactic alignment with the original chinese sentence. ## language models we implement both a transformer model and an rnn model to handle sequence-to-sequence tasks using the encoder-decoder architecture. (see experiment of figure 4 .) this architecture supports the processing of both input sequences and output sequences of varying lengths, which is crucial for accommodating sentences with different structures yet similar meanings. this section explores why these language models can assist us identify structural priming. we train and test our rnn model and transformer using amd epyc 75f3 8-core processor and 1 nvidia a100 gpu. ## experimental setup since structural priming effects are sometimes not symmetrical, our study only includes a structural priming experiment with mandarin to english bilinguals, while existing literature strongly supports the presence of structural priming effects in both language directions. to assess the effectiveness of our model in chinese-english, we adopt the standard bilingual evaluation understudy (bleu) metric @xcite , which ranges from 0 to 1, indicating the similarity of predicted text against target text: here, 𝑁 is the maximum n-gram order (typically 4), 𝑤 𝑛 is the weight assigned to each n-gram precision score (@xmath1, 𝑝 𝑛 is the precision score for n-grams of order 𝑛, and bp is the brevity penalty which penalizes shorter results. after generating predicted outcomes and assembling a test set, we analyze the relationship between predictions and four types of reference sentences: (1) correct mappings with the same structure; (2) semantically similar but structurally different sentences; (3) semantically different but structurally identical sentences; and (4) sentences that differ both semantically and structurally. we divide the comparisons into two groups based on semantic similarity. in the first group of sentences with identical meanings, we hypothesize that effective structural priming would result in higher bleu scores between the predicted sentences and the reference sentences that share the same structure, compared to those with different structures. this comparison aims to establish whether the model prefers to reproduce structures that are syntactically aligned with the ground truths when the semantic content remains constant. the second category, with sentences differing in meaning, is crucial for demonstrating structural priming, as it eliminates the influence of semantic similarity. if sentences with identical structures receive higher bleu scores than those with different structures, it suggests the model's predictions are driven by structure, regardless of semantic changes. this methodology rigorously tests for structural priming, offering insights into how models process and replicate language structures. ## results and analyses we present the performance of the gru-based rnn and standard transformer model @xcite demonstrating their crosslingual structural priming effect in chinese-english scenarios. ## summary and conclusions this study evaluates cross-language structural priming effects in rnn and transformer models in a chinese-english context. the models are trained on sentence pairs from both languages. our research aims to compare the structural priming abilities of different models. even when using the same training set, which contains structurally primed sentences, rnns and transformers still exhibit differences in their ability to achieve this effect. we find evidence for abstract crosslingual grammatical representations in these models, which operate similarly to those found in prior research. our results show that bleu scores decrease as n-gram length increases, consistent with findings in sentence-similarity evaluation @xcite . longer n-grams (e.g., bigrams and trigrams) capture more specific contexts, making exact matches less likely unless the target sentence is very precise. moreover, minor errors in word choice or sequence can disrupt the alignment of these n-grams. importantly, our results indicate that transformer models outperform rnns in modeling chinese-english structural priming, a finding that is intriguing given prior research. traditionally, rnns have been effective in modeling human sentence processing, explaining garden-path effects and structural priming through their sequential processing capabilities, which are thought to mirror aspects of human cognitive processing @xcite . our results show that the transformer model is more effective at preserving structural information than the rnn. the standardized accuracy rates for the transformer model exceed those of the rnn by 25.84% for the po structure and by 33.33% for the active structure. this offers guidance for selecting base models in future computational linguistics research aimed at implementing or enhancing structural priming effects. this superiority of transformers raises questions about the efficacy of rnns as human sentence processing models, especially if they are surpassed by a model considered less cognitively plausible. however, these results could also be seen as supporting the cognitive plausibility of transformers, particularly due to the attention mechanism. while the concept of unlimited working memory in transformers seems implausible, some researchers argue that human working memory capacity is much smaller than traditionally estimated, limited to only two or three items. they suggest that language processing involves rapid, directaccess retrieval of items from memory @xcite , a process compatible with the attention mechanism in transformers. this mechanism assigns weights to past inputs based on their relevance to the current input, consistent with cuebased retrieval theories, where memory retrieval is influenced by the similarity of current cues to stored information @xcite . our study on translation models extends the traditional rnn and transformer comparisons in cognitive science, typically applied to language models for predictive coding. @xcite have shown transformers often better capture human language structure. while distinct from pure language modeling, our translation-focused ap-proach offers insights into structural representations in neural networks and lays groundwork for refined language production models.