Are BERT s Sensitive to Native Interference in L 2 Production? (2021)

Abstract: With the essays part from The International Corpus Network of Asian Learners of English (ICNALE) and the TOEFL11 corpus, we fine-tuned neural language models based on BERT to predict English learners’ native languages. Results showed neural models can learn to represent and detect such native language impacts, but multilingually trained models have no advantage in doing so.

## introduction with globalization progress, more and more people start acquiring more than one language. therefore, computational models and theories to better understand multilingualism among machines and humans start receiving more and more attention. even though language models (lms) succeeded in a wide range of tasks in natural language understanding and processing (i.e. @xcite @xcite , how to do such models process and represent languages and knowledge remain unclear. we also have few ideas about these models' abilities when facing various types of language input, such as input from non-native speakers or speakers with lower proficiency. in this study, we implemented state-of-the-art neural lms to predict non-native (l2) speakers with different language backgrounds through native language identification tasks. furthermore, we investigated whether cross-lingual training can help identify a language learner's writing patterns, which his native languages (l1s) can cause interference. this study wants to expand current knowledge on representations and crosslingual components in transformer-based lms. ## related work with developments in computer science, data sciences, and cognitive and language sciences, artificial language models (lms) based on language representations and embeddings have received attention in multiple areas. current deep neural lms based on various architectures can capture syntactic information through natural language input and utilize language knowledge to complete different tasks, such as identifying specific sentence structures @xcite @xcite and predicting grammar acceptability @xcite . beyond training on a single language, multilingual neural language models could process cross-lingual tasks such as language identification and translation. studies found that language representations in these types of models are transferable or overlapped across languages to create universal representations of grammatical structures @xcite . models can also develop language-neutral components for better word alignment and improve performance on several cross-linguistic tasks @xcite . however, studies indicated that these lms have lower performance in crosslinguistic tasks such as machine translations @xcite . comparing to monolingual models, multilingual lms have relatively lower performance and unbalanced performance in syntacticrelated tasks between english and other languages @xcite . even though transformer-based lms showed impressive abilities in multiple tasks, few studies tried to analyze whether and how multilingual lms identify and distinguish writers' native interference in their l2 production caused by their l1s. one task for detecting such interference across speakers is the native language identification task, which predicts a speaker's l1 using his language production in l2. by sharing representations and transferring knowledge across languages @xcite , multilingual speakers can have different structures, preferences, and variations for language comprehension and production than monolingual speakers, result in cross-linguistic behaviors such as cross-lingual priming effects @xcite @xcite and syntactic preferences in language processing @xcite . previously, statistical-based language models (n-gram models) dominated the tasks @xcite , as statistical models are more sensitive in detecting syntactic and morpho-syntactic patterns. embedding models, on the other hand, are more sensitive in capturing semantic and lexical information and less sensitive in classifying syntactic patterns and differences @xcite . given the high performance in syntactic-related tasks @xcite , comparing to previous embedding-based models, transformer-based models may perform better in detecting native language interference. such advantages can help models better identify specific patterns in l2 learners. furthermore, lms with multilingual pre-training may be benefited in identifying different non-native speakers using additional knowledge across languages. however, existed results contradicted such assumptions, indicating that multilingual lms generally performed worse than monolingual models in the same tasks @xcite . to better understand transformerbased lms, multilingual pre-training experiences, and language representations in lms, our study aims to use native language identification task to answer the following research questions: ## results performance. the models fine-tuned with toefl11 data did not beat the state-of-the-art native language identifier using statistical models. for models fine-tuned with icnale data, controlling learners' proficiency did not critically impact models in identification. this showed that speaker proficiency did not strongly interfere with language background identification, indicating the robustness of native language interference across learners' l2 written production. shuffling word orders did impact all models severely in prediction ( multilingual pre-training. when using the corpus full shuffled icnale 0.86 (0.87) 0.24 (0.29) toefl11 0.72 (0.68) 0.46 (0.26) ## general discussion while transformer-based lms reached impressive results in language processing and generation, we still know little about how lms acquire, process, and store language representations. our study used native language identification task to investigate 1) whether transformer-based models (bert & mbert) can detect native interference in written production; 2) whether multilingual pre-training help in identifying native interference. overall, the results were negative, indicating 1) compared to statistical models, transformer-based models are less capable of capturing native impacts and interference, and 2) additional knowledge across multiple languages in mbert did not help predict such impacts across multilingual speakers. ## summary while transformer-based language models and multilingual language models start dominating the natural language process area, little is known about the knowledge structures within such models. in this study, we explored bert-like models' performance in identifying a speaker's native language. our models did not outperform the state-of-the-art native language identifier based on statistical-based models. multilingual pre-training did not improve the performance in predicting native language, indicating the insufficiencies for detecting writing styles and structural patterns among different multilingual speakers in current lms. further explorations are needed to investigate how multilingual lms store, process, communicate, and transfer representations and knowledge across languages.