Identifying negative language transfer in learner errors using POS information (2021)

Abstract: A common mistake made by language learners is the misguided usage of first language rules when communicating in another language. In this paper, n-gram and recurrent neural network language models are used to represent language structures and detect when Chinese native speakers incorrectly transfer rules from their first language (i.e., Chinese) into their English writing. These models make it possible to inform corrective error feedback with error causes, such as negative language transfer. We report the results of our negative language detection experiments with n-gram and recurrent neural network models that were trained using part-of-speech tags. The best performing model achieves an F1-score of 0.51 when tasked with recognizing negative language transfer in English learner data.

## introduction advances in grammatical error correction (gec) research allow writing editors to provide real-time corrective feedback to language learners. gec systems are trained on parallel erroneous and corrected learner data to detect and suggest corrections for user errors. state-of-the-art gec systems apply neural machine translation to learn how to correct grammatical errors in english learner data @xcite . these systems also use metadata, such as the learners' native languages and their proficiency levels in the target language, to support the gec task @xcite . the direct corrective feedback derived from gec systems' predictions helps learners improve their writing and increase their language understanding. however, direct corrective feedback is not the only type of information that can aid language learning @xcite . learners also benefit from metalinguistic feedback. this feedback type can support learners' reflections on their lan-guage usage and, consequently, increase their grammatical awareness @xcite . one metalinguistic phenomenon that often causes confusion and errors in learner writing is the occurrence of language transfer. the language transfer phenomenon is characterized by learners reusing rules from their first languages (l1s) when communicating in a second one foot_0 (l2) @xcite . when the l1 and l2 grammars diverge, learners who apply l1 rules make mistakes. this particular version of language transfer is called negative language transfer, and it is a significant source of errors in second language learner speaking and writing. this paper explores the application of language models to represent language structures and detect negative language transfer in learner essays written by chinese native speakers. the proposed methods aim to identify incorrect learner utterances that have structural patterns that are more similar to the learners' l1 (chinese) than to their l2 (english). in this paper, we demonstrate that a recurrent neural network trained to identify a part-of-speech (pos) tag sequence's source language outperformed pos n-gram models in negative language transfer detection. the rnn model achieved an f1-score of 0.51 on the negative language transfer detection task analysing pos tag sequences extracted from learner errors. the output of this language model can be used to inform metalinguistic feedback, tying the incorrect utterances to differences between the l1 and english rules. by enabling negative language transfer detection in learner writing, language learners will have access to interpretable information about their errors' potential causes along with direct corrective feedback for those errors. ## related work when writing in another language, writers are prone to leave behind certain traces of their native languages. this fact is the foundation for the native language identification task. this task makes use of computational models to determine the l1 of a text's author @xcite . native language identification models aim to uncover usage patterns that distinguish native speakers of a certain language when writing in english. @xcite explored the usage of cognates in non-native english writing to cluster the authors' text according to their native languages' families. cognates are words from distinct languages that are similar in meaning and form. as demonstrated by @xcite , non-native english writers show a preference for using english words that have cognates in their l1. furthermore, @xcite have shown that it is possible to employ the authors' error patterns to discriminate between l1s, as english learners' writing errors are highly correlated to their native languages. learner writing errors are useful in yet another computational task. the aforementioned grammatical error correction task aims to detect and correct errors in learner data @xcite @xcite . state-of-the-art gec systems model the task as a neural machine translation procedure in which the erroneous learner data is the source language, and its corrected version is the target language @xcite . throughout the years, researchers have explored the impact of using l1-specific learner data to train gec systems: see @xcite , and @xcite for examples. these researchers have found that whether gec models are exclusively trained with l1-specific data or whether this data is only used to fine-tune the model, gec benefits from information about learners' l1s. in addition to that, @xcite explored fine-tuning gec systems to learners' english proficiency levels and l1s. they found that information about both learner aspects improves gec performance. english non-native speakers transfer patterns and rules from their l1s into english writing. this phenomenon has been explored by native language identification and grammatical error correction systems. moreover, the phenomenon has been demonstrated in experimental investigations of the con-trastive analysis hypothesis @xcite , as reported in papers by @xcite and @xcite . @xcite found a correlation between structural error distribution in english as a second language learners' writing and typological differences between the learners' l1s and english. their work used the l1's structural properties to predict challenging grammatical areas in english as a second language writing. one of the challenges for language learners is that they are often unaware that they reuse structures from their l1s (wanderley and demmans epp, 2020). one way to alleviate this difficulty is to provide error feedback that contains information about possible error sources @xcite . linguists and education researchers have long debated what is the best way to provide feedback in a second language setting. some argue that no feedback should be given, while others posit that learners should have access to a different range of error feedback types, such as direct, indirect, comprehensive, and focused @xcite . more recent research has found that providing metalinguistic feedback can increase language learners' writing accuracy @xcite . unlike direct corrective feedback, which simply highlights and corrects learner errors, metalinguistic feedback can help learners understand their errors by calling attention to the errors and describing possible causes @xcite . to understand whether students value metalinguistic feedback about negative language transfer, @xcite examined japanese students' familiarity with the phenomenon. these learners highlighted how being aware of language transfer effects helped improve their english writing. while natural language processing (nlp) tasks and language learners benefit from information about negative language transfer, we are not aware of previous research that automatically detects this phenomenon. recently, monaikul and di eugenio (2020) applied traditional and neural natural language processing methods to detect preposition omission errors with the aim of informing negative language transfer feedback. preposition errors are one of the most common types of negative language transfer errors in second language learning. in their paper, the authors proposed applying the preposition error detection output to generate metalinguistic contrastive feedback for language learners and teachers @xcite . we envision a similar application of our results. however, our focus is on detecting additional negative language transfer errors. we want to detect all those that come from learners inappropriately applying structural patterns from their l1s when writing in english. ## results the n-gram baseline paired with the padded error representation was more precise in classifying negative language transfer than the rnn when it analysed errors represented with the error + unigram span. however, the recall scores yielded by these two experimental combinations show that the rnn approach was able to retrieve more nega-tive language transfer errors than the n-gram baseline. although both n-gram and rnn approaches achieved analogous precision results, especially when analysing padded errors, the n-gram baseline was consistently outperformed on recall. these results indicate that comparable precision came with a lower recall cost for the baseline. the high precision compared to low recall rates achieved by the n-gram baseline can be attributed to the fact that these models compute their output values from independent probability distributions derived from the training data. this may indicate that these models excel in classifying negative language transfer when there is a clear distinction in usage of the error structure between the languages. for example, if a learner applied a language structure that is frequent in chinese but extremely rare in english, the chinese n-gram model would assign a much higher probability than the english n-gram model, and the error would be correctly classified as negative language transfer. however, these very prominent usage differences are scarce in our data. as the fce test-takers are learning english, they may not fully apply chinese rules but instead combine rules from the two languages. this interlingual process makes the error structures ambiguous to our independent n-gram models. the results in ## discussion the error coding used by the fce annotators is useful to our error analysis as it allows the learner errors to be grouped by error subtype @xcite . analysing which error subtypes cause the shallow syntactic language models to misclassify negative language transfer errors provides insights as to how our methods could be revised to better support the task. our models performed poorly when detecting negative language transfer in replacement errors. replacement errors commonly occur when the learner uses the correct part-of-speech, but incorrect word in a sentence. for example, the word "many" in the sentence "how many you charge?" was flagged as a quantifier replacement error in the fce dataset. both n-gram and rnn shallow syntactic language models performed poorly in identifying whether replacement errors were related to negative language transfer. this may have occurred due to the pos tag representation of these errors not conveying a lot of information about the error's source. the erroneous and correct pos tag sequences follow the same pattern, a pattern that is acceptable in english. the pos tag representation of errors also hindered negative language transfer detection for errors involving verb tenses, forms, and agreement. the universal dependencies tagset annotates verbs with one of two tags, "verb" or "aux". none of these tags contains information about verb features, such as tense, form, or person. these features would be useful in detecting verb usage patterns that are more common in chinese than english and could be related to negative language transfer. these error subtype results indicate that the models would benefit from a more detailed pos tagset. a tagset that encapsulates more word features in their tags, such as the penn treebank tagset @xcite . the challenge in using a more detailed tagset is that it needs to be common between english and the learners' l1 (in our case, chinese). the universal dependencies tagset is the only tagset we are aware of that fits this commonality requirement. chinese speakers who are learning english as a second language usually have trouble with punctuation marks, as these symbols are employed differently in chinese @xcite . for example, in chinese, commas are used to mark the end of a complete thought @xcite . in our experiments, the pos tag span selection was fundamental to the detection of negative language transfer in missing punctuation errors. both the n-gram and rnn approaches achieved the highest f1-scores when detecting negative language transfer in missing punctuation errors when using the padded error span. when evaluating the padded error span for missing punctuation errors, the rnn model achieved an f1-score of 0.48. the error + unigram and error + bigram spans yielded f1-scores of 0.43 and 0.41, respectively. a few possible explanations for this disparity in scores are that the errors' previous contexts helped the models correlate the errors to the chinese language, or the lack of punctuation in between the spans' pos tags is divergent from english grammar rules. another common mistake for chinese speakers who are learning english is determiner omission. as determiners do not exist in the chinese language, learners tend to neglect their application in english @xcite . contrary to our initial assumption, missing determiner errors were better classified by the rnn model when using the error + unigram span. the previous context from the padded error span did not help the classification. one hypothesis for this behaviour is that it is common for determiners, as opposed to other parts-ofspeech that usually follow determiners, to appear at the beginning of sentences in english. as there are no determiner equivalents in chinese, whenever the rnn analysed a pos tag sequence that should begin with a determiner in english, but would not in chinese, the sequence was classified as negative language transfer. in these cases, the padded error span contained another pos tag preceding the position where the determiner should have been. the rnn model might have been confused by this preceding pos tag, as there are mid-sentence situations in english in which determiners are not necessary. the hypothesis that the rnn expected determiners to be at the beginning of english sequences as opposed to chinese ones is corroborated by the error + bigram span also yielding a higher f1-score than the padded error span in this setting. the error + bigram span is analogous to the error + unigram span as it begins with the pos tag extracted from the word that should be preceded by a determiner. to further investigate errors in model classification of negative language transfer, we charted the overall task performance and f1-scores for some specific subtypes of learner transfer (figure 1 ). this figure shows that although the error + unigram representation yielded the highest overall f1-score in the n-gram baseline setting, certain predefined sub-types of negative language transfer were better represented by the padded error span. this difference was seen when the learner had not used punctuation or determiners even though they were needed. in the missing punctuation case, the n-gram models with the padded error span outperformed all of the rnn results. this indicates that the padded error span and n-gram models were more capable of representing the distinction between chinese and english punctuation usage. in figure 1 : f1-scores for shallow syntactic language models and error spans combinations this experimental setting, the n-gram models were able to detect when a pos tag sequence was meant to contain a punctuation mark in english but that punctuation mark was not necessary in chinese. as for the missing determiner errors, the padded error span also outperformed the other error spans in the n-gram shallow syntactic language models setting. unlike the rnn, n-gram models do not have such a defined concept of sentence beginning. this distinction may help explain the padded error span's performance in the n-gram setting. the n-gram models' computation is based on how common a pos tag sequence is. in that case, the padded error span pos tags were able to indicate that while a determiner would follow and precede certain tags in english, it would not in chinese. the n-gram models correctly identified a chinese structural pattern in missing determiner errors' pos tag sequences because their pos tags were not followed or preceded by determiners. overall, the language modelling techniques used in our experiments were able to model the distinction between english and chinese structures when analysing pos tag sequences extracted from sentences in those languages. when applied to the edits extracted from chinese native speakers' english writing errors, the models achieved good precision in identifying errors related to negative language transfer. however, they were unable to detect a large portion of the negative language transfer related errors, i.e., low recall scores were observed. as a future step, it would be interesting to attempt to detect negative language transfer with more powerful language modelling techniques, such as transformer models. unlike rnn-based language models, transformer-based approaches do not rely solely on recurrent features to represent language structures @xcite . the selfattention mechanisms employed by these models could be beneficial in detecting language patterns in learner text that are more similar to the learners' l1s than to english. ## conclusion our experiments show that shallow syntactic language models can identify some negative language transfer related errors in english learner writing. while improvements can be made regarding the level of detail of the structural representation, our experiments indicate that it is possible to detect aspects of transferred language structures. negative language transfer detection methods, such as the ones we described, can support the generation of error feedback that connects learners' mistakes to their first languages. this type of information can help language learners become more aware of rule divergences between english and their l1s, and as a result increase their writing accuracy. in the future, we hope to apply the methods described above to other first languages and learner datasets, such as the lang-8 corpus of learner english @xcite .