Native-like Expression Identification by Contrasting Native and Proficient Second Language Speakers (2020)

Abstract: We propose a novel task of native-like expression identification by contrasting texts written by native speakers and those by proficient second language speakers. This task is highly challenging mainly because 1) the combinatorial nature of expressions prevents us from choosing candidate expressions a priori and 2) the distributions of the two types of texts overlap considerably. Our solution to the first problem is to combine a powerful neural network-based classifier of sentence-level nativeness with an explainability method that measures an approximate contribution of a given expression to the classifier’s prediction. To address the second problem, we introduce a special label neutral and reformulate the classification task as complementary-label learning. Our crowdsourcing-based evaluation and in-depth analysis suggest that our method successfully uncovers linguistically interesting usages distinctive of native speech.

## introduction we propose a novel task of native-like expression identification (nlei) by contrasting texts written by native and proficient second language (l2) speakers. our primary motivation lies in the observation that native english speakers often fail to be understood even by proficient l2 speakers @xcite . take the following sentence for example: could you give me a ballpark figure ? ballpark figure is a fairly common american english idiom meaning "an approximate figure or quantity". despite being a simple combination of two basic words, this expression is enigmatic to many l2 speakers, leading to communication breakdowns. we will refer to such expressions as native-like expressions. we believe that native speakers, no less than l2 speakers, would do well to adapt their speech in an international setting so as to maximize mutual comprehension, and that one effective strategy is to avoid native-like expressions of this sort. however, the hegemony of english in international communities has led to many monolingual english speakers lacking the notion of what it is like to speak a second language, making it particularly difficult for them to identify native-like expressions on their own. for this reason, a system that automatically detects native-like expressions would help native speakers improve their international outlook. nlei itself has other potential applications. it could prove useful as a tool for advanced language learners to find new, fluent expressions to acquire in any given text. it can also be used as a method of linguistic inquiry for examining the differences between first-and second-language acquisition. if we manage to draw cross-lingually valid generalizations from english data, they would have a significant impact. in many other languages, there has been an increasing recognition of the importance of effective communication with linguistic minorities, especially in emergency situations @xcite , but texts written by l2 speakers are not large enough to enable data-driven approaches. we train a neural network-based sentence classifier that gives a native score to a given input. after that, we feed a candidate expression, in addition to the input sentence, to the classifier to approximately divide the native score into the contribution made by the candidate expression and that made by the rest of the sentence. nlei poses two key technical challenges. first, native-like expressions are often context-sensitive, and more importantly, can consist of several words. in traditional word-based approaches, simple frequencybased statistical tests such as a chi-squared test @xcite can be used to contrast text written by native and proficient l2 speakers. such statistical tests are not applicable to our task because the combinatorial nature of native-like expressions prevents us from choosing candidate expressions a priori. second, the distributions of the two types of texts cannot totally be separated because, as illustrated in figure 1 (a), they do overlap considerably. this hampers discriminative approaches to the task that are known to be more powerful than simple statistical tests. to address the first challenge, we combine a neural network with an explainability method. we build a sentence-level classifier using a bilstm with subword embeddings as input, with the hope that it is expressive enough to implicitly exploit native-like expressions for classification. after narrowing down the targets to native-like sentences, we use a method named contextual decomposition @xcite to measure an approximate contribution of a given expression to the classifier's prediction (figure 1(b) ). we repeat this for multiple candidate expressions in a given sentence to choose the most appropriate one. we address the inseparability problem by introducing a special label neutral to indicate the overlapping region. the resulting three-way classifier can be trained under the framework of complementarylabel learning @xcite . by letting non-distinctive sentences be absorbed into neutral, the classifier is able to choose distinctively native-like sentences from sentences written by native speakers. due to the exploratory nature of the task, it is hard to evaluate the proposed method. nevertheless, we performed crowdsourcing for quantitative evaluation, which was followed by in-depth manual analysis of the detected expressions. we found that the scores given by the proposed method weakly correlated with aggregated ratings provided by l2 crowdworkers. remarkably, the proposed method often identified expressions that consisted of words so basic that the traditional word-based models would have deemed them easy for l2 speakers. ## native-like expression identification given texts written by native and proficient l2 speakers, the task of nlei is to find any possible nativelike expressions in them. we assume that each sentence is tied to a writer-attribute label, native writer or l2 writer . it is important to note that we assume no annotation of native-like expressions themselves. due to the exploratory nature of the task, it is difficult to give a precise definition to native-like expressions. any word or sequence of words is considered native-like if it is more commonly used by native speakers compared to l2 speakers. 2 an additional condition is that native-like expressions must not include domain-specific words or named entities. since topics that native speakers write about are often different from l2 speakers, domain-specific expressions such as baseball, and electoral wipeout, as well as country names and so on, could help reliably distinguish the two groups. however, since they do not constitute distinctive usages that we are aiming to identify, we do not treat them as native-like expressions. there are no clear-cut criteria for determining the boundary of a native-like expression. from the sentence, what on earth are you yammering on about? ## proposed method we build a neural network-based classifier that receives a sentence as the input and returns a threedimensional vector representing the labels native, neutral, and l2 (section 4.1). it is trained under the framework of complementary-label learning (section 4.2). the classifier is then combined with an explainability method named contextual decomposition to locate a native-like expression in a given native-like sentence (section 4.3). ## conclusion in this paper, we proposed a novel task of native-like expression identification. we showed that even if no expression-level annotation is available, a powerful neural network combined with an explainability method can detect native-like expressions by contrasting sentences written by native and proficient l2 speakers. our analysis on expressions detected by the proposed method suggests that even basic words may give some pause to l2 speakers if they are arranged in certain ways. although the proposed method in its current form is sufficiently useful for mining native-like expressions, further work is needed to mitigate the effect of domain-specific expressions. an interesting future direction is to create a user interface through which the system gets human feedback. such direct signals are expected to make the detected expressions more closely match human intuitions @xcite . figure 3 : the task screen for a native crowdworker. than 0.5. 2) we created another set of 10,000 sentence-expression pairs for which the top cd score was not necessarily greater than 0.5 (i.e., we selected random expressions with random scores). to make the data more appropriate for human evaluation, we removed the sentence-expression pairs in the following cases: next, we asked native crowdworkers to check the sentences. we hired native crowdworkers on amazon mechanical turk. workers residing in the above-listed six countries were treated as native speakers of english. the task screen for native crowdworkers is shown in figure 3 . the instruction at the top was the same as that given to l2 workers, but this time we provided six options, roughly ranging from least natural to most natural. as expected, a large majority of the expressions were familiar and natural to the native crowdworkers, as both the readers and the writers were native speakers. the workers chose the most-natural option in four out of five cases (3968 out of 5000 answers total). this indicates that most of the sentences and expressions in the selected subset were very familiar, natural-sounding, and easy to understand for native speakers of english. some of the sentences did get bad scores, however. because we are not interested in sentences that even native speakers cannot make sense of, we removed these from the subsequent evaluation. specifically, we removed 26 sentences for which at least two workers out of five gave the first three options. we also removed several sentences for which less than five answers were given (due to issues with the crowdsourcing interface). as a result, 958 sentences remained. as described in section 5.3, we removed sentences for which none of the words in the target expression matched svl12000. this reduced the number of sentences to 890. applying the cd score threshold of 0.7, we chose 287 sentences as the final dataset.