Investigating Linguistic Features for A rabic NLI (2024)

Abstract: Native Language Identification (NLI) is concerned with predicting the native language of an author writing in a second language. We investigate NLI for Arabic, with a focus on the types of linguistic information given that Arabic is morphologically rich. We use the Arabic Learner Corpus (ALC) foro training and testing along with a linear SVM. We explore lexical, morpho-syntactic, and syntactic features. Results show that the best single type of information is character n-grams ranging from 2 to 6. Using this model, we achieve an accuracy of 61.84%, thus outperforming previous results (Ionesco, 2015) by 11.74% even though we use an additional 2 L1s. However, when using prefix and suffix sequences, we reach an accuracy of 53.95%, showing that an approximation of unlexicalized features still reaches solid results.

## introduction native language identification (nli) task is concerned with predicting the native language of texts written by learners of a second language (l2). nli relies on the assumption that speakers of the same native language display certain linguistic patterns in their l2 texts which can be used as traces in nli to predict their l1. work on nli has exploited various types of these linguistic features such as function words, character n-grams, pos n-grams, syntactic structure, and spelling mistakes @xcite . previous work (e.g., @xcite ; @xcite ) has argued for unlexicalized features as opposed to lexicalized features because they are less biased to the prompt and domain of the data. lexicalized features consist of word n-grams while unlexicalized features are contentindependent and non-lexical (such as pos tags or function words) and are thus less dependent on text vocabulary. lexicalized features are considered less desirable because they cause topic bias since they depend on the topic of the text @xcite and may thus not be useful for texts on different topics (e.g., different prompts in toefl). in our current work, we investigate this argument using the arabic learner corpus (alc), a rather small collection of texts written by learners of arabic. the contribution of this paper is to examine which types of features will be more accurate and informative for arabic nli: lexicalized, unlexicalized, morpho-syntactic, syntactic features, or their combinations. we also provide a comparison between our system and three other systems that have used the same corpus. to our knowledge, this is the first attempt in arabic nli that examines the performance of lexicalized features and compare them to the unlexicalized ones using alc data. we also propose a new feature set, consisting of prefix-suffix sequences, which is close to an unlexicalized feature set and can be useful in situations with domain drift. the rest of the paper is organized as follows: section 2 describes related work, section 3 describes the data, section 4 explains the methodology, section 5 shows the results and a discussion, and section 6 presents our conclusions and future work. ## related work in text analysis tasks such as nli, various types of linguistic features have been exploited. certain linguistic features such as words, lemmas, tokens, and characters n-grams have been shown to be effective (e.g., @xcite ; @xcite ). however, these types of linguistic features can be problematic since they are content/domain dependent. by using such features, topic bias can occur, especially when prompts or topics are not equally distributed across texts. this will cause a classifier to indirectly learn to recognize topics. therefore, other research in nli (e.g., @xcite ; wong and dras (2011); malmasi et al. (2015) ) reported the importance and usefulness of using features that are content independent to avoid these issues. @xcite , who introduced the first application of nli using arabic learner data, employed content independent features such as function words, pos n-grams, and context-free grammar production rules. they used an svm classifier and the second version of the arabic learner corpus (alc) @xcite . more specifically, they used a subset by choosing the top seven l1s represented in the corpus: chinese, english, french, fulani, malay, urdu, and yoruba. their results show that pos n-grams were the most useful feature (37.60%) as a single feature type. however, the highest accuracy (41% against the baseline of 23%) was achieved when all features were combined. these results show that those features were effective in discriminating l1 groups. similarly, @xcite introduced a deep learning approach using a gated recurrent unit network (gru) to identify the l1s of arabic learners. they utilized the same subset (top 7 l1s) of arabic learner corpus (alc) and the same feature sets used by @xcite . they employed standard deviation for feature selection. their classification accuracy reached 45%, which outperformed the system by @xcite . ionescu (2015) proposed a string kernel system based on local rank distance metric (lrd), which was applied on the character level. the author used the same subset of arabic learner corpus used by @xcite in order to compare results. since lrd works on character level, ionescu (2015) did not perform any pre-processing except for normalization (removing any tabs, spaces, and new lines). a kernel based on lrd was implemented by using the kernel ridge regression (krr) classifier, using a combination of 3-5 character n-grams as features. comparing the results of this study with the results by @xcite showed that this system achieved an accuracy of 50.1%, which was 10% higher than the results by malmasi and dras (2014a) (41%). as described above, some nli studies argued for unlexicalized or syntactic features since they are content independent. almost all work done on arabic nli focused only on syntactic or morphosyntactic features while ignoring lexical features. for this reason, we investigate a wide range of lexicalized and unlexicalized features, to determine the preprocessing for preprocessing, we cleaned the transcribed spoken data for consistency with the written data. punctuation, such as double dash (marking incomplete sentences), is removed; other punctuation, such as period, comma, and question marks, is preserved. filled pauses and hesitations, marked by and , are removed. furthermore, partial words transcribed with a single dash are also deleted. additionally, mispronounced words transcribed using a preceding plus sign, are removed. the original corpus is anonymized, all personal information such as names, email addresses were replaced by an arabic phrase. we replaced this phrase by the english term 'unknown' to indicate the anonymization without overly interfering with the n-grams. in addition, any non-arabic words are manually removed during preprocessing as long as removing them will not affect the structure of the sentence. ## methodology machine learner we use linear support vector machines (linear svm), in the implementation of scikit-learn @xcite . we select one-versus-rest multi-class classification, and we also use mutual information (mi) for feature selection. feature weights consisting of tf-idf weights are employed; other parameter tuning is performed using a 10-fold cross validation setting. results are reported under stratified 10-fold cross validation. ## conclusion and future work in our work, we explored different types of feature sets: lexical, morpho-syntactic, and syntactic features in order to determine the best feature type given that arabic is a morphologically rich language and that the corpus is small in comparison to english nli corpora. our results indicate that lexical features are more informative and predictive, even for a morphologically rich language, compared to unlexicalized features. the best single feature type is character n-grams ranging from 2 to 6 (61.84%). combining lexicalized and unlexicalized features does not result in an improvement of results. however, using prefix and suffix n-grams shows promising results in case we need to address topic shift. while these results do not reach those of the lexicalized version using prefix-stem-suffix n-grams, they are close, thus showing that the stem does not provide much signal for nli. for future work, we are planning to investigate those linguistic features using a large scale data to determine to what degree our results are due to the small corpus size. we also need to broaden the spectrum and look at other l2s that are morpho-logically rich in order to see if similar regularities hold.