DETERMINING THE ETHNO-NATIONALITY OF WRITERS USING WRITTEN ENGLISH TEXT
Anonymous authors
Paper under double-blind review
A BSTRACT
Ethno-nationality is where nations are deﬁned by a shared heritage, for instance it can be a membership of a common language, nationality, religion or an ethnic ancestry.
The main goal of this research is to determine a person’s country-oforigin using English text written in less controlled environments, employing Machine Learning (ML) and Natural Language Processing (NLP) techniques.
The current literature mainly focuses on determining the native language of English writers and a minimal number of researches have been conducted in determining the country-of-origin of English writers.
Further, most experiments in the literature are mainly based on the TOEFL, ICLE datasets which were collected in more controlled environments (i.e., standard exam answers).
Hence, most of the writers try to follow some guidelines and patterns of writing.
Subsequently, the creativity, freedom of writing and the insights of writers could be hidden.
Thus, we believe it hides the real nativism of the writers.
Further, those corpora are not freely available as it involves a high cost of licenses.
Thus, the main data corpus used for this research was the International Corpus of English (ICE corpus).
Up to this point, none of the researchers have utilised the ICE corpus for the purpose of determining the writers’ countryof-origin, even though there is a true potential.
For this research, an overall accuracy of 0.7636 for the ﬂat classiﬁcation (for all ten countries) and accuracy of 0.6224 ∼ 1.000 for sub-categories were received.
In addition, the best ML model obtained for the ﬂat classiﬁcation strategy is linear
SVM with SGD optimizer trained with word (1,1) uni-gram model.
I NTRODUCTION
Ethno-nationality can be determined as the identity in which jointly deﬁned by ethnicity and nationality (Sapp, 2012).
Identifying a writer’s country-of-origin is a part of identifying his ethnonationality (Jerry, 2008).
Foremost focus of ethno-nationality in this research is to identify a writer’s country-of-origin based on their writings in a non-native or second language.
In recent years, identiﬁcation of ethno-nationality of a writer has gained a growing interest.
In author proﬁling demographic features (such as age, gender, education, native language, country of origin, etc.,) of an author from a written text will be identiﬁed, which are commonly needed in forensic linguistics.
For instance, Intelligence to build a proﬁle of their suspect, to identify the author of an anonymous email threat.
Therefore, this will enable to limit the search space as well (Estival et al., 2007).
Moreover, for business applications this can be useful; for example, in marketing where the demographic features as stated above of customers is important to predict behaviors, upgrade the current products and to develop new products.
People from different ethno-nationality make various language errors when learning a language.
Identiﬁcation of writer’s country of origin could have an impact on educational applications designed towards non-native speakers of a language.
Besides, this can be used as a plug-in to online tutor systems to provide more tailored feedback to the students about their mistakes (Tetreault et al., 2013).
This can help researchers to identify speciﬁc teaching and learning issues in different ethnonationalities.
This will enable them to develop pedagogical learning materials to address and solve those issues.
In this research we focus on English as the second language.
As English has become a ‘universal communication’ language due to the globalization.
It is no longer restricted to the native countries such as England and United States (Eric, 2013).
English language is now available in numerous emerging ﬁelds, and has become an essential requirement of labour market and further it is considered to have a cultural importance (Marko, 2009).
We deﬁne, A controlled environment as a place or area where rules, regulations and norms are subject to strict enforcement.
For instance, in an examination setting, candidates are required to produce answers limiting the scope of the question.
R ELATED WORK
Distribution of English language around the world can be identiﬁed in two ways, mainly; based on the geographic distribution and based on the Kachru’s theory (Braj B, 2004).
In this study when segmenting according to the geographic distribution we have considered clusters namely; ‘Asia’, ‘non-Asia’, ‘South Asia’, ‘non-South Asia’ and ‘North America’.
According to the Kachru’s ‘Concentric Circles’, English speakers were segmented in to three main categories, namely; ‘inner circle’, ‘outer circle’ and ‘expanded circle’.
As stated by his model, the inner circle comprises of the countries where the English is the native language (i.e., UK, USA, Canada, New Zealand, etc.).
The outer circle includes countries where there are small communities of native English speakers however English is used as the second language in education and ofﬁcial purposes (i.e., Sri Lanka, India, Singapore, Nigeria, etc. ).
The expanding circle contains countries which considered English as foreign language (i.e., China, Indonesia,
Japan, Saudi Arabia, etc.).
The work of Koppel et al.
(2005) is one of the ﬁrst ethno-nationality identiﬁcation work where it involves categorizing users based on their native language employing Support Vector Machines (SVMs) on various stylistic features focused on identiﬁcation of common.
Authors achieved 0.
accuracy using ﬁve chosen languages (i.e., Czech, French, Bulgarian, Russian and Spanish) from
International Corpus of Learner English (ICLE)(Granger, 2014).
The ICLE dataset contains argumentative essays writings of the university students and less nativism and creativity of the writer involved.
Bykh & Meurers (2012) proposed use of recurring n-grams on three different classes (word based,
POS based, Open-Class-POS-based) as features for training SVMs.
Out of other SVM implementations LIBLINEAR produced best results.
For this study, random data from seven native languages selected from the ICLE corpus.
The highest performance was obtained for word-level n-grams with an accuracy of 0.8971.
Gebre et al.
(2013) employs linear SVM, logistic regression and perceptron (as baseline) for the native language identiﬁcation (NLI) and achieved accuracy of 0.814 for eleven languages of ‘The
Test of English as a Foreign Language’ (TOEFL) data set(Blanchard et al., 2013).
Features used includes; word n-grams, POS n-grams, character n-grams and spelling errors.
TOEFL11 has become standard benchmark in NLI tasks since its introduction for the NLI Shared Task 2013.
The main limitation of the TOEFL11 dataset is that it is collected in a more controlled environment (i.e., exam for English).
Cimino & Dell’Orletta (2017) utilizes a novel stacked classiﬁer approach where linear logistic regression based sentence feature classiﬁer is stacked with a SVM based document feature classiﬁer with standard lexical, stylistic and syntactic features.
However stacked classiﬁcation approach has gained a minor gain compared to unstacked.
Best results of the NLI shared task 2017 reported for this approach with F1-score of 0.8818 for the TOEFL11 dataset.
Kulmizev et al.
(2017) introduced ‘Groningen’ system for the NLI Shared Task 2017 which out performs employing linear SVM for character 1-9 n-grams with the F1-score of 0.8756.
Authors have reported that several experiments done with ensemble approach and other features such POS, word, lemma n-grams, skip-grams; and those failed to match the performance of character 1-9 ngram system.
Goutte & L´eger (2017) explored use of voting ensemble SVM models with character, word and POS n-grams.
Authors conﬁrms that ensemble methods provide minor but systematic predictive perfor-
Research
[Table]
Koppel et al.
ICLE
[Table]
Support Vector Machines (SVMs) on various stylistic features namely; function words, letter n-grams, and errors and idiosyncrasies
Bykh &
Meurers (2012)
ICLE 0.
SVMs with recurring n-grams of three different classes (word based, POS based,
Open-Class-POS-based) as features
Gebre et al.
TOEFL 0.
Linear SVM, logistic regressions and perceptron as the linear classiﬁers with word n-grams, POS n-grams, character n-grams and spelling errors
Cimino &
Dell’Orletta (2017)
TOEFL 0.
Two-stacked sentence- and documentfeature based classiﬁer architecture.
Output of the sentence-level linear regression model is being used by a document-level
SVM.
Kulmizev et al.
TOEFL 0.
Linear SVM and character 1-9 n-grams
Goutte &
L´eger (2017)
TOEFL 0.
Voting
ensemble SVM models
approach with character, word and POS n-grams features
Markov et al.
TOEFL and ICLE 0.
(TOEFL11) & 0.
(ICLE) (TOEFL11) & (ICLE)
SVM with one-vs-all (OvA) multiclass approach.
Abstract POS n-gram and punctuation marks (PM) features have been used.
Malmasi &
Dras (2018)
TOEFL11,
EFCAM
[Table]
mance gains.
Highest F1-score performance was with best-vote approach consisting 10 models is 0.8740 for TOEFL11 dataset.
Markov et al.
(2018) has used punctuation-based features with POS n-grams for his experiments and accuracies of 0.4883 (TOEFL11) and 0.6948 (ICLE) reported for best performing settings.
SVM with one-vs-all (OvA) multi-class classiﬁcation approach has been used to conduct these experiments.
Malmasi & Dras (2018) employed a supervised multi-class classiﬁcation approach and incorporated several corpora, including; TOEFL11, EF Cambridge Open Language Database Corpus (EFCAMDAT), ASK Corpus (Andrespr˙akskorpus, Second Language Corpus), and Jinan Chinese Learner
Corpus.
The features extracted were; word/lemma n-grams, character n-grams, function word ngrams, POS n-grams, dependencies, CFG rules, adaptor Grammars and TSG fragments.
Highest accuracy of 0.871 is reported for the TOEFL11 dataset.
According to the literature it is evident that most of the researchers in the computational linguistic community have employed ICLE in the early stages and TOEFL in the latest NLI tasks (refer Table 1) which both were collected in more controlled environments.
Finally, due to above controlled environments; being unable to capture real nativism in written texts certainly inﬂate the performance of ethno-nationality identiﬁcation.
On the other hand, identifying the nation of the English writer opposed to his native language will be equally beneﬁcial for author proﬁling as well.
Thus, we introduce use of International Corpus of English (ICE) for the ethno-nationality identiﬁcation.
To the best of our knowledge, none of the researches have been conducted to determine the writer’s country of origin based on the International Corpus of English (ICE) corpus.
Moreover, very limited number of researchers have identiﬁed the signiﬁcant features which helps to distinguish Sri Lankan
English writers using a large corpus like ICE.
One limitation of these researches raised by the authors
Tofıghı et al.
(2012) is that, since for most of the web-based applications, automatic spell-checker has been applied, idiosyncratic features including misspelling and other anomalies are ignored.
This may hide some of the features which will be useful in identifying the real categories.
Nevertheless, in current context, usage of spell checkers and grammar checkers can be seen frequently.
Thus, this will be a common limitation in similar types of researches.
R ESEARCH M ETHODOLOGY
Supervised learning approach was chosen as it is suitable in classifying text documents into classes more accurately if the classes are known and the data set is labelled (Slotte, 2018).
Hence, according to the literature most promising ML algorithms for the text classiﬁcation such as; Support
Vector Machines with Stochastic Gradient Descent Optimizer (linear SVM with SGD), Multinomial Na ¨ ıve Bayes (MNB), Decision Tree (DT) and Random Forest (an ensemble approach) have been employed.
For this research mostly Scikit-learn(Pedregosa et al., 2012) has been used for preprocessing, feature extraction and classiﬁcation tasks.
Scikit-learn is a Python module for machine learning which provides state-of-the-art implementations of many well-known machine learning algorithms, and maintains an easy-to-use interface(Pedregosa et al., 2012).
The workﬂow of the research is depicted in Figure 1 (Left).
The main assumptions of this research are authors from the same ethno-nationality share the same linguistic features in their writing and will often have an inﬂuence on the way they express themselves in writings (Jain et al., 2017).
The research questions addressed in this research are
; Q1.
How can texts produced by English writers in a given ethno-nationality be captured from existing corpora?
Q3.
Which machine learning techniques can gainfully employ the extracted data to identify countryof-origin of English writers?
3. D ATA C OLLECTION
The main dataset employed in this study is International Corpus of English (ICE) (Greenbaum &
Nelson, 1996; Kirk & Nelson, 2018) which comprises of several corpora from different countries.
This data-set was built with the intention of providing a resource to conduct comparative studies of
English used in different countries where English is the native language or the second language.
In order to maintain consistency among each country corpus, common corpus design, corpus size and a common scheme for grammatical annotation have been followed (Kirk & Nelson, 2018).
This dataset consists of both written and spoken English texts.
Only written texts have been considered for this study.
Written texts were gathered from many areas, such as; student writings, letters, academic writing, news reports, instructional writing, persuasive writing and creative writing (Kirk & Nelson, 2018).
The authors and speakers of the texts are aged 18 or above, educated through the medium of English, and were either born in the country in whose corpus they are included or spent the majority of their lives there, or moved there at an early age, and received their education through the medium of English in the country concerned.
Written English corpora from Sri Lanka, India, Philippines, Singapore, Canada, Hong Kong, Nigeria, Ireland, Jamaica and USA have been collected and used for this research.
Each country is consisted of ≈ 200 text documents ( ≈ 2,000 words per document) (refer Figure 1).
3. D ATA A NALYSIS
As the initial step, 30% (testing data) of the total data of each country corpus was kept as unseen or un-touched data.
The remaining 70% of data (training data) was analyzed in order to identify the pre-processing requisites.
3. D ATA P RE - PROCESSING
Raw text-ﬁles with speciﬁc markups are used as input.
These ﬁles are cleaned as per the ICE-Corpus markup guide Nelson (2002).
In addition, ‘strip accents’ parameter for the TF-IDF vectorizer and ‘Unicode data normalize NFKD’ were used.
Moreover, URLs, XML tags, e-mail addresses, censored data, line feeds were removed.
The data set contained HTML entity encodings and those were decoded(refer Figure 1).
In order to avoid country names, nationality, currency and popular cities being trained as features, all the country speciﬁc nationality, cities and country names are removed as a pre-processing step.
3. C LASSIFICATION M ODEL B UILDING
The training data set were used for model building, parameter tuning, training and selection.
This labelled training data was tokenized using the TF-IDF vectorizer.
Vectorized output was fed to
ML classiﬁer to train the model.
Further, k-fold (k=3) cross validation technique was used on the training data.
Testing data set was kept hold to feed and evaluate the classiﬁcation model in later stages.
Models were built based on two classiﬁcation strategies;
• Flat Classiﬁcation Strategy
Flat classiﬁcation refers to a single classiﬁer at the root level as the decision point as depicted in
Figure 2(Top).
This classiﬁer will handle the all classes as per its classiﬁcation approaches such as one-vs-rest, one-vs-one...etc.
In this study ‘one-vs-all’ approach has been followed.
Hence, the dataset was trained for 10 classes (countries) together.
• Sub-Category Classiﬁcation Strategy
Sub-category classiﬁcation employs N number of classiﬁers as depicted in Figure 2(Bottom Left) and Figure 2(Bottom Right).
For each sub-category specialized in solving a subset of the problem in which each classiﬁer is trained.
Each sub-category was trained separately.
As depicted, some are binary (marked with black boxes in Figure 2) and some are multi-class classiﬁcation models (marked with grey boxes in Figure 2).
All the experiments of the sub-categories which were tested are depicted in Figure 3.
T ESTING AND E VALUATION
A combination of both ‘hold-out validation’ and k-fold cross-validation was used in order to reduce biasness for training and testing data set (Slotte, 2018).
Hence, on the 70% of the training data set, 3-fold cross validation technique was used for model building, training and selection and 30% of the data was kept for hold-out validation to validate the machine learning model.
For each decision point (where a single classiﬁer needs to be selected) 16 experiments (4 ML x 4 n-grams) have been carried out.
For each ML algorithm, word level (1-1,2,3,4) n-grams were tested.
Further, performance accuracy, F1-score, precision and recall were calculated to select the best performing ML model.
R ESULTS AND D ISCUSSION
Q1:
How can texts produced by English writers in a given ethno-nationality be captured from existing corpora?
• Flat classiﬁcation strategy
The best ML model obtained for the ﬂat classiﬁcation strategy is linear SVM with SGD optimizer trained with word (1,1) uni-gram model.
Furthermore, overall balanced accuracy of 0.7620 and macro average F1-Score of 0.76 was obtained.
Score breakdown for each country is depicted in Figure 4(Left).
In addition, India has the highest F1-score of 0.94 while USA & Sri Lanka holding the lowest F1-score of 0.68.
These statistics are further veriﬁed by the confusion matrix depicted in Figure 4(Right).
Accordingly, all the documents of the India have been classiﬁed correctly.
The confusion matrix further depicts that the model makes most of the mistakes at the classiﬁcation of USA vs Jamaica pair.
Certainly, one reason could be the geo-graphical proximity between these two countries.
In sub-category classiﬁcation, each sub-category can be perceived as a Decision Point (a single classiﬁer) in the hierarchy of a decision tree.
Figure 5 (Left) depicts the test results of the selected ML models for each sub-category.
Accuracies of different sub-categories have varied in between 0.6224 ∼ 1.000 and F1-score ranges between 0.49 ∼ 1.00.
Based on the Kachru’s concentric theory, lowest F1-Score of 0.71 is for “Outer vs Expanded” sub-category.
On the other hand, ‘Expanded‘ sub-category (Hong Kong vs Jamaica) has the highest F1-score of 0.92.
In geo-graphical distribution-based model, lowest F1-score of 0.49 is for “Non-Asia” sub-category, while ‘South
Asian countries‘ sub-category holding the highest F1-score of 1.0.
Assessing the generalizability of the ML models is crucial.
Over-ﬁtting leads to poor generalizability of the model.
Hence, in order to detect and prevent over-ﬁtting, ‘hold-out validation’ and k-fold cross-validation used.
Further, learning curves on the training data set have been drawn to diagnose whether
for varying number of samples. Cross validated score curve is
ﬂattening against the training score curve, as depicted in Figure 5(Right
Thus , this model does not over ﬁtting or under ﬁtting for the dataset
• Most signiﬁcant features of the model
Most signiﬁcant features identiﬁed for the ﬂat classiﬁcation is depicted in Figure 6.
These features are computed base
on
the coefﬁcients assigned.
Some of the features are driven by the cultural and geo-graphical differences of the particular country.
For instance, Ireland: northern, christmas, groundwater, queen. and Singapore: business, batik, population. Nigeria: donut(for do not), federal, god..etc.
Q2: Which machine learning techniques can gainfully employ the extracted data to identify country-of-origin of English writers?
Comparison of ML algorithms based on the selected best parameters are analysed on top of the test data and results are depicted in Figure 7. Linear SVM with SGD optimizer seems to be outperforms in most cases.
Moreover, this behavior is also veriﬁed in the previous work by Kulmizev et al.
(2017), Koppel et al.
Ekaterina (2011), Gebre et al. (2013), Bykh & Meurers (2012).
Furthermore, it is noticeable that the DT is under-performing for most cases.
C ONCLUSION AND F UTURE W ORK
With various accuracy levels the literature has proven that the ethno-nationality of a person can be identiﬁed using their written English texts and this area of research has lot of practical applications and usage.
However, as discussed in the literature review still those researches are comprised with lot of limitations as stated above.
Therefore, those identiﬁed limitations have re-framed this research to obtain solutions to the identiﬁed research problem and deﬁned research questions.
Our work on Ethno-nationality Identiﬁcation conﬁrms that linear SVM with SGD optimizer trained with word n-grams can yield a higher level of performance.
When determining country-of-origin it is essential to identify set of features which are unique to each author or a particular group of authors.
One spectacular restraint would be the usage of spelling and grammar checkers when writing English.
This limitation can be overwritten when using spoken English text.
Hence, as a future work transcribed text can be considered to identify the country-of-origin of the English writers.
Further, needs to focus more on features which have more pedagogical value and cross corpus generalizability should be assessed to examine the extendibility of the model. A CKNOWLEDGMENT
We are deeply grateful for the International English of Corpus (ICE) project owners and maintainers of the respective countries for making available this corpus to the research community without any license fee.
E THICS S TATEMENT
The Contributors of the respective ICE Corpus countries have informed about the data collection procedures and objectives.
All the identiﬁable named entities of the data have been anonymized in order to disjoint any relation of individuals and organizations to the data.
Hence, no one, including the researchers, will be able to link data to a speciﬁc individual.
Outcomes of these type of researches on categorization of individuals based on ethno-nationality can raise concerns over its usage and discrimination.
However, author-proﬁling is gaining pace and the authors of this paper admire the true potential of such categorizations.
R EPRODUCIBILITY S TATEMENT
The implementation details this experiment is available as a supplementary material along with the submission. Kindly note that the supplementary submission does not include the complete dataset. Authors can produce the complete data-set used upon a formal request. All the raw-data for each country in ICE corpora is available as text ﬁles.
Files are pre-processed through a jupyter notebook and results were written into a csv called ‘ice-merged.csv‘.
These preprocessing details are also available with the supplementary materials and for further details please refer to README.txt ﬁle in the root level.