Understanding Native Language Identification for B razilian Indigenous Languages (2023)

Abstract: We investigate native language identification (LangID) for Brazilian Indigenous Languages (BILs), using the Bible as training data. Our research extends from previous work, by presenting two analyses on the generalization of Bible-based LangID in non-biblical data. First, with newly collected non-biblical datasets, we show that such a LangID can still provide quite reasonable accuracy in languages for which there are more established writing standards, such as Guarani Mbya and Kaigang, but there can be a quite drastic drop in accuracy depending on the language. Then, we applied the LangID on a large set of texts, about 13M sentences from the Portuguese Wikipedia, towards understanding the difficulty factors may come out of such task in practice. The main outcome is that the lack of handling other American indigenous languages can affect considerably the precision for BILs, suggesting the need of a joint effort with related languages from the Americas.

## introduction brazil is home to about 270 indigenous languages, referred to as brazilian indigenous languages (bils) hereafter. all of those language are endangered, spoken by at most 30 thousand people, and are quite understudied. serious effort should be put onto creating resources and tools to help vitalize the culture of such underrepresented communities. the creation of ai tools, in special language models and applications such as language translators, next-word predictors, spell checkers, can be key for this endeavour, since all could be used as learningaid tools. one main issue in building ai tools for understudied languages, which is the case of bils, is the lack of data. there is almost no data available in ready-to-use formats, such as parallel corpora and labelled datasets, even monolingual data is scarce. finding data for such languages is very difficult, since documents are stored in varied repositories and there is no indexing in search engines for such languages. native language identification (langid) represent of a crucial approach to help in the task of gathering and augmenting data for bils and many other indigenous languages. not only langid can be helpful to mine data from the web, it can be used as a tool to validate data that is generated synthetically with back-translation or self-training @xcite . before putting a langid system into practice, though, it is very important to have a clear understanding of its capabilities, such as the expected accuracy on unseen domains. apart from an evaluation of langid with indigenous languages in isolation @xcite , or the addition of some language in a publiclyavailable langid dataset @xcite , in both cases with only bible data, the potential of langid for bils in non-biblical, open-world data is quite understudied. that, again, owns to the lack of data, since the only source of data available to build a langid for bils is the bible. and that is quite limiting in terms of understanding of the usefulness of a langid for bils in varied domains. in this work we focus on expanding the horizons on a langid for bils, and present a deeper investigation on the quality and practical issues of bible data for langid on such languages. for that, we collected and appended 1.5m sentences from 51 bils to the existing wili2018 langid dataset, with 235 languages @xcite , to train a machine learning-based langid approach, and test it on different scenarios. we focused on answering two main research questions: rq1) what is the level of accuracy achieved by this bible-based langid on a sample of out-of-domain, non-biblical data sets? and rq2) if we apply this langid on a large set of texts in the wild, what are the main difficulty factors? for langid, we implemented an approach considering bag-of-words on tokens computed with sentencepiece, and support vector machines (svms) as the classifier. results show an accuracy of 73.3% and 95.1% for bils and non-bil languages, respectively. to answer rq1, we built a dataset with almost three thousand sentences, comprising seven monolingual dataset in six different bils. the results indicate that our langid classifier generalizes quite well to most languages, reaching up to about 90% accuracy. but we see also that there might be a drop to about 37% with apurinã, for which writing standards are not quite well established. to answer rq2, we applied our langid on about 13.5m sentences extracted from the portuguese wikipedia. as much as 3,821 sentences were pointed out with a bil as the most probable class, but most of them with very low probability scores, below 0.1. a further manual inspection showed a precision of 7% only, uncovering a fundamental issue that needs to be overcome in the future to improve the prediction of langid in in-the-wild data, which the need to handle other american languages to reduce false positive hits. ## a langid dataset for bils we built a dataset containing 51 bils, with data extracted from the bible. although this covers only a sample of the total of about 270 existing bils, according to the last comprehensive assessment of linguistic diversity in brazil (ibge, 2010) 1 , this set represents about a third of the estimate of 90 languages that have established standards of writing @xcite . additionally, we expect that the results expand to languages that belong to same families and branches in which the bils are organized @xcite . besides the languages spoken solely in brazil, we include languages that are mostly spoken outside of brazil but with some speakers in the country, such as the version of guarani spoken in paraguay, and languages that are relatives to some bils, such as the eastern and western versions of guarani spoken in bolivia. for data splitting, the test set was composed of all sentences from the matthews new testament book, for which we tokenized all chapters with the nltk sentence tokenizer. then we perform the same procedure to create the training set, 1 there is some discussion about the accuracy of those numbers, see @xcite . with all remaining books from the new testament, and books from the old testament, when available. as a result, the total number of training samples is 1,330,457 samples, and 199,128 test examples. the average number of samples per language is of 26,087 for the training set, and 3,904 in the test set. additional details are presented in appendix a. ## the langid classifier for bils we developed a langid system using a linear svm classifier with bag-of-words (bow) features, relying on the sentencepiece tokenizer foot_0 , with 100k tokens. note that we have evaluated different configurations for vocabulary size and other classifiers, but found that the linear svm with 100k tokens presented the highest mean accuracy in the two test sets available, i.e. one for the bils and another from wili-2018. detailed results are provided in appendix b. as the training set, we considered the concatenation of our bible-based dataset for bils and the wili-2018 dataset, which contains 235k samples, evenly distributed over the 235 languages in the dataset. the accuracy on those sets are, respectively, 73.3% and 95.1%. notice that our langid approach excels pretty well on the wili2018 test set, almost 5 percentage points better than the 89.42% accuracy reported in @xcite . but the accuracy presented on the bils test set is 22 percentage points lower, which we believe is related to the inherited difficulties of doing langid for such languages. ## accuracy on non-biblical datasets in order to validate the quality of the langid system proposed in this work, and to answer rq1, we performed an evaluation on non-biblical data. we built seven new datasets, comprising six different bils, to measure the accuracy of langid on domains that are quite unrelated to the training set. furthermore, this analysis also helps understand if the orthography of the training samples match what is expected in unseen domains. this data has been collected either from pdf files, available in repositories in the web, or from annotation efforts such as the universal dependencies parsing (udp). for the former, the task basically consisted of cleaning up any annotation and generating a file only with the sentences in the corresponding bil. but for the pdfs, we had to either copy and paste the contents in the pdf to text files, or even retype the content given the lack of standard in encoding for such languages and the lack of standard for pdfs files. in both cases, tough, manual inspection of the conversion results proved necessary to handle special characters such as some combinations of letters and accents that are not very usual in non-indigenous languages. once the blocks of texts have been inspected and converted to a text file, we then applied a sentence tokenizer to split paragraphs into individual sentences. finally, we filtered all sentences with less than three tokens, to avoid dealing with such very short sentences. in the accuracy rates, also presented in an additional evaluation was then performed to understand if the classification of bils is affected by non-indigenous languages. for that, we checked which languages where misclassified the most with gun in the respective datasets for this language. in the books dataset, from the 162 misclassifications, 63 (39%) were associated to languages belonging to the tupi-guarani family, which is the same family of gun. from those 63 samples, 38 were detected as kgk, and 25 as gug. similarly, in the tales dataset, from the 118 errors, 78 (66%) were from tupi-guarani family languages: 48 in kgk and 30 in gug. thus, considering the high similarity of such languages from the tupi family, it is likely that the results with gun can be improved with further development of the langid classifier, in order to handle better the classification among these similar languages. ## bringing langid closer to practice aiming at answering rq2, we expanded the evaluation of the previous chapter to a large, unsupervised set. our goal was to understand the main challenges in a scenario that is closer to practical application, which is applying our langid on in-thewild data, to mine for sentences written in one of the 51 bils. for that, we considered about 13.5m sentences extracted from the portuguese wikipedia. although that data presents limitations, since most pages are supposedly written in portuguese and a totally open set such as common crawl represents better the real world, that is also an advantage since we can discard all sentences detected as portuguese and manually inspect only the remaining smaller set. and the associated wikipedia pages can be used as ground-truth for the results of the classifier. this evaluation considered an exact total of 13,573,101 sentences, from which our proposed langid was able to identify 3,821 sentences as one of the 51 bils considered in this work. that corresponds to 0.03% of total sentences in the dataset. we observed, though, the very low prediction score for such detected sentences, with a mean of around 0.03, and decided to discarded all sentences with a prediction score below 0.1, resulting in a set of only 129 sentences. that is a quite small set, but this number was somewhat expected given the data in portuguese. one the other hand, that allowed us to conduct a manual inspection on the results. we manually inspected all of the detected 129 sentences, marking all sentences that 'looked like being correctly classified'. that is, we inspected the 129 sentences and marked all sentences that were written in a latin scripts, but with words that did not belong to any of the non-indigenous languages known by the authors, such as portuguese, english, spanish, german, and french, to name a few. with that approach, we found a total of 50 sentences that could likely be from a bil. then, for each of those 50 sentences, we searched for the original wikipedia page of the sentence, by using its text as a query on google, and inspected the resulting pages. the results were, on one hand, disappointing, since very few sentences were correctly classified. but on the other hand, they were quite useful in understanding some particular difficulties of this task, and how to approach this problem better in the future. details are provided next. the results were disappointing in the sense that very few sentences were correctly classified, i.e. very low precision. from the 50 sentences that we suspected were correct, only 9 sentences were extracted from a wikipedia page that related to the actual predicted language. that gives a precision of only 7%. besides, we uncovered that those nine sentences consisted of samples of the lord's prayer, which is a content that is very close to what is in the training for such languages, so these results do not help in clarifying the potential of langid in non-religious content. nevertheless, some interesting findings of this study consist of a better understanding of the main difficulties that we may face when applying langid to mine data for bils. one clear drawback of our proposed approach, is the limited handling of similar low-resource languages, such as indigenous languages from other south and north american countries besides brazil. most of the classification mistakes involved wikipedia entries of languages spoken in countries such as peru, colombia, mexico, and the united states. some other few mistakes involved languages from more distant locations, such as indonesia and the african continent. these results show that, in order to perform accurate langid for bils, it is important to include as much languages as possible in the training set to have a more precise classification, or to implement some mechanism to deal with out-of-scope detection. this evaluation also showed that searching for webpages using sentences in a target language as a query for a search engine can be helpful to find for additional data, such as pdfs with additional content such as the one found for the amarakaeri language foot_1 . even though amarakaeri is not included in the set of bils, with more accuracy in langid, we could search for pdf documents is such languages with greater precision. furthermore, we found that misclassifications can be useful to find content in additional related languages, such as the language cocama foot_2 , which is spoken in brazil and belongs to the tupi family, but was not included our set of bils for langid. ## conclusions and future work in this paper we present an evaluation of langid for brazilian indigenous languages (bils), using the bible as the only source for training data. we demonstrate that on non-biblical, labeled datasets, the approach is able to achieve even accuracy in languages with more establised written forms, such as guarani mbya and kaigang, but the performance may drop considerable for less studied languages. by applying the langid classifier in an almost inthe-wild dataset, we saw that the precision is quite affected by related american indigenous languages that are not handled by our langid approach, so a joint effort must be made to handle as much american languages as possible, together, to improve the quality of the langid in practice. as future work, we believe that expanding the langid training set, to consider as much languages as possible, is mandatory. furthermore, an inspection of the orthography of some languages should also be done, by partnering with linguists and/or native speakers. and we think that we could further develop the study in in-the-wild data, either by searching for bil data on a more comprehensive dataset, such as common crawl foot_3 and brwac foot_4 , and by including the search for pdf documents, which is the most commonly used format containing data for such languages.