Evaluation of Off-the-Shelf Language Identification Tools on B ulgarian Social Media Posts (2022)

Abstract: Automatic Language Identification (LI) is a widely addressed task, but not all users (for example linguists) have the means or interest to develop their own tool or to train the existing ones with their own data. There are several off-the-shelf LI tools, but for some languages, it is unclear which tool is the best for specific types of text. This article presents a comparison of the performance of several off-the-shelf language identification tools on Bulgarian social media data. The LI tools are tested on a multilingual Twitter dataset (composed of 2966 tweets) and an existing Bulgarian Twitter dataset on the topic of fake content detection of 3350 tweets. The article presents the manual annotation procedure of the first dataset, a dis- cussion of the decisions of the two annotators, and the results from testing the 7 off-the-shelf LI tools on both datasets. Our findings show that the tool, which is the easiest for users with no programming skills, achieves the highest F1-Score on Bulgarian social media data, while other tools have very useful functionalities for Bulgarian social media texts.

## introduction automatic language identification (li) is a welladdressed task, with many existing approaches, tools, and evaluation initiatives @xcite . li solves the problem of those users, who need to detect the language of a large number of texts, and thus cannot perform this task manually, as it will take them a large amount of time and manual efforts. such users (for example linguists), do not have the knowledge, skills, or interest to develop their own li tool or to train existing tools with their own data, and thus prefer using an existing off-the-shelf li tool. as a first step, they are naturally interested to know which is the best tool for the specific language (e.g. bulgarian, romanian, hindi) and type(s) of text of their interest (e.g. news articles or social media posts). however, there is no sufficient information about which off-the-shelf li tools are the best for all specific language/type-of-text combinations. for this reason, we are sharing our findings of the best off-the-shelf li tools and their functionalities for the specific language and type of text of our interest. by doing this we aim to assist other users or researchers, who need to use such tools for their language identification tasks. our language of interest is bulgarian, and the the type of text -social media posts, and in this article we are reporting the results of comparing several off-the-shelf li tools on bulgarian social media data. our work is motivated by the wish to solve the issue of filtering out any non-bulgarian tweets from social media corpora. following our task to collect and pre-process bulgarian social media datasets for detecting fake content, our first observation was that despite using the twitter api for collecting only posts in bulgarian, our dataset contained many tweets (see to be able to identify the most appropriate li tool and motivate our choices, we have to first understand and describe the characteristics of the language (bulgarian) and type of text (social media posts) of our interest. the bulgarian language is part of the south slavic languages' group within the indo-european language family. in lexical, phonetic and grammati-cal terms, bulgarian has both slavic and non-slavic features. it is officially written in cyrillic alphabet, but in social media and internet forums people often use several variants of latin transcription. bulgarian is the official language of the republic of bulgaria. it has a literary form, used in all spheres of public life, and a number of local dialects, some of which are similar to the languages of north macedonia and serbia. social media texts (including those in bulgarian) are known for being different from standard texts by being much shorter (e.g. a tweet can contain a maximum of 280 characters), frequently containing orthographic errors, internet slang, non-dictionary words, emoticons, hashtags, unfinished sentences, and broken or non-standard syntax, and thus being challenging for many natural language processing applications @xcite . in addition to that, social media posts may sometimes contain words and phrases, written in different languages -a phenomenon, known as codeswitching @xcite . a li tool, which would be perfect for recognizing bulgarian social media posts, thus, should: an over 98% f1-score); 2. be able to recognize bulgarian texts, written both in cyrillic alphabet and in the various latin transcriptions (typical for the bulgarian internet slang); in order to discover the most appropriate li tool for correctly identifying the bulgarian language posts in social media data, we have determined the most frequently used off-the-shelf li tools (osli), by examining publications and consulting other researchers. we have then tested them on two datasets -a multilingual (mostly bulgarian) dataset, collected from twitter on the topic of covid-19 with 2979 tweets, manually annotated for language(s), and a bulgarian language dataset @xcite , used for fake content detection initiatives, consisting of 3350 tweets. the article provides the results of the human annotation and of testing the tools, as well as shows which tools achieve the highest f1-scores on the two datasets, and which have the most useful functionalities for social media posts. the rest of the article lists the relevant related work (section 2), a description of the datasets that we used for testing the tools (section 3), our methodology (including human annotation and the tested tools -in section 4), the annotation and testing results and some discussion (section 5), and finally, the conclusions (section 6). ## related work automatic language identification (li) is a widely addressed task, but it still has some issues which are hard to resolve. among them @xcite are: all these issues apply to bulgarian social media posts. there have been a number of previous works which include bulgarian among other languages in their li tasks or datasets, for example @xcite @xcite @xcite . most of them, however, use datasets compiled from types of texts, which are different from social media (e.g. wikipedia, news articles, europarl, and the universal declaration of human rights). also, most of these works do not apply existing off-the-shelf li tools to detect bulgarian, but rather implement their own methods. the closest works to ours are those of @xcite , @xcite , @xcite , and @xcite . among them, however, there is no work which compares the performance of different off-the-shelf li (osli) tools on bulgarian social media posts and publishes the results. specifically, @xcite are similar to us as they use short forum texts, including such written in bulgarian, but no testing of osli tools is performed. @xcite compare li methods implemented by them with three off-theshelf tools (textcat, google cld and langid.py) proceedings of clib 2022 on a multilingual twitter dataset containing also bulgarian. their methods outperform the osli tools, but there are no results reported separately for bulgarian. @xcite also observes that twitter's accuracy for bulgarian language identification is not satisfactory, however, the author does not test any osli tools on bulgarian tweets. finally, there are publications on testing various off-the-shelf li tools on specific languages, but not on bulgarian. for example, @xcite ) compared 8 osli tools on manually annotated tweets in english, chinese, and japanese. while several osli tools include bulgarian, according to our knowledge, there is no other published comparison of off-the-shelf li tools for this language, especially for social media texts. ## data used we have used two datasets -a randomly selected subset of our own twitter dataset, and the bulgarian language dataset, made available for the clef2021 checkthat! lab, task 1 (checkworthiness). from now on we refer to this dataset as clef2021 dataset) 1 @xcite . the large original version of our dataset contains 52810 tweets, from which we extracted 3124 tweets, which were annotated for their language by human annotators. we have removed some non immediately noticeable duplicates and did some additional cleaning (based on our annotators feedback), and obtained 2966 final human-annotated tweets, on which we tested the li tools. respectively, the clef2021 checkthat! lab, task 1 dataset for bulgarian originally contained 3350 entries (tweets). we have decided to compare the results of the same off-the-shelf li tools on the subset of our dataset with those on the clef2021 bulgarian dataset, as they both had comparable number of tweets and are on the same topic . before testing the li tools on the clef2021 checkthat! lab, task 1 bulgarian dataset, we have merged the bulgarian versions of its train and dev datasets into one to have more data. after a quick analysis of the merged clef2021 dataset, we noticed two issues: unusually long entries (consisting in many tweets concatenated in one row) and a few tweets in other languages. we separated the long rows into single posts and removed the the tweets in our original large dataset foot_0 were collected via the twitter api for the period may 2020 -march 2021. the keywords used were "ваксина" ("vaccine", sg.) and "ваксини" ("vaccines", pl.) in bulgarian language and using the cyrillic alphabet. from this large dataset we have selected a smaller random subset from different time intervals. each tweet from this final dataset (from now on referred to as covid-19 dataset) was manually annotated by two annotators for its language. the annotation methodology is described in detail in section 4.1. the first thing that we noticed during manual annotation, was that our dataset contained posts in other languages besides bulgarian. as it can be seen in ## methodology in order to test the existing off-the-shelf li tools, we have performed manual language annotation of the 3124-tweets-subset of our dataset (the one containing duplicates), which is described in subsection 4.1, and selected a number of freely available and functioning li tools (described in subsection 4.3). the methodology, which we followed for testing the tools on both datasets is described in subsection 4.2. ## conclusions in this article we have presented the results from comparing 7 well-known off-the-shelf language identification (li) tools on identifying bulgarian language posts in two twitter datasets, composed of around 3000 tweets each. we provided a presentation of each tool along with its useful functionalities and eventual shortcomings. we are confident that this information will be of use to any researchers, who would like to know the performance of off-the-shelf li tools on bulgarian social media posts, without training them. our results show that the tool which has the highest scores is the detectlanguage() functionality of google sheets. the second best is fasttext. we have found out that cld3 has also the functionality to recognize bulgarian, written with latin letters, which is useful for social media and internet forums texts. testing its performance for this task has still to be done. we have also discovered that polyglot and (partially) spacy can be used to guess multiple languages, present within the same text, but their performance in executing this task needs to be properly tested too. we haven't discovered any li tool, which simultaneously has a high accuracy/f1-score, can recognize bulgarian written with latin letters, and recognizes the languages in multi-lingual posts. this presents an opportunity for creating such a tool. as future work, we plan to evaluate in more detail the above mentioned functionalities of polyglot, spacy, and cld3, and also to implement our own proceedings of clib 2022 proceedings of clib 2022