Publish or Hold? Automatic Comment Moderation in L uxembourgish News Articles (2023)

Abstract: Recently, the internet has emerged as the primary platform for accessing news. In the majority of these news platforms, the users now have the ability to post comments on news articles and engage in discussions on various social media. While these features promote healthy conversations among users, they also serve as a breeding ground for spreading fake news, toxic discussions and hate speech. Moderating or removing such content is paramount to avoid unwanted consequences for the readers. How- ever, apart from a few notable exceptions, most research on automatic moderation of news article comments has dealt with English and other high resource languages. This leaves under-represented or low-resource languages at a loss. Addressing this gap, we perform the first large-scale qualitative analysis of more than one million Luxembourgish comments posted over the course of 14 years. We evaluate the performance of state-of-the-art transformer models in Luxembourgish news article comment moderation. Furthermore, we analyse how the language of Luxembourgish news article comments has changed over time. We observe that machine learning models trained on old comments do not perform well on recent data. The findings in this work will be beneficial in building news comment moderation systems for many low-resource languages

## introduction in recent years, the internet has revolutionised how individuals access and consume news. with the popularity of smart devices such as phones and tablets, the internet has emerged as the primary medium for acquiring news and information @xcite . people often share news articles on social media using these devices and discuss them with their friends. at the same time, news websites also allow users to post comments and discuss stories @xcite . while the inclusion of comment sections provides users with a platform to engage in constructive discussions regarding news stories, these discussions can also devolve into the expression of offensive remarks and hate speech @xcite @xcite . furthermore, malicious users can exploit discussion platforms to intentionally spread misinformation, often in the form of fake news, to mislead and provoke readers @xcite . the wide spread of inappropriate comments motivates the use of content moderation to avoid further undesirable consequences. moderating comment sections is a difficult task, mainly due to how widely the content can range, including fake news @xcite and various forms of offensive speech @xcite @xcite . detecting these varied types of content is difficult for humans alone, and in addition, the sheer number of comments that can be generated by any comment section makes manual moderation an overwhelming and costly task @xcite . many approaches in nlp are dedicated to identifying fake news @xcite , hate speech @xcite , and related phenomena. however, as is often the case, these approaches focus on english and other high-resource languages @xcite . with the increasing prevalence of smart devices, a significant number of individuals prefer to express their thoughts and opinions in their native languages. consequently, there is a pressing demand for systems that can cater to each language. unfortunately, the lack of language resources poses a significant challenge in developing such systems, particularly for low-resource languages @xcite . in this paper, we experiment with automatic content moderation for luxembourgish, a west germanic language spoken by around 400,000 people, primarily in luxembourg. we use state-of-theart multi-and cross-lingual language models, as well as a recently released model for luxembourgish specifically. using a dataset provided by the main news broadcaster in luxembourg, we trained a number of models to predict whether a given comment should be archived or not, according to the internal policy of the dataset provider. as such, this presents the first real evaluation of such an approach in the field of automatic content moderation, as well as its sub-tasks, for luxembourgish. additionally, it has been demonstrated that the case of luxembourgish is unique, offering resources for research but being under-represented in research @xcite . this paper answers two research questions: the remainder of this paper is structured as follows. section 2 presents an overview of related work in the field. section 3 describes the dataset used for the experiments, followed by a description of the employed methodology in section 4. the results of the experiments are presented in section 5. finally, section 6 offers our future plans as well as concluding remarks. ## related work automatic content moderation is a challenging and interesting task which has attracted the attention of the nlp community for many years. content moderation involves a number of sub-tasks in nlp, mainly including racism and hate speech detection, as well as fake news detection and irony and sarcasm detection. offensive content detecting and classifying offensive content has been studied extensively both for news comments and social media posts. early approaches have applied traditional machine learning classifiers to the task, while more recent work has applied neural networks @xcite . most of the datasets and approaches have been based on english @xcite . nevertheless, research is also conducted on croatian @xcite , estonian @xcite , german @xcite , korean @xcite , and slovene @xcite on detecting offensive content in news media comments. there is also a rise in shared tasks on the topic, notably semeval 2019 task 6 (offenseval), which treated the identification and categorisation of offensive language on social media for english, attracting over 800 teams with 115 final submissions @xcite . moreover, there have been shared tasks for various languages, including german @xcite , bangla (kumar et al., 2020), hindi @xcite , as well as multilingual @xcite and code-mixed @xcite settings. misinformation misinformation detection in news media comments is another sub-task that has caught the attention of the nlp community, as many malicious users exploit discussion platforms to spread misinformation intentionally @xcite . however, not much work has been done on detecting misinformation in news media comments @xcite . on the other hand, there have been several works on misinformation detection in social media posts, which are also focused on english and other high-resource languages @xcite . however, fake news detection remains a complex task in nlp @xcite . while various current architectures have been trained for this task, it is said that these approaches require more complex ensembles of architectures to accurately predict fake news segments, particularly shorter ones @xcite . resources for luxembourgish in general, luxembourgish is said to be under-represented in nlp, particularly because it is a relatively small language, especially compared to its linguistic neighbours, french and german. this can be attributed to the relatively recent development of the written domain in luxembourgish that has largely been fostered by the advent of social media. however, resources are steadily increasing. @xcite developed a state-of-the-art pipeline for sentiment analysis based on the same dataset as our study. purschke (2020) published a pipeline for the automatic orthographic correction of text data, foot_0 i.a. based on correction data from spellchecker.lu, an online spellchecking tool for luxembourgish. 2 additionally, the luxembourgish online dictionary (lod) recently launched an open api to its lexical resources. 3 lothritz et al. (2021) introduced an intent classification dataset for luxembourgish, which contains 1006 instances divided into 28 different intents related to banking requests such as opening/closing a bank account or ordering/blocking a credit card. the winograd natural language inference task which is part of the glue benchmark collection @xcite ) contains more than 750 instances in luxembourgish. with recent advances in neural networks, there now exists a language model for luxembourgish, lux-embert @xcite , which we also use for the purposes of this paper. with luxem-bert, @xcite introduced several language resources for luxembourgish, including part-of-speech tagging, named entity recognition and news classification. at the time of writing, there is no published research on work related to content moderation in luxembourgish. ## data the dataset used for the purposes of this paper was provided by the rtl media group, the largest news provider in luxembourg. the dataset provided stems from their own news platform, foot_3 which has existed since 2008 and is the only news offering that is entirely in luxembourgish. given the recent expansion of luxembourgish into the written domain and the central role of rtl in the country's media system, for many luxembourgers, the rtl news platform has been one of their main points of contact with written luxembourgish, apart from private messaging. against this backdrop, our data represents not only the largest collection of written texts in luxembourgish currently available, but also a crucial source for studying the development of written luxembourgish in real time. for the purposes of this paper, we work exclusively with user comments, comprising over one million comments posted on around 61,000 news articles over the course of a 14 year time-span, starting in 2008. each comment includes manual content moderation information provided by a number of dedicated content moderators over the years, with labels assigned according to a step in the moderation process. while the label published should be clear, three others indicate that the given comment has been moderated or archived (and there may be other moderation steps to be taken). we treat these three labels here as archived (meaning not published). it should be made clear at this point, that for the years 2008-2010 all comments are labelled published. this is an error in our iteration of the dataset and has resulted in this data being excluded. in terms of preprocessing, the comments have to be cleaned of special characters, incorrect encodings and markup language. since the platform has undergone some changes in its technical implementation, various markup standards are represented and need to be removed. in addition, various text encodings need to be converted to unicode, and special characters and embedded content need to be removed. all preprocessing steps were carried out in a dedicated python pipeline. the mean comment length is 352 characters, with the median lying at 220 characters. the shortest comment is one character in length, with the longest comment being 34,597 characters in length. figure 2 shows the average length of comments over the years represented in the dataset, which highlights the fact that the comment length has gone down by almost 50% since 2008. interestingly, the lowest average comment length was recorded in 2020, the same year that has by far the highest number of comments on a yearly basis. luxembourg is a multilingual country, with german, french and luxembourgish recognised as official languages, although with different domain allocations in administration and everyday practice @xcite . while french and german are the main administrative languages, luxembourgish has the status of the national language. french is the language of legislation, and german serves as the language for alphabetisation. it also holds, for historical reasons, an important position in print media, whereas luxembourgish has only recently developed from a predominately spoken into a written variety that is suitable for all social domains @xcite . furthermore, due to the country's migration and industrial history, portuguese and italian are considered important minority languages. nowadays, cross-border commuting and the international workforce in the finance industry put pressure on the traditional language regime, with french and english gaining more ground. this complex multilingualism is, of course, reflected in the corpus, with instances of code switching on the comment level, but also answers in french or german to luxembourgish comments are not uncommon in the dataset. to investigate the language representation further, we processed all comments with the langdetect package available for python. 5 as luxembourgish is not available for this package, we used a custom profile, which has been trained previously for the recognition of luxembourgish, based on the rtl news articles @xcite . detection accuracy for luxembourgish works reliably (100%) using a random sample of 1,000 texts. for non-luxembourgish texts, accuracy is around 96% for texts longer than 200 characters, but drops to 64% for short texts that do not offer many languagespecific patterns. figure 3 shows the percentage of the top four languages detected automatically in the dataset, with all others grouped together. although these results are not necessarily representative: luxembourgish language detection is an area of ongoing research and can often be misclassified as french (due to many loan words) and german (due to the two being closely related). in addition, the full list of detected languages comprises about 30 languages, including languages such as chinese, which are not very likely, although it should not be dismissed entirely. further analysis has shown that many labels are assigned based on one word, hinting again at mislabelling. ## methodology to investigate the research questions posed in section 1, we carried out the following steps. first, the data was processed and cleaned. next, we trained various language models on the task of classifying the comments into two groups. following this, we experimented with the composition of the training set, limiting it to certain years and testing the effectiveness on the most recent year. ## results we first concatenated all the comments from 2011-2021 as the training set. the comments from 2022 were considered as the test set. we trained all the models described in section 4 under this setting. the results of these models are shown in as the label distribution is highly imbalanced, we evaluate and compare the performance of the different models using the macro f1-score. furthermore, a classifier that can correctly identify both classes would protect freedom of expression while moderating the unwanted texts. we further report per-class precision (p), recall (r), f1-score (f1), and weighted averages. finally, we compare the performance of the models against simple majority and minority class baselines. as can be seen in interestingly, the luxembert model @xcite , which was built on luxembourgish text, did not perform well compared to other models in this task. models such as xlm-r, which do not support luxembourgish, outperform lux-embert. we assume that this can be due to two reasons; (i) the texts used to train the models are heavily code-switched and code-mixed. xlm-r models have an advantage over this. this is further confirmed by the superior performance of byt5 models. byt5 models follow a tokenizer-free approach and, therefore, perform well in code-switched and code-mixed texts. (ii) xlm-r models provide stronger models compared to lux-embert. overall, we can see that it is advantageous to use xlm-r rather than language-specific luxembert. all the models we experimented with performed poorly in identifying the archived class. the best model, byt5 large, only had an f1 score of 0.31 for the archived class. scores of the published class were better and consistent across the models. we assume that identifying archived comments is challenging for machine learning models, as there are many reasons why a comment could have been archived, including but not limited to the sub-tasks of content moderation mentioned in section 2. it is clear that this requires more research input and some insight into the moderation policy. the byt5-large model took approximately 155 hours on an nvidia rtx a6000 48gb gpu 2 for each model, where the models were trained on all the instances from 2008-2021. to train. the xlm-r large model took 83 hours, and luxembert only took 44 hours to train on the same gpu. therefore, even though byt5-large provided the best result for our task, it is not the most computationally efficient model. with these results, we answer rq1: how do the state-of-the-art transformer models perform in automatic content moderation in luxembourgish? we showed that several transformer models perform fairly well in the task. however, the models do not provide impressive results, and this task requires more attention from the nlp community for low-resource languages such as luxembourgish. ## conclusion in this paper, we presented the first study on automatic comment moderation in luxembourgish news articles. our study involved a comprehensive qualitative analysis of over one million luxembourgish comments spanning a period of 14 years. the main objective was to evaluate the performance of various state-of-the-art multilingual, cross-lingual, and language-specific transformer models in the task of content moderation. among these models, byt5 large @xcite emerged as the best model, indicating that its tokenizer-free approach is particularly advantageous for handling the code-mixed and codeswitched nature of luxembourgish news comments. while the transformer models overall produced satisfactory results, there remains significant room for improvement, especially when it comes to the archived class. additionally, our findings revealed that machine learning models trained on old data exhibit poor performance when applied to recent data on content moderation. our findings in this study will be beneficial for researchers working on automatic content moderation in low-resource languages. in future work, we hope to enhance the interpretability of the recommended machine learning models to better assist human content moderators in their decision-making process. by pursuing these avenues, we aim to contribute towards the advancement of automatic content moderation techniques while ensuring their alignment with human moderation needs.