Universität Regensburg M ax S at G erm E val 2021 Task 1: Synthetic Data in Toxic Comment Classification (2021)

Abstract: We report on our submission to Task 1 of the GermEval 2021 challenge – toxic comment classification. We investigate different ways of bolstering scarce training data to improve off-the-shelf model performance on a toxic comment classification task. To help address the limitations of a small dataset, we use data synthetically generated by a German GPT-2 model. The use of synthetic data has only recently been taking off as a possible solution to ad- dressing training data sparseness in NLP, and initial results are promising. However, our model did not see measurable improvement through the use of synthetic data. We discuss possible reasons for this finding and explore future works in the field.

## introduction in recent years, social media platforms have become an integral part of our everyday lives. together with their enormous rise in use and popularity, they have also faced several troubles. these range from pr problems due to privacy concerns foot_0 to fake news @xcite . there have also been incidents related to deplatforming controversial individuals of public interest foot_1 . in 2018, for instance, facebook was used to incite a genocide against the rohingya people of myanmar foot_2 . the ongoing global pandemic has seen an increase in xenophobic and antisemitic hate @xcite . in the light of these and other developments, the task of detecting toxicity on the internet has seen increased attention in recent years. by now, the legislature of multiple countries is getting modified to accommodate laws counteracting the incitement of hatred online. the german netzwerkdurchsetzungsgesetz (network enforcement act), for instance, requires social media providers with over 2 million users registered in germany to report hate speech on their platform to legal authorities foot_3 . these steps try to prevent the marginalization of populations. users exposed to hate online may no longer take part in debates or discourse. this work aims to advance the state of the art in the field of toxicity detection by providing an additional avenue of working with scarce data. any code used for this work is available on github foot_4 . ## related work there have been numerous developments in the space of toxicity detection since germeval in 2019. @xcite give a well-composed overview of the state of affairs in 2019. we will therefore focus on recent work and concepts closely related to the challenge. ## methodology in this work, we investigate whether or not synthetically generated data can improve the baseline of a model solely trained on a scarce dataset. ## results and discussion as seen in first, the discrepancy in the validation and testing performance of the model using synthetic data is possibly due to the initial validation results being achieved on data that included synthetic data. this could be problematic, as the data generated by another issue is the selected data generation model, ggpt-2. the output of gpt-2, as seen in figure 1 , is composed of both english and german sentences. this composition is not the desired outcome, but english and german are sister languages. the output generated by ggpt-2, as seen in figure 2 , on the other hand, appears to be effective on the token level, but in some cases not capable of generating coherent sentences or words. therefore, we should have selected gpt-2 over ggpt-2. in light of these mishaps, we still deem the approach of using synthetic data to be successful. synthetic data is comparatively easy, cheap and fast to use. it did not negatively affect the baseline approach. furthermore, ethical and practical implications of synthetic data are major issues. privacy concerns of real datasets can be mitigated if synthetic data can be generated using real-world datasets that can not be published themselves. ## conclusion and future work we conclude that synthetic data does seem to be a promising avenue of research. however, this particular work did not find a measurable improvement over the baseline model using synthetic data. future work will use either gpt-3 or gpt-2, and we will rework the methodology. once a more robust method is formulated, we will test it on several different datasets. we suggest further research on synthetic data in classification tasks, as the comparatively poor performance of our model may be due to the limitations of our chosen methodology. other possible future avenues of research include the training and evaluation of a model solely trained on synthetic data.