Team Rouges at S em E val-2020 Task 12: Cross-lingual Inductive Transfer to Detect Offensive Language (2020)

Abstract: With the growing use of social media and its availability, many instances of the use of offensive language have been observed across multiple languages and domains. This phenomenon has given rise to the growing need to detect the offensive language used in social media cross-lingually. In OffensEval 2020, the organizers have released the multilingual Offensive Language Identification Dataset (mOLID), which contains tweets in five different languages, to detect offensive language. In this work, we introduce a cross-lingual inductive approach to identify the offensive language in tweets using the contextual word embedding XLM-RoBERTa (XLM-R). We show that our model performs competitively on all five languages, obtaining the fourth position in the English task with an F1-score of 0.919 and eighth position in the Turkish task with an F1-score of 0.781. Further experimentation proves that our model works competitively in a zero-shot learning environment, and is extensible to other languages.

## introduction the prevalence of social media has made public commentary a critical aspect in shaping public opinion. although freedom of speech is often advocated, offensive language in social media is unacceptable. nevertheless, social media platforms and online communities are laden with offensive comments. this phenomenon results in the need for computationally identifying offense, aggression, and hate-speech in user-generated content in multiple languages. this paper addresses the challenge put forward in the multilingual offensive language identification in social media shared task-organized at semeval 2020 @xcite . the theme of the problem is to identify the offensive language in tweets in arabic, danish, english, greek, and turkish. this shared task is further divided into three sub-tasks. the first task consists of the identification of offensive tweets in a multilingual setting, whereas the other two tasks consist of the categorization of offensive tweets and identification of targets in english. transfer learning is a methodology to utilize the knowledge acquired from one or more tasks to solve other related tasks. it is ubiquitous in the domain of natural language processing. it can be classified into multiple types, like transductive transfer learning and inductive transfer learning. the former is used when the tasks are the same, but the corresponding domains are different, such as in the case of the cross-lingual learning for similar tasks. on the other hand, the latter is used when the tasks are different, but the domains are similar such as in the case of finetuning of pretrained contextual word embeddings. cross-lingual inductive learning has been used for many downstream tasks like multi-lingual variants of question answering, text classification, and text generation @xcite @xcite . the rationale behind this is that the language with limited resources benefits from joint training over many languages. it also helps in performing zero-shot learning and handling of code-switched text, which otherwise are difficult to tackle. in this work, we propose the use of cross-lingual inductive learning to detect the offensive language in the given five languages. we use pretrained xlm-r @xcite cross-lingual embeddings and train a single cross-lingual model for all five languages in the multilingual offensive language identification dataset. artetxe and schwenk (2018) introduced an architecture for learning joint multilingual sentence representations, using a single bilstm encoder with a bpe vocabulary shared by all the languages. they use 93 languages, spanning across 30 different language families and 28 different scripts for training the sentence representation embeddings. experimental results on the xnli dataset @xcite for cross-lingual natural language inference and mldoc dataset for cross-lingual document classification show that the cross-lingual transfer of learned features helps in improving the performance of the classification models in a cross-lingual setting. chen and cardie (2018) proposed a fully unsupervised framework for learning multilingual word embeddings using only monolingual corpora. unlike prior work in multilingual word embeddings, these embeddings exploit the relations between all the language pairs. by mapping all monolingual embeddings into a shared multilingual embedding space via a two-stage algorithm consisting of multilingual adversarial training and multilingual pseudo-supervised refinement, the authors propose an effective method for learning the representation. recent works for the task of text classification exploit pretrained contextualized word representations rather than context-independent word representations. these pretrained contextualized word embeddings, like bert @xcite and roberta @xcite , have outperformed many existing techniques on most nlp tasks with minimal task-specific architectural changes and training resources. contextualized word embeddings have been used for detecting subjective bias @xcite , and detecting hate speech in tweets @xcite . to train multilingual variants of the contextualized word embeddings effectively, @xcite proposed a cross-lingual language modeling methodology termed as xlm. in their paper, they proposed two variants of their model: supervised and unsupervised multilingual language models. the supervised variant used parallel data with a new cross-lingual language model objective. @xcite proposed xlm-r extending xlm. xlm-r differs from xlm in its ability to allow pretraining of the cross-lingual language models at scale using larger datasets and better-tuned hyperparameters. their experiments on xnli, mlqa, and ner show significant increases over the previous state-of-the-art. we further finetune xlm-r, pretrained on 2.5 tb common crawl corpus spanning 100 languages. ## methodology in this section, we describe the preliminaries for the work: xlm-r, a cross-lingual contextual word representation, and cross-lingual inductive transfer. we then illustrate our sequential approach for finetuning the models. ## experiments and results in this section, we present details on the multilingual offenseeval dataset, experimental settings for the validation experiments, and specifications of the system runs. ## discussion xlm-r is an unsupervised cross-lingual representation pretrained using transformers on an enormous scale across 100 languages. it can be fine-tuned for different downstream tasks in multiple languages, allowing one to take advantage of cross-lingual transfer learning. like other transformer-based contextual word embeddings, xlm-r is highly scalable, and the process of fine-tuning takes minimum training efforts and resources. our proposed model, extending xlm-r, was trained in five different languages for the detection of offensive language. owing to the training of xlm-r on hundred languages, our model allows a direct extension to other commonly-used languages. our chain-like model exploits learnings from one language for detecting offensive language on another unrelated language. we further performed experiments for detecting offensive language in a language different from the ones that we have trained on without supervision. this form of learning, known as zero-shot learning, has been demonstrated in recent related works @xcite . to evaluate the efficacy of our model to perform zero-shot learning, we detect offensive language text in german using our final cross-lingual model. we perform the experiments in the same experimental setting as the germeval shared task on the identification of offensive language's subtask 1 @xcite , which entails coarse-grained binary classification of offensive language. ## conclusion this work presents a cross-lingual inductive transfer learning approach to detect the offensive language in tweets across five languages: english, turkish, greek, arabic, and danish. our proposed model performs competitively for all five languages. we further performed comprehensive experiments to show that our model performs well in a zero-shot setting, which can be useful in the case of low resource languages. our proposed model can be easily extended to other languages with minimum training efforts and resources, performing competitively even with significantly imbalanced data. future work may involve including multiple unrelated languages to enable a universal offensive language detection model and the application of the proposed approach in other cross-lingual tasks.