Classification of Censored Tweets in C hinese Language using XLN et (2021)

Abstract: In the growth of today’s world and advanced technology, social media networks play a significant role in impacting human lives. Censorship is the overthrowing of speech, public transmission, or other details that play a vast role in social media. The content may be considered harmful, sensitive, or inconvenient. Authorities like institutes, governments, and other organizations conduct Censorship. This paper has implemented a model that helps classify censored and uncensored tweets as a binary classification. The paper describes submission to the Censorship shared task of the NLP4IF 2021 workshop. We used various transformer-based pre-trained models, and XLNet outputs a better accuracy among all. We fine-tuned the model for better performance and achieved a reasonable accuracy, and calculated other performance metrics.

## introduction the suppression of words, images, and ideas is known as censorship. the government or the private organization can carry censorship based on objectionable, harmful, sensitive, or inconvenient material. there are different types of censorship; for example, when a person uses censorship for their work or speech, this type of censorship is known as self-censorship. censorship is used for many things like books, music, videos, movies, etc., for various reasons like hate speech, national security, etc. @xcite . many countries in their law provide protections against censorship, but there is much uncertainty in determining what could be censored and what could not be censored. however, nowadays, we know that most of the data and the information are available on the internet, so many governments strictly monitor the disturbing or objectionable content on the internet. we could not use any method other than the software like fraud censorship detection and disturbing and objectionable content monitor, which works continuously and maintains the same accuracy for monitoring this vast data size. this paper examines the methodologies and various machine learning domains that classify the censored and uncensored tweets associated with the workshop @xcite . we used multiple models such as bert (bidirectional encoder representations from transformers) @xcite , deberta (decoding-enhanced bert with disentangled attention) @xcite , electra @xcite , and xlnet (a generic autoregressive pre-training procedure) for binary classification of the tweets. "0" says that the tweet is uncensored, and "1" says that the tweet is censored. also, we have experimented with various phases, such as data preprocessing, tokenization, and fine-tuning for model prediction. further, we will go through various performance metrics such as accuracy, precision, and recall. we achieved a reasonable accuracy using xlnet as compared to other models. ## dataset description the dataset of the shared task has been built using a web scraper (kei yin ng and peng, 2020) that contains censored and uncensored tweets gathered for a duration of 4 months (august 29, 2018, to december 29, 2018). the dataset attributes contain tweets (represented by the text in the dataset) and label, where the "text" field contains the information collected in the chinese language, and "label" contains 0's and 1's where '0' signifies the tweet as uncensored and '1' signifies as a censored tweet. the first few lines and format of the dataset is shown in fig. 1 . the dataset comprises three sets, i.e. train, validation and test set. the train set comprises 1512 tweets, and the validation set comprises 189 tweets. the test set only comprises 189 tweets with no labels. ## methodology the xlnet @xcite is a transformer-based machine learning method for natural language processing tasks. it is famous for a generalized autoregressive pretraining method which is one of the most significant emerging models of nlp. the xlnet consists of the recent innovations in nlp, stating the solutions and other approaches regarding language modelling. xlnet is also known for the auto-regressive language model that promotes joint predictions over a sequence of tokens on transformer design. it aims to find the possibility of a word token's overall alterations of word tokens in a sentence. the language model comprises two stages, the pretrain phase and fine-tune phase. xlnet mainly concentrates on the pre-train phase. permutation language modeling is one of the new objectives which is implemented in the pre-train phase. we used "hfl/chinesexlnet-base" as a pre-trained model @xcite for chinese data that targets enhancing chinese nlp resources and contributes a broad category of chinese pre-trained model selection. initially, the dataset is preprocessed, and the generated tokens are given input to xlnet pre-trained model. the model trains the data over 20 epochs and further goes through a mean pool, passing through a fully connected layer for fine-tuning and classification, and predicts the data over a given test set. fig. 2 shows the architecture of the xlnet model. ## conclusion and future work in the paper, we investigated various pre-trained models and achieved a reasonable accuracy for xlnet. we cleaned the dataset during preprocessing, which is further given input to the model. xlnet seems to be influential in the classification problem moving deep into censorship detection. xlnet performs better than bert, deberta, and electra having its improved training methodology, where it uses permutation language modelling predicting the tokens randomly. the future work is to examine other nlp models and finetune them censorship detection in other languages.