CEIA - NLP at CASE 2022 Task 1: Protest News Detection for P ortuguese (2022)

Abstract: This paper summarizes our work on the document classification subtask of Multilingual protest news detection of the CASE @ ACL-IJCNLP 2022 workshok. In this context, we investigate the performance of monolingual and multilingual transformer-based models in low data resources, taking Portuguese as an example and evaluating language models on document classification. Our approach became the winning solution in Portuguese document classification achieving 0.8007 F1 Score on Test set. The experimental results demonstrate that multilingual models achieve best results in scenarios with few dataset samples of specific language, because we can train models using datasets from other languages of the same task and domain.

## introduction observing the prominent ease of use and variety of virtual media, such as social networks in general, and the exponential use of these for the organizational purpose of various manifestations, protests and social movements @xcite , a large amount of information is stored in databases of applications that are not properly analyzed for a socially beneficial purpose. therefore, it is important to explore alternatives for an analysis capable of classifying and even predicting the organization of social movements such as those mentioned above. considering the importance of detecting crises and sociopolitical events present in social networks @xcite . the practical application of extracting and classifying information and its importance in the field of collective social manifestations, in order to obtain several useful results for important political and economic decisions @xcite . in this paper, we investigate the performance of monolingual and multilingual language models for classification of documents in portuguese. the experiments are conducted on socio-political datasets and all models are transformer-based models. our submission achieved the 1st place in document level predictions for the portuguese language at first shared task of the case @ acl-ijcnlp 2022 workshop @xcite , the multilingual protest news detection subtask @xcite . this article is organized as follows. in section 2, reviews the related work. section 3 details of subtask and data. section 4 describes the methodology, while experiments results are discussed in section 5. section 6 brings the conclusions. francesco ignazio re (2021) presented a disruptive perspective with the exploratory analysis of the dataset. their conclusions approached differences in the use of state versus non-state conflict actors based on conditional probabilities, and also identified an outlier in the english corpus via the tf-idf-weighted principal component analysis (pca). ## subtask and data the dataset was provided by the organization of case 2022's first multilingual protest news detection shared task. ## methodology we used pretrained transformer-based models for portuguese to investigate the classification performance of monolingual across multilingual models in scenario with low dataset resources. for this study, we selected two models and their multilingual versions: these models were optimized with a grid search optimization on held-out development set with a combination of finetuning hyperparameters provided by ## results and evaluation all experiments were conducted on the hugging's face transformer library using one nvidia a100 gpu @xcite for classify whether a document in portuguese mentions an event or not. the models performance was evaluated by the macro f1-scores on the validation set, which were created by splitting the dataset. the dataset for multilingual models was created by combining training data from each languages into one dataset. the results for the test set are shown in ## conclusion in this paper, we have explored the capabilities of multilingual and monolingual language models on document classification of the case 2022 task 1: multilingual protest news detection. we demonstrate that multilingual transformer-based approach could be more competitive that monolingual transformer-based model in scenarios that have low data resources of a specific language and more data of other languages can help achieve a best performance. the proposed xlm-roberta model achieved the 1st place for the portuguese language with 0.8007 f1 score on test set. these results illustrate the importance of increasing the maximum sequence length for document classification. as future work, it would be interesting to extend the study to architectures with much longer input sequences. we also investigate other methodologies based on ensemble approach, data augmentation and few shot models.