Fine-tuning Neural Language Models for Multidimensional Opinion Mining of E nglish- M altese Social Data (2021)

Abstract: This paper presents multidimensional Social Opinion Mining on user-generated content gathered from newswires and social networking services in three different languages: English —a high-resourced language, Maltese —a low-resourced language, and Maltese-English —a code-switched language. Multiple fine-tuned neural classification language models which cater for the i) English, Maltese and Maltese-English languages as well as ii) five different social opinion dimensions, namely subjectivity, sentiment polarity, emotion, irony and sarcasm, are presented. Results per classification model for each social opinion dimension are discussed.

## introduction social opinion mining on data obtained from social sources is an evolving research domain tasked with the identification of several opinion dimensions, such as subjectivity, sentiment polarity, emotion, irony and sarcasm, from noisy user-generated social data spread across heterogeneous sources @xcite . currently, social opinion mining is used in several real-world scenarios, namely chatbots @xcite , adaptive customer online service based on identified customer sentiment and emotion @xcite , tracking of overall customer satisfaction for a product or service @xcite , and detection of changes in customer opinion towards a brand, product or service @xcite . this paper presents multidimensional social opinion mining on user-generated content gathered from newswires and social networking services in three different languages: english -a highresourced language, maltese foot_0 -a low-resourced language, and maltese-english -a code-switched language. our aim is use these initial results to improve cross-lingual performance of english-maltese neural language models. research applications of the developed classification models include opinion summarisation and fine-grained opinionated search of each dimension. this work is in line with malta's strategy and vision for artificial intelligence (parliamentary secretariat for financial services and innovation, 2019), with current investment being made in the development of maltese language resources and tools to counter the threat of "digital extinction" for the maltese language, which has low technological support available in comparison with other european languages @xcite . we leverage a novel multidimensional and multilingual social opinion dataset in the socioeconomic domain, specifically malta's annual government budget, which comprises social data from the 2018, 2019 and 2020 budgets to fine-tune pretrained neural language models for benchmarking purposes. ## related work nguyen et al. @xcite developed the first large-scale pre-trained language model bertweet for english tweets, which outperforms its baselines. experiments were conducted on three nlp tasks, namely part-of-speech tagging, named entity recognition and text classification, namely sentiment analysis and irony detection. for the latter task, the authors used the 3-class sentiment analysis dataset from semeval-2017 task 4a @xcite and the 2-class irony detection dataset from the semeval-2018 task 3a @xcite . the authors in @xcite propose gan-bert which extends the fine-tuning of architectures similar to bidirectional encoder representations from transformers (bert) @xcite , using unlabelled data in a generative adversarial setting. experimental results show that around 50-100 annotated examples can still produce good performance in sentence classification tasks. results are confirmed for sentiment analysis over the sst-5 dataset @xcite containing 5-class sentiment polarity categories. babanejad et al. @xcite propose two novel deep neural network models for sarcasm detection by including affective and contextual features in the extended bert architecture. certain studies focused on low-resourced languages, with @xcite investigating crosslingual sentiment classification where the lowresource language does not have any labels or parallel corpus, @xcite exploring the reduction of trainable model parameters for finetuning a model with a small amount of data, @xcite releasing a new pre-trained language model for indonesian which was evaluated on several tasks such as sentiment analysis, and (yimam et al., 2020) using roberta @xcite -a replication of bert developed by facebook-for exploring amharic sentiment analysis from social media text. demszky et al. @xcite ) conduct transfer learning experiments on existing emotion benchmarks to show that the goemotions dataset of fine-grained emotions generalises across domains and taxonomies. the authors demonstrate that if little target domain labelled data is available, this dataset can be used as a baseline for emotion understanding. similarly, the xed multilingual dataset for emotion detection catering for a total of 32 languages has been evaluated using languagespecific bert models @xcite . lastly, @xcite explore several off-the-shelf bert models, where they show that the complexity and computational cost of bert does not provide a guarantee for an improved predictive performance for classification tasks. this is especially relevant in cases where small domainspecific datasets are used, which datasets are also imbalanced due to the minority class being underrepresented. ## dataset the dataset of multidimensional and multilingual social opinions for malta's annual government budget foot_1 @xcite is used for the work carried out in this paper. this dataset contains 6,387 online posts for the 2018, 2019, and 2020 budgets, which user-generated content was collected from newswires and social networking services. in terms of languages, the majority of the online posts were in english (74.09%), maltese or maltese-english (24.99%). each online post is annotated for the following five social opinion dimensions: subjectivity, sentiment polarity, emotion, sarcasm and irony. ## experiments all experiments have been carried out on google colaboratory foot_2 using a tesla k80/tesla t4/tesla p100-pcie-16gb graphics processing unit (gpu). the baseline models experiments have been carried out in the python programming language using jupyter notebook foot_3 on a machine with an intel(r) core(tm) i7-8550u cpu @ 1.80hz 1.99 ghz processor and 8.00 gb (7.88 gb usable) installed memory (ram). ## results and discussion results per classification model for each social opinion dimension are presented in the following is an overview of the results and some observations. ## conclusions and future work we have leveraged a novel multidimensional and multilingual social opinion dataset in the socioeconomic domain to fine-tune neural language models targeting english-maltese social data for different opinion dimensions, namely subjectivity, sentiment polarity, emotion, sarcasm and irony. even though our results are a work-in-progress, we have been encouraged by @xcite to provide multilingual benchmarks which can be further used, evaluated and adapted for lowresourced languages. research applications for the developed classification models include opinion summarisation and fine-grained opinionated search of each dimension.