U o R - NCL at S em E val-2022 Task 6: Using ensemble loss with BERT for intended sarcasm detection (2022)

Abstract: Sarcasm has gained notoriety for being difficult to detect by machine learning systems due to its figurative nature. In this paper, Bidirectional Encoder Representations from Transformers (BERT) model has been used with ensemble loss made of cross-entropy loss and negative log-likelihood loss to classify whether a given sentence is in English and Arabic tweets are sarcastic or not. From the results obtained in the experiments, our proposed BERT with ensemble loss achieved superior performance when applied to English and Arabic test datasets. For the validation dataset, our model performed better on the Arabic dataset but failed to outperform the baseline method (made of BERT with only a single loss function) when applied on the English validation set.

## introduction in recent times, social media platforms such as twitter have turned out to be one of the most influential media for the expression of views, emotions, and information. to characterize sarcasm on twitter; @xcite proposed the following: (a) strife between negative situation and positive sentiment, (b) strife between positive situation and negative sentiment, (c) tweet starts with an interjection word, (d) likes and dislikes contradiction, (e) tweet conflicting ubiquitous facts, (f) tweets with positive sentiment and antonym pair, and (g) tweet contrasting facts that are time-sensitive. with a huge amount of content being churned on social media and the need to analyze and detect sarcasm closely, text classification methods have been widely introduced to deal with these sophisticated tasks. sarcasm has been shown to pose a major difficulty for sentiment analysis models @xcite , mainly because sarcasm acts as a form of verbal irony which enables the concealment of the true intention of denigration and negativity under a pretence of open a respectful representation such as; "the only thing i got from college is a caffeine addiction". as a result, the ability to detect sarcasm is crucial for understanding the real intents and beliefs of people @xcite . to address these issues and limitations, @xcite have organized the is-arcasmeval shared task for intended sarcasm detection in english and arabic where different tweets in english and arabic languages have to be classified as either sarcastic or not. several models for detection of sarcasm have been proposed in literature which incorporate statistical, machine learning, and rulebased approaches but predominantly (mandal and mahto, 2019). however, these techniques are not able to effectively perceive the figurative and ironic meaning of words @xcite . also, these methods require manually engineered features and are unable to understand the patterns in passive voice sentences @xcite . however, instead of utilizing manually engineered features, the incorporation of transformer models has the ability to automatically learn the important features. in this paper, we present our participating system to the isarcasmeval shared task @xcite . in this work, we seek to apply a novel transformer-based approach to detect sarcastic statements in both english and arabic languages and propose a novel approach to fine-tune the bert model for sarcasm detection of both arabic and en-glish tweets using ensemble loss. bert is a model that is gaining increasing popularity due to its outstanding performance for multiple nlp tasks. recently, the language model called bert has been widely used in several tasks. the pre-trained model can generate word/token or sentence representations that are enriched with prior knowledge. then, they can be fine-tuned specifically for many downstream tasks such as sarcasm detection. the traditional bert model only uses one loss function whilst our proposed model which combines a cross-entropy loss with a negative loglikelihood function leads to a better penalization of incorrect predictions that the model is confident about more than incorrect predictions the model is less confident about. this leads to better detection of sarcasm due to an improved f-1 sarcastic. one of the main contributions of this work is the use of ensemble loss with bert @xcite to address the task of sarcasm detection of tweets in both english and arabic languages with improved sarcasm detection ability. the rest of the paper is organized as follows: the related work in section 2 discusses the research works which are closely related to the current study. section 3 contains the system description part and highlights the description of the proposed approach and its working methodology. the dataset used for experiment setup is provided in the experiments section in section 4, whilst the results and discussion segment can be found under section 4.1 and contains the experimental results and the performance metrics. the conclusion and future work for this research work are all captured in section 5. ## related work the conversation aspect in sarcasm detection was investigated by @xcite , and sarcastic and non-sarcastic tweets were collected to create a dataset for their experiments. bert, which is a transformer-based machine learning model has been exploited to provide a deep contextual representation of words and used for sarcasm detection. they are pretrained language models and have been also been widely utilized in many nlp domains. these models have been proven to be effective since they are enriched with knowledge from the pre-training resources @xcite . @xcite recently surveyed automated sarcastic detection on twitter and found out that there was a shift towards the use of deep learning methods due to the benefit of using a model with induced instead of discrete features combined with the innovation of transformers. bert and glove embeddings were combined with machine learning classifiers to detect sarcasm in tweets in the work of (khatri and p, 2020). semi-supervised techniques were used by @xcite to detect sarcasm sentences in online product reviews whilst @xcite also deployed the use of automatic learning and exploiting word embeddings to recognize sarcasm. other approaches such as bi-directional gated recurrent neural network (bi-directional grnu) have also been used to detect sarcasm @xcite . bert has been used with a crowd layer on tweets with noisy labels to achieve improved results @xcite . ## system description for each sample in the english and arabic dataset, a tweeted text and a given token were first concatenated in the following format: ## experiments experiments were conducted on the respective english and arabic training and test sets provided by the task organizers. the training set of the english language had 3467 samples whilst that of the arabic language had 3120 data samples. during the development stage, the training datasets were divided into 90% and 10% splits sets respectively for the training and validation datasets. the test sets for both languages on the other hand had a total of 1,400 data samples used in the evaluation phase. all the models were fine-tuned by using the adam optimizer for 12 epochs with the batch size 32 and the learning rate of 0.000005 and 0.000000001 as the default epsilon value. for sub-task a, the models were evaluated by accuracy, recall, false positives, f-1 sarcastic, accuracy, and f1-macro. ## conclusion this work has proposed the use of ensemble loss on the bert model to detect whether a given sentence in english or arabic language is sarcastic or not. the results indicate that the use of our ensemble loss on the bert model exhibits superior performance over the baseline models when applied to the english and arabic test datasets. future work will involve the undertaking of further investigation to find the optimal proportion of each combination that yields the optimal results. it will also involve the use of ensemble loss on arabic-based bert models such as ara-bert and other bert-based models to find their performance. different types of losses will also be investigated and added to the ensemble loss portfolio in order to explore the different types with superior performance. since the two language datasets used were unbalanced in nature, future work will also explore other data balancing techniques to help improve the results of our model.