BERT ’s The Word : Sarcasm Target Detection using BERT (2021)

Abstract: In 2019, the Australasian Language Technology Association (ALTA) organised a shared task to detect the target of sarcastic comments posted on social media. However, there were no winners as it proved to be a difficult task. In this work, we revisit the task posted by ALTA by using transformers, specifically BERT, given the current success of the transformer-based model in various NLP tasks. We conducted our experiments on two BERT models (TD-BERT and BERT-AEN). We evaluated our model on the data set provided by ALTA (Reddit) and two additional data sets: ‘book snippets’ and ‘Tweets’. Our results show that our proposed method achieves a 15.2

% improvement from the current state-of-the-art system on the Reddit data set and 4

% improvement on Tweets.

## introduction sarcasm is a remark made by a certain person to ridicule or hurt another person's feelings @xcite . a unique property of sarcasm lies in the way words are used. the result digresses from the conventional word order and alters the meaning of the whole sentence @xcite . this very aspect also makes it very challenging to detect in a text. there has been a large number of studies that looked at automating sarcasm detection @xcite however there is less work done in identifying and extracting the target of sarcasm from the text. the problem of sarcasm target detection was originally coined by @xcite . the target of sarcasm is defined as an entity or a situation that is being ridiculed in a sarcastic text. the task of sarcasm target identification is to extract the subset of words that indicate the target of ridicule for a given sarcastic sentence. iden-tifying the target of ridicule can improve the detection of cyber-bullying and hate speech targeted towards minority communities such as people of colour, the lgbtq+ community and others @xcite . however, this task is particularly challenging because of the following factors: when the sarcasm target does is not present or it is unclear, it is marked as outside. there have been various attempts to improve the performance of sarcasm target detection @xcite @xcite such as through the use of deep-learning models and rulebased methods. given the successes of transformers, particularly bert @xcite , in nlp tasks such as aspect-based sentiment analysis (absa) @xcite , and summarisation @xcite , we hypothesise that bert-like models may also be good at this task. our experiments show that bert models outperform the current state-of-the-art system on our reddit data by 23.4% and give a 3% increase on our tweets data. sarcasm detection (i.e., distinguishing sarcastic texts from non-sarcastic texts) is widely studied in computational linguistics. eke et al. (2020); @xcite have presented a comprehensive overview in this field. to summarise, there are several approaches: semi-supervised learning @xcite @xcite , deep learning @xcite @xcite @xcite , and lately, with the advancement of transformers, researchers have used transformers to distinguish sarcastic texts from non-sarcastic texts @xcite @xcite . little work has been done to detect the target of sarcasm-in spite of the australasian language technology association (alta) organising a shared task challenge to encourage researchers to tackle this problem @xcite . the approaches taken in prior work include a rule-based system that looks at part-of-speech (pos) @xcite , deep learning @xcite , and an ensemble of machine learning and deep learning classifiers @xcite . to the best of our knowledge, there is no research exploring the use of bert models for sarcasm target detection, but we note that @xcite used embeddings from bert, but not the transformer. bert has shown success in aspect-extraction (ae) within absa tasks @xcite . our task is similar to aspect-extraction, but in our task, the targets may be absent from the given text, which makes it challenging. we consider two models from the absa literature for our experiments: td-bert @xcite and bert-aen @xcite . initially our choice was guided by the fact that bert-aen works well with a smaller data set @xcite and we chose td-bert as our second option as we have noticed similiar performance to bert-aen with a much simpler architecture by just extending it to include the aspect. our choice to use these models is further motivated by the availability of a public repository foot_0 with standard implementations using pytorch foot_1 (and therefore ease of reproducibility of our experiments). we also note that @xcite used the absa-pytorch repository as the basis for reproducing the results of absa approaches. ## data set we consider the data sets released by @xcite and @xcite . the sets consist of three different kinds of data: tweets (tweets), book snippets (books) and also reddit posts (reddit). ## methodology when predicting the target of sarcasm, like others ( patro et al. (2019) and parameswaran et al. (2021) ), we formulated the problem as a sequence labelling problem. we first represent a sarcastic sentence (s) as a sequence of words {w 1 , w 2 , . . . , w n }. we then append each word with a label indicating if it is a potential target. consider the following example, "he is a good cook" with 'he' as the potential target. the sentence is represented as {'he', 'is', 'a', 'good', 'cook'} and its label sequence is {'he' t , 'is' ø , 'a' ø , 'good' ø , 'cook' ø }, where t is a potential target and ø is not. we feed both of these sequences as training input for our two bert models (td-bert and bert-aen). ## experimental setup we ran our experiments on an intel xeon e5-2690 v3 @ 2.00 ghz cpu with an nvidia tesla t4 (cuda version 11.2, driver version 460.73) running on debian 10 (buster). we forked commit 9acab7e of absa-pytorch and modified it to suit our task. our source code can be found on our github page. 3 we consider the current state-of-the-art models from @xcite and @xcite as baselines for this task, that we called baseline 1 and baseline 2, respectively. we implemented the approaches of each author and compared our results to theirs. a one-way anova showed no statistically significant difference at the 0.05 level, providing confidence in our implementations of their approaches. we use dice score to measure the accuracy as it has been used in past works @xcite . all the results reported used five-fold cross-validation. ## results we report our results in surprising to us, td-bert performs best in all our tasks. we believe that the simple method of just incorporating a target's position helped the model to better understand the context of the sentences. although the multiple attention mechanisms in bert-aen could be expected to outperform td-bert, it is unclear why this is not the case in our experimental results. one possible explanation is that the data set is small and the model has learned more noise. we leave for future work the exploration using larger data sets. ## conclusion we presented our approach to sarcasm target detection. we used two different publicly available bert models: td-bert and bert-aen, and fine-tuned them to the task using extra examples of data from the domains we explore. finally, we evaluated our models on three publicly available data sets: tweets, books, and reddit. our empirical results show that this approach outperforms the current state-of-the-art on all three data sets. despite setting a strong baseline, we believe that there remains plenty of room for further work in this area. firstly, we conducted our experiments on a small data set, therefore our proposed methodology needs to be tested when applied to a larger data set. secondly, the use of user profiles, user history, context, and so on, might improve performance for reddit and tweets as detecting sarcasm is a difficult task and it requires more than content alone. some users are more prone to sarcastic quips than others, and that could be mined from a person's past posts @xcite .