M ar S an at S em E val-2022 Task 6: i S arcasm Detection via T5 and Sequence Learners (2022)

Abstract: The paper describes SemEval-2022’s shared task “Intended Sarcasm Detection in English and Arabic.” This task includes English and Arabic tweets with sarcasm and non-sarcasm samples and irony speech labels. The first two subtasks predict whether a text is sarcastic and the ironic category the sarcasm sample belongs to. The third one is to find the sarcastic sample from its non-sarcastic paraphrase. Deep neural networks have recently achieved highly competitive performance in many tasks. Combining deep learning with language models has also resulted in acceptable accuracy. Inspired by this, we propose a novel deep learning model on top of language models. On top of T5, this architecture uses an encoder module of the transformer, followed by LSTM and attention to utilizing past and future information, concentrating on informative tokens. Due to the success of the proposed model, we used the same architecture with a few modifications to the output layer in all three subtasks.

## introduction sarcasm is a sophisticated form of expression that implicitly conveys the content of a sentence. automated sarcasm detection focuses mainly on the lexical, syntactic, and semantic levels of text analysis @xcite . natural language understanding(nlu), dialogue systems, and text mining can benefit from sarcasm detection. arguably, the most challenging part of sarcasm is its rarity, infrequency, difficulty in detecting, and ambiguity in meaning. sarcasm, for instance, can imply a negative meaning with the use of positive words. for example, "taxes are just the best, and i cannot wait to pay more " a sarcastic sentence that uses positive words but carries a negative meaning of "i dislike paying taxes." as a result, detecting sarcasm poses a * equal contribution. listing order is random. a sarcasm detection algorithm goes beyond sentiment analysis, and instead of looking at sentiment in a sample, it focuses on sarcasm. the purpose of this field is to identify whether a given text is sarcastic or not. recently, it has been shown that neural language models trained on unstructured text can implicitly store and retrieve knowledge. studies have revealed that the text-to-text transfer transformer (t5) architecture @xcite can achieve high performance for various nlp applications. an essential step in creating nlp models is choosing an appropriate embedding vector. in this research, the t5 and multilingual t5 (mt5) @xcite encoder module for english and arabic was implemented, respectively. our task is comprised of three subtasks: there are insufficient instances in the dataset for this task, particularly the low number of sarcastic instances which make deep learning models unsuitable for extracting text features. fine-tunning the language model on two large open-source datasets in english and arabic is the key to solving this challenge. a fine-tuning step may provide the model with valuable awareness of task context. for the english task, the dataset of 4,484 tweets named isarcasm (oprea and magdy, 2019) was selected, while for the arabic task, the dataset of 10,547 tweets farha and magdy (2020) was chosen. this research uses the t5 language model as a word embedding layer to design the sarcasm detection model. after t5, the encoder module of the transformer, the long short term memory (lstm) @xcite layer, and scaled dot-product attention @xcite were implemented sequentially for extracting the informative knowledge. lastly, the model was fed an output from max-pooling. the same model has implemented all three subtasks, changing the input and output layer types. to participate in task 6 of semeval-2022 abu farha et al. (2022), we submitted the results of 5 subtasks. the ranking of our model in subtask a ranked 7th in english and 31st foot_0 in arabic. our model placed 9th out of 22 teams in subtask b. in subtask c, our model placed 7th out of 16 teams in english and 3rd out of 13 teams in arabic. our code is available at github foot_1 for researchers. the remaining of this paper is organized as follows: section 2 reviews related work. section 3 describes both tasks and provided dataset. section 4 presents the theoretical background of the proposed neural model. implementation details are provided in section 5, while experiments and results are presented in section 6. section 7 presents both quantitative and qualitative error analysis. section 8 contains paper conclusions. ## background it is pointed out in @xcite that sarcasm can alter the meaning of a phrase, making opinion analysis error-prone. subsequently, a model by bert and aspect-based sentiment analysis is employed to address the issue. based on the context dialogue sequence, this system can determine whether a response is sarcastic or not, with an f1score of 0.73 on twitter. in @xcite , as in javdan et al. (2020), the researcher used two reddit and twitter datasets and applied roberta-large to detect sarcasm in both datasets. @xcite investigates lexical (like uni-grams and dictionary-based) and pragmatic (like positive or negative emotions) features and compares the performance of machine learning techniques and human judges. since the meaning of sarcasm differs for each individual and may lead to misunderstandings in everyday communications, @xcite claims that user embedding can encode the stylistic and personality attributes of users, and combined with convolutional neural networks (cnns) @xcite that extracts localized information, the results are reasonable. @xcite introduce a binary classification deep learning model for sarcasm detection. @xcite bi-lstm and multi-head attention mechanism to obtain sentence embedding and classify input text with a softmax layer. roberta network architecture is utilized in @xcite to map words onto a rich embedding space efficiently. to improve roberta performance and capture temporal reliance information, use the rcnn network. ## task description task 6 of semeval-2022 presents a sarcasm detection dataset in english and arabic. statistical information on the number of samples in the train, dev, and test data is shown in ## system overview contextual features can be extracted very efficiently with pre-trained language models. in nlp tasks, t5 has proven to be an efficient encoderdecoder framework. using the encoder layers within the t5 language model, we can fine-tune pre-trained encoder-decoder t5 models efficiently for classification and regression tasks. pre-trained models ease fine-tuning downstream tasks by reducing the reliance on large task-specific training datasets. their results can be further enhanced if other deep learning architectures, such as lstm, cnn, and attention, are applied on top of them @xcite . the framework we developed uses the encoder module of t5 and transformer, bi-lstm layer, and scaled dot-product attention to determine whether a text is sarcastic. also, we apply the same architecture to subtasks b and c but make some changes to the output and input layers, respectively. the proposed model architecture is shown in figure3. as part of subtasks a and b, the model gets a sequence of @xmath0, s 2 , s 3 , ..., s n }, where s n is the nth token of input text. in subtask c, the inputs are sequence of @xmath1, p 2 , p 3 , ..., p i }, and @xmath2, q 2 , q 3 , ..., q j }, where p j is the jth token of the first text and q i is the ith token of the second. the input sequence @xmath3, < /s >, q} are the final inputs for subtask c. subtask a estimates a sarcastic label based on the probability distribution of p r (y|s). within subtask b, the model estimates the probability distribution p r (y|s) for each category of ironic speech. finally, subtask c estimates the probability distribution p r (y|p, q), predicting whether p indicates sarcasm or q. ## experimental setup in this section, we first describe data pre-processing. following that, we discuss the implementation details. pre-processing data analysis revealed some samples containing urls and user mentions. due to the possibility of these items confusing the model in identifying the correct label, these items were removed in the pre-processing phase. tokens such as punctuation and emojis were not removed since they could be excellent indicators of sarcasm. implementation details pytorch was used to implement the model, and we trained it on nvidia v100 gpus. each subtask's hyper-parameters were tuned using the development set. the adamw optimizer with the learning rate of 2e -5 is used to train the network using the back-propagation algorithm. a training method of early stopping with the patience of 5 and monitoring validation loss of sarcastic class in min mode is used. as for regularization, we evaluate the effect of the dropout on the transformer encoder and the bi-lstm layer; the model has a better performance when the dropout rate is set as 0.2, 0.3 respectively. each subtask has a batch size of 32. the number of attention heads in the transformer encoder is eight, and the hidden size in the position-wise feedforward layer is 2048. one layer of bi-lstm with 128 lstm units was used. over the first fully connected layer, we applied tanh, and the output size of this layer was 256. in the case of data imbalance, the cross-entropy loss can be used with class weights. the maximum length used in t5 and mt5 tokenizers is 100. other parameters are randomly initialized. ## results this section reviews the different baselines and compares them with the proposed model. due to the success of language models in recent years, we have evaluated different language models to select the most appropriate language model. since there was no official test data at the evaluation phase, experiments were conducted on the dev set to select the best architecture. finally, the performance of the models on the test data is also evaluated. ## error and performance analysis in this section, we analyze the performance of several components of our system. this section will examine our model's ability to correctly label samples in sarcastic or non-sarcastic contexts. ## conclusion we implement a novel deep learning approach based on language models. the last hidden state of t5 is used as an embedding layer in this architecture. on top of this layer, a bidirectional lstm is used to extract future and past contexts as representations of the input text. lstm output is processed using an attention mechanism, which focuses more on the valuable tokens to predict. the main challenge in this study was that there were not enough samples in the dataset. to solve this, we fine-tuned the t5 language model with other large open-source datasets to have a language model that had a pre-awareness of the task and a higher accuracy. this fine-tuned language model was then used as an embedding representation in our deep architecture. to evaluate the performance of the developed model and find the best language model, many experiments were conducted. the experimental results show that the t5 language model covers a reasonable range of results and is most appropriate for our architecture. the same architecture was used in all three subtasks, as mentioned earlier. due to the multi-label nature of subtask b, six separate classifiers were used instead of one to produce the output. we identified specific sarcasm challenges through error analysis, creating immediate future tasks. as a final point, our architecture has already been created for english and arabic, but it could be easily extended to other languages.