S a R o C o: Detecting Satire in a Novel R omanian Corpus of News Articles (2021)

Abstract: In this work, we introduce a corpus for satire detection in Romanian news. We gathered 55,608 public news articles from multiple real and satirical news sources, composing one of the largest corpora for satire detection regardless of language and the only one for the Romanian language. We provide an official split of the text samples, such that training news articles belong to different sources than test news articles, thus ensuring that models do not achieve high performance simply due to overfitting. We conduct experiments with two state-of-the-art deep neural models, resulting in a set of strong baselines for our novel corpus. Our results show that the machine-level accuracy for satire detection in Romanian is quite low (under 73% on the test set) compared to the human-level accuracy (87%), leaving enough room for improvement in future research.

## introduction according to its definition in the cambridge dictionary, satire is "a humorous way of criticizing people or ideas" foot_0 . news satire employs this mechanism in the form of seemingly legitimate journalistic reporting, with the intention of ridiculing public figures, politics or contemporary events @xcite @xcite . although the articles pertaining to this genre contain fictionalized stories, the intent is not to mislead the public into thinking that the discussed subjects are real. on the contrary, satirical news articles are supposed to reveal their nature by the writing style and comedic devices employed, such as irony, parody or exaggeration. thus, the intention behind the writing differentiates satirical news @xcite from fake news @xcite @xcite . however, in some rare cases, the real intent might be deeply buried in the complex irony and subtleties of news satire @xcite , which has the effect of fiction being deemed as factual @xcite . even so, there is a clear distinction between satirical and fake news. in fake news, the intent is to deceive the readers in thinking that the news is real, while presenting fake facts to influence the readers' opinion. since our study is focused on satire detection, we consider discussing research on fake news detection as being out of our scope. at the same time, we acknowledge the growing importance of detecting fake news and the fact that an accurate differentiation of satirical from legitimate journalistic reports might be seen as a starting point in controlling the spread of deceptive news @xcite . satire detection is an important task that could be addressed prior to the development of conversational systems and robots that interact with humans. certainly, the importance of understanding satirical (funny, ridiculous or ironical) text becomes obvious when we consider a scenario in which a robot performs a dangerous action because it takes a satirical comment of the user too literally. given the relevance of the task for the natural language processing community, satire detection has already been investigated in several well-studied languages such as arabic @xcite , english @xcite @xcite @xcite , german @xcite , spanish @xcite and turkish (toc ¸oglu and onan, 2019). through the definition of satire, the satire detection task is tightly connected to irony and sarcasm detection. these tasks strengthen or broaden the language variety with languages such as ara-data set language #articles regular satirical total @xcite english 4,000 233 4,233 @xcite english 1,705 1,706 3,411 @xcite english 10,921 1,225 12,146 (ionescu and chifu, 2021) french 5,648 5,922 11,570 @xcite english 6,000 4,000 10,000 @xcite french 2,841 2,841 5,682 @xcite german 320,219 9,643 329,862 @xcite english 1,272 393 1,665 @xcite arabic 3,185 3,710 6,895 (toc ¸oglu and onan, 2019) turkish 1,000 1,000 2,000 @xcite in this work, we introduce saroco foot_1 , the satire detection romanian corpus, which comprises 55,608 news articles collected from various sources. to the best of our knowledge, this is the first and only data set for the study of romanian satirical news. furthermore, saroco is also one of the largest data sets for satirical news detection, being surpassed only two corpora, one for english @xcite and one for german @xcite . however, our corpus contains the largest collection of satirical news articles (over 27,000). these facts are confirmed by the comparative statistics presented in along with the novel data set, we include two strong deep learning methods to be used as baselines in future works. the first method is based on low-level features learned by a character-level convolutional neural network @xcite , while the second method employs high-level semantic features learned by the romanian version of bert @xcite . the gap between the human-level performance and that of the deep learning baselines indicates that there is enough room for improvement left for future studies. we make our corpus and baselines available online for ## corpus saroco gathers both satirical and non-satirical news from some of the most popular romanian news websites. the collected news samples were found in the public web domain, i.e. access is provided for free without requiring any subscription to the publication sources. the entire corpus consists of 55,608 samples (27,628 satirical samples and 27,980 non-satirical samples), having more than 28 million tokens in total, as illustrated in we provide an official split for our corpus, such that all future studies will use the same training, val after crawling satirical and non-satirical news samples, our first aim was to prevent discrimination based on named entities. the satirical character of an article should be inferred from the language use rather than specific clues, such as named entities. for example, certain sources of news satire show preference towards mocking politicians from a specific political party, and an automated system might erroneously label a news article about a member of the respective party as satirical simply based on the presence of the named entity. furthermore, we even noticed that some romanian politicians have certain mocking nicknames assigned in satirical news. in order to eliminate named entities, we followed a similar approach as the one used for the moroco @xcite data set. thus, all the identified named entities are replaced with the special token @xmath0. besides eliminating named entities, we also substituted all whitespace characters with space and replaced multiple consecutive spaces with a single space. a set of processed satirical and regular headlines are shown in ## baselines fine-tuned ro-bert. our first baseline consists of a fine-tuned romanian bert @xcite , which follows the same transformer-based model architecture as the original bert @xcite . according to @xcite , the romanian bert (ro-bert) attains better results than the multilingual bert on a range of tasks. we therefore assume that the romanian bert should represent a stronger baseline for our romanian corpus. we use the ro-bert encoder to encode each text sequence into a list of token ids. the tokens are further processed by the model, obtaining the corresponding 768-dimensional embeddings. at this point, we add a global average pooling layer to obtain a continuous bag-of-words (cbow) representation for each sequence of text, followed by a softmax output layer with two neural units, each predicting the probability for one category, either non-satirical or satirical. to obtain the final class label for a text sample, we apply argmax on the two probabilities. we fine-tune the whole model for 10 epochs on mini-batches of 32 samples, using the adam with decoupled weight decay (adamw) optimizer @xcite , with a learning rate of 10 -7 and the default value for . character-level cnn. the second baseline model considered in the experiments is a convolutional neural network (cnn) that operates at the character level @xcite . we set the input size to 1,000 characters. after the input layer, we add an embedding layer to encode each character into a vector of 128 components. the optimal architecture for the task at hand proved to be composed of three convolutional (conv) blocks, each having a conv layer with 64 filters applied at stride 1, followed by scaled exponential linear unit (selu) activation. from the first block to the third block, the convolutional kernel sizes are 5, 3 and 1, respectively. max-pooling with a filter size of 3 is applied after each conv layer. after each conv block, we insert a squeeze-and-excitation block with the reduction ratio set to @xmath1, following @xcite . to prevent overfitting, we use batch normalization and alpha dropout @xcite with a dropout rate of 0.5. the final prediction layer is composed of two neural units, one for each class (i.e. legitimate and satirical), with softmax activation. we use the nesterov-accelerated adaptive moment estimation (nadam) optimizer @xcite with a learning rate of 2 • 10 -4 , training the network for 50 epochs on mini-batches of 128 samples. ## experiments evaluation. we conducted binary classification experiments on saroco, predicting if a given piece of text is either satirical or non-satirical. as evaluation metrics, we employ the precision and recall for each of the two classes. we also combine these scores through the macro f 1 and micro f 1 (accuracy) measures. results. in legal terms "asasinat" "assasinated" "l-au denunt ¸at pe autorul atacului" "denounced the perpetrator" weather "temperatura în timpul nopt ¸ii a scȃzut" "the temperature has dropped during the night" political terms "scrutinul prezident ¸ial" "presidential election" "prefectura informeazȃ cȃ" "prefecture informs that" in tables 6 and 7 , we present a few examples of interesting patterns considered relevant for predicting satire versus regular news, respectively. a broad range of constructions covering a great variety of styles and significant words are underlined via grad-cam in the satirical news samples. the network seems to pick up obvious clues such as slang, insults and popular sayings rather than more subtle indicatives of satire, including irony or exaggeration. at the same time, for the real news in saroco, there are fewer categories of predictive patterns. in general, the cnn deems formal, standard news expressions as relevant for regular news. these patterns vary across topics and domains. the cnn also finds that the presence of numbers and statistical clues is indicative for non-satirical content, which is consistent with the observations of @xcite . our analysis reveals that the discriminative features are appropriate for satire detection, showing that our corpus is indeed suitable for the considered task. deep models versus humans. given 100 satirical and 100 non-satirical news headlines (titles) randomly sampled from the saroco test set, we asked ten romanian human annotators to label each sample as satirical or non-satirical. we evaluated the deep learning methods on the same subset of 200 samples, reporting the results in if a satire-specific pattern is not identified, the respective sample is labeled as regular, increasing the recall for the non-satirical class. although humans and machine seem to share the same way of thinking, there is a considerable performance gap in satire detection between humans and machines. indeed, the average accuracy of our ten human annotators is around 87%, while the state-of-the-art deep learning models do not surpass 68% on the same news headlines. even on full news articles (see ## conclusion in this work, we presented saroco, a novel data set containing satirical and non-satirical news samples. to the best of our knowledge, saroco is the only corpus for romanian satire detection and one of the largest corpora regardless of language. we trained two state-of-the-art neural models as baselines for future research on our novel corpus. we also compared the performance of the neural models with the averaged performance of ten human annotators, showing that the neural models lag far behind the human-level performance. our discriminative feature analysis confirms the limitations of state-of-the-art neural models in detecting satire. although we selected a set of strong models from the recent literature as baselines for saroco, significant future research is necessary to close the gap with respect to the human-level satire detection performance. designing models to pick up irony or exaggerations could pave the way towards closing this gap in future work.