WMD at S em E val-2020 Tasks 7 and 11: Assessing Humor and Propaganda Using Unsupervised Data Augmentation (2020)

Abstract: In this work, we combine the state-of-the-art BERT architecture with the semi-supervised learning technique UDA in order to exploit unlabeled raw data to assess humor and detect propaganda in the tasks 7 and 11 of the SemEval-2020 competition. The use of UDA shows promising results with a systematic improvement of the performances over the four different subtasks, and even outperforms supervised learning with the additional labels of the Funlines dataset.

## introduction in times when information is able to circulate freely via the internet with limited to non-existing quality control, organized groups and governments are using various tools of propaganda to spread missinformation and push their own agenda to achieve financial and political gains @xcite . with millions of press articles written every month, it became unrealistic for humans to perform any form of regulation. machine learning (ml) can help with the automatic processing of large volumes of data, but paradoxically also requires a significant amount of examples in order to apprehend high-level concepts such as humor or propaganda. more and more solutions to this problem are emerging, for example transfer learning @xcite which allows the use of already existing resources to increase the performance of a given task. it can be achieved by using a larger and closely related dataset, or simply by fine-tuning a pre-trained model. a recent neural architecture proposed for natural language processing in @xcite and achieving state-of-the-art results in numerous tasks is called bidirectional encoder representations from transformers (bert). one of the reasons for its popularity is the availability of models pre-trained with large resources in different languages. it is based on the combination of bidirectional long short-term memory (lstm) networks, described in @xcite and attention mechanisms, described in @xcite . both techniques have been discussed and considered as promising subjects of study in previous works @xcite on sentiment analysis during the semeval-2018 challenge. another solution to deal with the data collection is to exploit raw data with semi-supervised approaches. both humor assessment in task 7 @xcite and propaganda detection in task 11 @xcite have the advantage of a wide availability of samples in online press articles. by avoiding the use of labels, a lot more resources can be used through techniques like unsupervised data augmentation (uda) proposed in @xcite . the idea is to draw similarities between labeled and unlabeled data during a semi-supervised training, making the overwhelming millions of un-regulated articles into a strength as an exploitable resource. in this paper, we propose to combine the two solutions along with other practices with the objective of using the bare minimum of human-prepared resources while simultaneously improving the performances. the purpose of the task, described in @xcite is to estimate the funniness of micro-edited headlines, where a micro-edit is defined as one of the following word-wise replacements: entity to noun, noun to noun, and verb to verb. each member of a jury, consisting of at least five people, assigns a grade to the headlines. the grade is an integer from 0 to 3 where 0 means "not funny", 1 means "slightly funny", 2 means "moderately funny" and 3 means "funny". all the scores for a given modified headline are then averaged to obtain a unique real value. however, each of the individual scores are also available in the official dataset described in @xcite . the task itself is split into two subtasks to which we refer as 7-1 and 7-2. the first subtask is a regression based approach with the aim of directly predicting the averaged funniness of the edited headlines. two alternatives exist using classification, since the prediction should only take a small, finite number of values: the system could be trained to only predict the few possible scores, or even simply to learn the preferences of each judge. however, the number of judges can vary and the individual scores are ordered by value and not by person, making these two approaches less straightforward. we decided to stick to the regression as the task was thought. the second subtask is a ternary comparison between two modifications of the same headline in order to find the funniest. one micro-edit can be considered more fun, less fun or equally fun to another, but the last possibility was not considered for the evaluation. once again there are a few ways to tackle the problem: we chose the first proposition in order to focus on subtask 7-1, as it was advantageous for both subtasks. in addition, it allowed to considerably reduce the number of trainings, and so the overall computational resources required. ## system overview the submission is based on the unsupervised data augmentation (uda) technique described in @xcite applied to a pre-trained bert architecture @xcite in order to use external, unlabeled resources in addition to the official datasets. the pipeline is composed of the following steps, with more details for most of them in specific subsections: ## conclusion in this competition we decided to highlight a few practices that could benefit many domains at lower cost. the performances in the various tasks covered by this article have all been improved by the use of semi-supervised learning with uda, and in some cases even more than supervised learning with additional, closely related, external resources. the availability of systems pre-trained with large resources such as bert also enhanced the performances through transfer learning. the common point between these two techniques is to turn incomplete or unrelated resources into knowledge for the neural network architectures. the different propositions were discussed and most of them have room for improvements. the limited computational resources available constrained the experiments, as for example just 2.4% of the external datasets found were actually used. moreover, in an effort to keep the pipeline simple, a few steps discussed in @xcite were avoided but could be used. in general, more specific approaches of each task will be subject to further research.