A pplica AI at S em E val-2020 Task 11: On R o BERT a- CRF , Span CLS and Whether Self-Training Helps Them (2020)

Abstract: This paper presents the winning system for the propaganda Technique Classification (TC) task and the second-placed system for the propaganda Span Identification (SI) task. The purpose of TC task was to identify an applied propaganda technique given propaganda text fragment. The goal of SI task was to find specific text fragments which contain at least one propaganda technique. Both of the developed solutions used semi-supervised learning technique of self-training. Interestingly, although CRF is barely used with transformer-based language models, the SI task was approached with RoBERTa-CRF architecture. An ensemble of RoBERTa-based models was proposed for the TC task, with one of them making use of Span CLS layers we introduce in the present paper. In addition to describing the submitted systems, an impact of architectural decisions and training schemes is investigated along with remarks regarding training models of the same or better quality with lower computational budget. Finally, the results of error analysis are presented.

## introduction the idea of fine-grained propaganda detection was introduced by @xcite , whose intention was to facilitate research on this topic by publishing a corpus with detailed annotations of high reliability. there was a chance to propose nlp systems solving this task automatically as a part of this year's semeval series. it was expected to detect all fragments of news articles that contain propaganda techniques, and to identify the exact type of used technique @xcite . the authors decided to evaluate technique classification (tc) and span identification (si) tasks separately. the purpose of the tc task was to identify an applied propaganda technique given the propaganda text fragment. in contrast, the goal of the si task was to find specific text fragments that contain at least one propaganda technique. this paper presents the winning system for the propaganda technique classification task and the second-placed system for the propaganda span identification task. ## systems description systems proposed for both si and tc tasks were based on roberta model @xcite with task-specific modifications and training schemes applied. the central motif behind our submissions is a commonly used semi-supervised learning technique of self-training @xcite @xcite , sometimes referred to as incremental semi-supervised training @xcite or selflearning @xcite . in general, these terms stand for a process of training an initial model on a manually annotated dataset first and using it to further extend the train set by automatically annotating other dataset. usually, only a selected subset of auto-annotated data is used, however neither selection of high-confidence examples nor loss correction for noisy annotations is performed in our case. this is why it can be considered a simplification of mainstream approaches-the naïve self-training. figure 1: self-training stands for a process of training an initial model on manually annotated dataset first and using it to further extend train by means of annotating other dataset automatically. ## ablation studies since different random initialization or data order can result in considerably higher scores, 3 models with different random seeds were trained for the purposes of ablation studies. in the case of the si task, results were evaluated on the original development set. in contrast, in the case of tc, where fewer data points are available, we decided to use cross-validation instead. (1) (6) 72.3 ± 1.7 (1) (2) 72.9 ± 1.8 (3) (5) 73.6 ± 1.5 (1) (5) (8) ## error analysis in addition to providing an overview of problematic classes, the question of which shallow features influence score and worsen the results was addressed. this problem was analyzed in a no-box manner, as proposed by @xcite . the main idea is to create two dataset subsets for each feature considered (one for data points with the feature present and one for data points without the feature), rank subsets by per-item scores, and use mann-whitney rank u (mann and whitney, 1947) to determine whether there is a non-accidental difference between subsets. a low p-value indicates that feature reduces the evaluation score of the model. ## discussion and summary the winning system for the propaganda technique classification (tc) task and the second-placed system for the propaganda span identification (si) task has been described. both of the developed solutions used a semi-supervised learning technique of self-training. although crf is barely used with transformerbased language models, the si task was approached with roberta-crf architecture. an ensemble of roberta-based models has been proposed for the tc task, with one of them making use of span cls layers we introduce in the present paper. analysis conducted afterward can be applied in a rather straightforward manner to further improve the scores for both si and tc tasks. it is because some of the decisions we have made given lack of or uncertain information, during the post-hoc inquiry turned out to be sub-optimal. these include the proportion of data from self-training in the si task, and the possibility of providing a better ensemble in the case of tc. the ablation studies conducted, however, have some limitations. the same subset of openwebtext was used in experiments conducted within one self-training iteration. this means a random seed did not impact which sentences were used during the first, second, and third self-training phase, and in each, we were manipulating only the data order. moreover, an analysis we reported was limited to few hyperparameter combinations and no extensive hyperparameter space search was performed. finally, only one and a rather simple method of cost-sensitive re-weighting was tested, and there is a great chance it was sub-optimal. it would be interesting to investigate other schemes, such as the one proposed by @xcite . the error analysis revealed propaganda techniques commonly confused in tc task, and the techniques we were unable to detect effectively within the si input articles. in addition to providing an overview of problematic classes, the question of which shallow features influence score and worsen the results was addressed. a few of these were identified and our remarks can be used to slightly improve results on si task with simple post-processing. this is not the case for tc task, where one is unable to propose how to improve the final results with our findings. an interesting future research direction seems to be the application of the crf layer and span cls to transformer-based language models when dealing with other tasks outside the propaganda detection problem. these may include named entity recognition in the case of roberta-crf, and an aspectbased sentiment analysis that can be viewed through the lens of span classification with span cls we proposed.