SATL ab at S em E val-2022 Task 4: Trying to Detect Patronizing and Condescending Language with only Character and Word N-grams (2022)

Abstract: A logistic regression model only fed with character and word n-grams is proposed for the SemEval-2022 Task 4 on Patronizing and Condescending Language Detection (PCL). It obtained an average level of performance, well above the performance of a system that tries to guess without using any knowledge about the task, but much lower than the best teams. To facilitate the interpretation of the performance scores, the F1 measure, the best level of performance of a system that tries to guess without using any knowledge is calculated and used to correct the F1 scores in the manner of a Kappa. As the proposed model is very similar to the one that performed well on a task requiring to automatically identify hate speech and offensive content, this paper confirms the difficulty of PCL detection.

## introduction this paper presents the satlab's participation in semeval-2022 task 4: patronizing and condescending language detection. it is a very recent task that aims at identifying passages in texts in which a person is condescending to a vulnerable community @xcite . the task organizers have proposed to identify this type of discourse in paragraphs extracted from news stories published in english-speaking media. the patronizing and condescending language (pcl) detection task is part of the large family of tasks that aim at automatically identifying problematic language, whether in media or on social networks. examples include the detection of hate speech and offensive content, fakenews and hyperpartisan news @xcite . the pcl challenge, however, differs from these cases in one important characteristic. as pointed out by @xcite , using pcl is not always conscious and the intention of its author is often positive. nevertheless, pcl tends to indirectly demean the community in question and reinforce negative stereotypes about it. it is therefore important to develop procedures to identify it. recently, the satlab performed well on a task requiring to automatically identify hate speech and offensive content in tweets using a classical supervised algorithm only fed with character n-grams @xcite . it was especially effective for resource-poor languages (e.g., marathi). for english, its performance (f 1 = 0.782) was average, but nevertheless relatively close to the best team (f 1 = 0.831) that used complementary resources, pre-trained embeddings and deep learning approaches. these performances suggest that the proposed approach could be a interesting baseline to evaluate the benefits of these more complex approaches. the goal of satlab's participation in the pcl detection task was to evaluate whether this finding can be generalized to this new task. a priori, one might think that the performance of the system should be significantly worse here for two reasons. first, this task seems to be much more difficult because of the complex language means of expression on which it relies and the considerable amount of world knowledge and common sense reasoning required to understand this type of language (pérez-almendros et al., 2020). secondly, this task is only proposed for english, a language for which many additional resources are available. the rest of this document presents the task (see @xcite for more details), the proposed system and the performances reached in the challenge. ## task for this challenge, the organizers extracted paragraphs mentioning predefined vulnerable communities, such as homeless people, migrants or poor families, in news stories from various media in dif-ferent countries, all in english. each paragraph was annotated according to whether or not it contained one or more instances of pcl. annotators were also asked to identify the type of pcls present in a paragraph using seven categories. on this materials, the satlab participated in the two tasks proposed by the organizers. the first task consists in deciding whether a paragraph contains some kind of pcl or not. it is thus a classical binary decision task. the second task is a fine categorization task in seven positive and one negative categories. the seven positive pcl categories are: a) 1 unbalanced power relations, b) shallow solution, c) presupposition, d) authority voice, e) metaphor, f) compassion, g) the poorer, the merrier. it is important to note that this categorization is non-exclusive: a single instance can be an example of two, three, four and even five categories. i treated this second problem as task 1 and thus as seven independent binary problems, the predictions of the seven models being simply concatenated in the final submission. for this second task, the organizers also provided the precise position of the text areas that had been identified by the annotators as warranting the assignment of a given category. this information was not used. the dataset provided by the organizers to develop the systems consisted of 10,469 instances (but one, @xmath0, contained no text). it was highly unbalanced for task 1 with only 9.5% positive instances. the distribution was even more unbalanced in task 2 since it contained the same proportion of negative cases. the most frequent positive category (a) represented 6.84% of the total instances and the least frequent only 0.38%. for the system development phase, the organizers proposed a division of the paragraphs into a learning set (80%) and a development set. the test set used in the final evaluation of the systems, whose responses were therefore unknown, consisted of 3,832 paragraphs. the measure of effectiveness chosen by the organizers was the f1-score on the positive category for task 1 and the unweighted average of the f1-scores on the seven pcl categories for task 2. 1 the letters are used to identify these categories in the proposed system is adapted from the satlab's participation at hasoc 2021. it is based on the following components. ## results the system just described is identical for the two tasks, the only differences being in the parameters of the logistic regression which were optimized independently for each task and for each target category in task 2. for the two final submissions, as can be seen in this table, the parameter values are often very different in the two versions while the performance of the models was very close. tables 3 and 4 show the performance of the better of the two submissions on the test set, as provided by the organizers. for task 1, submission 1 performed better while submission 2 performed better for task 2. the differences between the two submissions for the two tasks are however very small, less than 0.005. these two tables also give the performance of the first team in the challenge as well as that of the roberta-based baseline proposed by the organizers. the proposed system outperformed the robertabased baseline in task 2, but not in task 1. it ranked in the middle of the participants at best, very far from the challenge winners. it is unfortunate (for the satlab at least) that the task was not proposed in languages with less resources and precomputed embeddings, making the use of deep learning much more easier. it is interesting to note that the performances obtained on the test set is only slightly lower than those obtained when optimizing the parameters by a cross-validation procedure on the training set. for task 1, this performance on the training set was only 0.03 higher and, on task 2, only 0.008. this suggests that the optimization did not produce an overfit. the comparison of the system's performance to that of the best team for the seven categories ( while the f1-score has become the standard of evaluation for nlp categorization tasks, its interpretation is not obvious. one may wonder whether a mean f1 of 0.256 (task 2) represents a prediction of at least an acceptable value. a priori, this does not seem to be the case. on the other hand, the large differences in the frequency of the categories and the imbalance in favor of the negative category may explain the relatively low f1. one way to answer this question more objectively is to determine the best level of performance a system that tries to guess without using any knowledge about the task (best guess) can expect to achieve. ansgar grüne in his blog post available at https://inside. getyourguide.com/blog/2020/9/30/ what-makes-a-good-f1-score shows foot_0 that in the case of a binary task this is the f1 obtained by a system that always predicts the positive category. if q is the actual proportion of instances belonging to this positive category, this value can also be obtained by submitting the responses of a system that always predicts the positive category, the approach i used here since the challenge participants still ignore the proportion of each category in the test set. in the present challenge, this f1 best guess is 0.153 for task 1 and 0.041 for task 2. a system that would not exceed these values therefore does no better than a system that always predicts the positive category. satlab's performance is five times better than this baseline for task 2, but less than three times better for task 1. in absolute value, the gain compared to the best guess is 0.274 for task 1 and 0.205 for task 2. so the n-grams bring some information, but it is obvious that it is not enough to perform well in this task. this best guest can also be used to correct participants' scores on the task using a formula derived from the kappa coefficient proposed by @xcite to adjust the degree of agreement between two judges for agreement due to chance alone. the formula for this correction is: 1 and 2 show the normalized performance of all teams on both tasks. the best system for task 1 achieved a corrected f1 slightly below 0.60. it thus performed 60 % of the way between doing nothing (i.e., the best guess) and a perfect performance. this again highlights the complexity of the task. the normalized f1 of the satlab is only slightly higher than 0.30. for task 2, all the corrected performances are even worse. figure 2 shows also that the satlab normalized f1 is quite far from the team directly ahead of it. ## conclusion the satlab's proposed system for semeval-2022 task 4: patronizing and condescending language detection relies solely on the character and word ngrams present in the paragraphs to be categorized. it does not use any additional data and employs a classical supervised learning procedure (i.e., logistic regression). it obtained an average level of performance, well above the performance of a system that tries to guess without using any knowledge about the task, but much lower than the best teams. compared to the performance obtained in the hasoc task @xcite and in the detection of hyperpartisan news articles @xcite , this system is clearly further from these best teams. these results confirm, if it were necessary, the much greater difficulty of pcl detection compared to hate speech and offensive content identification @xcite . they thus suggest, unless other teams using a similar approach were more successful, that the use of much more complex approaches is essential for the pcl task.