From Ground Trust to Truth: Disparities in Offensive Language Judgments on Contemporary K orean Political Discourse (2025)

Abstract: Although offensive language continually evolves over time, even recent studies using LLMs have predominantly relied on outdated datasets and rarely evaluated the generalization ability on unseen texts. In this study, we constructed a large-scale dataset of contemporary political discourse and employed three refined judgments in the absence of ground truth. Each judgment reflects a representative offensive language detection method and is carefully designed for optimal conditions. We identified distinct patterns for each judgment and demonstrated tendencies of label agreement using a leave-one-out strategy. By establishing pseudo-labels as ground trust for quantitative performance assessment, we observed that a strategically designed single prompting achieves comparable performance to more resource-intensive methods. This suggests a feasible approach applicable in real-world settings with inherent constraints.

## introduction offensive language has emerged as a persistent linguistic issue in online discourse, broadly encompassing expressions intended to insult, demean, or ridicule others foot_0 @xcite . this concern is particularly evident in social media comments, where users articulate and exchange diverse opinions @xcite . the form and content of such language frequently depend on the underlying factual context @xcite , and they are recognized as key factors in intensifying social tensions and driving the polarization of public opinions @xcite . in this study, we focus on korean online news platforms as a forum for public discourse in the context of rapidly shifting political dynamics @xcite . most prior research on offensive language in korean has relied on outdated datasets, with the latest collected in early 2022, limiting their ability to capture recent political developments and emerging patterns of social conflict @xcite @xcite @xcite . to advance the field, we constructed a new dataset by curating political news articles and user comments posted throughout 2024 on the most widely used news platform, ensuring it reflects the evolving sociopolitical landscape @xcite . the news articles are categorized into six topics: presidential office, national assembly / political parties, north korea, administration, national defense / foreign affairs, and general politics. through the fine-grained framework, we constructed the politicalk.o dataset (political comments for korean offensive language), comprising 114,000 articles and 9.28 million user comments. since the collected comments contained no ground truth for offensiveness, we employed a diverse set of established offensive language detection methods within a carefully designed framework to assign predicted labels to each comment. we introduce three main judgments foot_1 : supervised ensemble judgment (sej), promptvariants ensemble judgment (pej), and multidebate reasoning judgment (mrj), which are outlined in figure 1 . first, sej utilizes five existing offensive language datasets to fine-tune plms to an optimal configuration, combining their predictions through majority voting. despite the datasets being outdated, we intended to maximize the utilization figure 1 : we adopt three distinct judgments for offensive language detection: supervised ensemble judgment (sej) on the left, prompt-variants ensemble judgments (pej) in the center, and multi-debate reasoning judgment (mrj) on the right. each constitutes a distinct approach to label inference, characterized by its methodology for decision guidance and aggregation, tailored to the newly collected comments from the politicalk.o . we finally derived offensive labels for the unseen comments across three judgments with optimized configurations. although numerous studies have explored offensive language detection, their applicability to emerging data remains underexplored. to fill this gap, we conducted a large-scale analysis on the politicalk.o by implementing judgments tailored to the current sociolinguistic landscape. this framework enabled us to examine offensive tendencies of each judgment, identify potential shared decision criteria, and assess label consistency when certain judgments were excluded. through this analysis, we obtained a granular understanding of how offensive language is perceived in unseen comments from diverse perspectives. furthermore, in the absence of ground truth labels for the newly collected comments, we evaluated performance of each judgment through pseudo-labeling @xcite @xcite . by treating the aggregated judgments as a ground trust † , we conducted an analytic evaluation of the three judgments and their respective components. we then examined how combinations of prior datasets and prompt variants yielded reliable offensive language detection results by leveraging proxy ground trust † even without human annotations. our analysis revealed practical scenarios where a single prompt achieves comparable performance to more resource-intensive methods, highlighting practical approaches for real-world use cases. to facilitate future research in this aspect, we release our dataset and the labels generated by each judgment as opensource resources foot_2 . ## method 3.1 dataset construction to capture the dynamics of recent political discourse, we constructed politicalk.o from naver foot_3 , south korea's largest news platform, covering all political news articles published in 2024. the dataset includes 114,000 articles and 9.28 million user comments, along with article summaries and comment threads. the detailed statistics of the dataset are provided in appendix a. ## exploratory analysis of offensiveness across judgments we employed the three judgment methods-sej, pej, and mrj-on the entire set of comments in the politicalk.o to obtain corresponding labels. to examine detection tendencies and potential biases, we first compared the label distributions generated 7 the performance of the three llms on each of the prior datasets is reported in appendix b.3. while the main analysis focuses on a single model trillion, pilot results from the other models are also included in appendix c.2. algorithm 3 multi-debate reasoning judgment (mrj) 1: ## model selection 2: {old1, . . . , old5} ← offensive language datasets 3: for i, @xmath0, ..., old5} do 4: evaluate llms on test set 5: end for 6: best_llm ← llm with highest f1 score across olds 7: ## persona alignment 8: llmo ← initialize best_llm to discuss offensive 9: llmn ← initialize best_llm to discuss non-offensive 10: llmjudge ← initialize best_llm to make final decision 11: for @xmath1.o do 12: ## rationale generation 13: summary ← summarized source article 14: from §algorithm 2 15: rationaleo ← argument(llmo, summary, comment) 16: rationalen ← argument(llmn, summary, comment) 17: ## discuss on each side 18: stanceo ← debate(llmo, comment, rationalen) 19: stancen ← debate(llmn, comment, rationaleo) 20: ## final judgment 21: label ← instruct llmjudge based on 22: the label distributions and pairwise label overlap ratios of judgments across topics are presented in notably, sej inferred from prior datasets yielded a comparatively lower rate of offensive labels with values ranging from approximately 50% to 60%. although the models were trained on explicit examples of offensive language, their reliance on outdated data raises concerns regarding their ability to generalize to contemporary discourse. while it is plausible that llm-based judgments pej and mrj may overestimate offensiveness, we observed the consistently high label distributions between 70% and 80% across a range of prompt variants and multi-debate settings. these results suggest that such outputs are unlikely to result solely from over-detection of previously unseen comments. we also found that pej and mrj exhibited over 80% pairwise label overlap, suggesting a notable degree of alignment in decision criteria informed by shared contextual cues and an interconnected reasoning process. in contrast, the overlap ratio decreased when sej was included, indicating that its decision boundaries diverge from those of llmbased judgments, which demonstrate more adaptive and nuanced predictive behavior. when aggregating the judgments across all topics, the overall overlap ratio was approximately 66%, reflecting a moderate level of consistency. to assess the impact of individual judgments, we employed a leave-one-out strategy @xcite . we first calculated the agreement score based on the complete set of judgments, then iteratively excluded each judgment to evaluate its effect on the overall score. an increase in agreement upon exclusion (∆ -@xmath2. through this analysis, we aimed to identify whether the aggregated judgment was disproportionately influenced by specific components and to determine more reliable judgment for collective assessments. from a total of m comments based on n judgments, each label was defined as x i,j . the agreement score f was computed using krippendorff's α 8 @xcite . let the agreement difference be denoted as ∆ -k obtained by excluding the 8 we scaled the obtained scores by multiplying them by 100 to enable more intuitive comparison in our study. the agreement score total for the three main judgments and within each of sej and pej are presented in one noteworthy aspect is that, despite the large volume of collected comments, especially reaching up to 5.61 million for the topic general politics, the score of pej exceeded 60. this indicates significantly greater consistency compared to sej, which remained at around 30. these results suggest that the prompt variants used in our experiment provided more consistent evaluation criteria than those derived from prior datasets, and also imply that prompt-based approaches may ease evaluation and enhance consistency. figure 3 : leave-one-out agreement score differences by excluding each of the component: training dataset or prompt variant, in sej and pej. the agreement score differences ∆ -k across main and component judgments of sej and pej are presented in figure 2 and 3. although it is not possible to conclusively determine which judgment aligns most closely with the ground truth, figure 2 shows that all judgments exhibited substantial discrepancies, with marked shifts upward and downward for sej and pej. this pattern suggests that relying on pre-existing datasets-even when synthesized from five distinct sources-falls short of the predictive stability demonstrated by more llm-based approaches. in contrast, mrj displayed intermediate agreement differences, indicating comparatively greater stability. we then analyzed each component of sej and pej, as shown in figure 3 . larger deviations appeared more often in sej than in pej, suggesting that the choice of training dataset is a more sensitive factor than prompt design in judgment aggregation. notably, we found that kodoli and k-mhas yielded substantially lower scores, indicating degraded judgment quality. in contrast, un-smile led to a significant increase, underscoring its importance in aligning inference consistency. in pej, we observed that defn achieved the highest score by providing an explicit definition of offensive language grounded in political context, followed by fewshots, which offered topic-relevant samples that enabled more contextual inferences. in contrast, the summ resulted in the lowest score, despite the article summaries exhibiting sufficient factual consistency 9 . this suggests that the model may have been biased by the article content, potentially leading to under-or overestimation of offensiveness. moreover, political comments are often driven more by the author's ideological stance rather than the actual content @xcite , limiting the effectiveness of ar-ticle summaries in supporting nuanced judgments in such contexts. ## establishing ground trust † for analytical evaluation in the absence of ground truth labels for offensiveness in our newly constructed dataset, we assessed the performance of each judgment using pseudolabels @xcite @xcite . we treated each carefully designed judgment as a reference point within a trustworthy range, referred to as ground trust † . this construct served as a practical proxy for ground truth, enabling systematic evaluation and comparison across offensive language judgments. the construction of ground trust † from each judgment is illustrated in figure 4 . individual labels were obtained from sej, pej, and mrj (indicated by red check marks), and subsequently aggregated through majority voting (represented by the red box) to form the ground trust † . this procedure reflects the integration of multiple optimized conditions to assign a representative label to each comment in the absence of ground truth. although ground trust † may not serve as an absolute reference equivalent to conventional ground truth, we regarded it as the most robust pseudolabeling strategy available in our study, developed through the careful integration of multiple judgments tailored to newly collected comments. given the ongoing demand for up-to-date dataset con- we further evaluated component-level predictions from sej and pej (indicated by yellow check marks in figure 4 ), with results presented in in the case of pej, which showed consistent performance across prompt variants, no single case outperformed the aggregated results of all variants. this indicates that while each prompt exhibits robustness on unseen comments, combining outcomes from multiple prompts yields the most reliable performance. each prompt's performance was comparable to mrj, suggesting that under constraints on external resources or iterative inference with llms, a strategic prompting approach leveraging contextual cues can effectively detect offensiveness in unseen comments foot_6 . ## conclusion we constructed the politicalk.o dataset-one of the most extensive collections of offensive language with 9.28 million user comments-to capture the dynamics of contemporary political discourse in korea. in the absence of ground truth labels, we designed three refined judgments sej, pej, and mrj grounded in widely adopted offensive language detection methodologies. our large-scale analysis revealed notable label distributions and agreement scores on leave-one-out strategy. notably, when each judgment was evaluated against the ground trust † , strategic prompting based on explicit contextual cues consistently achieved performance comparable to more resource-intensive approaches. this result offers a valuable reference point for handling previously unseen comments under real-world constraints, and provides guidance for future scenarios where annotated data or repetitive model access may be limited.