Expanding the UNSC Conflicts Corpus by Incorporating Domain Expert Annotations and LLM Experiments (2025)

Abstract: In this work we expand the UN Security Council Conflicts corpus (UNSCon) (Zaczynska at al. 2024) on verbal disputes in diplomatic speeches in English.

## introduction the unsc conflicts corpus (unscon) presented in our previous work @xcite aims to serve as a resource for understanding verbal conflicts in united nations security council (unsc) speeches. it is novel in its attempt to operationalise conflicts defined as verbal disputes and critique in a diplomatic setting, and works on disagreement detection for speeches that are mostly pre-written. we developed an annotation scheme of conflicts including content and linguistic markers, allowing for the detection of different types of conflicts without requiring expert knowledge of the topic. the annotations were performed by computational linguists, and had not yet been compared to those from political scientists. to address this, in this work we conduct experiments with a un security council expert, identify key disagreements and suggest modifications to the annotation guidelines to improve the corpus. limited to debates on two topics and speeches from 2014 and 2016, unscon covers a restricted range of targets and periods. we expand the corpus by adding 40 new speeches on the subject iraq from the years @xcite , in order to increase the diversity in topics and targets. with the expanded corpus, we perform classification experiments on conflict types and compare them to results from the original unscon paper. we see that although the increasingly imbalanced label distribution between conflicts and no conflicts in the new dataset poses a challenge for the models, we improve scores by using roberta models trained on argumentation and the political domain. detecting lexical markers of negative evaluation within conflict spans is a crucial part of annotating these spans and is required for certain conflict labels. currently, annotations are applied to elementary discourse units (edus), which are typically sentences or clauses. these annotations define conflict types within the edus but do not specify the lexical markers themselves. to enhance the corpus' granularity, we conduct a pilot study using large language models (llms) to identify the lexical markers inside the conflict spans (edus) and categorise different types of lexical markers that indicate negative evaluation. to summarise our contributions, we expand the corpus on two levels, qualitatively and quantitatively: the updated dataset and the code for experiments are available in our github repository. 1 the remainder of the paper is structured as follows: first, we present related work and detail the annotation scheme for conflict types as defined in @xcite . next, we describe the annotation experiments conducted with a political scientist ( §3) and the updated conflicts annotation scheme based on identified disagreements. then, we introduce our expanded dataset with new annotation guidelines and the additional speeches included ( §4). we outline the experiments and classification setups ( §5), discuss the results ( §6), and, finally, draw conclusions ( §7). ## background in our former work presenting the unscon (zaczynska et al., 2024), we define conflicts as verbal disagreements or critique directed at someone present at the unsc debate, without necessarily referring to a military or physical conflict. there are different types of conflict: for an edu to be a conflict, it must be possible to identify a target (addressee) of the critique by examining the speech. the annotation scheme specifies a set of target types for the conflict, along with the specific countries being targeted. the unscon includes 87 speeches from debates discussing two topics: the ukraine conflict, and the women, peace and security agenda (wps) focusing on gender (in)equality and crimes committed during peace keeping missions. the annotation spans are elementary discourse units (edus) based on rhetorical structure theory @xcite . edus are usually sentences or clauses. the work on the unscon is based on transcriptions of meetings in the unsc @xcite , which serve as a foundation for various analyses in linguistics, computational linguistics, and political science. for example, @xcite examine how diplomats convey evaluative speech using appraisal theory @xcite for their analysis. other studies focus on extracting country mentions in unsc discussions using wikidata for named entity linking @xcite and named entity recognition @xcite . network analyses have also been conducted on unsc topics from afghanistan debates @xcite . scartozzi (2022) look at discourse related to climate change in the unsc. reinig et al. (2024) created a new resource of german parliamentary debates, annotated with finegrained speech act types distinguishing between cooperation and conflict communication. focusing on discourse in political debates around the us election 2016, visser et al. (2020) annotated argument relations using the relation classes inference, conflict, and rephrase. focussing on dialogues they use the term conflict differently than in the unscon, indicating incompatible propositions. ## evolution of the annotation scheme based on domain expert annotations in this section, we compare parallel conflict annotations of the unscon speeches made by a un security council expert with the original ones made by computational linguists. the analysis is the basis for the refined annotation scheme we present in the following sections. we first present the inter-annotator agreement (iaa), along with some general observations, followed by a detailed analysis of the most common disagreements in the annotations. ## corpus extension by size in this section we describe the extension of the unscon not only through applying the refined annotation guidelines to existing speeches but also by including new speeches from new debates. to broaden the scope of the unscon, which concentrates on ukraine and the wps agenda, we included debates on iraq. these debates focus on an (imminent) military conflict in iraq, highlighting a crisis in international relations and the formation of opposing factions within unsc countriesone supporting the military operation (including the us and great britain), and another opposing it (russian federation, france, and others). we also included 2019 and 2020 debates on iraq covering topics like the formation of a new iraqi government, the violent response of the previous iraqi government to demonstrations, and the threat posed by islamic state (is) terrorist groups in iraq. having a broader range of topics not directly related to military conflicts is more representative of other unsc discussions, though they have a smaller total amount of conflicts. ## experiments the next section outlines our setups for two sets of experiments: first, a pilot study on half of the dataset to incorporate lexical marker annotations for unscon, and second, an experiment utilising transformer models for fine-tuning on the conflict type classification task. ## conclusion this paper presents an extended version of the unsc conflicts corpus as introduced by @xcite , by expanding both the annotation guidelines and corpus size, and incorporating more detailed annotations of lexical markers of conflicts using an llm. working with diplomatic texts, and being annotated by computational linguists, we provide a detailed evaluation of political scientist annotations on the corpus and discuss identified disagreements. annotating communicative phenomena in language within nlp, especially in a domain with its own culture and rules such as the diplomatic setting, presents a balancing act regarding annotation guidelines. one must choose between creating guidelines that target diplomatic language usage only interpretable by people with advanced political science backgrounds, and linguistically marked verbalisations that are relatively domain-independent and possible to pick up on by nlp classifiers. we refined the annotation scheme and kept both the original notion of a mandatory lexical verbalisation of conflict, and also included conflict labels that might need cultural knowledge to detect, like figurative language. our classification experiments on conflict types using transformer models show that integrating a model trained on a similar task and domain improves the performance. despite this, the results indicate that smaller conflict types like chal-lenge correction (cc) (which involves detecting when someone claims another speaker is lying, and the correction of this alleged lie), and figurative language (figl) (which includes sarcasm and rhetorical questions) require more data to achieve satisfactory outcomes. looking at the classification results for each conflict label, we observe that all models struggled to accurately classify less frequent classes. in addition to the small number of training samples, this also may be attributed to the inherent difficulty of the task. detecting figurative language, for instance, remains a challenge in nlp @xcite . however, training on dedicated task-specific datasets might enhance performance @xcite . for future work we will conduct a further qualitative analysis of the lexical markers and types extracted by the llm and will expand the experiments to the full dataset. additionally, we plan to broaden the current limited list of emotive and instructive words by @xcite into a larger taxonomy, using the list of lexical markers found in the experiments by the llm, including terms expressing negative assessments found in the speeches.