Hybrid Knowledge Engineering Leveraging a Robust ML Framework to Produce an Assassination Dataset (2022)

Abstract: Social and political researchers require robust event datasets to conduct data-driven analysis, an example being the need for trigger event datasets to analyze under what conditions and in what patterns certain trigger-type events increase the probability of mass killings. Fortunately, NLP and ML can be leveraged to create these robust datasets. In this paper we (i) outline a robust ML framework that prioritizes understandability through visualizations and generalizability through the ability to implement different ML algorithms, (ii) perform a comparative analysis of these ML tools within the framework for the coup trigger, (iii) leverage our ML framework along with a unique combination of NLP tools, such as NER and knowledge graphs, to produce a dataset for the the assassination trigger, and (iv) make this comprehensive, consolidated, and cohesive assassination dataset publicly available to provide temporal data for understanding political violence as well as training data for further socio-political research.

## introduction peace and conflict researchers have identified several large-scale structural conditions that make state-led mass killings more likely, such as ongoing political instability or histories of state violence against vulnerable groups @xcite . however, the timing of mass killing onset is less understood. @xcite identifies nine potential triggering events for state mass killings, such as coups and assassinations, but before socio-political researchers can conduct systematic analysis to examine whether, and if so when, certain patterns of trigger-type events actually increase the probability of mass killings, it is necessary for political researchers to obtain political event datasets for each of these potential triggering events. processing the massive amount of information in available data in order to create socio-policial event (spe) datasets for events such as the triggers described above takes extensive time, money, and human power. fortunately, natural language processing (nlp) and related machine learning (ml) tools can be harnessed to classify the rapidly growing, but often poorly structured and unlabeled, data as to whether they contain an event or not. ml classification tools have been increasing combined with other nlp tools such as named entity recognition (ner) and knowledge graphs (kgs) to engineer these datasets. although these ml and nlp tools have become more robust, it is important for the ai research community to acknowledge that each tool comes with limitations and a scope of use. with this in mind, our project seeks to uniquely leverage a combination of these tools in order to mitigate their drawbacks to create an spe dataset. the most cited challenge for political event extraction is small labeled training datasets @xcite @xcite which become an issue when working with ml classification algorithms. therefore, our first task is to provide a clear, efficient, and accessible machine learning framework that future social scientists may utilize when implementing nlp-focused algorithms to classify large quantities of text documents given a small labeled training dataset. we prioritize a framework that is reproducible, understandable, and generalizable by both including essential visualizations of the input data and results and structuring the framework in such a way that fellow researchers can implement different ml algorithms, such as support vector machines (svms) or bidirectional encoder representations from transformers (bert). we demonstrate that different ml algorithms are most suitable for a given optimization problem by performing a thorough comparative analysis of these different ml algorithms for the coup trigger in the process of refining our framework. after explaining our ml framework, we demon-strate how we implement this framework to create a dataset for a new trigger: assassinations. we describe the process of deciding which ml tool to implement within the framework and subsequently leverage our robust ml framework along with a combination of additional nlp tools, such as ner and kgs, to create the spe dataset. by mitigating the drawbacks and uniting the strengths of both machine-based and human-centric approaches we create the most comprehensive (targeting all known assassination events), consolidated (a single dataset solely focused on assassination events), and cohesive (easily filterable and readable) assassinations dataset to provide temporal data for understanding political violence as well as training data for further socio-political research 1 . ## hybrid knowledge engineering to create a spe dataset: assassinations after refining our ml framework, our next step was to implement the framework on one of spe of interest in order to create the desired dataset. we switched to assassination events to create our spe dataset because we saw a lack of assassination datasets in literature (section 2.1) and assassinations are the most clearly defined trigger foot_3 for the triggers laid out in @xcite with one keyword ('assassination'). we leveraged our flushed out machine learning pipeline, along with existing assassination datasets, kgs, and ner to enhance our new assassinations dataset (fig 8 ). ## results: the assassination dataset by uniting the strengths of each tool within our hybrid approach we created an assassination dataset with 7,457 assassination events. for each entry we collected available information on name, date, country, position, and success status (successful vs. attempted) of each assassination event along with the unique identifiers from the source(s) it was identified from. figure 11 highlights the unique information and number of assassination events contributed by each tools. this shows that despite each method's limitations, ambiguous event definitions for humans, incomplete datasets, missing wikidata properties, and small training datasets for ml, it is evident that each tool benefited the dataset. existing databases provided a starting point for our dataset, wikidata enhanced our repository, and the ml pipeline allows us to extract assassination events from 169,637 articles with only a 165 article training set. ## conclusion & future work we have contributed to ongoing spe research by providing a robust ml framework for small training datasets, performing a comparative analysis of ml tools, presenting a novel hybrid knowledge engineering approach to curate a dataset, and releasing our comprehensive, consolidated, and cohesive assassination dataset which will provide temporal data for understanding political violence as well as training data for further socio-political research. although our framework and hybrid knowledge engineering approach will not perfectly transfer for every spe dataset curation task, our focus on understandability visualizations, replicable frame-works, and explanation of challenges will allow future researches to incorporate our work for a variety of spe extraction tasks. in future work, we hope to apply the knowledge engineering approach, encompassing our ml framework, to the remaining triggers of interest while continuing to improve and automate our ml framework.