Classifying Organized Criminal Violence in M exico using ML and LLM s (2023)

Abstract: Natural Language Processing (NLP) tools have been rapidly adopted in political science for the study of conflict and violence. In this paper, we present an application to analyze various lethal and non-lethal events conducted by organized criminal groups and state forces in Mexico. Based on a large corpus of news articles in Spanish and a set of high-quality annotations, the application evaluates different Machine Learning (ML) algorithms and Large Language Models (LLMs) to classify documents and individual sentences, and to identify specific behaviors related to organized criminal violence and law enforcement efforts. Our experiments support the growing evidence that BERT-like models achieve outstanding classification performance for the study of organized crime. This application amplifies the capacity of conflict scholars to provide valuable information related to important security challenges in the developing world.

## introduction recent advancements in natural language processing (nlp) have revolutionized political science analyses by enabling efficient and accurate analysis of large volumes of text. these tools have demonstrated impressive capabilities in tackling complex text analysis tasks, leading to their increasing adoption by political scientists including conflict scholars specialized in the study of violence and crime @xcite @xcite . in this paper, we present an application of various machine learning (ml) algorithms and large language models (llms) to analyze a variety of behaviors by organized criminal groups in mexico by processing text written in spanish. by leveraging these state-of-the-art nlp techniques, we aim to make significant contributions to the study of organized criminal violence and law enforcement efforts in developing countries. based on a set of high-quality annotations, we train and evaluate different ml algorithms and llms to determine their effectiveness in detecting and categorizing organized crime violence and law enforcement. furthermore, we extend our analysis beyond simple document-level classification by analyzing the relevance of specific sentences within documents and then analyzing the specific types of events described in the narratives. in this way, we move beyond the identification of organized criminal groups as named entities @xcite @xcite , and focus on analyzing criminal behaviors. results show the high levels of performance of bert-like models to effectively classify relevant news articles, as well as relevant sentences within them. the models also have remarkable results for classifying a variety of violent and non-violent actions perpetrated by criminal groups or conducted by law enforcement forces. overall, our research emphasizes the advantages of leveraging nlp tools in political science research, particularly in the domain of political violence and organized crime analysis. by exploiting their remarkable capabilities for document, sentence, and class classification, researchers can extract valuable insights from vast corpora in local languages, thus enabling a more comprehensive understanding of complex social behaviors. the elements advanced in this research can pave the way toward the development of a fully integrated ml crime analysis system in spanish. computerized text analysis has a long trajectory in the study of international conflict, but recent ml developments are just gaining traction among conflict scholars. early efforts to identify incidents of political conflict or cooperation using text analysis relied on complex systems of rules @xcite @xcite @xcite . unfortunately, these rule-based systems are too rigid and expensive to update, and the algorithms showed limited performance when tackling even basic nlp tasks. due to the limitations of rule-based approaches, recent nlp developments such as conflibert @xcite and polecat @xcite bring more flexibility and effectiveness in analyzing political violence. unfortunately, those tools are focused exclusively on the english language. to address this challenge, scholars such as @xcite , @xcite , and @xcite have been advancing multilingual ml tools and llm to study conflict. within this constellation of research, scholars have been using nlp tools to study organized crime in mexico by processing text written in spanish. early efforts relied on rule-based approaches to track the territorial presence of organized criminal groups @xcite @xcite . a common limitation of these studies is their exclusive focus on tracking the location of criminal groups. unfortunately, this only provides information about "who" is present but does not say much about their behavior. although @xcite and @xcite have been incorporating ml approaches to study organized crime, these ml applications have only focused on a narrow set of behaviors. to address these limitations, this study provides a fully integrated ml application to identify a broad range of behavioral trends of criminal groups and state authorities from news stories written in spanish. ## training data computational social scientists have paid increasing attention to the quality of training data annotations @xcite @xcite . due to the need for high-quality annotations to maximize ml performance, this study implements a rigorous annotation protocol. to generate the training data, the study relied on a group of three human annotators supervised by the principal investigator (pi). the meticulous training, supervision, and validation protocols implemented in this project allowed generating high-quality annotations. the protocol consisted of human annotators classifying information from high a level of aggregation to progressively fine-grained annotations in three stages: document classification (task 1), sentence relevance (task 2), and event type (task 3). task 1: document relevance. to ensure the validity of the data at the highest level of aggregation, the first task consists of identifying news articles conveying information on organized criminal violence and law enforcement efforts against criminal groups. failing to discriminate the domainspecificity of the documents increases the risk of including false positives which are likely to undermine the ml performance and the output validity. this study relies on the document classification originally conducted by @xcite , who used a team of human annotators to classify news articles as "relevant" or "not relevant". the first step consisted of using a query to gather news articles from 110 national and local newspapers in mexico. then, annotators classified as "relevant" news reports that provide descriptions of factual incidents of criminal violence or law enforcement against criminal groups. these incidents include armed confrontations between criminals; armed clashes between criminals and government authorities; arrests of members of criminal groups; drug seizures; seizures of assets (e.g. vehicles, money, real state); seizures of weapons; or the capture of high-profile targets. the team of annotators classified as "not relevant" news stories that do not make direct reference to organized criminal violence events; editorial opinions about criminal violence; statements or claims from victims, civil society organizations, or government officials; or summaries from government authorities providing a cumulative report of law enforcement activities. the training data at the document level consists of 60,837 news articles in spanish, out of which 61% are "relevant" and the other 39% are "not relevant" (see figure 1 ). this large training data was produced with high inter-annotator reliability (@xmath0.904), reported by @xcite . task 2: sentence relevance. this study goes beyond the document-level classification initially implemented by @xcite and analyzes the relevance of specific sentences. to do so, we selected a random sample of relevant documents from task 1 and disaggregated them into individual sentences. a team of human annotators classified sentences as "relevant" or "not relevant" following the criteria proposed by @xcite . an initial group of six annotators underwent a three-week training program to gain familiarity with the ontology. in this process, the annotators labeled the same corpus in several rounds. then, the pi selected the three annotators with the highest inter-annotator reliability score (@xmath1.8) to work on task 2. we used www.tagtog.com , a web-based annotation platform, to annotate a collection of 12,252 sentences. under the pi's supervision, the team of annotators independently classified each sentence and implemented a cross-validation process consisting of several rounds of revision to ensure the consistency of their labeling decisions. after each sentence received three validated classifications, the team generated the gold standard record (gsr). to do so, the pi randomly assigned a set of sentences to each annotator, who evaluated the set of anonymous annotations from the previous round and determined the most accurate one as the gsr. figure 2 shows the binary annotation that produced a balanced collection of 51.7% of the sentences as "relevant" and the other 48.3% as "not relevant", with a high inter-annotator agreement of @xmath2.9982. task 3: event type. the next step consists of annotating the type of event in the relevant paragraphs derived from the previous step. to do so, the annotators relied on a detailed codebook to classify 11 different types of events: (i) criminal violence vs. criminals, (ii) criminal violence vs. state, (iii) criminal violence vs. civilians, (iv) drug trafficking or production, (v) state violence vs. criminals, (vi) state arrest of criminals, (vii) state seizure of drugs, (viii) state seizure of guns, (ix) state seizure of assets, (x) state violence vs. civilians, and (xi) civilian violence vs. criminals. classifying unstructured text using a large number of categories can be challenging, particularly when the narrative conveys information about multiple events. identifying multiple actors conducting different actions in the same sentence can generate intractable annotation schemes. to address this challenge, the pi modified the annotation space to enable multiple-actor-action classification. figure 3 shows the interface using the same sentence to classify three different actions: an arrest (event 1), an attack on the police (event 2), and violence against civilians (event 3). the interface allows coding up to four distinct types of actions from the same sentence. 1 according to the annotation output in figure 4 , about 51.7% of the sentences contain a single event, 12.2% sentences have two events, 3.9% include three events, and 1.3% contain four events. the inter-annotator reliability assessment also indicates a high level of agreement between annotators with an @xmath3.998 in event 1, @xmath4.997 in event 2, and @xmath5. the team of annotators classified a total of 8,466 figure 4: event annotations among the events initiated by criminal groups, criminal violence against civilians stands out as the most common category with 20.9% of the cases. in contrast, violence between criminal groups is less frequent (3.2%). in practice, it is difficult to distinguish between criminal violence against the population and against rival criminals using news articles. the reason is that news reports tend to provide a generic description of the incident without giving details about the victims. for example, an article may just indicate that "a group of hitmen conducted an attack and killed three men." in this case, the criminal character of the perpetrators is clear, but there is no information about the victims. the annotation protocol used in this research classifies this type of event as violence against the population. foot_1 following the codebook, annotators classified events of violence between criminals when the news reports explicitly mention the criminal character of the perpetrator as well as the victim (e.g. name of the criminal group or the person's role such as hitman or lieutenant). criminal violence against the state constitutes the second most frequent annotation of criminal behavior (9.4%). in addition, annotators detected instances of criminal violence against the state in 3.3% of the sentences. finally, the annotation output indicates that violence from the population against criminals is very rare, with only 0.2%. overall, as figure 5 shows, the distribution of annotations in the training data is not balanced. there are some categories with a substantial number of annotations (particularly, criminal violence against civilians), while others do not have many annotations. this could affect the performance of ml algorithms and it is not plausible to expect good performance in categories with scarce annotations. section 6.1 below discusses future research to address this challenge. ## experiment setting this study analyzes organized criminal violence in mexico using a set of experiments to progressively process finer-grained information. the first stage focuses on classifying relevant documents. the second stage consists of classifying relevant sentences extracted from the relevant documents identified in the previous step. the final stage classifies the specific type of organized criminal violence or law enforcement actions contained in the relevant sentences. this sequential approach helps to ensure the validity of the output data based on the concatenated focus on documents, sentences, and events. the large number and high quality of the annotations included in the training data provide a strong empirical foundation to assess the performance of the different ml algorithms and llm considered. task 1: document relevance. to address the challenge of determining which news articles are relevant to the topic of organized criminal violence, the study approaches this problem as a binary classification task at the document level. based on the annotations provided in the training data, a positive outcome is operationalized as "relevant" and a negative outcome as "not relevant." in line with standards in computer science research, the experiment setup takes an agnostic approach and puts a variety of algorithms to compete in this binary classification task. this experiment considers a set of five traditional ml algorithms and three llms: multinomial naive bayes (nb), logistic regression (lr), random forest classifier (rf) @xcite , support vector machine (svm) @xcite , extreme gradient boosting (xgb) @xcite , beto @xcite , and multilingual bert (mbert) @xcite . the experimental setting uses this whole set of ml algorithms and llm to assess the performance of tasks 2 and 3. first, we pre-processed the data. we lemmatized the text in the corpus using the es_core_news_sm model from spacy. then, we removed latin diacritics and stop-words. next, we trained the traditional ml algorithms using their default settings in scikit-learn, and fine-tuned the llms using the hugging face library. we split the corpus into 90% for training and 10% for evaluation. to test the performance of our classifiers, we used the metrics implemented in scikit-learn. task 2: sentence relevance. based on the selection of relevant news stories derived from the previous stage, the experiment setup then focuses on classifying relevant sentences within the relevant documents. to do so, the study approaches this task as a binary classification at the sentence level. based on the annotations, a positive outcome is operationalized as a "relevant" sentence, while a negative outcome indicates "not relevant" sentences. for the automatic classification, we follow the pipeline described for task 1 and the experiments evaluate the full set of ml and llm. task 3: event type. the final set of experiments use a variety ml algorithms and llm to classify different types of events at the sentence level. to do so, the study considers the 11 types of actions discussed in section 3 as a multi-class classification task. for each type of event, the algorithms classify as a "positive" outcome a specific type of event mentioned in the sentence, and "negative" otherwise. this phase applies the same pre-processing steps as in tasks 1 and 2. to perform the classification, we generated individual binary subsets for each event label, enabling a binary classification for every label. this means that, from the 11 available classes, we created 11 subsets, each one with only two classes: positive and negative. this allows us to evaluate the performance of the learning algorithms with respect to each event type. 5 results ## conclusions this study presents an application to classify information related to organized criminal violence from unstructured text written in spanish using ml and llm. the results from this study enable researchers and government authorities to track the violent behavior of organized criminal groups in mexico and assess the effects of law enforcement activities. this research allows for the generation of data on a large scale, in a timely manner, and with an unprecedented degree of granularity and accuracy. by analyzing criminal and state behaviors, the study goes beyond previous efforts exclusively focusing on tracking the territorial presence of criminal groups using rule-based approaches @xcite @xcite . tracking the territorial presence of criminal groups only provides information about who is present, but does not say anything about what are they doing. thus, this study provides valuable tools to identify behavioral trends of criminal groups and state authorities from news stories with unprecedented accuracy. the methodological approach in this research focuses on a sequence of classification tasks of increasing levels of detail. based on a large collection of documents and a robust set of high-quality annotations in the training data, the first task focuses on classifying the relevance of entire news articles related to organized criminal violence and law enforcement. to do so, the experimental setting puts a variety of ml algorithms to compete in the binary classification task. the algorithm reporting the best performance is multilingual bert -uncased, with an f1 score of 0.9630. this high level of performance provides a strong indicator of the effectiveness of this ml algorithm. the second stage evaluates the different algorithms for the identification of relevant sentences. results show that beto presents the highest level of performance for the binary sentence classification task with an f1 score of 0.8588. finally, the study focuses on classifying the specific types of events of organized criminal violence and law enforcement contained in the data. this application considers 11 different types of events of lethal and non-lethal violence initiated by criminal groups, government authorities, and the civilian population. given the variations in the distribution of annotations across event categories, results show varying degrees of performance in the multi-class classification of event types. in general, the family of bert-like models shows a strong performance when classifying different types of organized criminal violence and law enforcement efforts. results of these nlp tasks report f1 scores ranging from 0.8440 to 0.9259. in particular, beto consistently presents high performance in many categories. beyond the technical performance evaluated in this application, results provide great confidence about the use of nlp tools to accurately extract and classify a broad range of behavioral information related to organized criminal violence from text written in spanish. these results offer valuable contributions to researchers, security analysts, and government agencies in spanish-speaking countries in their efforts to understand organized criminal behavior using high-quality data.