Leveraging Codebook Knowledge with NLI and C hat GPT for Zero-Shot Political Relation Classification (2024)

Abstract: Is it possible accurately classify political relations within evolving event ontologies without extensive annotations? This study investigates zero-shot learning methods that use expert knowledge from existing annotation codebook, and evaluates the performance of advanced ChatGPT (GPT-3.5/4) and a natural language inference (NLI)-based model called ZSP. ChatGPT uses codebook’s labeled summaries as prompts, whereas ZSP breaks down the classification task into context, event mode, and class disambiguation to refine task-specific hypotheses. This decomposition enhances interpretability, efficiency, and adaptability to schema changes. The experiments reveal ChatGPT’s strengths and limitations, and crucially show ZSP’s outperformance of dictionary-based methods and its competitive edge over some supervised models. These findings affirm the value of ZSP for validating event records and advancing ontology development. Our study underscores the efficacy of leveraging transfer learning and existing domain expertise to enhance research efficiency and scalability.

## introduction event coding is a crucial task in political violence research for both academic and policy communities. it transforms unstructured text from news articles into structured event data, represented as source-action-target triplets, achieved through entity extraction and relation classification. it provides a structured record of interactions among political actors and serves as input for monitoring, understanding, and forecasting political conflicts and mediation processes worldwide @xcite @xcite @xcite @xcite @xcite @xcite ). 1 https://github.com/snowood1/zero-shot-plover however, manually coding events from extensive datasets is labor-intensive. to streamline this, experts have developed event ontologies and knowledge bases @xcite @xcite @xcite @xcite @xcite . yet, traditional pattern-matching models based on static dictionaries suffer from inflexibility, low recall, and high maintenance costs. recent advancements in deep learning and pretrained language models (plms) offer promising supervised learning solutions @xcite @xcite @xcite @xcite . yet, their reliance on extensively annotated datasets introduces significant challenges, especially for in-depth and subnational studies requiring nuanced categorization and non-exclusive labeling within political event ontologies. moreover, labeled datasets lack flexibility and may require frequent relabeling as ontologies evolve. thus, much of the current plm-based research in event coding targets broad, coarse-grained categorizations, often constrained by limited evaluation sets. costs, time, and effort associated with developing training data have foiled the large-scale adoption and ready deployment of plms by government security agencies, researchers, and practitioners in need of monitoring and understanding rapidly-changing conflict processes around the world. in light of these challenges, we pose the following questions: (1) is it possible to leverage existing expert knowledge to enhance the efficiency of event coding without extensive annotation of new data? (2) is it possible to create an interpretable and adaptable system that easily accommodates ontology or schema changes? to tackle these questions, our paper focuses on relation classification, a key aspect of event coding. the goal is to categorize events in a source- in sum, the untapped potential of gpt-4 and the success of zsp encourage experts to reevaluate the value of existing knowledge bases and inspire innovative uses of this knowledge to expedite research within the political science community. ## approach we start by discussing the discovery of nli as a potential solution and the construction process of the nli-based zsp framework, followed by the deployment of chatgpt. ## conclusion future event coding tools should prioritize ease of interpretation and flexibility, making them more practical than annotating new datasets for blackbox supervised models. therefore, we explored the potential of zero-shot relation classification using chatgpt (gpt-3.5/4) and introduced our zsp model. while gpt-3.5 struggled with fine-grained classification, gpt-4 showed promise in mitigating instability issues. our zsp offers an even more cheap, precise, and adaptable solution. the key is structuring the complex problem into an interpretable, three-level tree framework, integrating mode-aware nli, and incorporating class disambiguation rules from the codebooks. overall, our study highlights the value of integrating transferred knowledge with expert linguistic insights to streamline the process of verifying event records for the political science community.