Corpus Development Based on Conflict Structures in the Security Field and LLM Bias Verification (2024)

Abstract: This study investigates the presence of biases in large language models (LLMs), specifically focusing on how these models process and reflect inter-state conflict structures. Previous research has often lacked the standardized datasets necessary for a thorough and consistent evaluation of biases in this context. Without such datasets, it is challenging to accurately assess the impact of these biases on critical applications. To address this gap, we developed a diverse and high-quality corpus using a four-phase process. This process included generating texts based on international conflict-related keywords, enhancing emotional diversity to capture a broad spectrum of sentiments, validating the coherence and connections between texts, and conducting final quality assurance through human reviewers who are experts in natural language processing. Our analysis, conducted using this newly developed corpus, revealed subtle but significant negative biases in LLMs, particularly towards Eastern bloc countries such as Russia and China. These biases have the potential to influence decision-making processes in fields like national security and international relations, where accurate, unbiased information is crucial. The findings underscore the importance of evaluating and mitigating these biases to ensure the reliability and fairness of LLMs when applied in sensitive areas.

## introduction in recent years, advancements in artificial intelligence (ai) have significantly improved large language models (llms) in natural language processing (nlp). notably, openai's gpt series (openai, 2023) and meta's llama series @xcite have achieved human-like performance in tasks like text generation, translation, and question answering. these models have also expanded to handle multimodal data, such as images and audio @xcite . however, llms may inherit biases from their training data, reflecting prejudices related to race, gender, religion, and nationality @xcite . these biases present risks when llms are deployed in critical areas like national security. mikhailov (2023) highlighted the importance of llms in security decision-making, and the u.s. department of defense (2024) has already integrated llms to enhance military strategies. in japan, the ministry of economy, trade, and industry is developing domestic llms through the geniac project (ministry of economy, trade, and industry, 2024). despite their growing use, these initiatives often overlook bias evaluation. existing methods for detecting biases, particularly in security contexts, are limited and lack standardized corpora @xcite . this gap can lead to the deployment of discriminatory llms, potentially exacerbating international tensions. to address this, the study aims to create a corpus that reflects inter-state conflict structures and assesses biases in llms. the corpus will include texts that portray two countries with contrasting sentiments, enabling sentiment analysis to reveal inherent biases. the development process involves four phases: text generation with conflict-related keywords, diversity enhancement with varying emotional intensities, validity checks using a next sentence prediction (nsp) model, and quality assurance through manual review. this approach aims to simplify bias verification and foster more accurate assessments. finally, the study will use this corpus for sentiment analysis to identify biases in llms, contributing to discussions on mitigating these issues. the contributions of this paper are as follows: ## related works numerous studies have highlighted that llms inherently possess biases related to gender, race, political ideology, and other attributes. for example, @xcite and @xcite reported that llms might exhibit discriminatory behavior based on users' attributes, leading to inequality and system imbalance, which poses challenges for the societal implementation of llms. technologies to align llms with human values are currently emphasized to address these unintended biases @xcite . specific techniques for bias reduction include reinforcement learning from human feedback (rlhf) @xcite and reinforcement learning from ai feedback (rlaif) @xcite . additionally, @xcite proposed reducing gender bias by using debiased data during fine-tuning, while @xcite have focused on improving fairness through prompt engineering and in-context learning. addressing political biases in llms remains challenging. @xcite demonstrated that llms like chatgpt tend to lean towards specific political ideologies, with gpt models showing liberal tendencies and llama models exhibiting authoritarian ones. such research is crucial for understanding political biases in llms but is limited when it comes to verifying biases in interstate conflict structures related to national security. @xcite confirmed that llms possess extensive world knowledge but did not verify biases related to specific inter-state conflicts in detail. inoshita (2024) found a positive bias towards ukraine and a negative bias towards russia using artificially created data, which lacked objectivity and diversity. the absence of standardized corpora for bias verification in interstate conflicts compromises the accuracy of llm bias assessments and risks missing critical issues. this study addresses this gap by developing a standardized corpus and demonstrating its effectiveness. ## conclusion this study found that biases in llms are rooted in cultural and political influences from their training data. while the biases in gpt-4o-mini were generally subtle, there were slight negative biases towards eastern bloc countries, particularly on topics like "war" and "resources." even minor biases can significantly impact decision-making in national security and international relations, highlighting the need for careful evaluation and mitigation. to address these issues, it is essential to diversify training datasets and use fairness-aware methods during model development. incorporating bias detection algorithms into the llm evaluation process is also crucial, both during development and post-deployment, to ensure ongoing fairness. additionally, scenario-based analyses are necessary to understand how biases affect realworld applications, allowing for more practical mitigation strategies. by implementing these strategies, the fairness and reliability of llms can be enhanced, supporting the ethical use of ai in sensitive areas like national security. this study emphasizes the importance of tackling these challenges and provides a roadmap for future research in this critical field.