Verification of Reasoning Ability using BDI Logic and Large Language Model in AIW olf (2024)

Abstract: We attempt to improve the reasoning capability of LLMs in werewolf game by combining BDI logic with LLMs. While LLMs such as ChatGPT has been developed and used for various tasks, there remain several weakness of the LLMs. Logical reasoning is one of such weakness. Therefore, we try to introduce BDI logic-based prompts to verify the logical reasoning ability of LLMs in dialogue of werewofl game. Experiments and evaluations were conducted using “AI-Werewolf,” a communication game for AI with incomplete information. From the results of the game played by five agents, we compare the logical reasoning ability of LLMs by using the win rate and the vote rate against werewolf.

## introduction one of the important goals of artificial intelligence research is to realize human reasoning abilities on computers. from the early days of artificial intelligence research to the present, various studies on logical reasoning have been conducted, establishing research areas such as theorem proving and logic programming. with the recent development of deep learning, the integration of deep learning and logical reasoning @xcite , @xcite ) has become an issue. large language models (llms) such as chat-gpt have made it possible to generate human-like natural sentences. today, llms are used in a wide variety of domains. however, llms have several challenges, one of which is their inference capability, and research has been conducted on the inference capability of llms, including commonsense inference @xcite , @xcite ). on the other hand, there has been little research on the ability of llms to detect intentional deception. in this study, we investigate the ability to detect intentional lies in werewolf game. the purpose of this study is to improve the inferential ability to detect intentional lies in werewolf game. "werewolf game" is an incomplete information communication game commonly known as "mafia". in this paper we call it a "werewolf game". in incomplete information games, some important information is hidden from the players, and the players play games such as bluffing against each other. therefore, it is necessary to have higherorder logical reasoning ability to handle the opponent's lies. in this study, we propose a method using bdi logic @xcite , @xcite ) to improve the logical reasoning ability of llms in werewolf game. bdi logic is considered effective for higher-order logical reasoning for lies because it allows for the explicit description of agents' mental states. in this study, we aim to improve the logical reasoning ability of llms in werewolf game using methods based on bdi logic. for this purpose, we used the aiwolf server provided by the aiwolf project and conducted experiments and evaluations. ## experiments this chapter describes the actual experiments conducted with the aiwolf platform described in chapter 3. ## conclusions we proposed a methodology to introduce bdi logic representation into llms inferences to improve logical inference capability on communication games that contain lies in the conversation. we compared the inference performance of llms by conducting experiments using the ai-wolf server. in the experiment using gpt-4, gpt-3.5, keldic, and aiwolfnlagentpython as opponents, partkey_manyex outperformed gpt-3.5's vote rate for werewolf + possessed, showing that it has better inference ability than gpt-3.5. in the experiment using only gpt-4 as the opponents, the proposed method outperformed gpt-4 in voting for werewolves and werewolves + possessed, and the proposed method significantly outperformed gpt-4 in voting on day 2 only, suggesting that the more information the proposed method has, the better its inference ability becomes. this result shows that the proposed method outperforms gpt-4 in the werewolf game when the opponent is only gpt-4.