Towards Rationality in Language and Multimodal Agents: A Survey (2025)

Abstract: This work discusses how to build more rational language and multimodal agents and what criteria define rationality in intelligent systems.Rationality is the quality of being guided by reason, characterized by decision-making that aligns with evidence and logical principles. It plays a crucial role in reliable problem-solving by ensuring well-grounded and consistent solutions. Despite their progress, large language models (LLMs) often fall short of rationality due to their bounded knowledge space and inconsistent outputs. In response, recent efforts have shifted toward developing multimodal and multi-agent systems, as well as integrating modules like external tools, programming codes, symbolic reasoners, utility function, and conformal risk controls rather than relying solely on a single LLM for decision-making. This paper surveys state-of-the-art advancements in language and multimodal agents, assesses their role in enhancing rationality, and outlines open challenges and future research directions. We maintain an open repository at https://github.com/bowen-upenn/Agent_Rationality.

## introduction rationality remains a critical and urgent topic in the research of artificial intelligence, particularly as intelligent systems become increasingly involved in high-stakes decision-making processes, such as healthcare, finance, science, and legal services @xcite @xcite @xcite where reliability is paramount for human end-users utilizing these agents for decision-making. unlike reasoning that aims to draw conclusions from premises, rationality ensures that those conclusions are reliably consistent, have an orderability of preference, and are aligned with evidence from various sources and logical principles. however, recent studies reveal that even stateof-the-art large language models (llms) exhibit limitations in rationality. because a single llm relies solely on its internal parametric representations of textual knowledge, while lacking real-world grounding and feedback mechanisms necessary to develop rationality @xcite @xcite , it shows bounded knowledge, inconsistent responses, and susceptibility to biases and framing effects @xcite @xcite @xcite @xcite . these limitations raise concerns about their practical reliability in critical sectors, highlighting the need for more reliable and coherent systems capable of rational behaviors. to address these challenges, research is shifting toward multimodal agents and multi-agent systems, as complex problems in real life often require collaborations of experts across fields @xcite and data from diverse sources. formally, we refer "agent" @xcite as an artificial intelligent entity that perceives and understands its environment through various inputseither natural language or multimodal information like vision, audio, and codes -and acts to achieve specific goals or tasks within natural language domains, while the term "agentic" to describe such behaviors @xcite . this can encompass a range of systems, from a single llm, a multimodal foundation model with instruction following capabilities @xcite , to a multi-agent system that integrates multiple ai agents, traditional machine learning or symbolic reasoning modules, external knowledge bases, and tools working together towards a collective goal within the same environment. given this, this survey explores how current literature helps address the limitations of a vanilla llm in achieving rationality, with an emphasize on lan-figure 1 : this survey identifies four necessary, though not sufficient, axioms that a rational agent should fulfill. meanwhile, we reinterpret various research domains related to agents and agent systems through the lens of rationality, examining how their underlying algorithms contribute to each of these axioms. ## scope existing surveys in the field of agent systems, such as those on multimodal @xcite @xcite @xcite and multi-agent systems @xcite @xcite , primarily focus on their components, architectures, profiling, planning, communication strategies, memory mechanisms, and applications. additionally, many works @xcite @xcite @xcite @xcite @xcite @xcite and surveys @xcite @xcite @xcite @xcite explore the reasoning capabilities of llms. although reasoning plays an important role in ensuring rationality, especially in complex scenarios, it remains parallel to our focus, as mentioned in section 1. furthermore, some works touch the aspect of rationality in llms @xcite @xcite ), but they fo-cus on one specific algorithm or application domain. to the best of our knowledge, this survey is the first to comprehensively explore the notion of rationality in language and multimodal agents. we aim to bridge the gap between rationality and agent system, analyzing how designs in these agents and agent systems contribute to advancing certain key axioms of rationality. ## defining rationality in agents drawing on foundational works in cognitive science about rational decision-making @xcite @xcite , this section presents four necessary, though not sufficient, axioms we expect a rational agent or agent systems to fulfill: information grounding a rational agent's decision-making should be grounded in physical and factual reality, incorporating information it perceives from multimodal formats and sources. in contrast, an irrational agent generates hallucinations @xcite , producing false or misleading information that is not grounded in facts. logical consistency logical consistency refers to an agent's ability to avoid self-contradictions in reasoning and ensure that its conclusions logically follow from its premises. a rational agent should deliver consistent decisions in its final responses, producing invariant decisions across equivalent representations of the same problem. invariance from irrelevant context a rational agent should not be swayed by irrelevant contextual information, focusing instead on the logical essence of the problems and relevant data. ## towards rationality in agents this section discusses how existing language and multimodal agent systems are advancing the concept of rationality. we survey a range of research domains, reinterpret their contributions through the lens of the necessary axioms of rationality outlined earlier, and present a novel perspective that bridges existing methodologies with rational principles. ## evaluating rationality in agents while there are numerous reasoning benchmarks @xcite @xcite @xcite @xcite @xcite @xcite , they do not directly measure rationality. the amount of studies for evaluating rationality in agent or agent systems remains scant, despite the growing interest in the field. in this section, we explore potential evaluation methods and benchmarks aligned with each of the proposed axioms of rationality. ## open problems and future directions towards inherent rationality despite significant research efforts in this survey aimed at achieving one or more axioms of rationality, most existing algorithms rely on external tools, thereby not inherently enhancing the rationality of artificial intelligence. in other words, current methods are neither sufficient nor necessary to achieve human-level rationality, but they serve as instrumental tools that bridge the gap between an llm's response and rationality. these approaches enable agent systems to more closely mimic rational thinking in their output responses from the end-user's perspective. however, how to effectively close the loop and bake these more rational outputs back into foundation models themselves @xcite beyond mere fine-tuning remains an open question. it remains a question if we can leverage these more rational outputs, or training the model to verify against rational axioms for reward scores, to inherently enhance a single foundation model's rationality in its initial responses without external assistance. ## discussion rationality adds another crucial dimension to the performance of intelligent agents besides reasoning capabilities. more rational agents -capable of providing coherent orderability of preferences and grounded in knowledge beyond their bounded parametric memory -could enhance their roles in automatic evaluation and preference alignment, where human involvements are expensive yet unstable. it becomes increasingly important for human users applying these agents in critical sectors like health care and finance that expect consistent and reliable decision-making. for example, in financial domains, decision-making by llms must operate within an acceptable risk threshold. by adapting conformal risks, we can enhance the orderability of preference and create a theoretical framework for parameterizing risk preferences into utility models. this approach enables rational financial agents to optimize decisions to max utilities while controlling risks through loss functions that bound expected risks. medical domains also present unique multimodal challenges, particularly in medical imaging interpretation. for instance, diagnostic methods must withstand irrelevant information like background noise in mri scans. an vlm could provide diagnostic explanations by looking at these images, and an llm could collaborate with it as a multi-agent system to verify against pre-defined illness criteria, incorporating external knowledge retrieval to improve information grounding. besides, a multi-agent collaboration could involve diverse medical-domain personas, such as surgeons, nurses, physicians, radiologists, pharmacists, offering diagnostic recommendations from different perspectives, potentially reducing hallucinations through collaborations. as a result, we hope our survey could serve as a tool-box for agent developers who want to build more rational agents in diverse domains. this survey investigate current approaches in a range of related literature that advance towards more rational language and multimodal agents. these approaches include the integration of multimodal inputs, reflections and multi-agent collaborations, rlaif, utility functions, conformal risk controls, and modules that perform deterministic execution like tools -including programming codes -and neuro-symbolic reasoning modules. we believe in the value of further investigations into the rationality of ai agents, particularly through a collaboration between the ai research community and cognitive psychologists for a deeper understanding of rationality. ## limitations the fields of language and multimodal agents are rapidly evolving. despite our best efforts, it is inherently impossible to encompass all related works within the scope of this survey. our discussion also possesses limited mention of the reasoning capabilities, theory of mind in machine psychology, cognitive architectures, and rationality models like formal logic and probability theory, all of which lie beyond the scope of this survey but are crucial for a deeper understanding of the agent systems. furthermore, we present necessary but not sufficient axioms of rationality; no methodologies mentioned in our survey could sufficiently guarantee a genuine rationality in agents. the concept of rationality in human cognitive science may encompass more principles and axioms than those defined in our survey, such as completeness, transitivity, monotonicity, decompoability (poole and mackworth, 2010), which are more theoretical and fundamental in nature and not directly related to language and multimodal agents discussed in our survey.