C og GPT : Unleashing the Power of Cognitive Dynamics on Large Language Models (2024)

Abstract: Cognitive dynamics, which refer to the evolution in human cognitive processes, are pivotal to advance human understanding of the world. Recent advancements in large language models (LLMs) highlight their potential for cognitive simulation. However, these LLM-based cognitive studies primarily focus on replicating human cognition in specific contexts, overlooking the inherently dynamic nature of cognition. To bridge this gap, we explore the cognitive dynamics of LLMs and present a corresponding task inspired by longitudinal studies. Toward the task, we develop CogBench, a novel benchmark to assess the cognitive dynamics of LLMs and validate it through participant surveys. We also design two evaluation metrics for CogBench, including Authenticity and Rationality. Recognizing the inherent static nature of LLMs, we further introduce CogGPT for the task, which features an innovative iterative cognitive mechanism to develop lifelong cognitive dynamics. Empirical results demonstrate the superiority of CogGPT over several existing methods, particularly in its ability to facilitate role-specific cognitive dynamics under continuous information flows. We will release the code and data to enable further research.

## introduction cognitive dynamics refer to the continuous evolution of human cognitive behavior within environmental context @xcite . these dynamics are essential for human advancement, facilitating learning, innovation, and adjustment in everchanging environments @xcite . a prime example of human cognitive dynamics is well exemplified by our ability to adapt our viewpoints based on environmental explorations @xcite . as illustrated in figure 1 , there has been a progressive shift in our understanding of the universe, evolving from geocentric to the earth is at the center of the universe, and all celestial bodies, including the sun and stars, revolve around it. ## task definition in this section, we present the formal definition of the task to assess the cognitive dynamics of llms. given the inherent static nature of llms, the task focuses on the cognitive dynamics of an llmdriven agent a, denoted as @xmath0, c 1 , . . . , c n }, over n iterations. here, c i corresponds to the cognitive state of a at the i-th iteration and @xmath1. the task input consists of: (1) a specific profile p that establishes the initial cognitive state of the agent a; here, (r 0 j , s 0 j ) represents the rating r 0 j and reasoning s 0 j for a question q @xmath2. at the t-th iteration, where 1 ≤ @xmath3, starting from its current cognitive state c t-1 , the agent a perceives an information flow i t , updates its cognitive state to c t , and formulates responses to q. the t-th cognitive process is captured by the function f : (c, i, q) → c, where: here, c @xmath4, s t 1 ), (r t 2 , s t 2 ), . . . , (r t m , s t m ); p t } details the cognitive state of a at the t-th iteration, where each (r t j , s t j ) reflects the adjusted rating r t j and reasoning s t j for a question q @xmath5. ## cogbench this section introduces cogbench, which is constructed through a semi-automated methodology. we validate cogbench through participant surveys and further design two essential evaluation metrics: authenticity and rationality. ## related work cognitive benchmarks towards llms. various distinguished cognitive benchmarks are employed in cognitive studies towards llms @xcite @xcite . instruments such as the big five personality trait @xcite and myers-briggs type indicator (mbti) @xcite indicate the personality traits of llms. the theory of mind (tom) benchmark @xcite explores in-context cognitive capabilities of llms. the cognitive reflection test (crt) reveals that the thinking abilities of llms are comparable to humans @xcite . additionally, the situational evaluation of complex emotional understanding (seceu) showcases that llms may understand human emotions and values @xcite . diverging from these static benchmarks, cogbench incorporates multi-source information flows, thereby supporting the explorations towards the cognitive dynamics of llms. llm-based cognitive modeling. recent work emphasizes the importance of prompt engineering in enhancing the cognitive abilities of agents @xcite @xcite . by incorporating comprehensive descriptions into prompts, such as hobbies and skills, users can customize agents for specific behaviors and responses @xcite . vector databases gain popularity for simulating human memory mechanisms due to their generality and efficiency @xcite @xcite . for cognitive decision-making, methods like chain-of-thought (cot) @xcite @xcite and self-validation @xcite enhance the logical thinking abilities of llms through intermediate reasoning steps. nevertheless, these efforts fall short in synthesizing an iterative cognitive mechanism to model the cognitive dynamics of llms, which is pivotal for coggpt to outperform other baselines under dynamic information flows. ## conclusion in this work, we investigated the cognitive dynamics of llms and presented a formally defined task, addressing a notable gap in llm-based cognitive studies. to facilitate this task, we developed an innovative benchmark, cogbench, and validated it through extensive participant surveys. meanwhile, we designed two evaluation metrics to ensure thorough assessments. recognizing the inherent limitations of llms, we introduced coggpt, an llm-driven agent featuring a novel iterative cognitive mechanism, tailored for the task. empirical results demonstrated that coggpt outperformed baseline agents in promoting lifelong cognitive dynamics. in the future, we plan to explore more advanced methods that facilitate direct interactions between llms and humans in a sandbox, further deepening our insight into the cognitive dynamics of llms.