Capturing Human Cognitive Styles with Language: Towards an Experimental Evaluation Paradigm (2025)

Abstract: While NLP models often seek to capture cognitive states via language, the validity of predicted states is determined by comparing them to annotations created without access the cognitive states of the authors. In behavioral sciences, cognitive states are instead measured via experiments. Here, we introduce an experiment-based framework for evaluating language-based cognitive style models against human behavior. We explore the phenomenon of decision making, and its relationship to the linguistic style of an individual talking about a recent decision they made. The participants then follow a classical decision-making experiment that captures their cognitive style, determined by how preferences change during a decision exercise. We find that language features, intended to capture cognitive style, can predict participants’ decision style with moderate-to-high accuracy (AUC 0.8), demonstrating that cognitive style can be partly captured and revealed by discourse patterns.

## introduction while language models grow in sophistication, nlp tasks increasingly focus on understanding the people behind the language @xcite . such social and psychological nlp studies still rely primarily on annotations for evaluation. for example, recent social tasks have depended on annotated datasets for, e.g., emotions @xcite , empathy @xcite , politeness @xcite , humor @xcite , dissonance @xcite , and reasoning abilities @xcite . however, while annotation-based work has pushed nlp towards capturing cognitive states of the language generators (i.e. people), it falls short of offering ground truth of psychological processes because annotations reflect perception of another person's state. @xcite for cognitive style recently i had to move to a different country for a job … it's difficult without friends … but i love my job. discourse features writing to evoke cognitive style cognitive experiment post-experiment measurement cognitive experiment validate pre-experiment measurement (inducing a cognitive process) figure 1 : an alternate evaluation framework for validation of cognitive processes with language: the participants are first prompted to write about their experiences, eliciting their thought process. then they are subjected to an experiment that would measure their behavior. the behavior is a ground truth measure of their cognitive style that can be tied to the expressed language. we introduce an experimental framework that collects linguistic data alongside induced cognitive phenomena to evaluate the feasibility of discourse modeling approaches for capturing cognitive styles in decision making. by associating linguistic patterns with specific cognitive phenomena, we aim to understand individuals' unique cognitive styles, which are largely unseen and often only observable through the final decision @xcite . our study follows modern psychologyexperimental designs to quantify how language use signals cognitive styles, or habitual patterns of thought related to various cognitive phenomena. our key contributions include: our study draws from previous nlp research that validates author state measurements through annotations or self-report questionnaires. for example, past work has compared affective states with self-reported mental health by analyzing selfdisclosures @xcite , while others have examined cognitive styles in the context of discourse @xcite @xcite . however, annotations and self-reports are subject to perceptual biases, such as those observed in dialogue evaluations @xcite or when assessing constructs such as humor, empathy, or offensiveness @xcite @xcite . to address these limitations, we adopt an experimental approach that aims to objectively capture cognitive states, focusing on how individuals manage disso-1 for dataset and code: https://github.com/humanlab/cog_style_validation nance and consistency in their decision-making. discourse structures provide a theoretically grounded link between cognitive processes and communication patterns, serving as a window into how individuals construct and convey explanations @xcite . research in psychology has established strong connections between linguistic patterns and cognitive styles, particularly in how individuals process and communicate information @xcite . the analysis of discourse relations is especially valuable because they capture both explicit and implicit connections between text segments, revealing deeper patterns in explanatory styles such as reasoning @xcite and rhetorical structures @xcite ) that may not be apparent from lexical-level features alone @xcite . it serves as a powerful indicator of explanatory and rhetorical patterns in text, offering insights into how ideas are connected and presented @xcite . in this work, we explore discourse features as well as state-of-the-art llms to model the outcomes of the cognitive experiment. ## experiment a total of 514 participants were recruited in person for the study; 12 were excluded due to incomplete or invalid responses, resulting in a final dataset of 502 participants. data collection was performed in 2 stages (see figure 1 ). the questionnaire has been described in detail in appendix a. writing task participants received 2 writing prompts to elicit language relevant to their decisionmaking cognitive style: 1) "please describe a recent important and difficult decision that you have made" (20-100 words), and 2) "what were the considerations that you thought about while making the decision? when answering, please consider all of the circumstances and details that went into the difficult decision" (100-300 words). these questions were chosen to elicit detailed descriptions of a recent decision-making process, encouraging participants to discuss options and explain their reasoning. the elicited essays to the two questions were concatenated for all further analysis. we henceforth call the collection of essays from the participants the decisions dataset. ## methods we explored the following theoretically relevant discourse relations: ## results we explore results for our primary application of the experimental validation framework: do discourse relation models, which capture explanatory styles and coherence in language of individuals, predict the cognitive style of a decision that an individual makes? to explore language-specific patterns that relate to each type of cognitive style, we also extracted for theoretically-relevant lexical and discourse relation features in predicting each class of cis_inf. results are presented in we find that individuals who use more causal explanations and dissonant statements in their description of a recent past decision are less likely to change their minds about a decision due to external influence, and are less likely to change their preferences after making a decision, whereas, individuals who use less consonant statements in describing their decisions are more likely to switch their preferences after making a decision in the experiment. interestingly, higher linguistic dissonance is associated with less change in preferences / tendency to be influenced, which may signal difficulty in resolving dissonance surrounding one's decision. higher change in preferences with low tendency to be influenced also seems to be signaled by linguistic anxiety, and each of the cognitive styles have a distinct signature across personality and well-being dimensions. this indicates that individual decisionmaking cognitive styles derived from simulated real-life experiments can be gleaned from personal discourse and the explanatory style of the person. recommendations: as an initial step in developing this evaluation framework, we recommend incorporating direct behavioral measurements into linguistic analyses, moving beyond traditional annotation-based methods. while annotations pro-vide useful approximations of cognitive states, they rely on external judgments rather than direct psychological evidence. in contrast, experimental paradigms-widely used in psychology-allow researchers to systematically measure cognition and behavior under controlled conditions, offering a more reliable way to validate language-based models. to ensure ecological validity, language data should be collected before the experiment to prevent unintended influence on participants' responses. to capture a fuller picture of cognitive processes, researchers should combine linguistic features with behavioral metrics such as response times (e.g., questionnaire completion speed), clickthrough rates, and dynamic shifts in participant responses. this multimodal approach provides stronger evidence for the relationship between language and cognition, allowing nlp models to be evaluated against real psychological processes rather than relying solely on subjective annotations. by integrating experimental methods, this framework strengthens the scientific grounding of language-based models and enhances their validity for applications in cognitive science, decisionmaking research, and human-computer interaction. ## conclusion we demonstrated that experimentally-evoked cognitive styles can indeed be captured by language, offering a more solid "ground truth" compared to annotations of perceived behavior, which often fail to reflect a person's true state. this framework emphasizes methodological rigor through controlled psychological experiments, enabling researchers to establish robust connections between language patterns and realistic estimates of cognitive states. our framework's effectiveness is demonstrated with language-based features having strong predictive power for objective cognitive styles, especially discourse features successfully capturing experimentally measured cognitive styles. this approach not only enhances statistical validity but also has practical applications in the use of llms for mental health therapy, agent engagement systems, and cognitive science. by moving beyond the limitations of annotation-based or questionnaire-based labels, this paradigm represents a crucial step toward more rigorous evaluation in nlp, suggesting promising directions for future research in understanding the relationship between language and cognition.