Cognitive Feedback: Decoding Human Feedback from Cognitive Signals (2025)

Abstract: Alignment from human feedback has played a crucial role in enhancing the performance of large language models. However, conventional approaches typically require creating large amounts of explicit preference labels, which is costly, time-consuming, and demands sustained human attention. In this work, we propose Cognitive Feedback, a framework that infers preferences from electroencephalography (EEG) signals recorded while annotators simply read text, eliminating the need for explicit labeling. To our knowledge, this is the first empirical investigation of EEG-based feedback as an alternative to conventional human annotations for aligning language models. Experiments on controlled sentiment generation show that Cognitive Feedback achieves performance comparable to explicit human feedback, suggesting that brain-signal-derived preferences can provide a viable, lower-burden pathway for language model alignment.

## introduction human alignment for large language models (llms) is crucial for generating safe and preference-aligned outputs. previous work has shown that this process helps llms better follow human instructions and mitigate harmful behaviors @xcite . a traditional posttraining approach involves supervised fine-tuning (sft) on a pretrained llm, followed by reinforcement learning from human feedback (rlhf) @xcite . direct preference optimization (dpo) @xcite is an alternative to rlhf that skips the reward model and offers more stable training. many state-of-the-art models, such as openai's o-series, continue to adopt the sft + dpo paradigm @xcite , demonstrating that it remains an effective strategy. however, creating the preference labels necessary for dpo and related preference optimization methods remains labor-intensive. tasks such as selecting and training annotators, establishing trust, and coordinating large-scale annotation efforts incur substantial costs @xcite . to address these challenges, reinforcement learning from ai feedback (rlaif) @xcite leverages llm-generated synthetic feedback to substitute for explicit human feedback. this approach offers lower costs, easier large-scale data collection, and strong scalability compared to traditional human-driven methods @xcite @xcite . however, several drawbacks remain. depending on the task, humans may disagree with ai-generated judgments @xcite @xcite , indicating that synthetic feedback may fail to capture genuine human intentions. moreover, there is a bootstrapping issue: ensuring the model that produces feedback is itself properly aligned is non-trivial @xcite , theoretically undermining ai feedback as a complete solution to alignment. finally, while ai-generated feedback can reduce cost, it does so at the expense of direct human involvement, raising concerns about whether such signals faithfully reflect nuanced human values. the question of which feedback signals, or combinations of such signals, most effectively align llms with human goals remains open @xcite . in this work, we propose cognitive feedback, a framework for obtaining preference information directly from human brain activity. specifically, we investigate whether preference signals extracted from electroencephalography (eeg) can be integrated into preference optimization methods such as dpo. if feasible, this approach could offer a more direct and potentially less cognitively demanding means of capturing individual responses than conventional annotation pipelines, as participants only need to read the presented text without providing explicit ratings. we focus on the con- cognitive signals decoder zuco sr dataset text a: i love this movie. text b: this is so terrible... natural reading i love this movie. this is so terrible... text a text b text a text b a is better than b a is better than b human judges cognitive feedback human feedback implicit feedback explicit feedback trolled sentiment generation task. this task is well suited to our study for two reasons: (1) it serves as a foundational benchmark for preference optimization, as it was one of the benchmark tasks originally employed in dpo or other derivative methods @xcite @xcite , and (2) previous works have demonstrated that eeg is effective at capturing emotional responses in nlp @xcite . to operationalize this idea, we introduce cognitive preference optimization (cpo), a method that estimates preference information from eeg data collected while participants read text. by relying on implicit cognitive feedback instead of explicit human feedback, cpo aims to significantly reduce the need for manual annotation (figure 1 ). alongside falling costs and growing accessibility in mobile eeg, recent large-scale decoding results show clear data-performance scaling, reinforcing the practical path for eeg-based alignment @xcite . in our experiments, we compare two forms of feedback: standard human feedback requiring explicit labeling, and implicit feedback inferred from eeg. our results show that the cpo-trained model not only produces more positive outputs than a baseline model but also achieves performance comparable to conventional human feedback settings. these findings highlight the potential for eegbased feedback signals to serve as a novel approach for llm alignment. we summarize our main contributions: ## cognitive preference optimization as outlined in section 2, cognitive signals on their own can be noisy; however, they serve to enrich nlp embeddings by providing more detailed information. we adopt this paradigm for ai feedback: cognitive signals function as an implicit form of human feedback, capturing user preferences with minimal burden on the annotators, while reinforcing the input information used in ai feedback. in so doing, we attempt a novel feedback approach that alleviates the limitations of both human and ai feedback. figure 2 is an overview of the proposed method. step 1: training cognitive decoder let @xmath0, x 2 , . . . , x t ) be the sequence of combined feature vectors for a text of length t . following previous work @xcite , each x t is formed by concatenating the eeg feature vector e @xmath1, i.e. x @xmath2; h t ]. this approach has been shown to yield robust representations (figure 3 ). we then define a cognitive decoder s ϕ (x) ∈ r, where ϕ denotes its trainable parameters. for training, suppose we have n pairs {(x to achieve this, we minimize: ## experiments in this section, we empirically evaluate the performance of our proposed method by examining three questions: (1) to what extent can we decode feedback from eeg signals? (2) does the proposed method perform at a level comparable to conventional, explicit human feedback? (3) does its performance scale with the size of the eeg datasets we use? although no eeg dataset currently exists for the purpose of llm preference optimization, if initial experiments demonstrate the method's effectiveness even under limited data conditions, this would provide motivation for creating larger, more realistic datasets. this work serves as a first step toward assessing whether cognitive signals can supplement or even replace traditional forms of human feedback. ## discussion examples of text generated by the proposed method are presented in ## conclusion in this paper, we proposed cognitive preference optimization (cpo), a novel framework for aligning large language models (llms) with human preferences inferred from electroencephalography (eeg) signals. by training a cognitive decoder to extract pairwise preferences from a natural reading corpus, we introduced a method that reduces reliance on explicitly labeled data. our results suggest that eeg-derived feedback can successfully guide policy optimization for sentiment generation, producing outputs that match or even rival models trained with conventional human feedback. the proposed method can use the scalability of traditional ai feedback while obtaining human feedback in the form of readings that are less burdensome for the operator. future experiments in a more realistic setting will require the construction of a large dataset of cognitive signals for the purpose of reinforcement learning of llms.