RLKGF : Reinforcement Learning from Knowledge Graph Feedback Without Human Annotations (2025)

Abstract: Reinforcement Learning from Human Feedback (RLHF) has been shown to effectively align large language models (LLMs) with human knowledge. However, the lack of human preference labels remains a significant bottleneck when applying RLHF to a downstream domain. Humans in RLHF play a critical role in injecting reasoning preferences into LLM, and we assume the reasoning process underlying human assessments may potentially be replaced by reasoning pathways derived from Knowledge Graphs (KGs). Inspired by this assumption, we propose Reinforcement Learning from Knowledge Graph Feedback (RLKGF), a novel method that leverages KG semantics and structure to derive RL rewards in the absence of manual annotations. Unlike Reinforcement Learning from AI Feedback (RLAIF), RLKGF directly integrates human priors encoded in KGs as the reward model, aligning LLM responses with expert knowledge without additional preference labeling or reward model training. RLKGF structures context-relevant facts into knowledge subgraphs and defines rewards by simulating information flow across semantic and logical connections between question and candidate response entities. Experiments on three public and one private medical dialogue dataset demonstrate that RLKGF significantly outperforms the competitive RLAIF in improving LLM diagnostic accuracy. The code is available at https://github.com/YanPioneer/RLKGF .

## introduction large language models (llms) like chatgpt @xcite have shown remarkable potential in tasks such as knowledge-based questionand-answer (q&a) @xcite and intelligent decision-making @xcite . as llms advance in specialized domains like medicine (zhang et al., 2023a), agriculture @xcite , and law @xcite , the demand for factually accurate and helpful responses grows. reinforcement learning from human feedback (rlhf), regarded as a key driver of chatgpt's success, aligns llm outputs with human preferences and enhances generation quality @xcite . its effectiveness has also been validated in domainspecific llm adaptations @xcite . however, rlhf involves a complex training process. first, a reward model is learned from ranked human preference data. subsequently, scores generated by the reward model are used to apply policy optimization @xcite . despite its benefits, the high cost of human annotation, inconsistent annotation standards, and potential biases from subjective judgments hinder the widespread application of rlhf. both self-reflection @xcite and cot @xcite approach in llms highlight the advantage of leveraging their embedded knowledge to enhance task performance. meanwhile, several works mention llms have demonstrated human-like judgment capabilities in certain aspects @xcite . thus, automating preference selection for model responses through llms is a natural progression. self-refine @xcite and refiner @xcite employ llms to evaluate and iteratively refine outputs through feedback. additionally, anthropic @xcite and google @xcite directly use llms to filter response data and train reward models with the selected results to aid model training, essentially conducting reinforcement learning from ai feedback (rlaif). although rlaif can distill the evaluation ability of advanced llms into reward models, its reliability remains limited by potential knowledge gaps and hallucinations, particularly in high-accuracy domains like medicine. current evaluations of llm-generated responses primarily emphasize the semantic relevance between responses and question contexts and the correctness of logical reasoning chains @xcite . these criteria align with the implicit semantic relationships and explicit structural connections among entities in knowledge graphs (kgs). since the inception of llms, kgs have been instrumental in tasks such as evaluation @xcite , knowledge injection @xcite , and knowledge augmentation @xcite , due to their structured fact storage and annotation-free advantages. however, these approaches predominantly treat kgs as static knowledge repositories and leave llms to filter and select relevant facts. this overlooks the semantic associations between facts and fails to fully exploit the structured connectivity of kgs. considering that entities with high linkage criticality are more likely to reach each other during inference and engage in greater semantic interactions, both semantic relevance between factual entities and the strength of logical connections in kgs can serve as natural scoring mechanisms @xcite @xcite . building on this insight and inspired by rlaif, we propose reinforcement learning from knowledge graph feedback (rlkgf), which directly derives reward signals from kgs without manual annotations. rlkgf treats the kg itself as a reward model and assigns reinforcement learning (rl) rewards to llm responses by simulating semantic information flow and logical link transmission between question and candidate response entities on relevant subgraphs-without the need for preference labeling or reward model training. the scoring process integrates local seman-tic aggregation and global path reasoning among factual entities. at the semantic level, rlkgf employs graph neural networks (gnns) for nodelevel information exchange and computes semantic relevance scores between question and candidate response entities. structurally, rlkgf initiates reasoning from the question entities via random walks across connected paths and transparently calculates the criticality of path-connected entities based on reachability probabilities. we validate rlkgf in medical dialogue diagnosis tasks. experimental results demonstrate that rlkgf outperforms rlaif in disease prediction accuracy, which proves rlkgf's effectiveness as a viable alternative to rlhf. further comparisons with supervised fine-tuning and kg-based prompts highlight rlkgf's advantages in aligning llms with knowledge. besides, to eliminate potential contamination from existing datasets, we also construct a new medical dialogue diagnosis dataset (med-d) from unpublished electronic medical records. the contributions of this study are: ## method in this section, we define the task and describe our method, which directly utilizes the structural and semantic information among factual entities in kgs to provide feedback on model responses. ## conclusion the semantic correlations and link criticality inherent in kgs closely mirror the semantic and logical relevance humans use to evaluate llm responses. building on this, we propose rlkgf, which directly employs kgs as a reward model to provide feedback to llms without the need for human annotation or separate reward model training. rlkgf utilizes both local semantic interactions and global path reachability to define reinforcement learning rewards. in the context of medical dialogue diagnosis, rlkgf outperforms rlaif, which relies on model-embedded knowledge. we also compare various knowledge injection methods, such as sft and kg-based prompts, offering valuable insights into the effective use of kgs. although this work highlights the potential of rlkgf, several limitations remain. first, its generalization to other tasks and domains has not been explored. additionally, we employ ppo to train llms, and there may be more suitable reward structures and training methods to explore.