Generative Models for Automatic Medical Decision Rule Extraction from Text (2024)

Abstract: Medical decision rules play a key role in many clinical decision support systems (CDSS). However, these rules are conventionally constructed by medical experts, which is expensive and hard to scale up. In this study, we explore the automatic extraction of medical decision rules from text, leading to a solution to construct large-scale medical decision rules. We adopt a formulation of medical decision rules as binary trees consisting of condition/decision nodes. Such trees are referred to as medical decision trees and we introduce several generative models to extract them from text. The proposed models inherit the merit of two categories of successful natural language generation frameworks, i.e., sequence-to-sequence generation and autoregressive generation. To unleash the potential of pretrained language models, we design three styles of linearization (natural language, augmented natural language and JSON code), acting as the target sequence for our models. Our final system achieves 67% tree accuracy on a comprehensive Chinese benchmark, outperforming state-of-the-art baseline by 12%. The result demonstrates the effectiveness of generative models on explicitly modeling structural decision-making roadmaps, and shows great potential to boost the development of CDSS and explainable AI. Our code will be open-source upon acceptance.

## introduction currently, the development of clinical decision support systems (cdss) relies heavily on manual enumeration of medical decision rules @xcite @xcite . although this paradigm brings cdss interpretability and reliability, its request of extensive labor poses a challenge on scaling, given the huge amount of potential medical decision rules @xcite . and the fact that some medical decision rules get occasionally updated make the challenge even worse. this motivates researchers to explore the automation of medical decision rules construction. inspired by the fact that human doctors acquire medical decision rules from textbooks and clinical guidelines, a recent study proposes to imitate this process via deep learning methods @xcite . there exist two typical formulations of medical decision rules: first-order predicate logic formulas @xcite and medical decision trees @xcite , where the latter is an extension of the former. formally, a medical decision tree is a binary tree consisting of condition nodes and decision nodes. each node is a relation triple or multiple relation triples combined by logical operators ("or", "and"). the decision nodes are leaf nodes of the tree, whereas the condition nodes are internal nodes. and the transition from one node to another represents judgment or decision-making. a first-order predicate logic formula in conjunctive normal form can be viewed as a special case of a medical decision tree where there is only one condition node and one decision node. hence, we adopt the tree-form formulation in this paper. different from traditional information extraction tasks, e.g., name entity recognition @xcite , relation triple extraction @xcite and event extraction @xcite , where the target output is a set of unitary/dual/multivariate tuples, the target output of medical decision tree extraction is a logically combined complex of relation triples. the logical coherence exhibited by such complexes mimics that of human language. this motivates us to adopt generative approaches for medical decision tree extraction, so as to better model the intrinsic logical connection among the relation triples inside a medical decision tree. reflecting on the exciting success within the field of natural language generation, we can observe that two paradigms (sequence-to-sequence, autoregressive generation) along with the idea of pretraining play the crucial roles. in this work, we try to replicate the success of sequence-tosequence/autoregressive generation on the task of medical decision tree extraction. in order to maximally elicit the potential of pretrained generative language models, three designs of medical decision tree linearization are trialed: 1) natural language (nl) style of linearization, where the relation triples are verbalized and naturally assembled with conjunctions; 2) augmented natural language (augnl) style of linearization, where each relation triple is represented as an augmented token, sharing equal status with natural language tokens; 3) json style of linearization, the most widely used data interchange format that represents data objects as key-value pairs. the linearized medical decision trees act as the target sequences during training, and are generated then parsed into tree structure during inference. the proposed sequence-to-sequence models employ an encoder-decoder architecture with a pair of pretrained language encoder and decoder, as well as a query-based entity-relation extractor. under this paradigm, relation triple extraction is treated as a sub-task and the models fulfill it via the entityrelation extractor. whereas the proposed autoregressive models are instantiated from decoder-only large language models (llms). in this discipline, relation triple extraction is treated as an auxiliary task for multi-task learning without introducing extra parameters. benchmarking on text2dt @xcite , a comprehensive chinese dataset, we find that generative models are much more capable of extracting medical decision tree than state-of-the-art (sota) discriminative models. our experiments also show that a carefully designed sequence-tosequence model (section 2.2) is competitive to a llm-based autoregressive model (section 2.3) that is 10+ times larger. our contributions are summarized as follows: ## conclusion in this study, we present several generative models to extract medical decision trees, which are valuable for cdss but costly to acquire manually. the proposed models inherit two mainstream text generation paradigms, i.e. sequence-to-sequence generation and autoregressive generation, which bring advantage in modeling both source text and the intrinsic logical connection among tree components. experiments show that our method wins the sota discriminative method by a large margin, es-tablishing new sota with 67% tree accuracy and 78% path f1 score. besides, an in-depth analysis of error distribution reveals the pros and cons of different models, paving the way for future research on this area. another direction for future efforts is the evaluation of predicted decision tree's clinical usefulness in real-world scenario, which requires consideration of potential ethical risks and careful experimental designs.