CHARD : Clinical Health-Aware Reasoning Across Dimensions for Text Generation Models (2023)

Abstract: We motivate and introduce CHARD: Clinical Health-Aware Reasoning across Dimensions, to investigate the capability of text generation models to act as implicit clinical knowledge bases and generate free-flow textual explanations about various health-related conditions across several dimensions. We collect and present an associated dataset, CHARDat, consisting of explanations about 52 health conditions across three clinical dimensions. We conduct extensive experiments using BART and T5 along with data augmentation, and perform automatic, human, and qualitative analyses. We show that while our models can perform decently, CHARD is very challenging with strong potential for further exploration.

## introduction pretrained language models (plm) have seen increasing popularity for nlp tasks and applications, including text generation. researchers have become interested in the extent to which plms can: 1) act as knowledge bases, 2) reason like humans. rather than using external databases, exposure to large amounts of data during training combined with their large number of parameters, has given plms the ability to store knowledge that can be extracted through effective probing strategies such as text infilling @xcite , prompting @xcite , and qa @xcite . plms imitate a more high-level information store, allowing for greater abstractness, flexibility, and generalizability. they are also able to better exploit contextual information than simple retrieval. studies have also shown that as plms scale up, they have have emergent abilities @xcite , including reasoning. there has been increasing attention on their commonsense reasoning through works like comet @xcite . however, studies show that even large plms strug- * work done while at cmu. ## results and analysis we report automatic results on chardat test of the best models (for bart-base, bart-large, t5base, t5-large) trained on chardat compared to retr in our best overall chard model is t5-large based on automatic results and qualitative analysis. we break down results of t5-large on char-dat test-seen and chardat test-unseen in graphs displaying models' rouge-2 on chardat val for 2x da across various bt tmp and different da amounts can be found in figures 3 and 4 , respectively. tables 8 and 9 ## directions for improvement we see that our models are decent and generate readable text, but can improve on medical accuracy and informativeness. while they are not nearly ready for real-world use, they show potential. as stated, the purpose of chard is to assess the capabilities of plms to act as implicit clinical knowledge bases that can reason through several dimensions. how can we improve our models, and possibly our dataset and task formulation? dataset and task formulation: we introduce chard and initially frame the task using a template infilling approach which is more constrained. more flexible formulations may better leverage the knowledge and generation capabilities of plms. our current approach involves generating explanations about a single condition and dim attribute at a time. we can possibly improve chardat by annotating for more complicated input queries. this is because a plm may be more effective at answering more complicated queries, e.g. comparing and contrasting conditions and dim and multi-hop reasoning. it is likely easier to make complicated inferences and connections over the abstract plm embedding space than over retrieved text passages. further, we can expand chardat to include more dimensions and topics in the health domain. these improvements may allow for the training of a single system that is able to make complicated clinical inferences across various topics and dim. ## conclusion and future work in conclusion, we proposed and investigated the task of chard: clinical health-aware reasoning across dimensions, to explore the capability of text generation models to act as implicit clinical knowledge bases and generate explanations across several health dimensions. we presented a dataset, chardat, and conducted experiments with bart and t5. extensive evaluation and qualitative analysis demonstrated that our models are decent, especially for generating concise and readable text, but can be improved on medical accuracy and informativeness, and that chard is challenging with much potential for further exploration. we highly encourage the research community to further investigate and improve upon chard. future directions are discussed in §6. some additional ideas include trying more data augmentation strategies and decoding strategies for text infilling.