On The Persona-based Summarization of Domain-Specific Documents (2024)

Abstract: In an ever-expanding world of domain-specific knowledge, the increasing complexity of consuming, and storing information necessitates the generation of summaries from large information repositories. However, every persona of a domain has different requirements of information and hence their summarization. For example, in the healthcare domain, a persona-based (such as Doctor, Nurse, Patient etc.) approach is imperative to deliver targeted medical information efficiently. Persona-based summarization of domain-specific information by humans is a high cognitive load task and is generally not preferred. The summaries generated by two different humans have high variability and do not scale in cost and subject matter expertise as domains and personas grow. Further, AI-generated summaries using generic Large Language Models (LLMs) may not necessarily offer satisfactory accuracy for different domains unless they have been specifically trained on domain-specific data and can also be very expensive to use in day-to-day operations. Our contribution in this paper is two-fold: 1) We present an approach to efficiently fine-tune a domain-specific small foundation LLM using a healthcare corpus and also show that we can effectively evaluate the summarization quality using AI-based critiquing. 2) We further show that AI-based critiquing has good concordance with Human-based critiquing of the summaries. Hence, such AI-based pipelines to generate domain-specific persona-based summaries can be easily scaled to other domains such as legal, enterprise documents, education etc. in a very efficient and cost-effective manner.

## introduction in the rapidly expanding digital world, the exponential growth of domain-specific knowledge has posed unprecedented challenges in efficiently storing and consuming vast information repositories. ## proposed framework we describe here our framework for training our small-size domain-specific llm, generation of 2 https://chat.openai.com/ ## evaluation and results we evaluate the performance of our finetuned models (l-f-7b and l-f-13b) in terms of generating high quality persona-based summaries for medical articles and compare against the baseline models. evaluation metrics: our evaluation relies on two different approaches: the generated persona-based summary correctly covers important key points described in the gold standard persona-based summary of the document. ## conclusion in this paper, we propose a framework for the efficient training of a small foundation llm on aigenerated datasets to obtain high quality domainspecific persona-based summaries. our focus is on training a finetuned version of llama2 on a corpus related to healthcare domain such that the trained model captures the intricacies of personabased summaries in healthcare domain. we also demonstrate the effectiveness of using ai-based critiquing for the evaluation of the model generated summaries, providing a more automated and scalable solution. our experiments also reveal the superior quality of persona-based summaries generated by our finetuned model compared to contemporary baselines. further, ai-based critiquing of the summaries show high inter-annotator agreement with human-based critiquing methods, further confirming the effectiveness of our proposed approach. we plan to extend our work in generating accurate persona-based summaries for documents in other domains such as legal, enterprises, education and more, which is the focus of our future work.