A Multi-Agent Framework with Diagnostic Feedback for Iterative Plain Language Summary Generation from Cochrane Medical Abstracts (2025)

Abstract: Plain Language Summaries PLS improve health literacy and enable informed healthcare decisions but writing them requires domain expertise and is time-consuming. Automated methods often prioritize efficiency over comprehension and medical documents unique simplification requirements challenge generic solutions. We present a multi-agent system for generating PLS using Cochrane PLS as proof of concept. The system uses specialized agents for information extraction writing diagnosis and evaluation integrating a medical glossary and statistical analyzer to guide revisions. We evaluated three architectural configurations on 100 Cochrane abstracts using six LLMs both proprietary and open-source. Results reveal model-dependent trade-offs between factuality and readability with the multi-agent approach showing improvements for smaller models and providing operational advantages in control and interpretability.

## introduction health literacy is the ability of an individual to access, understand, and apply health information. this is essential for informed decision-making and effective navigation of healthcare systems. inadequate health literacy remains a global challenge, contributing to poor treatment adherence, higher hospitalization rates, and health disparities @xcite @xcite . plain language summaries (pls) is a way to reduce health literacy gaps by translating medical texts into clear, accurate, and accessible language for non-technical audiences @xcite . however, producing high-quality pls manually is resource-intensive and requires expertise in both medical content and health communication. recent advances in llms offer new opportunities to automate pls generation. while early efforts showed that llms can produce readable and semantically faithful summaries, most approaches relied on single-pass generation and lacked systematic guardrails for factuality, readability, and regulatory compliance @xcite . the increasing complexity of biomedical content and the need for domain-specific standards underscore the importance of structured, multi-step workflows over monolithic generation. this work builds on our previous research in llm-based pls generation @xcite , which showed the potential of single-prompt models to translate cochrane abstracts into pls and clinical trials into protocol plain language summaries (ppls). however, generating these kind of structured pls that meet professional standards differs from generic text simplification; it needs adherence to specific templates and guidelines, diagnostic feedback, and systematic quality control. unlike general simplification tasks that focus only on reducing complexity, structured pls generation requires writing documents with welldefined structures that balance accessibility with medical accuracy. current llms struggle to balance simplicity with factual accuracy, often oversimplifying complex medical content or preserving meaning at the cost of readability @xcite . our prior work revealed similar domain-specific challenges, requiring distinct prompts for different document types (cochrane pls and ppls). also, this approach did not provide mechanisms to identify specific problems in the generated pls drafts or provide targeted corrections. based on these limitations, this research aims to: (1) develop tools that allow a better understanding of what makes a text non-compliant with pls standards and how to systematically address these issues, and (2) create a multi-agent framework supported by diagnostic tools that can both generate structured pls and evaluate their quality through some iterative refinement. we propose a framework that decomposes pls generation into specialized subtasks, each handled by dedicated agents: information extraction, writing, diagnostic, and evaluation (figure 1 ). the key component is a diagnostic feedback loop where evaluation agents identify specific complexity issues and guide targeted revisions using verifiable criteria. we instantiate this framework for cochrane pls generation, as their guidelines @xcite provide an useful template ideal for testing structured document generation. ## methodology we present the methodology for developing and evaluating a multi-agent system for automatic generation of pls from cochrane medical abstracts. the approach leverages linguistic analysis, medical glossaries, domain-specific glossaries, and advanced language models to produce accessible medical communication. the methodology follows six main steps: (1) data collection and processing, (2) pls linguistic feature extraction, (3) percentiles for pls diagnosis, (4) develop diagnostic tools, (5) define specialization, and (6) define architecture for the multi-agent system (see appendix a for the complete workflow diagram). ## evaluation and results we evaluated the three architectural configurations (baseline, v 1 , v 2 ) described in section 3.4.4 across multiple language models to assess the impact of multi-agent decomposition and iterative refinement on pls generation quality. ## discussion our evaluation reveals inconclusive results regarding which approach is superior, with both baseline and multi-agent configurations showing distinct advantages depending on use case requirements. the baseline proves remarkably effective when properly designed with comprehensive instructions based on cochrane guidelines, demonstrating that systematic prompt engineering grounded in domain standards can produce high-quality pls. the multiagent architecture, while not completely superior, provides specific benefits in certain contexts. for smaller models, the multi-agent approach shows improvements in relevance and factuality metrics. gpt-oss-20b improves in semantic similarity and factual alignment when using the v 2 configuration, though with a slight deterioration in readability. notably, smaller models achieve competitive or higher factuality scores compared to larger models, something we noticed in our prior work @xcite , suggesting these models may be more conservative in adhering to source material. a potential strategy to address readability limitations would involve using a smaller model for content extraction followed by a larger model for final linguistic refinement, potentially offering cost-effective generation while preserving factual accuracy. the multi-agent architecture provides operational advantages in terms of control and inter-pretability. separating extraction, assembly, and evaluation into distinct agents makes each component transparent and independently modifiable. while the baseline consolidates all instructions into a single comprehensive prompt, the decomposed approach allows for targeted refinement of specific subtasks without affecting the entire pipeline. this modularity also produces intermediate outputs for each section, which are stored separately in our repository and can be inspected individually for diagnostic purposes. however, the multi-agent approach incurs higher token costs due to multiple agent invocations and memory preservation through context repetition across agents. the v 1 configuration, which includes the medical glossary tool, is particularly token-intensive, with processing costs reaching approximately 10m tokens (input + output combined) per 100-abstract batch. most tokens are input tokens, which are typically cheaper than output tokens, though iterative configurations may generate similar text multiple times, increasing output costs. the baseline, requiring only a single model invocation per abstract, proves more cost-effective in terms of api usage. this cost-benefit trade-off must be considered when selecting an approach for production deployment. the effectiveness of smaller models like gpt-oss-20b with the multi-agent approach could enable local deployment in medical contexts where data privacy is critical. while the current task of simplifying published cochrane reviews does not involve sensitive information, other medical text simplification scenarios could benefit from local processing. for instance, healthcare providers might need to simplify patient-specific medical reports or treatment explanations without transmitting sensitive data to external apis. in such contexts, the ability to run smaller models locally while maintaining reasonable quality through multi-agent decomposition could provide a viable solution. beyond technical performance metrics, this work addresses the practical need of facilitating medical writers' work and improving health information accessibility. our framework provides diagnostic tools and automated first drafts that meet professional pls standards, potentially reducing the manual effort required to produce accessible health communication materials. ## future work future work could explore alternative evaluation approaches beyond percentile-based diagnostics, including different metrics and quality assessment methods for iterative refinement. multi-agent hybrid systems where different agents use specialized models could balance cost and quality. extending the architecture to other plain language formats such as ppls or documents following the federal plain language guidelines (williams, 2025) would test its generalizability. additionally, incorporating full-text papers as input sources rather than abstracts alone could address content coverage limitations, potentially leveraging datasets like cochrane-auto for improved alignment between source and simplified text. ## limitations our system may perpetuate suboptimal information prioritization patterns from the training data @xcite . computational constraints and api rate limits restricted experimentation scope, and we did not conduct formal statistical hypothesis testing. gpt-5 could not be tested with v 2 due to api rate limits, only supports temperature 1.0 foot_1 preventing deterministic generation, and exhibited inconsistent behavior (sometimes the model did not use the evaluation tool for v 1 ). gpt-oss-120b v 1 had implementation issues with the evaluator tool. v 1 intermediate outputs were lost due to storage issues; v 2 outputs are available in the repository. the v 2 example in appendix a.2 represents a single cherry-picked case. our percentile-based evaluation framework represents statistical conformity rather than absolute quality, and strict percentile ranges could penalize innovative plain language strategies. ## lay summary medical research papers often contain complex language that makes them difficult for patients and the general public to understand. plain language summaries help solve this problem by explaining research findings using everyday words. organizations like cochrane create these summaries for their systematic reviews, which combine results from many studies to answer health questions. however, writing plain language summaries requires medical expertise and takes considerable time. this creates a bottleneck in making health information accessible to everyone. we wanted to find out whether computer systems using ai could automatically generate highquality plain language summaries. specifically, we tested whether breaking down the writing task into smaller steps handled by specialized ai agents would work better than using a single comprehensive instruction. we also wanted to know if this approach would be more helpful for some ai models than others. we built a system that divides summary writing into four stages: extracting information from the medical abstract, assembling it into a draft, checking for medical terms that need simplification, and evaluating readability. the system uses a medical dictionary with over 20,000 terms and their plain language alternatives. it also includes a statistical analyzer that compares the generated text against patterns found in human-written summaries. we tested this system using 100 cochrane medical abstracts and six different ai models, ranging from large commercial models to smaller open-source ones. our results are mixed and there is no clear winner among the designed architectures. the baseline approach, which uses a single well-designed instruction, performed surprisingly well. the multi-agent system did not consistently outperform the baseline, but it showed specific advantages. smaller ai models improved notably when using the multi-agent approach, achieving better accuracy in preserving medical facts, though sometimes at the cost of readability. we also found that the multi-agent system provides greater control and less black-box effect, allowing users to inspect and modify individual steps separately. healthcare organizations and research institutions working with limited computational resources could benefit from these findings. the results suggest that smaller, locally-run ai models combined with the multi-agent approach could generate reasonably accurate summaries while maintaining data privacy. the modular design also makes it easier to adapt the system for different types of medical writing beyond cochrane summaries. however, more work is needed to improve readability when using smaller models and to reduce the computational costs of the multi-agent approach.