An introduction to realist evaluation and synthesis for kidney research
Authors: Dominic M. Taylor, Ailish M.S. Nimmo, Barnaby Hole, G.J. Melendez-Torres
Journal: Kidney International (2023)

Realist research describes a methodological approach that aims to explore how and why interventions work, for whom, and under what circumstances. Rather than quantifying how well an intervention works under specific conditions, realist theory explores the function of interventions in detail and specifically considers how the contexts in which interventional components are delivered influence the mechanisms that lead to outcomes. Realist methods can be applied to primary data (realist evaluation) or secondary data (realist synthesis). Although realist techniques are increasingly being used in the evaluation of complex interventions, there are relatively few published studies in the field of kidney care. In this review, we outline the theory and principles behind realist methods through discussion of a published realist synthesis describing complex interventions promoting delivery of optimal chronic kidney disease care. We discuss other kidney studies that have used realist methodology and situations where realist techniques could be applied to advance our understanding of how to best deliver care to patients with kidney disease. Realist research describes a methodological approach that aims to explore how and why interventions work, for whom, and under what circumstances. Rather than quantifying how well an intervention works under specific conditions, realist theory explores the function of interventions in detail and specifically considers how the contexts in which interventional components are delivered influence the mechanisms that lead to outcomes. Realist methods can be applied to primary data (realist evaluation) or secondary data (realist synthesis). Although realist techniques are increasingly being used in the evaluation of complex interventions, there are relatively few published studies in the field of kidney care. In this review, we outline the theory and principles behind realist methods through discussion of a published realist synthesis describing complex interventions promoting delivery of optimal chronic kidney disease care. We discuss other kidney studies that have used realist methodology and situations where realist techniques could be applied to advance our understanding of how to best deliver care to patients with kidney disease. Realist research is a theory-driven methodology that aims to understand why, how, for whom, and under what circumstances interventions, programs, services, or policies work.1Pawson R. Tilley N. Tilley N. Realistic Evaluation. Sage, 1997Google Scholar Realist evaluation involves primary data collection, whereas realist synthesis is a form of systematic review using secondary (already collected) data, such as published trials. These techniques are increasingly used in health care as they provide practical information on how and why complex medical interventions lead to observed outcomes in nonexperimental settings.2Marchal B. van Belle S. van Olmen J. et al.Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research.Evaluation. 2012; 18: 192-212Crossref Scopus (279) Google Scholar This review aims to describe realist evaluation and synthesis in the context of kidney care. This will be of use to those planning kidney service evaluation or research that may benefit from a realist approach. It will also assist those appraising realist literature to understand findings, assess whether the choice of research methodology was appropriate, and appreciate any limitations. Through discussion of our recent realist synthesis of complex interventions aiming to improve delivery of chronic kidney disease (CKD) care, we explain the principles and terminology of realist research, how it works, and when it may be applicable. We then provide a brief overview of previously published realist syntheses and evaluations of kidney care, before reflecting on questions in kidney care where realist techniques could answer important questions but have not yet been applied. Kidney care involves interventions that aim to reduce individual and population risks associated with kidney dysfunction delivered at various stages of kidney disease, including acute kidney injury (AKI), CKD, and kidney failure. A diverse range of interventions are employed, including medications, dialysis, surgical interventions, patient support, and education. Thus, kidney care is inherently complex, involving multiple components interacting with each other and with the conditions in which they are delivered.1Pawson R. Tilley N. Tilley N. Realistic Evaluation. Sage, 1997Google Scholar,3Fletcher A. Jamal F. Moore G. et al.Realist complex intervention science: applying realist principles across all phases of the Medical Research Council framework for developing and evaluating complex interventions.Evaluation. 2016; 22: 286-303Crossref PubMed Scopus (152) Google Scholar Realist evaluation and synthesis are particularly suited to the evaluation of complex systems, including health care systems, and may enhance the understanding of how kidney care should be developed to better deliver favorable clinical outcomes. The emphasis of realist research is to provide an understanding of why outcomes arise from interventions, by asking "How does this work?" By comparison, randomized controlled trials (RCTs) ask "Does this work?" by measuring the aggregated clinical effectiveness of medical interventions in controlled circumstances. Although these approaches are complementary, they invoke different scientific philosophies.1Pawson R. Tilley N. Tilley N. Realistic Evaluation. Sage, 1997Google Scholar Trials employ inferential evaluation, where differences in experimental group outcomes are seen to reflect successionist causation. Realist methods involve explanatory evaluation, where generative causation is used to explain intervention functioning.1Pawson R. Tilley N. Tilley N. Realistic Evaluation. Sage, 1997Google Scholar This terminology is summarized in Table 1.4Retroduction in realist evaluation: the RAMESES II Project.https://www.ramesesproject.org/media/RAMESES_II_Retroduction.pdfDate: 2017Date accessed: April 8, 2023Google ScholarTable 1Glossary of terms used within realist methodologyTermDefinitionInferential evaluationProcess by which the outcomes from treatment groups are used to assess the success of a treatment and extrapolate/make generalizations to a larger population of individuals.Explanatory evaluationProcess that aims to explain how and why an intervention works, in which context, and for whom.Successionist causationProcess by which an association between a causal agent/treatment and an outcome is examined. An estimate of the effect size of the observed causal influence is made. Typical analysis performed in a randomized control trial.Generative causationProcess by which an association between a causal agent and an outcome occurs via the mechanism of its action. Typical analysis performed in realist research.Retroduction4Retroduction in realist evaluation: the RAMESES II Project.https://www.ramesesproject.org/media/RAMESES_II_Retroduction.pdfDate: 2017Date accessed: April 8, 2023Google ScholarProcess in realist analysis that considers the likely hidden causal pathways resulting in observed patterns of results or effects; "why do things appear as they do." This process requires the use of common sense, expertise, and imagination to build and test theories about underlying causal processes. Open table in a new tab The results of realist research are described in terms of context, mechanisms, and outcomes. Realist methods aim to explain how contexts (such as people's beliefs or characteristics) influence the firing of mechanisms (processes and behaviors) through which interventions result in desired or undesired outcomes (Table 2).5Pawson R. The Science of Evaluation: A Realist Manifesto. Sage, 2013Crossref Google Scholar,6Westhorp G. Realist impact evaluation: an introduction. Overseas Development Institute, London, UK2014https://odi.org/en/publications/realist-impact-evaluation-an-introduction/Date accessed: April 8, 2023Google Scholar This is important because the empirical evidence used to inform practice is often derived from studies performed in more controlled environments than those seen in routine practice.7Skivington K. Matthews L. Simpson S.A. et al.A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance.BMJ. 2021; 374: n2061Crossref PubMed Scopus (1188) Google ScholarTable 2Outline of CMOCsTermDefinitionInterventionIn a single study, there can be single or multiple interventions. Interventions are resources, for example, training and education, information, services, or treatments, which are intended to be used in a certain way but can be responded to in different manners depending upon the conditions and setting in which they are applied. Thus, an intervention in isolation does not lead to a successful outcome; rather the outcome occurs as a result of how recipients use and apply it. Interventions (typically a health program or policy) are the focus of analysis in realist evaluation and synthesis.ContextContext refers to the environment that existed before the intervention was initiated. This can include the physical, cultural, or economic environment but also the background and characteristics of the individuals who take part in the intervention. The realist approach assumes that the contexts within which interventions take place are complex and open social systems. Within CMOCs, contexts are influencers of whether a specific mechanism acts.MechanismMechanisms are the causal pathways that are activated by the intervention and lead to the study outcome. Mechanisms are not usually measurable or directly observed, for example, a mechanism could be changing participants' attitudes or decision-making processes.OutcomesRealist synthesis and evaluation focus on real outcomes, as in observational studies and randomized controlled trials. Outcomes result from the intervention strategy, via described mechanisms, in the presence of specific contexts. Outcomes can be intended or unintended, with desired outcomes being those that would reflect a successful intervention.CMOCsRealist analyses link the above elements into CMOCs. CMOCs begin as ideas or hypotheses, before being tested and refined or refuted to form a final program theory, which demonstrates the processes that led to the outcome of the intervention.CMOC, context-mechanism-outcome configuration. Open table in a new tab CMOC, context-mechanism-outcome configuration. CKD affects up to 850 million people worldwide,8Jager K.J. Kovesdy C. Langham R. et al.A single number for advocacy and communication—worldwide more than 850 million individuals have kidney diseases.Kidney Int. 2019; 96: 1048-1050Abstract Full Text Full Text PDF PubMed Scopus (204) Google Scholar the majority of whom are managed in primary care. CKD management involves treatments to reduce the risk of kidney disease progression and cardiovascular complications. Although the benefit of individual treatments in improving these outcomes is clear from trial data, a challenge to effective population health is in delivering this evidence-based treatment at scale. Even in countries with well-resourced health care systems, a large proportion of people with CKD do not receive established treatments such as targeted blood pressure control, renin-aldosterone-angiotensin system inhibitors, and statins.9Major R. Shepherd D. Warwick G. Brunskill N. Prescription rates of cardiovascular medications in a large UK primary care chronic kidney disease cohort.Nephron. 2016; 133: 15-22Crossref PubMed Scopus (7) Google Scholar Newer treatments such as sodium-glucose cotransporter-2 inhibitors and finerenone are likely to also face barriers with optimal use. In an attempt to improve the delivery of evidence-based CKD treatments to populations, many complex interventions have been developed. These include automated prompts to primary care to alter patient management or refer to secondary care, alongside education for primary care clinicians, and novel routes to secondary care input such as non–patient-facing nephrologist review. Traditional systematic reviews that synthesized quantitative evidence of the effectiveness of these interventions found that evidence was critically lacking, existing interventions were poorly designed, and that the interventions tested were found to have "minimal impact" on clinical outcomes.10Galbraith L. Jacobs C. Hemmelgarn B.R. et al.Chronic disease management interventions for people with chronic kidney disease in primary care: a systematic review and meta-analysis.Nephrol Dial Transplant. 2018; 33: 112-121PubMed Google Scholar(p112),11Kamath C.C. Dobler C.C. McCoy R.G. et al.Improving blood pressure management in primary care patients with chronic kidney disease: a systematic review of interventions and implementation strategies.J Gen Intern Med. 2020; 35: 849-869Crossref PubMed Scopus (7) Google Scholar This lack of unified quantitative evidence of benefit may relate to the complexity of these interventions, variation in their design, and the influence of the context(s) in which they are delivered, including geographical factors, organizational factors, and the complex interactions between members of multidisciplinary teams across primary and secondary care. Therefore, as a step toward optimization of these interventions, we performed a realist synthesis of published evidence to describe how these interventions can work to improve population health.12Taylor D.M. Nimmo A.M. Caskey F.J. et al.Complex interventions across primary and secondary care to optimize population kidney health.Clin J Am Soc Nephrol. 2023; 18: 563-572Crossref PubMed Google Scholar This work aims to inform the identification of optimal intervention structures that warrant detailed evaluation for effectiveness and cost-effectiveness. We performed an initial scoping review to identify key components of these interventions. We then used an iterative process of electronic database searches13Booth A. Briscoe S. Wright J.M. The "realist search": a systematic scoping review of current practice and reporting.Res Synth Methods. 2020; 11: 14-35Crossref PubMed Scopus (36) Google Scholar to identify published evidence of relevant interventions. We identified 60 relevant studies, which we allocated randomly to the 6 researchers in our team, with backgrounds in nephrology, primary care, and realist methods. Each researcher reviewed their allocated studies, aiming to depict how the intervention described worked by describing contexts, mechanisms, and outcomes. These relationships are expressed as context-mechanism-outcome configurations (CMOCs), each proposing how, in specific context, mechanisms are activated, which then lead to specific outcomes. Although quantitative data can be used to infer causation to support a specific CMOC within a realist program theory, all types of data are considered, including qualitative data and informal evidence,14Sutcliffe K. Thomas J. Stokes G. et al.Intervention Component Analysis (ICA): a pragmatic approach for identifying the critical features of complex interventions.Syst Rev. 2015; 4: 140Crossref PubMed Scopus (51) Google Scholar such as descriptions made by the authors of strengths or weaknesses of intervention components or postulating why certain interventions were or were not found to be successful. Outcomes, mechanisms, and contexts are now described with reference to this example. Realist synthesis and evaluation focus on real outcomes, as in observational studies and RCTs. In our realist synthesis, the composite outcome of interventions to improve delivery of CKD care was population health, which included blood pressure control, medical therapy, medication concordance, and waiting times for clinic appointments.12Taylor D.M. Nimmo A.M. Caskey F.J. et al.Complex interventions across primary and secondary care to optimize population kidney health.Clin J Am Soc Nephrol. 2023; 18: 563-572Crossref PubMed Google Scholar We examined the mechanisms that underlie how and why particular approaches to CKD care deliver these outcomes and the contexts that must be present for those mechanisms to function. Rather than describing individual components of interventions, mechanisms are the processes through which these components function, contingent on behavioral and relationship changes involving human actors in social systems. Mechanisms are highly sensitive to context. Our realist synthesis described learning while doing—the mechanism by which clinicians (predominantly in primary care) learn through the process of engaging with CKD management interventions. This mechanism is enabled by multiple contexts, including clinical data delivery (clinical systems that provide all the data required for clinical decision-making) and clinical judgment (a clinician characteristic allowing implementation of guidance while considering patients' needs and preferences).12Taylor D.M. Nimmo A.M. Caskey F.J. et al.Complex interventions across primary and secondary care to optimize population kidney health.Clin J Am Soc Nephrol. 2023; 18: 563-572Crossref PubMed Google Scholar Subgroup, stratified, and adjusted analyses of experimental studies can explore variation in clinical effectiveness and may explain unexpected findings or inform implementation. However, realist evaluation and synthesis seek greater depth—attempting to go beyond interventional conditions that can be controlled for or stratified, which are rarely replicated in nonexperimental settings.15Bhaskar R. A Realist Theory of Science. Routledge, 1975Google Scholar In realist terms, contexts are critical not in terms of what they are, but in terms of what they cause.1Pawson R. Tilley N. Tilley N. Realistic Evaluation. Sage, 1997Google Scholar Contexts may be conceptualized as concrete "things": "tangible, fixed, observable features that trigger mechanisms," and as relational "forces": "dynamic features that shape the mechanisms through which the intervention works."16Greenhalgh J. Manzano A. Understanding 'context' in realist evaluation and synthesis.Int J Soc Res Methodol. 2022; 25: 583-595Crossref Scopus (51) Google Scholar(p584) Our realist synthesis described the contexts "clinical data delivery" and "clinical judgment," which are required in order for the mechanism "learning while doing" to function, which (combined with other mechanisms, enabled by other contexts) can result in the intervention's success.12Taylor D.M. Nimmo A.M. Caskey F.J. et al.Complex interventions across primary and secondary care to optimize population kidney health.Clin J Am Soc Nephrol. 2023; 18: 563-572Crossref PubMed Google Scholar In realist synthesis, hypothesized CMOCs are tested using evidence extracted from a systematic literature search. CMOCs are collated and coded to identify higher-level configurations that are generalizable across multiple studies.17Wong G. Greenhalgh T. Westhorp G. Pawson R. Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (Realist And Meta-narrative Evidence Syntheses—Evolving Standards) project.Health Serv Deliv Res. 2014; 2Crossref Google Scholar,18Greenhalgh T. Wong G. Westhorp G. Pawson R. Protocol—Realist and Meta-narrative Evidence Synthesis: Evolving Standards (RAMESES).BMC Med Res Methodol. 2011; 11: 115Crossref PubMed Scopus (178) Google Scholar These CMOCs are then refuted, refined, or accepted, and used to build a realist "program theory"—the evidence-based logic model that makes explicit causal connections between the "things done" in an intervention and the outcomes it achieves (Figure 1). In our realist synthesis of interventions in CKD, we performed a series of "data session" meetings, during which a long list of CMOCs was produced, alongside the intervention components linked to each CMOC. This process identified CMOCs that are active across multiple interventions and developed a description of how mechanisms interact with each other to result in desired outcomes from the interventions. The resulting model of interacting CMOCs forms a program theory for these interventions. A key strength of the realist approach is that these results can inform the development and implementation of future interventions across different contexts, in contrast to previous syntheses of quantitative data that led to uncertain conclusions on interventions' effectiveness.10Galbraith L. Jacobs C. Hemmelgarn B.R. et al.Chronic disease management interventions for people with chronic kidney disease in primary care: a systematic review and meta-analysis.Nephrol Dial Transplant. 2018; 33: 112-121PubMed Google Scholar,11Kamath C.C. Dobler C.C. McCoy R.G. et al.Improving blood pressure management in primary care patients with chronic kidney disease: a systematic review of interventions and implementation strategies.J Gen Intern Med. 2020; 35: 849-869Crossref PubMed Scopus (7) Google Scholar Forming the program theory can also identify areas where evidence is missing. For instance, in our realist synthesis, evidence of patients' perceptions of the interventions was not readily available, and it remains unclear how interventions should be designed to promote patient engagement with treatment. Without this information, the confidence with which we can assume that existing interventions will be successful is reduced. Future work will incorporate patients' views on the function of these interventions into this program theory, including defining patient-centered outcome criteria. In realist evaluation, primary data collection is targeted at testing CMOCs,19Wong G. Westhorp G. Manzano A. et al.RAMESES II reporting standards for realist evaluations.BMC Med. 2016; 14: 96Crossref PubMed Scopus (352) Google Scholar,20Greenhalgh T. Wong G. Jagosh J. et al.Protocol—the RAMESES II study: developing guidance and reporting standards for realist evaluation.BMJ Open. 2015; 5e008567Crossref PubMed Scopus (84) Google Scholar which often includes qualitative interviews with participants, personnel, and other stakeholders.21Mukumbang F.C. Marchal B. Van Belle S. van Wyk B. Using the realist interview approach to maintain theoretical awareness in realist studies.Qual Res. 2020; 20: 485-515Crossref Scopus (62) Google Scholar,22Manzano A. The craft of interviewing in realist evaluation.Evaluation. 2016; 22: 342-360Crossref Scopus (289) Google Scholar In realist interviewing, interviewees may be invited to comment on the proposed program theories, where "the theoretical postulates/conceptual structures under investigation are open for inspection in a way that allows the respondent to make an informed and critical account of them."23Pawson R. Theorizing the interview.Brit J Sociol. 1996; 47: 295-314Crossref Scopus (127) Google Scholar(p313) The final program theory contains the CMOCs that remain after the review process and identifies the circumstances under which intervention components are likely to work again.5Pawson R. The Science of Evaluation: A Realist Manifesto. Sage, 2013Crossref Google Scholar,24Lawson T. Economics and Reality.Vol. 9. Psychology Press, 1997Crossref Google Scholar There has been a growing trend for RCTs to adopt a more pragmatic approach, by being embedded as far as possible within normal clinical practice,25Bonell C. Fletcher A. Morton M. et al.Realist randomised controlled trials: a new approach to evaluating complex public health interventions.Soc Sci Med. 2012; 75: 2299-2306Crossref PubMed Scopus (354) Google Scholar and the use of mixed methods is increasingly seen in nephrology and other clinical research areas.26Bailey P.K. Hole B.D. Plumb L.A. Caskey F.J. Mixed-methods research in nephrology.Kidney Int. 2022; 101: 895-905Abstract Full Text Full Text PDF PubMed Scopus (2) Google Scholar This may represent a move toward deepening the understanding of how interventions work, rather than relying solely on RCT evidence to assess whether an intervention works. Even outside formal realist evaluation and synthesis, this may enrich the literature with pragmatic, inclusive, and relevant evidence, and is likely to enhance realist syntheses of published material and prepare the ground for future realist evaluations. The authors of this article believe this to be a positive development, and this approach is likely to increase the reliability of evidence used to justify the implementation of interventions in practice. Standards for reporting realist studies are described in the RAMESES (Realist And Meta-narrative Evidence Syntheses: Evolving Standards) guidance for realist evaluation19Wong G. Westhorp G. Manzano A. et al.RAMESES II reporting standards for realist evaluations.BMC Med. 2016; 14: 96Crossref PubMed Scopus (352) Google Scholar and realist synthesis,17Wong G. Greenhalgh T. Westhorp G. Pawson R. Development of methodological guidance, publication standards and training materials for realist and meta-narrative reviews: the RAMESES (Realist And Meta-narrative Evidence Syntheses—Evolving Standards) project.Health Serv Deliv Res. 2014; 2Crossref Google Scholar along with reporting checklists. Aiming to explore the use of realist evaluation and synthesis in kidney care, we undertook a search for published literature describing realist studies in kidney care. This approach relied on authors declaring realist methodology. Although it does not permit capture of studies using realist methods that are not declared, our intention was to broadly describe studies that report using realist methods to describe or investigate kidney care rather than performing a comprehensive systematic review. Searches were performed on February 21, 2023 (search strategy shown in Supplementary Table S1 and results in Supplementary Figure S1). Two authors (AN and GJM-T) screened titles and filtered these based on whether they described a realist evaluation or synthesis relevant to kidney care. We found 9 published studies, 4 protocols, and 3 conference abstracts. The published studies are summarized below and in Table 3.27Greenhalgh T. Humphrey C. Hughes J. et al.How do you modernize a health service? A realist evaluation of whole-scale transformation in London.Milbank Q. 2009; 87: 391-416Crossref PubMed Scopus (236) Google Scholar, 28de Lusignan S. Informatics as tool for quality improvement: rapid implementation of guidance for the management of chronic kidney disease in England as an exemplar.Healthc Inform Res. 2013; 19: 9-15Crossref PubMed Scopus (4) Google Scholar, 29Schick-Makaroff K. Levay A. Thompson S. et al.An evidence-based theory about PRO use in kidney care: a realist synthesis.Patient. 2022; 15: 21-38Crossref PubMed Scopus (9) Google Scholar, 30Conway J. Lawn S. Crail S. McDonald S. Indigenous patient experiences of returning to country: a qualitative evaluation on the Country Health SA Dialysis bus.BMC Health Serv Res. 2018; 18: 1010Crossref PubMed Scopus (16) Google Scholar, 31Harwood L. Pye D. Clinton E. et al.Innovations in hemodialysis care: an evaluation of quality and the patient experience.J Patient Exp. 2020; 7: 1278-1285Crossref PubMed Google Scholar, 32Tsang J.Y. Blakeman T. Hegarty J. et al.Understanding the implementation of interventions to improve the management of chronic kidney disease in primary care: a rapid realist review.Implement Sci. 2016; 11: 47Crossref PubMed Google Scholar, 33O'Halloran P. Noble H. Norwood K. et al.Advance care planning with patients who have end-stage kidney disease: a systematic realist review.J Pain Symptom Manage. 2018; 56: 795-807.e18Abstract Full Text Full Text PDF PubMed Scopus (33) Google Scholar, 34Carswell C. Reid J. Walsh I. et al.Complex arts-based interventions for patients receiving haemodialysis: a realist review.Arts Health. 2021; 13: 107-133Crossref PubMed Scopus (10) Google Scholar, 35Carswell C.E. Reid J. Walsh I. et al.Development of a complex arts-based intervention for patients with end-stage kidney disease whilst receiving haemodialysis.Pilot Feasibility Stud. 2021; 7: 127Crossref PubMed Scopus (3) Google Scholar They come from different groups and geographical locations, explore various subspecialty areas, and all except 1 study was published within the last 10 years, demonstrating that this research technique has only been employed relatively recently.Table 3Examples of context-mechanism-outcome configurations for published studies using realist evaluation and synthesis in the field of kidney careStudySummary of paperIntervention examplesSettingOutcome examplesGreenhalgh et al., 200927Greenhalgh T. Humphrey C. Hughes J. et al.How do you modernize a health service? A realist evaluation of whole-scale transformation in London.Milbank Q. 2009; 87: 391-416Crossref PubMed Scopus (236) Google ScholarRealist evaluation of interventions to modernize health careCross-boundary project managers, shared IT systems, networking meetings, visits to other centersFour health care organizations spanning 3 specialties in London, UKModernizing services: improved efficiency, effectiveness and patient-centered health carede Lusignan, 201328de Lusignan S. Informatics as tool for quality improvement: rapid implementation of guidance for the management of chronic kidney disease in England as an exemplar.Healthc Inform Res. 2013; 19: 9-15Crossref PubMed Scopus (4) Google ScholarRealist synthesis (using nonsystematic search) of the use of informatics to implement national guidance on the management of chronic kidney disease (CKD)Health informatics including computerized health records, recording on disease registers, frequently asked questions monographEng