POS1515-HPR EXAMINING THE QUALITY OF SURVEYS USED IN RHEUMATOLOGY HEALTH PROFESSIONS EDUCATION RESEARCH
Authors: M. O’sullivan, M. Battistone, A. Barker, N. Schlesinger
Journal: Annals of the Rheumatic Diseases (2024)

<h3>Background:</h3> Health professions education (HPE) is an evolving discipline with a growing body of research. HPE research frequently utilizes surveys of learners and/or teachers to collect data and capture information about abstract concepts. Unfortunately, a recent examination of the methodologic rigor of surveys published in three leading HPE journals has raised concerns about the quality of this work [1]. Surveys that do not follow best practices in design often include questions that are difficult to interpret and answer; this can lead to measurement errors and negatively impact study reliability and validity, resulting in low-quality data. An interprofessional community of clinician scholar educators and HPE researchers is becoming recognized within the European Alliance of Associations for Rheumatology (EULAR) and the American College of Rheumatology (ACR), and HPE scholarship is increasingly published in rheumatology journals. The relative prevalence and quality of HPE surveys presented in these journals have not been examined. This represents a gap in our knowledge of the state of the discipline. This is important to at least three stakeholder groups: 1) rheumatology education scholars who plan and conduct research relevant to the future workforce, 2) sponsors and funding agencies that support this work, and 3) reviewers and journal editors who critically appraise and determine the scientific merit of a report. <h3>Objectives:</h3> This study had three objectives: 1) to determine the 5-year trend of numbers of HPE-themed studies that used surveys and were published in leading rheumatology journals, 2) to assess the quality of these surveys using a previously described rubric, and 3) to determine the frequency of response rate reporting and the nonresponse bias as a potential limitation. <h3>Methods:</h3> Two authors (MO, MJB) performed a search of all articles in 10 leading rheumatology journals, plus the recently introduced <i>ACR Open Rheumatology</i>, published over a 5-year period from January 2019 through December 2023. Titles and abstracts were independently screened for eligibility, and inclusion decisions were made in consensus between MO and MJB. Articles were included if they used at least 1 survey tool and included health professionals as respondents. Quality was assessed using a coding rubric developed by Artino. The rubric evaluated the reliability and validity of the reported evidence and examined the methodologic rigor of specific survey items. All authors independently coded 4 papers then met to compare ratings and resolve differences; 3 authors (MO, AMB, MJB) divided the remaining publications so that each article was independently coded twice. <h3>Results:</h3> 20 articles met the inclusion criteria for evaluation using the coding rubric. Of these, 11 (55%) included a complete copy of the fully formatted survey for review. 6 (30%) were adapted from previously published surveys. Response rates were reported (or could be calculated) in 13 (65%) of studies, ranging from 5% to 100%. Reliability and validity evidence were reported for 11 (55%) and 1 (5%) of the surveys, respectively. 9 of the 11 (81%) surveys with a fully formatted copy available for review violated at least one best practice for design, with the most common violations being agreement-response and multi-barreled items. <h3>Conclusion:</h3> Despite the widespread use of surveys in medical education, the majority of HPE articles published in leading rheumatology journals failed to report validity and reliability evidence, and a substantial proportion of the survey instruments violated established best practices in survey design. It is imperative that surveys are carefully designed or adapted in accordance with an established coding rubric and evaluated for reliability and validity to ensure the highest quality data from these studies in the evolving field of rheumatology HPE research. <h3>REFERENCES:</h3> [1] Artino AR Jr, Phillips AW, Utrankar A, Ta AQ, Durning SJ. "The Questions Shape the Answers": Assessing the Quality of Published Survey Instruments in Health Professions Education Research. <i>Acad Med</i>. 2018;93(3):456-463. <h3>Acknowledgements:</h3> The project described was supported in part by the Health Professions Education Evaluation and Research (HPEER) Fellowship, through the VA Office of Academic Affiliations Advanced Fellowships Program (3HPE05201B). The content is solely the responsibility of the authors and does not necessarily represent the official views of the HPEER Program or the Department of Veterans Affairs. <h3>Disclosure of Interests:</h3> Maddie O'Sullivan: None declared, Michael Battistone: None declared, Andrea Barker: None declared, Naomi Schlesinger Serves or has served on an advisory board or as a consultant for Novartis, Horizon Pharma, Selecta Biosciences, Olatec, Mallinckrodt Pharmaceuticals, Celgene, AstraZeneca, Horizon Pharmaceuticals, Swedish Orphan Biovitrum (SOBI) and Bristol-Myers Squibb (BMS)., Has received grants from Pfizer, AMGEN, and AstraZeneca.