POS1094 THE EULAR TOOL IS SUPERIOR TO A GENERIC TOOL IN THE ASSESSMENT OF MUSCULOSKELETAL ULTRASOUND COMPETENCE: A VALIDITY STUDY
Authors: S. M. D. Carstensen, S. A. Just, M. Pfeiffer-Jensen, M. Østergaard, L. Konge, L. Terslev
Journal: Annals of the Rheumatic Diseases (2023)

<h3>Background</h3> Musculoskeletal ultrasound (MSUS) is a widely implemented tool in rheumatological routine care. It has been established which competencies are relevant for rheumatologists[1], and competency levels have already been developed within European League Against Rheumatism (EULAR)[2] and European Federation of Societies for Ultrasound in Medicine and Biology (EFSUMB)[3]. Furthermore, the content and conduct of MSUS courses have been published [2]. Thus, the only component missing is an objective assessment tool supported by a contemporary validity framework to determine when residents have acquired the competencies relevant for independent practice[1]. A generic assessment tool for the assessment of ultrasound competence, The Objective Structured Assessment of Ultrasound skills (OSAUS)[4], has been developed and is already used in clinical practice in several other medical fields[5,6]. EULAR has developed another assessment tool specific for MSUS. Although both OSAUS and the EULAR tool are consensus-based, the tools' reliability and their discriminatory ability have not been explored. <h3>Objectives</h3> To explore the validity of the generic OSAUS tool and the EULAR tool for the assessment of MSUS operator performance. Moreover, to examine the optimal number of cases and raters needed to ensure a reliable assessment of residents. <h3>Methods</h3> We included 30 physicians, representing MSUS novices, intermediates, and experts. Validity evidence was gathered using the gold standard contemporary framework of Messick[8] All participants performed MSUS examinations of four joint areas: the wrist, fingers, ankle, and shoulder on the same patient with rheumatoid arthritis. All performances were video recorded and subsequently assessed in completely random order by two blinded raters using first the OSAUS assessment tool and then one month after, followed by new randomization of the videos, the EULAR tool. <h3>Results</h3> A total of 120 videos were assessed twice by two assessors. The inter-rater reliability was high for both tools: 0.807 (OSAUS) versus 0.848 (EULAR), and both tools demonstrated excellent inter-case reliability of 0.970 (OSAUS) versus 0.964 (EULAR). The ability to discriminate between different experience levels and MSUS performance was significant (p &lt;0.001) for both the OSAUS and the EULAR tool. Moreover, by conducting a series of Decision-studies, we estimated the optimal number of cases and raters needed to ensure a reliable assessment (generalizability coefficient &gt;0.8). A reliable assessment could be reached by using either one rater and three cases or two raters and one case (OSAUS), whereas the EULAR assessment tool only needed one rater assessing two cases or two raters assessing one case (Figure 1). <h3>Conclusion</h3> MSUS operator competences can be assessed in a reliable and valid way using either the OSAUS or the EULAR assessment tool, thereby allowing a uniform competency-based MSUS education in the future. Although both tools demonstrated high inter-rater reliability, the EULAR tool was superior to OSAUS. Using the EULAR tool in clinical practice will allow a single assessor to provide a reliable MSUS assessment of the trainee by observing only two different MSUS examinations. <h3>References</h3> [1]Brown AK et al. Rheumatology; 2007. p. 391–7. [2]Naredo E et al. Ann Rheum Dis. 2008 Jul;67(7):1017–22. [3]Terslev L et al. Ultraschall der Medizin. 2013;34(5):475–7. [4]Tolsgaard MG et al. PLoS One. 2013 Feb 28;8(2):e57687. [5]Todsen T et al. Laryngoscope. 2018 Jun 1;128(6):1346–52. [6]Todsen T et al. Ann Surg. 2015;261(2):309–15. [7]Tolsgaard MG et al. Ultrasound Obstet Gynecol. 2014;43(4):437–43. [8]Messick S. Am Counc Educ Macmillan. 1989;13–104. <h3>Acknowledgements</h3> We would like to express our appreciation to the physicians and the healthy volunteer who donated their time to participate in this study. We would like to thank GE Healthcare for technical support. <h3>Disclosure of Interests</h3> None Declared.