OP0004 AUTOMATED RECOGNITION AND MONITORING OF DORSAL SKIN FOLDS BY A CONVOLUTIONAL NEURAL NETWORK AS A POTENTIAL DIGITAL BIOMARKER FOR JOINT SWELLING IN PATIENTS WITH RHEUMATOID ARTHRITIS
Authors: T. Hügle, L. Caratsch, M. Matteo Caorsi, J. Maglione, D. Dan, A. Dumusc, M. Blanchard, G. Kalweit, M. Kalweit
Journal: Annals of the Rheumatic Diseases (2022)

Background To monitor rheumatic diseases, digital biomarkers such as wearables are of increasing interest, but they lack disease specificity. Objectives In this study, we apply convolution neural networks (CNN) to real world hand photographs in order to automatically detect, extract and analyse dorsal finger folds lines as a correlate of proximal interphalangeal (PIP) joint swelling in patients with rheumatoid arthritis (RA). Methods Hand photographs from 190 RA patients were taken by a smartphone camera in a standardized manner. PIP joints were categorised as either swollen or not swollen based on clinical judgement and ultrasound. Images were automatically preprocessed by cropping PIP joints and extracting dorsal finger folds. Subsequently, metrical analysis of dorsal finger folds was performed and a CNN was trained to classify the dorsal finger lines into swollen versus non-swollen joints. Representative horizontal finger folds were also quantified in a subset of patients before and after resolution of PIP swelling and in patients with disease flares, respectively. Results In swollen joints, the number of automatically extracted double-contoured, deep skinfold imprints was significantly reduced compared to non-swollen joints (1.3, SD 0.8 vs. 3.3, SD 0.49). The joint diameter / deep skinfold ratio was significantly higher in swollen (4.1, SD 1.4) versus non-swollen joints (2.1 SD 0.6). The CNN model successfully differentiated swollen from non-swollen joints based on finger fold patterns with a validation accuracy of 0.84. A heatmap of the original images obtained by an extraction algorithm confirmed finger folds as the region of interest for correct classification. After significant response to DMARD +/- corticosteroid therapy, longitudinal metrical analysis of eight representative deep finger folds showed a decrease of the mean diameter/ finger fold length (finger fold index, FFl) from 3.03 (SD 0.68) to 2.08 (SD 0.57). Conversely, the FFI increased in patients with a flare of joint swelling. Conclusion Automated preprocessing and the application of CNN algorithms in combination with longitudinal metrical analysis of dorsal finger fold patterns extracted from real world hand photos might serve as a digital biomarker in RA. Figure 1. Automated finger fold recognition to monitor rheumatoid arthritis (RA). Hand photographs are taken by a smartphone (A). Hands, and subsequently proximal interphalangeal (PIP) joints, are automatically recognized and extracted. Finger fold lines are isolated from the images, measured and related to the joint diameter (B,C). A convolutional deep neural network was used to train a model for classification of extracted finger fold patterns into swollen vs. non swollen joints (D). On cropped PIP joint images, the heatmap of the same classification task confirms finger folds as the region of interest (E). Disclosure of Interests Thomas Hügle Shareholder of: Atreon SA., Speakers bureau: Multiple. Not relevant for this work., Grant/research support from: Multiple. Not for this work., Leo Caratsch: None declared, Matteo Matteo Caorsi Employee of: MC is an employee of L2F., Jules Maglione: None declared, Diana Dan: None declared, Alexandre Dumusc Speakers bureau: Multiple. Not relevant for this work., Marc Blanchard Shareholder of: Atreon SA., Gabriel Kalweit: None declared, Maria Kalweit: None declared.