Sumotosima : A Framework and Dataset for Classifying and Summarizing Otoscopic Images (2024)

Abstract: Otoscopy is a diagnostic procedure to examine the ear canal and eardrum using an otoscope. It identifies conditions like infections, foreign bodies, eardrum perforations, and ear abnormalities. We propose a novel resource-efficient deep learning and transformer-based framework, Sumotosima (Summarizer for Otoscopic Images), which provides an end-to-end pipeline for classification followed by summarization. Our framework utilizes a combination of triplet and cross-entropy losses. Additionally, we use Knowledge Enhanced Multimodal BART, where the input is fused textual and image embeddings. The objective is to deliver summaries that are well-suited for patients, ensuring clarity and efficiency in understanding otoscopic images. Given the lack of existing datasets, we have curated our own OCASD (Otoscopy Classification And Summary Dataset), which includes 500 images with 5 unique categories, annotated with their class and summaries by otolaryngologists. Sumotosima achieved a result of 98.03%, which is 7.00%, 3.10%, and 3.01% higher than K-Nearest Neighbors, Random Forest, and Support Vector Machines, respectively, in classification tasks. For summarization, Sumotosima outperformed GPT-4o and LLaVA by 88.53% and 107.57% in ROUGE scores, respectively. We have made our code and dataset publicly available at https://github.com/anas2908/Sumotosima

## introduction otoscopy is a medical procedure using an otoscope to visually inspect the ear canal and eardrum. it is performed by healthcare professionals such as otorhinolaryngologists @xcite . this examination helps identify conditions like ear infections, blockages, eardrum perforations, and other abnormalities. otoscopy is crucial for detection in treatment of ear-related issues, ensuring better auditory health and preventing complications that can arise from untreated conditions. our objective is to develop efficient summaries for otoscopic ear images that are both clear and well-defined for patients. a common complaint from patients is the lack of interaction and explanations from healthcare professionals, often due to the high workload and limited availability of medical staff, which limits the time doctors can spend with each patient. by improving the clarity of information through ai-generated summaries, we aim to enhance patient understanding and interaction with their healthcare providers. these summaries identify the disease if any and describe various conditions, including areas of redness, infection spots, ear drum perforations, etc. further details are provided in section 3. to facilitate our research, we identified lack of availability in open-source otoscopic image datasets, and none of them accompanying summaries or captions. to address this, we curated our dataset comprising 500 images equally distributed into 5 categories : 'acute otitis media', 'cerumen impaction', 'chronic otitis media', 'myringosclerosis' and 'normal' from sources such as @xcite , and publicly available images on google. it was was then filtered to remove noisy and redundant data that could introduce bias into the supervised learning results. additionally, the images were summarized by an otorhinolaryngologist to ensure the dataset's robustness. further details and annotation guidelines are provided in section 3.1. our framework is divided into two stages: classification followed by summarry generation. we observed that separating the classification task by applying a combination of triplet loss @xcite and cross-entropy loss and passing resultant information alongside the prompt to a multimodal bart @xcite yielded superior results. the main contributions of our proposed research are as follows: for the summarization task, it demonstrated an improvement of 88.53% compared to gpt-4o ## related works the classification of otoscopic images using traditional machine learning architectures has been extensively studied. notable and recent research by @xcite @xcite has demonstrated the potential of machine learning for diagnosing ear conditions using otoscopic images. in recent years, convolutional neural networks (cnns) have become a popular approach for analyzing otoscopic images. these networks have shown better accuracy in detecting common ear diseases such as otitis media, tympanic membrane perforations, and cerumen impaction, as highlighted by studies from @xcite @xcite . moreover, both machine learning (ml) and deep learning (dl) algorithms have been employed to classify various ear conditions and segment areas of interest in otoscopic images, aiding in more precise diagnosis and treatment planning, as demonstrated by @xcite . additionally, the integration of artificial intelligence (ai) with smartphone-connected otoscopes has facilitated remote diagnosis by analyzing images captured using smartphone attachments, providing real-time feedback to healthcare providers and patients, as shown by @xcite . despite significant advancements in classification and segmentation tasks of otoscopic images using ai, there remains a lack of open-source datasets in this field. furthermore, no substantial work has been done on summarizing otoscopic findings for better patient understanding. addressing these gaps could significantly enhance the application of ai in otoscopy, which is the focus of our current research. ## dataset our dataset, ocasd (otoscopic classification and summary dataset), was curated in response to the scarcity of available datasets and is derived from previously existing open-source datasets (po-lat, 2021; @xcite . instances in @xcite were found to be extensively redundant for the categories 'cerumen impaction,' 'chronic otitis media,' 'myringosclerosis,' and 'normal.' to address this, we manually removed redundant or nearly identical instances, and an otorhinolaryngologist handpicked 100 images for each of the four categories, ensuring they were the most informative and contributive. a similar approach was followed for (polat, 2021) for category 'acute otitis media' however, due to a lack of informative images, some were additionally sourced from the internet. the distribution of the curated collection is shown in ## conclusion in this work, we introduced a novel pipeline for classifying and generating summaries for otoscopic images of the middle ear, with the objective of developing summaries that are both well-defined and patient-friendly, addressing the challenge of insufficient explanations from medical professionals due to their hectic schedules and limited time per patient. unlike previous approaches that relied on tra-ditional machine learning algorithms or straightforward deep learning architectures, our model utilizes a combination of triplet loss and cross-entropy loss, built upon the resnext-18 architecture, achieving a classification accuracy of 98.03%, surpassing all baselines on our ocasd dataset, which comprises 500 images across 5 different classes. as this is the first work on summarization for otoscopic images, we addressed the lack of annotated summary datasets by creating summaries for all 500 images, combining class-enriched prompts with image embeddings obtained from a fine-tuned clip model, and feeding these into a multimodal bart model for summary generation. our framework, sumotosima, significantly outperformed gpt-4o and llava in summarization, with improvements of 88.53% and 107.57% in rouge scores, respectively. looking ahead, we plan to extend this work by incorporating additional metadata, such as patient age, gender, medical history, and educational background, to generate more robust and patient-specific summaries.