Overview of the MEDIQA - M 3 G 2024 Shared Task on Multilingual Multimodal Medical Answer Generation (2024)

Abstract: Remote patient care provides opportunities for expanding medical access, saving healthcare costs, and offering on-demand convenient services. In the MEDIQA-M3G 2024 Shared Task, researchers explored solutions for the specific task of dermatological consumer health visual question answering, where user generated queries and images are used as input and a free-text answer response is generated as output. In this novel challenge, eight teams with a total of 48 submissions were evaluated across three language test sets. In this work, we provide a summary of the dataset, as well as results and approaches. We hope that the insights learned here will inspire future research directions that can lead to technology that deburdens clinical workload and improves care.

## introduction driven by long patient wait times, high medical costs and physician burnout, remote patient care delivery (e.g. e-visits, e-mails) provides a cost effective solution that lowers facility expenditures, allows flexible schedules for both clinicians and patients, and expands health care access @xcite . the trend, already in motion due to the maturation of telecommunication technologies and the proliferation of health portals, was massively accelerated by the onset of the global covid 19 epidemic in 2019. five years later, today, while remote technologies allow for the conveniences of patient care delivered conveniently from one's own home, this poses new challenges for providers who need to meet the new demand, where patients can request services at any time of the day, creating a perception of "never-ending work" @xcite . automatic response generation may alleviate doctor burden by providing suggestions when answering patient queries, speeding up response throughput. in this work, we present the mediqa @xcite multilingual & multimodal medical answer generation (m3g) shared task, which is focused on the problem of multimodal answer generation in the space of dermatology, evaluated in multiple languages (english, chinese, and spanish). specifically, a health related question along with one or more images is posed; the expected task is to generate an appropriate answer response. previous editions of the mediqa shared tasks have featured radiology-related visual question answering @xcite ) and text-only consumer health answer generation @xcite . other prior work in medical vqa includes images in the space of pathology and gi-tract @xcite . meanwhile, previous work in dermatological image classification focused on image-only input and multi-class classification @xcite . this is the first shared task to incorporate visual question answering for user generated health queries and images. ## conclusion and future work open-ended consumer health visual question answering remains a challenging problem. this shared tasks highlights several areas for future work. one aspect is related to the generation of a dermatology common problem gold standard. here we used a dataset with multiple references, some with varying opinions. for the dermatological specialty, a true gold standard with pathological lab confirmation is difficult to obtain in real life. this reflects the realities of current healthcare technology and costs -biological sampling and assays are only reserved for the most severe cases. thus, datasets with biopsy observations are highly biased towards problems suspected to be malignant; whereas the plethora of other common-place maladies will remain unconfirmed. textbook images and diagnosis labels, on the other hand, will not include usergenerated queries. this is a non-trivial hurdle if an unequivocal dermatological vqa gold standard beyond medical doctor opinion is to be achieved. furthermore, the dataset here limits responses to queries to a single turn -however multiple turns are necessary for clarification purposes in real clinical settings. another future direction is the development of mature evaluation methods when multiple references of varying quality is available. in past trec competitions, one evaluation strategy included the employment of expert humans who would annotate each participant system based on answer quality @xcite . ratings include categories: (a) correct and complete answer, (b) correct but incomplete, (c) incorrect but related, and (d) incorrect. in this task, we sought to incorporate this automatically in terms of weighing response answers for bleu. however, although this sidesteps a need for a human expert to rate each system output, this method still relies on some human annotation of the gold standard instances. as well, the final scoring depends heavily on the quality and variety of existing answers; this leaves room for metric exploitation given the data biases. for example, on observation of the test set, although responses may include a variety of responses including recommended diagnosis, treatments, and test suggestions; since most responses at least give a diagnosis, it is advantageous to optimize for a short disease response instead of try to add more details and possibly incur penalties with an incorrect suggestion. furthermore, mentioned medical concepts may have hierarchical relations with those the gold standard for current metrics do not take into account for well. for example, atopic dermatitis is equivalent to eczema and is a subtype of dermititis -however, eczema is not the same as contact dermatitis. depending on the available combinations of gold responses, the same system output may receive different scores. in this shared task, a variety of solutions were explored to provide solutions for the dermatological vqa. we hope that the benchmarks provided here, the insights from different systems, and the identified methological problems will inspire future research directions.