COMETA : A Corpus for Medical Entity Linking in the Social Media (2020)

Abstract: Whilst there has been growing progress in Entity Linking (EL) for general language, existing datasets fail to address the complex nature of health terminology in layman’s language. Meanwhile, there is a growing need for applications that can understand the public’s voice in the health domain. To address this we introduce a new corpus called COMETA, consisting of 20k English biomedical entity mentions from Reddit expert-annotated with links to SNOMED CT, a widely-used medical knowledge graph. Our corpus satisfies a combination of desirable properties, from scale and coverage to diversity and quality, that to the best of our knowledge has not been met by any of the existing resources in the field. Through benchmark experiments on 20 EL baselines from string- to neural-based models we shed light on the ability of these systems to perform complex inference on entities and concepts under 2 challenging evaluation scenarios. Our experimental results on COMETA illustrate that no golden bullet exists and even the best mainstream techniques still have a significant performance gap to fill, while the best solution relies on combining different views of data.

## introduction social media has become a dominant means for users to share their opinions, emotions and daily experience of life. a large body of work has shown that informal exchanges such as online forums can be leveraged to supplement traditional approaches to a broad range of public health questions such as monitoring suicidal risk and depression @xcite , domestic abuse @xcite , cancer @xcite , and epidemics @xcite . one of the widely exercised steps to establish a semantic understanding of social media is entity linking (el), i.e., the task of linking entities within a text to a suitable concept in a reference knowledge graph (kg) @xcite @xcite . however, it is well-documented that poorly composed contexts, the ubiquitous presence of colloquialisms, shortened forms, typing/spelling mistakes, and out-of-vocabulary words introduce challenges for effective utilisation of social media text @xcite . these challenges are exacerbated in el for user generated content (ugc) in the health domain for two main reasons: lack of dedicated annotated resources for training el models, and entanglement of the aforementioned challenges in general social media with the inherent complexity of the health domain and its terminology (see for example, in figure 1 we show sentences taken from social media where the semantics of the concept linking is complex and context-dependent. in the first case, "diagnosed with gad where by ## the cometa corpus the cometa corpus satisfies multiple properties which we will explain throughout this section: consistency. cometa has been annotated by biomedical experts to a high quality using snomed ct concepts (sctids) -a standard for clinical information interchange ( §3.2); scale and scope. to the best of our knowledge, with at 20k concept mentions, it is the largest ugc corpus for medical el. annotated entities cover a wide range of concepts including symptoms, diseases, anatomical expressions, chemicals, genes, devices and procedures across a range of conditions ( §3.3); distribution. we release the full corpus along with two sampling strategies (stratified and zeroshot) to prevent over-optimistic reporting of performance @xcite : while stratified is designed to show the ability of systems to recognise known concepts with possibly novel mentions, zero-shot is designed to test for recognising novel concepts ( §3.4). ## experiments and results in this section we conduct a diverse set of el experiments, where we apply different simple and complex paradigms to link the annotated entities (and the sentences in which they appear) with the corresponding snomed concepts. we follow previous works in biomedical entity linking and use top-k accuracy (@xmath0, 10}) to evaluate performance of el systems (d'souza and ng, 2015). note that acc@10 is only computed for systems returning a ranked list and measures if the correct concept is contained within the top 10 concepts returned by the system. we also report mean reciprocal rank (mrr, craswell (2018)), which instead measures the position of the correct concept in the list of concepts returned by the system. details about training as well as model and hardware configurations are available in appendix a.5. our baselines cover both string/dictionary-based algorithms ( §4.1) which are good at capturing surface-level similarities, and neural models capable of incorporating contextual information ( §4.2), where we experiment with a new multi-level attention mechanism based on bert to allow more efficient incorporation of context. finally, to achieve the best possible performance, we combine these models in a back-off setting where we leverage the benefits of each paradigm ( §4.3). when describing the results, we will report the results on the general split and place the results on the specific split in parentheses. ## discussion the cometa corpus introduces a challenging scenario for entity linking systems from both ml and nlp perspectives. in this section we summarise these challenges, our findings, and shed light on aspects that demand future attention: domain-specific language. el systems similar to our baselines are not uncommon in the biomedical domain @xcite used a similar dictionary-bert ensemble model to achieve the best performance in the 2019 craft shared task @xcite on biomedical literature. however, in their case, the neural component offered a much higher contribution highlighting the underlying challenges in medical layman's language. additionally, probing our proposed multi-level attention for bert, we observed that a more flexible utilisation of context is effective in understanding the diverse contextual cues. ## conclusion we presented cometa, a unique corpus for its scale and coverage which is curated to maintain high quality annotations of medical terms in layman's language on reddit with concepts from snomed knowledge graph. different evaluation scenarios were designed to compare the performance of conventional dictionary/string-matching techniques against the mainstream neural counterparts and revealed that these models complement each other very well and the best performance is achieved by combining these paradigms. nonetheless, the missing performance of 28-46% (depending on the evaluation scenario) encourages future research on this area to take this corpus as a challenging yet reliable evaluation benchmark for further development of models specific to this domain. cometa is available by contacting the last author via e-mail or following the instructions on https://www.siphs.org/ . we release the pre-trained embeddings and the code to replicate our baselines online at https://github.com/ cambridgeltl/cometa .