CLIMATELI : Evaluating Entity Linking on Climate Change Data (2024)

Abstract: Climate Change (CC) is a pressing topic of global importance, attracting increasing attention across research fields, from social sciences to Natural Language Processing (NLP). CC is also discussed in various settings and communication platforms, from academic publications to social media forums. Understanding who and what is mentioned in such data is a first critical step to gaining new insights into CC. We present CLIMATELI (CLIMATe Entity LInking), the first manually annotated CC dataset that links 3,087 entity spans to Wikipedia. Using CLIMATELI (CLIMATe Entity LInking), we evaluate existing entity linking (EL) systems on the CC topic across various genres and propose automated filtering methods for CC entities. We find that the performance of EL models notably lags behind humans at both token and entity levels. Testing within the scope of retaining or excluding non-nominal and/or non-CC entities particularly impacts the models’ performances.

## introduction climate change (cc) is a well-established and omnipresent concept influencing daily lives. natural language processing (nlp) tasks such as entity linking (el) facilitate knowledge base population and empower individuals to enhance their cc understanding. for example, after reading the news about "young indigenous women from mexico and morocco unite for cop27," one could click the following wikipedia links and explore further about cop27 or climate _ of _ mexico. el provides easy access to cc-related knowledge and endows downstream applications like extracting information regarding stakeholders: policymakers, scientists, administrators, and etc. @xcite . to gain valuable and broad access to relevant cc information, a major challenge is that el needs to robustly process different texts, both in terms of we present climateli, foot_0 the first manually annotated cc el corpus covering five english genres in §3. §4 introduces three el systems for evaluation and applies an automatic filtering mechanism for cc-related and nominal entities. §5- §6 analyze models' overall performance and on specific genres and threshold conditions. §8 concludes the paper. ## related work entity-level information is gaining increasing attention in cc-related research. different from named entity recognition (ner) applications on cc texts @xcite @xcite @xcite , entity linkers (el) further disambiguate textual mentions and associate them to knowledge bases (kbs), e.g., wikipedia @xcite , dbpedia @xcite , wikidata @xcite , via candidate selection and ranking @xcite @xcite . el evaluations also spread from news, e.g., tac-kbp @xcite , to more genres @xcite @xcite , and topic domains @xcite . despite a lack of human evaluation benchmarks for el on cc data, there is work on integrating el as an automatic pipeline for information extraction. weichselbraun et al. (2015 ) present recognyze, an el platform confined to named entities adopted by two cc web applications on swiss business news, i.e., the media watch on climate change and the climate resilience toolkit. ruiz fabo et al. (2016) propose an nlp pipeline including el for identifying supporting and opposing propositions in cc data. diggelmann et al. (2021) employ el to extract relevant documents for evidence candidates in cc claim verification. pérez ortiz et al. (2022) conduct el on video transcriptions surrounding two topics-"machine learning" and "climate change"-to speed up video search. toulet et al. (2022) provide an el-integrated pipeline and a visualization tool for analyzing scientific articles, including a case study on co-occurring "climate change" and "health" topics. pita costa et al. (2024) rely on el as a semantic annotation to analyze the impact of water-related climatic disasters. however, no evaluation of el performance on english cc data across genres exists. ## climateli: climate entity linking we present climateli, the first el evaluation corpus on cc data. we include ten english documents across five genres-wikipedia pages, academic articles, foot_1 web news, foot_2 united nations' intergovernmental panel on climate change reports, 4 and youtube transcriptions-consisting of 3,087 (1,161 unique) entity links across 12,802 tokens. entity linking manual el annotation from scratch is time-consuming and might result in low recall. this is mostly due to annotators' inability to encompass all contents from ever-expanding knowledge bases (kbs). we pick wikipedia as our target knowledge base due to its enormous size foot_4 and its integration into our daily life. firstly, we manually correct tokenization and pos-tagging from stanza @xcite dictions. we then create climateli by opting to manually correct pre-tagged linked entities from wikifier's @xcite threshold 1.0 (highest recall) predictions, which provide an extensive baseline for gold annotations while also adding missing entities. we use the markdown style [document tokens](link _ url) to ease annotation as shown in figure 1 . annotations include removing or correcting wrongly detected entity spans and links predicted by wikifier and adding missing entities not annotated by wikifier. we also verify whether individual links exist on wikipedia and resolve various ambiguities. we next also discuss several climateli guidelines. firstly, we follow the flat schema of wikipedia webpages to annotate only the longest entity without the shorter nested ones. for example, if "climate change mitigation" appears, we only link the entirety to climate _ change _ mitigation, without annotating the shorter climate _ change or climate. secondly, we determine the associated wikipedia link based on an entity's contextual reading. for example, when "it" refers to "climate change", we annotate "[it's effect](effects _ of _ climate _ change)". similarly, if the kyoto protocol is in context, we annotate the subsequent "[the protocol](kyoto _ protocol)." lastly, since el kbs are not tailored for cc, we follow wikipedia to annotate els on generic verbs, adjectives, adverbs, conjunctions, and etc., if such entries exist. these include "refer" (a verbal form of reference), "possible" (an adjectival form of logical _ possibility), "successfully" (an adverbial form of success), "while" (while), and etc. inter-annotator agreement two authors of this paper, fluent english speakers, split the ten documents, each taking ∼3 hours to annotate. we evaluate inter-annotator agreement (iaa) on one wikipedia article, paris _ agreement, which includes 1,371 tokens, and 334 or 307 entity link annotations by two annotators, respectively. ## evaluation setup this section presents the experiment setups for evaluating entity-linking models on climateli data. ## results this section evaluates four filtering scenarios, comparing valid nom and valid ctext versions (n-only and c-only) and their intersections (nc-only) to unfiltered (orig) entities. for wikifier and tagme, we use the default confidence threshold with the highest recall. ## quantitative analysis we examine two impacting factors on el model scores: confidence thresholds and text genres. wikifier and tagme thresholds figure 2 visualizes how different wikifier and tagme confidence thresholds affect model performances. unsurprisingly, wikifier beats tagme on all thresholds. on the more filtered data, the recall of both models increases with the threshold. in contrast, precision fluctuates: wikifier dips around 0.2 and peaks around 0.7; tagme reaches its best precision around 0.4 and decreases thereafter. moreover, it is intriguing that the precision and recall of both models converge at threshold 1.0 under the nc-only filter, which means the classifiers are balanced in terms of sensitivity and specificity. genres climateli contains documents from 5 genres with different linguistic styles and discourse structures. given our interest in cc, all three models achieve the highest performance in the vlog genre. though tagme and cao et al. perform inferior to wikifier, their advantages are remarkable in vlogs compared to other genres. figure 3 visualizes the distribution of entity lengths across various genres and illustrates that most (68.39%) entity spans in vlogs include only one token. this reduces the likelihood of entity nesting and decreases the difficulty of determining the positions of entity spans. meanwhile, academic, ipcc, and wiki have considerably longer entities, possibly explaining tagme and cao et al.' sub-optimal performances in these three genres. ## error types this section further exemplifies common error types of the el models' predictions. misinterpreting polysemous words as false nominals polysemous tokens occurring in nonnominal positions were sometimes wrongly linked to their nominal interpretations. for example, the adjective "current" (i.e., belonging to the present time) receives a wrong link to ocean _ current; the coordinating conjunction "both" gets wrongly linked to a trap song both _ (song); the modal verb "will" gets falsely associated with free _ will (i.e., capacity to make decisions independently). models genre precision recall f1 wikifier aca. 58.35 57.47 57.91 ipcc 55.93 55.59 55.76 news 57.77 57.19 57.48 vlog 64.85 69.03 66.88 wiki 55.73 51.48 53.52 tagme aca. 19.95 20.51 20.22 ipcc 24.84 23.56 24.19 news 26.51 26.42 26.47 vlog 43.71 42.58 43.14 wiki 15.21 14.53 14.86 cao et al. aca. 47.26 24.05 31.88 ipcc 51.20 19.34 28.07 news 61.02 36.12 45.38 vlog 66.40 53.55 59.29 wiki 46.27 30.54 36.80 _ union) and [its member states](member _ state _ of _ the _ european _ union)" when "it" refers european _ union. however, the coreference of "the"-headed definite common nouns is difficult. for example, when "the accord" refers to copenhagen _ accord, it gets falsely linked to prices _ and _ incomes _ accord in australia. similarly, a coreferring "the agreement" to paris _ agreement is always mis-interpreted as joint _ comprehensive _ plan _ of _ action, i.e., the iran nuclear deal. the more tricky situation is when some els occur nearby but are irrelevant to the current entity. still, el models are falsely influenced by these contexts. in "the 195 unfccc par-figure 3: nc-only entity length distributions per genre. ticipating member states and the european union", tagme annotates "member states" wrongly as member _ state _ of _ the _ european _ union, and wikifier annotates them as european _ union, while the gold label should be list _ of _ parties _ to _ the _ united _ nations _ framework _ convention _ on _ climate _ change. ambiguity caused by the flat schema annotating the longest entity and ignoring the nested ones is common in named entity linking and recognition (mcnamee and dang, 2009; ji et al., 2010; tjong kim sang and de meulder, 2003). however, this results in human label variation in annotating overlapping els. for example, "adapt to climate change effects" can be annotated as "[adapt to climate change effects](climate _ change _ adaptation)", but can also be annotated as "[adapt to climate change](climate _ change _ adaptation) [effects](effects _ of _ climate _ change)". both interpretations are equally reasonable and converge with the annotation guidelines. 8 conclusion this paper presents climateli, an entity linking corpus for english climate change data on five genres. we analyze existing el systems and propose filters to focus the evaluation on nominal climate entities. we show that els struggle to detect long entity spans and link them to corresponding wikipedia pages. future work includes nested el annotations since cc-related terminologies exhibit nesting and overlap, expanding annotated texts and wikipedia links to more languages, and training a cc-adapted el model for downstream nlp tasks.