E vent Coreference Data (Almost) for Free: M ining Hyperlinks from Online News (2021)

Abstract: Cross-document event coreference resolution (CDCR) is the task of identifying which event mentions refer to the same events throughout a collection of documents. Annotating CDCR data is an arduous and expensive process, explaining why existing corpora are small and lack domain coverage. To overcome this bottleneck, we automatically extract event coreference data from hyperlinks in online news: When referring to a significant real-world event, writers often add a hyperlink to another article covering this event. We demonstrate that collecting hyperlinks which point to the same article(s) produces extensive and high-quality CDCR data and create a corpus of 2M documents and 2.7M silver-standard event mentions called HyperCoref. We evaluate a state-of-the-art system on three CDCR corpora and find that models trained on small subsets of HyperCoref are highly competitive, with performance similar to models trained on gold-standard data. With our work, we free CDCR research from depending on costly human-annotated training data and open up possibilities for research beyond English CDCR, as our data extraction approach can be easily adapted to other languages.

## introduction cross-document event coreference resolution (cdcr) is the task of identifying and clustering mentions of real-world events in a given collection of documents. for example, cdcr systems need to decide whether the sentences "on monday, lindsay lohan checked into rehab in malibu" and "ms. lohan entered a rehab facility" from two different documents refer to the same event, by taking temporal, spatial and other cues from the document contexts into account. cdcr is a vital preprocessing step for downstream multi-document tasks such as question answering or fact checking. to overcome this gap, we leverage hyperlinks in online news articles: when referencing a significant real-world event in the body of an article, writers often add a hyperlink to a different article covering this event. we conjecture that by collecting hyperlinks which point to the same article(s) and interpreting anchor texts as mention spans, highquality cross-document event coreference links can be retrieved quickly and in large quantity (see figure 1 ). to this end, we devise a data extraction pipeline which mines such datasets automatically from common crawl 2 and apply it to create the hypercoref corpus, consisting of 40 news outlets with over 2m mentions in total, far exceeding the size of existing cdcr corpora. hypercoref achieves broader coverage in event types compared to manually annotated corpora. in experiments with a state-of-the-art cdcr model @xcite , we evaluate the relation between the amount of gold training data and test performance across three cdcr corpora: ecb+, fcc-t, and gvc. we make the remarkable observation that models trained entirely on silver-standard data from hypercoref perform on a similar level as models trained on gold-standard data (between 4 pp. conll f1 worse and 4 pp. better, depending on the corpus at hand). overall, our findings lift the dependency on gold data for training cdcr systems and pave the way for large, robust and potentially multilingual systems, as our data extraction approach can be easily adapted to any language found on the web. our contributions are: c1. a novel approach for acquiring silverstandard cross-document event coreference links from hyperlinks, c2. hypercoref, a large corpus created with this approach, and its analysis compared to 2 https://commoncrawl.org/ gold-standard cdcr corpora, c3. out-of-domain transfer experiments with a state-of-the-art cdcr system, certifying the quality of this data. ## fundamentals we define cross-document event coreference resolution (cdcr) and its relation to hyperlinks in news. task definition cdcr consists of two steps: (1) identifying mentions of real-world or hypothetical events in a collection of documents (event mention detection), and (2) recognizing which of these mentions refer to the same events, thereby producing a cross-document clustering of mentions (coreference resolution). event mentions are commonly defined by four components: their action (checked into), participants (lindsay lohan), time (on monday) and location (rehab in malibu) @xcite . actions are the centerpiece of event mentions, and their token span is the main representative of an event mention in text. we also refer to this span as a mention span. hyperlinks in news to establish the context of a recent news development, news journalists make reference to other events which have caused, influenced or are otherwise related to the recent newsworthy event. in online news, such references are often marked with a hyperlink to another article which covers the referenced event in greater detail. these hyperlinks can (with some margin of error) be interpreted as cross-document coreference links: the hyperlink's anchor text (its clickable text region) corresponds to an event mention's action, and the target url identifies the referenced event. a pair of hyperlinks which point to the same url but are located in different articles then correspond to two event mentions connected by a cdcr link. this is exemplified in figure 1 . we propose to collect cdcr data by mining hyperlinks from online news. in the next section, we explain our data pipeline creating such data and the key issues one needs to overcome in the process. ## data extraction following an explanation of our data extraction pipeline, we describe its application for creating the hypercoref corpus which we then compare to expert-annotated cdcr corpora. ## experiments annotating gold-standard cdcr corpora is a laborious and expensive process, raising the question to which extent such data can be replaced with cheaper to obtain silver-standard (i.e. automatically generated) data for training cdcr models. we investigate this question by evaluating the stateof-the-art cdcr system of cattan et al. (2021) on three gold corpora (ecb+, fcc-t, gvc) and the abc news and bbc news subcorpora of the silver hypercoref corpus. we first describe the aforementioned cdcr system and explain how we prepare each corpus for cdcr experiments, then report our results obtained for the coreference resolution and mention detection tasks. sota system we choose cattan et al. (2021) for our experiments since it is the most recent stateof-the-art approach on ecb+ with available implementation. at test time, this system predicts a similarity score for each possible pair of mentions. using these similarities, agglomerative clustering referenced article article excerpts with hyperlink reference to the article shown left israeli police and the israeli securities authority announced the conclusion of a joint investigation into the conduct of prime minister benjamin netanyahu on sunday, recommending indictments for both netanyahu and his wife sara. a1 israeli prime minister benjamin netanyahu spoke with un secretary general antonio guterres on wednesday to update him on the details of a military operation that will destroy cross-border tunnels between israel and lebanon [...]. a2 israel's prime minister benjamin netanyahu should be indicted for bribery, fraud and breach of trust, the country's attorney general said thursday. a3 netanyahu is the first israeli prime minister criminally charged while in office. [2015-06-24] st helens legend paul wellens has announced his immediate retirement from rugby league because of a hip injury. [...] wellens will take up a coaching position at langtree park as part of keiron cunningham's staff. b1 [2015-06-24] they are huge accolades for a quiet man who dedicated his career to one club, and who has announced his retirement at the age of 35 because of a hip problem. b2 [2015-07-01] the 22-year-old australian has agreed to join saints as full-backs shannon mcdonnell and jonny lomax are out, while paul wellens retired last week. b3 [2015-12-03] paul wellens has taken up a player performance coach role at st helens after his retirement. he announced he was quitting playing in june because of a hip injury and was told he would take up a coaching position which is now confirmed. ## discussion and future work expensive data annotation constitutes a bottleneck for research on scalable, open-domain cdcr systems. addressing this gap, we found that hyperlinks extracted from online news have tremendous potential when used as proxy training data for cdcr: the fact that a model trained on hyperlinks from bbc sports news outperforms indomain models trained on fcc-t (sports) and gvc (gun violence), and that a model trained on ecb+ performs only 4% pp. conll f1 better than a model trained on hyperlinks from abc news (see nevertheless, there are downsides to our proposed approach. by nature, it cannot be applied to text types without hyperlinks such as works of fiction @xcite , e-mail conversa-tions @xcite or dialogue (eisenberg and sheriff, 2020). data scraped from the web is inherently noisy, and while filtering steps can mitigate this issue, noise from imperfect markup cleaning, hyperlinks not referring to eventive content, or different issues remains. similarly, hyperlink anchor texts oftentimes resemble the mention spans of event mentions as defined in traditional annotation guidelines, but this is not guaranteed (see section 3.3). future work may investigate humanin-the-loop annotation @xcite to resolve such cases on a subset of the data. training event mention detection systems entirely on hyperlink data is possible (see section 4.2), however with a deficit in precision and particularly recall since hyperlinks appear less frequently in news data than event mentions do in gold-standard corpora. doing so can nonetheless be an effective fallback solution for languages for which no gold-standard mention detection corpus is available. ## related work harvesting nlp datasets from the web has a long history. @xcite extract sequences of verb constructions from webpages to learn common preconditions of actions and events, and cham-bers and jurafsky (2008) extract narrative chains of events from the gigaword corpus @xcite . however, these works extract event knowledge from raw newswire text, omitting signals from hyperlinks. the wikilinks corpus @xcite is an entity coreference corpus created by collecting hyperlinks to wikipedia pages from a web crawl. while large in volume, the dataset does not target events and suffers from low mention ambiguity, since a mention-pair string identity baseline can reach 82% f1. as part of the wikification task @xcite , links in the body of wikipedia articles and their anchors were collected to produce multilingual entity coreference resolution data. closest to our approach is wikipedia event coreference (wec) @xcite , a recent cdcr corpus created from wikipedia articles on real-world events and cross-page links. while wikipedia-based corpora are easier to create than newswire corpora (such as hypercoref) due to standardized markup, their key downside is that encyclopedic text lacks the temporal and spatial anchoring present in newswire (as in "today, the white house announced" or "ed sheeran is coming to town") which considerably lessens their usefulness for event-related tasks. furthermore, compared to hypercoref, event coverage in wec is limited to events which the wikipedia community deemed significant enough to warrant a dedicated article (for each separate language). this excludes high-frequency, high-ambiguity events (resignations, stocks surging, arrests, etc.) foot_4 which are most challenging to resolve for cdcr systems and therefore crucial to have as training data. these corpus differences have a direct impact on results, with our broad-coverage abc s ss model outscoring the identical roberta-based architecture trained by @xcite on wec by 5.9% conll f1 when tested on ecb+ (75.04 f1 vs. 69.10 f1). recently, choubey and huang (2021) investigated automated retrieval of annotations for withindocument event coreference resolution. their method is applicable on plaintext news articles, but requires a database of lexical paraphrases for mention identification and a discourse parsing system for filtering. it is unclear whether this method can be successfully applied for cdcr, since the employed discourse-based filtering rules may not be transferable to the cross-document case. to the best of our knowledge, we are the first to propose a cheap and high-quality data extraction approach specifically for cross-document event extraction and coreference which does not depend on pre-existing resources, combining past work on event extraction from raw newswire text and mining of hyperlinks. ## conclusion to overcome the prevalent data bottleneck of the cdcr task, we proposed a new method for cheaply and automatically collecting silver-standard data from hyperlinks in online news. we used this approach to create hypercoref, a large dataset with over 2m mentions and show that a system trained on a subset of this dataset achieves equivalent performance as the same system trained on expert-annotated corpora. our data collection approach opens up many avenues for future work, particularly for languages where gold-standard cdcr data is currently scarce or non-existent. arie cattan, alon eirew, gabriel stanovsky, mandar joshi, and ido dagan. 2021. cross-document coreference resolution over predicted mentions. in findings of the association for computational linguistics: acl-ijcnlp 2021, pages 5100-5107, online. association for computational linguistics. nathanael chambers and dan jurafsky. 2008. unsupervised learning of narrative event chains. in proceedings of acl-08: hlt, pages 789-797, columbus, ohio. association for computational linguistics. prafulla kumar choubey and ruihong huang. 2017. event coreference resolution by iteratively unfolding inter-dependencies among events. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 2114-2123, copenhagen, denmark. association for computational linguistics. prafulla kumar choubey and ruihong huang. 2021. automatic data acquisition for event coreference resolution. in proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pages 1185-1196, online. association for computational linguistics. agata cybulska and piek vossen. 2014a. guidelines for ecb+ annotation of events and their coreference. technical report, vu university amsterdam. agata cybulska and piek vossen. 2014b. using a sledgehammer to crack a nut? lexical diversity and event coreference resolution. in proceedings of the ninth international conference on language resources and evaluation (lrec'14), pages 4545-4552, reykjavik, iceland. european language resources association (elra). parag pravin dakle, takshak desai, and dan moldovan. 2020. a study on entity resolution for email conversations. in proceedings of the 12th language resources and evaluation conference, pages 65-73, marseille, france. european language resources association.