R eference N et: a semantic-pragmatic network for capturing reference relations. (2018)

Abstract: In this paper, we present ReferenceNet: a semantic-pragmatic network of reference relations between synsets. Synonyms are assumed to be exchangeable in similar contexts and also word embeddings are based on sharing of local contexts represented as vectors. Co-referring words, however, tend to occur in the same topical context but in different local contexts. In addition, they may express different concepts related through topical coherence, and through author framing and perspective. In this paper, we describe how reference relations can be added to WordNet and how they can be acquired. We evaluate two methods of extracting event coreference relations using WordNet relations against a manual annotation of 38 documents within the same topical domain of gun violence. We conclude that precision is reasonable but recall is lower because the WordNet hierarchy does not sufficiently capture the required coherence and perspective relations.

## introduction synonyms from the same synset @xcite are assumed to be exchangeable in contexts. similarly, word embeddings are based on sharing of contexts represented as vectors @xcite . both synsets and word embeddings capture some variation in language, but they do not fully capture variation in reference and coreference. reference relations are different in that they cross local (sentence) contexts. we typically tell stories in discourse in which entities or events play different roles and reflect different phases in relation to the same incident (the topic of the story). furthermore, authors may frame these entities and events differently either within the same story or across different stories. we can thus consider a story as a larger topical context within which co-referring expressions occur in different local contexts. each local context of a co-referring expression may represent a different concept. the set of local contexts within a topical context is therefore expected to express not only similarity, but also topical coherence and author framing and perspective. the next two examples show two fragments from two news articles that make reference to the same incident (topical coherence) in which a man shot several people in a bar in pittsburgh. the first fragment is published shortly after the incident when the suspect has not yet been identified. the second fragment is published later after the suspect was identified, found guilty and sentenced to prison (changing perspective). investigators continue to look for suspects after one person was killed and four others were injured when gunfire erupted overnight at a bar in homewood . ..... several witnesses , [...] they believe the gunman was not searched by the four security guards who left the business before police arrived . man gets 15 -30 years for deadly shooting at homewood bar . pittsburgh ( kdka ) a man has pleaded guilty in a 2014 shooting that left four men injured and one dead . cornell poindexter , 30 , appeared in court monday and pleaded guilty to one count of 3rd degree murder , four counts of aggravated assault and one count of person not to possess a firearm . ... according to our partners at the pittsburgh post -gazette , 23-year -old corey clark was originally accused of being the gunman , but those charges were dropped . the following words and expressions are used to make reference to the incident or parts of the incident: killed, injured, gunfire erupted (first fragment) and deadly shooting, shooting, left injured and dead, murder, aggravated assault (second fragment). the references to the shooter are made through suspects, gunman and through man, cornell poindexter, person not to possess a firearm, 23-year -old corey clark and gunman respectively. references to the events differ across the text due to the legal view, e.g. murder, whereas the entity references differ due to having more knowledge on the identity of the suspects and the fact that one suspect turned out to be innocent and another was convicted. making reference is more than similarity of meaning, as it is also governed by pragmatic principles related to information sharing, relevance, salience, and framing. in the different sentences of a discourse, we tend to tell different things about the same referents. these sentences thus represent different local contexts, which are connected through the topical context of the story that is told. from a language understanding and generation perspective, wordnet synsets and word embeddings are not expected to provide sufficient information to predict usage of one expression over the other, or to infer from the referential usage of expressions what is the semantic implicature (coherence or framing). we therefore propose to add a layer to wordnet, that captures variation in making reference within a topical context across different synsets or word embeddings that represent local contexts. in this paper, we describe how these relations can be acquired as a ref-erencenet. the relations in a referencenet exceed the notion of synonymy and partially also hyponymy and capture a broader range of roles, perspectives, and also different phases of processes. referential relations can not only help detecting coreference and coherence relations, but also help distinguishing roles from rigid types which is important for further ontologisation of semantic networks, and capturing different ways of framing the same things. this paper is structured as follows. in section 2, we discuss related work and present the motivation for adding referential relations to wordnet. in section 3, we define the model for expressing these relations. we present two approaches to acquire these relations in sections 4 and 5. section 6 describes the evaluation data created and section 7 contains the evaluation results. finally, we conclude and discuss future work in section 8. ## related work and motivation reference and identity have been discussed extensively in the philosophical literature @xcite @xcite @xcite . the linguistic field of lexical pragmatics @xcite @xcite tries to explain variation in reference as a function of pragmatic principles such as the gricean maxims @xcite : be maximally informative but no more informative than necessary. variation of form is partly explained through pragmatic licensing: the least complex form that yields the most salient implicatures is preferred among all forms that can potentially yield these implications. such principles may predict how we make reference to real-world situations using certain words and expressions and not others, given the shared knowledge we have about these situations. the way we make reference is however not only determined by efficiency, salience and information sharing, but also by the framing of referents by the author. framenet @xcite is a large resource that describes different ways in which situations can be framed. frames and frame elements in framenet are very specific and typically model the specific realisation of lexical units in texts. it is not clear how to generalise over the specific frames (what do they share or have in common) nor to derive from the database which combinations of frames can be expected within specific topical contexts. we believe it is worthwhile to investigate empirically the actual referential relations that occur within topical contexts at a large scale, as well as to describe the observed lexical variation according to both pragmatic principles of quality and efficiency and framing principles. we therefore propose a referencenet as a data structure that captures the observed coherence and framing relations between wordnet synsets. ref-erencenet therefore extends wordnet with a new orthogonal relation, which is less strict and limited than framenet, and more specific than for instance word-net domains @xcite . we argue that such data can be potentially very valuable, as it enables our community: 1. to investigate the semantic-pragmatic implications of making reference 2. to learn about the contextual roles and perspectives that govern the usage of these words and expressions, and 3. to improve the detection of these relations by coreference systems. ## the referencenet model we define a referencenet as a collection of reference-sets. a referenceset consists of: as synsets represent concepts, the variety of synsets reflects the range of things or denotation that is captured in a single referenceset. as this range is not ontologically defined, it reflects the typical ways in which we frame and conceptualize individuals in topical situations. typically, these synsets cannot be disjoint (mutually exclusive): they should either belong to the same hypernym chain (being more or less specific), or should be orthogonal according to formal ontological criteria @xcite . a referenceset may consist of one or more synsets and the same synset may participate in more than one reference set, thus constituting a 'manyto-many' relation. in addition to the synset of the expression, we also need to record the actual form or synonym from the synset that was used to make reference. 1 as the constraints for making reference with different expressions and different concepts are mildly ontological, it make sense to register the referential usage of expressions and synsets using counters. finally, referencesets include an attribute to mark the type of topical context within which referential variation is observed. the topical context underlies the coherence relations within a discourse. moreover, it explains the variation in making reference to the same entities and events either through shifting roles, phases, and aspects, or through framing by the author. the topical context allows us to abstract from references to individual entities and events, by generalising the observations to the surface forms and synsets. for example, the same person may be referenced during school, family, leisure, or at work. it makes little sense to combine all the references to the same person in a single referenceset. instead, we gather reference to individuals across all different incidents within the same type of topical context. this captures our general ways of framing persons and events within these topics and according to some topical schema. referencesets thus will reflect which synonyms from which synsets are used how frequently to make reference within the same topical context. figure 1 shows two examples of a referenceset for the two texts in the introduction that report on the same incident and thus the same topical context of gun-violence. we see separate referencesets for the shooter and for the shooting. each referenceset consists of a list of synset-ref elements. 2 the synset-ref element has attributes for the cili identifier iid @xcite , the language specific wordnet synset, and the corefcount attribute to express how often this entity was mentioned in the text. each synset-ref contains a list of surface-form elements with the surface form and its observed token frequency of making reference. we can see that the words span different synsets and also different parts-of-speech tags. the first reference-set exhibits the perspective of the shooter and the suspect before the trial. we abstracted from the actual names of the people through a separate element and counter proper-name. the second referenceset shows different granularities of the event: the overall incident, the shooting, hitting and the outcome, and it shows the legal judgments: murder, assault. this illustrates that the reference relations are often orthogonal to hypernym relations. referencesets as in figure 1 can be derived from collections of texts in which coreference relations are resolved across documents making reference to the same incident, involving the same entities and events. referenceset can then be formed by aggregation across incidents of the same topic type, based on sufficient overlap between surface forms and synsets of incidents. we discuss methodologies for building a referencenet in detail in the next section. ## methodologies for building a referencenet semantic parsing aims at generating a representation of entities and events from their mentions throughout this text. it relies on a broad range of nlp techniques such as tokenization, parsing, named-entity recognition and linking, and semantic role labeling. coreference modules often operate on top of the output of such modules. words and phrases that make reference to the same individual or event are coreferential. if different documents report on the same entities, these would ideally result in cross-document coreference. applying coreference modules to large collections of texts potentially gives us the different ways in which people make reference to the same entities and events in the world. if for all these referential expressions, we would also know the wordnet synsets, we can abstract from coreferential mentions to their synsets and derive referencesets for the semantic types of referents. this requires word sense disambiguation (wsd) to run in addition to establishing coreference relations. the feasibility of this approach depends on the quality of all the underlying modules (among which wsd) as well as the quality of the coreference modules. a distinction can be made between nominal/entity and event coreference, as they are defined and approached differently by different research groups. as we are primarily interested in the topical coherence underlying texts in this paper, we focus in this paper on event coreference and leave nominal or entity coreference for future work. we discuss two methods for obtaining event referencenet data from text collections using semantic parsing: 1) text-to-data and 2) data-to-text. text-to-data involves semantic text parsing without knowing the referents a priori and without knowing which texts report on the same incident. it therefore relies on high-quality cross-document event coreference resolution and it is computationally very expensive as it requires comparing all event mentions with each other (within and across documents). automatic event coreference is a difficult task @xcite and made little progress over the years. to compare different approaches on the ecb+ dataset ( cybulska and vossen, 2014 @xcite and @xcite , as well as their own approach. they report 58.7 conll-f1 @xcite on ecb+ for their own approach, compared to 53.6 conll-f1 for (bejan and harabagiu, 2010) and 55.2 conll-f1 for @xcite . they obtained their results however only after boosting event detection from an original 65f to 95f by training a separate event detection system on part of the ecb+ data. without such nearly perfect event detection, their results are much lower. all three approaches are clustering approaches over the dataset using event mentions as input. likewise, they can only recover coreference relations between mentions that match local structural features, hence exhibit limited variation. another approach implemented by @xcite , logically matches semantic representations of the action mentions, the participants, the time, and the place. assuming again near-perfect event detection, this approach results in a conll-f1 score of 67.13. for comparison, a baseline system that applies a one-lemmaone-referent heuristics already scores 53.4 conll-f1. as argued in @xcite , the ecb+ dataset is very limited in terms of referential variation and within each topic there are only two potential referents to choose between. concluding, we observe that event coreference systems perform poorly, especially with respect to recall. applying these corpora to large collections of texts is not likely to give us reliable referential data to derive a referencenet and will not capture sufficient variation. however, the advantage of this approach is that it can be applied to any collection of text. the data-to-text method starts from structured data in which the referents are predefined and searches for texts that make reference to this data, so-called reference texts. structured event data paired with reference texts appear to exist and are publicly available: gunviolencearchive (gva), 3 fireincidentre-3 http://gunviolencearchive.org/ reports/ ports (fr), foot_2 railwaysarchive (ra), 5 gun violence database (gvdb), foot_4 asn incident database, foot_5 asn wikibase. 8 these resources register event incidents with rich properties such as participants, location, and incident time, and often even provide pointers to one or more reference texts. the number of events and documents is usually high, i.e. there are ∼ 9k incidents in ra, and ∼ 30k incidents in gva. in the data-to-text approach, we convert the structured data from such archives to what we call a microworld. a microworld is an rdf 9 representation of the referents related to a specific event (e.g. human calamities or economic events) but no more than that. reference texts are then news, blogs, and wikipedia pages that report on this data. given the a-priori pairing of microworlds with reference text, we can apply the simple one-mention-one-referent principle to obtain reference relations for event mentions for free with a relatively high confidence. by increasingly mixing microworlds and reference texts, we approximate the complexity of reference relations in reality across large volumes of text. by collecting news from different sources on the same or similar events, we approximate true variation in making reference from different perspectives. for example, we can not only take news from different sources with different stances but also vary the time between the event date and the publication date to get articles with different historical perspective. furthermore, the fact that the data on events from which we start has been created from the perspective of general human interest (e.g. gun violence incident reports) avoids discussion on what establishes an event in text, as we consider only those mentions that directly refer to the reported incident or salient subevents of these incidents. although this method may result in more precise reference relations as there is little ambiguity for paired microworlds and reference texts, its downside is the dependency on the availability of the structured data coupled with such reference texts. while for certain types of events such as calamities, sports, and business there may be sufficient data, for others people are less inclined to register events for longer periods. alternatively, structured event data can also be obtained from dbpedia @xcite , wikidata @xcite , and yago2 @xcite . as these databases are often linked to wikipedia articles, references in these articles can be used to find reference texts. note that we only need the structured data to reconstruct a minimal representation of the referents and we do not need the full representation of the event. another downside of this approach is that the granularity of the incident is more coarse than the granularity at which the events are reported in the associated news articles. to illustrate this, the gva collection provides a summary on the incident outcome, whereas the corresponding news documents report on the process that led to this outcome: firing, hitting, killing, getting injured, dying, etc. ## the newsreader event coreference system we used the newsreader system @xcite to simulate both a text-to-data and data-to-text process. in both cases, we apply generic semantic parsing to articles, obtaining representations of entities, events, and roles. the output is represented in the natural language annotation format (naf) @xcite . coreference for events within a single naf file is based on a number of steps described in @xcite ): foot_8 1. all predicates from the semantic role layer in naf are considered as event mentions; in addition, we include the lowest-commonsubsumer synset that was responsible for the similarity match. (b) if below the threshold, we keep the sets distinct 6. we iterate over the reference sets until there are no changes for wsd, newsreader uses the ukb system @xcite , as well as the supervised it-makes-sense system of @xcite . the output of both systems is used to vote for the most dominant synsets associated with a mention of a predicate. for wordnet similarity, newsreader uses the wordnet distance measure proposed by @xcite . 11 to be able to capture similarity across nouns and verbs, we extended the wordnet hypernym relations with morphological relations of the type event across noun and verb synsets, obtained from the princeton wordnet website. 12 below we show two examples of event coreference sets in naf obtained from two text fragments on the same incident, where the similarity threshold was set to 1.0 and the dominant sense threshold was set to the 80% best-scoring synsets in wsd. curry bryson , the father of the 11-yearold who police say shot and killed a 3-year -old , appeared in court today for a hearing . ... barney says it is not the charges against him that have torn his client apart . it is the fact bryson 's 11-year -old son is accused of shooting and killing 3-year -old elijah walker . in the first fragment, the software lumped together verbal mentions of shooting, shot, and torn. the first two share the same lemma, while they were matched with torn through the lowest-commonsubsumer (@xmath0: buck;charge;shoot down;shoot;tear. the similarity score was 1.38. setting the similarity threshold to 1.5 would prevent merging these mentions. in the second fragment, there is only one mention of the noun shooting. we can see that across the documents the verbal and nominal senses will not match on the basis of just the synset identifiers. however, they may still be merged through the form shooting or using crosspart-of-speech similarity. from all the mentions, we obtain the most dominant synsets (@xmath1. the lowest-common-subsumer and the dominant synsets form the basis to compare event coreference sets across documents. in order to match reference sets across documents, newsreader first converts naf representations to sem-rdf, in which each coreference set represents a unique instance of an event (represented by a unique uri). each event instance is described with the semantic information associated from all mentions throughout the document. however for the cross-document comparison reported here, we have chosen to match coreference sets only in terms of the overlap of word-net synsets and surface forms, thus ignoring participants, roles, and temporal relations. the proportion of overlap across instances of events can be set through a parameter. in our experiment, 5% of the synsets or surface forms (in case a lemma has no synsets) need to match for merging instances across different documents. to simulate the text-to-data approach, all the rdf representations of events are compared across all the documents. in order to simulate a data-to-text approach, we applied the above cross-document strategy in such a way that events are only compared when the reference texts report on the same incident according to the structured data. this means that shootings in documents reporting on different incidents are never compared and cannot constitute coreference relations. ## evaluation data to evaluate both these methodologies, we manually annotated 38 news articles associated with 20 incidents from the gva data set. the articles were grouped by the incident on which they report together with the structured data on the incidents, e.g. which people got injured or died. we used an annotation schema that differentiates events at different levels of granularity and with respect to the most salient implication derived from the event mention: for each mention of these events, the annotator creates a unique instance identifier based on the incident, the assigned event type, and the affected victims. when annotating events in documents reporting on the same incident, identity results from same type and victims assigned to mentions whereas non-identity results from a difference in type and/or victim. documents that report on different incidents never result in identity regardless of the type or victims annotated. shooting the same person in different incidents is not the same, as well as shooting a different person in the same incident. the annotation resulted in 138 event instances and 874 mentions in 38 documents. in total, 77 different lemmas were used to make reference to these events. given these annotations, we can abstract from the instances and group lemmas that make reference to the same type of event. ## evaluation results we automatically generated referencesets from the 38 annotated documents using the newsreader pipeline. we used standard settings for dominant-senses (80% top-scoring senses) and similarity (similarity of 2 or higher). following the methodologies described in section 4, we processed the data twice: this method is enriched with data-to-text knowledge. we see that with-i (incident pairing) performs better in terms of mention recall (+2), precision (+14) and f-score (+6) than without-i. for lemma coverage, the recall is the same, but the incident-aware version with-i has much higher precision (+16). overall recall is significantly lower than precision for both methods. the precision of the data-to-text approach with incident pairing is reasonable (around 60%), though not very high. this can be improved by using better wsd and/or by making the cross-document matching more strict. in the current setting only 5% of the synsets or phrases need to match across documents. in the recall for both methods is really low: around 20% for mentions and 50% for lemmas. error analysis on the missed recall shows that most of these are not detected as predicates by the semantic role labeler: pronouns @xcite , adjectives (fatal, fatally, reckless, injurious, accidental, deadly), and nouns (dead, incident, surgery, felony, autopsy). predicate detection is based on the mate tool @xcite , 13 the only exception are the referencesets that include take, where the incident pairing generated 4 referencesets and included more wrong mentions than without incident pairing. ## conclusions and future work in this paper, we present referencenet: a network of referential relations between synsets that is complementary to wordnet and word embeddings. refer-encenet consists of referencesets that group synsets and words that make reference to similar entities and events within similar topical contexts. typically, ref-erencesets reflect different local contexts and perspectives within a shared topical context as opposed to synsets and word embeddings that capture similar local contexts. we described two methods to derive ref-erencesets from textual data. we evaluated the approaches against a manually annotated data set. we concluded that precision is reasonable, whereas recall is low, mainly due to poor recall of predicates. we also observed that coreference relations are missed because wordnet does not sufficiently capture coherence and perspective relations, resulting in smaller reference-sets. the evaluation further showed that reference-sets created with a data-to-text approach have higher recall and precision. in future work, we want to capture more referential variation. event coreference can be improved using other coherence measures, especially when comparing coreference sets across documents. the fact that wsd already restricts the association of concepts by part-of-speech limits the matching in the current system. we will also extend to other types of events and contexts. finally, entity coreference can be included by exploiting semantic matches of noun phrases and semantic roles.