Adapting Coreference Resolution for Processing Violent Death Narratives (2021)

Abstract: Coreference resolution is an important compo-nent in analyzing narrative text from admin-istrative data (e.g., clinical or police sources).However, existing coreference models trainedon general language corpora suffer from poortransferability due to domain gaps, especiallywhen they are applied to gender-inclusive datawith lesbian, gay, bisexual, and transgender(LGBT) individuals. In this paper, we an-alyzed the challenges of coreference resolu-tion in an exemplary form of administrativetext written in English: violent death nar-ratives from the USA’s Centers for DiseaseControl’s (CDC) National Violent Death Re-porting System. We developed a set of dataaugmentation rules to improve model perfor-mance using a probabilistic data programmingframework. Experiments on narratives froman administrative database, as well as existinggender-inclusive coreference datasets, demon-strate the effectiveness of data augmentationin training coreference models that can betterhandle text data about LGBT individuals.

## introduction coreference resolution @xcite is the task of identifying denotative phrases in text that refer to the same entity. it is an essential component in natural language processing (nlp). in real world applications of nlp, coreference resolution is crucial for analysts to extract structured information from text data. like all components of nlp, it is important that coreference resolution is robust and accurate, as applications of nlp may inform policy-making and other decisions. this is especially true when coreference systems are applied to administrative data, since results may inform policy-making decisions. in this paper, we describe an approach to adapting a coreference model to process narrative text * kwchang@cs.ucla.edu from an important administrative database written in english: the national violent death reporting system (nvdrs), maintained by the centers for disease control (cdc) in the usa. violent death narratives document murders, suicides, murder-suicides, and other violent deaths. these narratives are complex, containing information on one or more persons; some individuals are victims, others are partners (heterosexual or same-sex), family members, witnesses and law enforcement. specifically, we apply the end-to-end coreference resolution (e2e-coref) system @xcite , which has achieved high performance on the ontonotes 5.0 @xcite corpus. we observe that when a model trained on ontonotes is applied to violent death narratives, the performance drops significantly for the following reasons. first, despite the fact that ontonotes contains multiple genres foot_0 , it does not include administrative data. administrative text data is terse and contains an abundance of domain-specific jargon. because of the gap between training and administrative data, models trained on ontonotes are poorly equipped to handle administrative data that are heavily skewed in vocabulary, structure, and style, such as violent death narratives. second, approximately 5% of the victims in the nvdrs are lesbian, gay, bisexual, or transgender (lgbt). this is a vulnerable population; for example, existing data show lgb youth are 5 times more likely to attempt suicide than heterosexual youth @xcite and are more likely to be bullied prior to suicide 2 . it is essential that data-analytic models work well with these hard to identify but highly vulnerable populations; indeed correctly processing text data is an important step in revealing the true level of elevated risk for primary_victim is a 50 year old male . ... primary_victim's partner states that he and primary_victim had been living together for three years. ... lgbt populations. this remains challenging because of limitations of existing coreference systems. close relationship partners provide a marker of sexual orientations and can be used @xcite by social scientists to identify relevant information in lgbt deaths. however, ontonotes is heavily skewed towards male entities @xcite and e2e-coref relies heavily on gender when deciphering context (cao and @xcite . consequently, e2e-coref has a trouble dealing with narratives involving lgbt individuals where gender referents do not follow the modal pattern. figure 1 illustrates a scenario where coreference systems struggle. the model mislabels the pronoun "he" and this error will propagate to downstream analysis. specifically, the model takes the context and resolves the coreference based on gender; it makes a mistake partially due to an incorrect presumption of the sexual orientation of the 50 year old male victim. to study coreference resolution on violent death narratives (vdn), we created a new corpus that draws on a subset of cases from nvdrs where cdc has reported the sex of both victims and their partners. we assigned ground truth labels using experienced annotators trained by social scientists in public health. 3 to bridge the domain gap, we further adapted e2e-coref by using a weakly supervised data creation method empowered by the snorkel toolkit @xcite . this toolkit is often used to apply a set of rules to augment data by probabilistic programming. inspired by snorkel, we designed a set of rules to 1) bridge the vocabulary difference between the source and target domains and 2) to mitigate data bias by augmenting data with samples from a more diverse population. because labeling public health data requires arduous human labor, data augmentation provide a promising method to enlarge datasets while covering a broader range of scenarios. we verified our adaptation approach on both the in-house vdn dataset as well as two publicly available english datasets, gicoref (cao and daumé iii, 2020) and map (cao and @xcite . we then measured the performance of our approach on documents heavily skewed toward lgbt individuals and on documents in which gendered terms were swapped with non-gendered ones (pronouns, names, etc.). on all datasets, we achieved an improvement. for lgbt specific datasets, we see much larger improvements, highlighting how poor the ontonotes model performed on these underrepresented populations before. models trained on the new data prove more applicable in that domain. our experiments underscore the need for a modifiable tool to train specialized coreference resolution models across a variety of specific domains and use-cases. ## related work researchers have shown coreference systems exhibit gender bias and resolve pronouns by relying heavily on gender information (cao and @xcite @xcite @xcite . in particular, cao and daumé iii (2020) collected a gender-inclusive coreference dataset and evaluated how state of the art coreference models performed against them. as nlp systems are deployed in social science, policy making, government, and industry, it is imperative to keep inclusivity in mind when working with models that perform downstream tasks with text data. for example, named entity recognition (ner) was used in processing portuguese police reports to extract people, location, organization, and time from the set of documents @xcite . these authors noted the need for a better training corpus with more ner entities. other nlp models face challenges in domain-adaptation like the one demonstrated in this paper. one example from the biomedical field is biobert @xcite , in which the authors achieved better results on biomedical text mining tasks by pretraining bert on a set of biomedical documents. likewise, even when evaluating a model on a general set, @xcite showed that many general-domain datasets include as much bias as datasets designed to be toxic and biased. all these cases required re-evaluation of the corpus used to train the model. this underscores the need for methodology that can evaluate, debias, and increase the amount of data used. ## annotating violent death narratives we first applied for and were given access to the cdc's national violent death reporting-system's (nvdrs) restricted access database. from this, we sampled a total 115 of violent death cases 4 each over 200 words in length. in these 115 cases, we had a total of 6,134 coreference links and 44,074 tokens, with a vocabulary size of 3,653. each case had information about the victim, the victim's partner, and the type of death. we randomly sampled 30 cases from three strata: 1) the victim is male and the partner is female, 2) the victim is female and the partner is male, and 3) it was an lgbt case. we also included 25 cases that were particularly challenging for the general e2e model. the cases used were spell-checked and cleaned thoroughly. to obtain gold-standard labels, we tasked a team of three annotators 5 to label the coreference ground truth, under the guidance of senior experts in suicide and public health. annotators were told that every expression referring to a specific person or group was to be placed into that person's or group's cluster. from there, we resolved the three label sets into one by a majority voting method -if two out of three annotators put the phrase in a cluster, we assigned it to that cluster. two of the annotators had previous experience with coding the nvdrs narratives for other tasks, while one was inexperienced. agreement was typically unanimous. reproducibility to get access to the nvdrs, users must apply for access and follow a data management agreement executed directly with cdc. we cannot release vdn or the annotations but we will provide the augmentation code and instructions on how reproducing the experiments. to allow reproduction of our approaches on data without access-restriction, we perform evaluations on map and gicoref which are readily available. ## weakly-supervised data augmentation for domain adaption our next step was to build a pipeline for adapting e2e-coref to resolve coreference on vdn. the key component of this pipeline is the snorkel toolkit 4 homicides and suicides 5 all annotators signed the release form for accessing the nvdrs data. and its capacity to design rules that programmatically label, augment, and slice data. we looked to adapt e2e-coref to process domain-specific data by creating a set of augmentation rules that would improve training data performance. our rules can generate augmented data with diverse genders and then challenged our model to predict the coreference clusters. ## experiments and results we conducted experiments to analyze e2e-coref on vdn and verified the effectiveness of the data augmentation method. we used the following corpora foot_3 . figure 2 : the proposed rules for gi data applied to a sample paragraph. by domain experts and used as the test set for measuring model performance. we split vdn into train/dev/test with a 20/5/90 document split. we are interested in the setting where only a small set of training data is available, to emulate use-cases in which annotating a large amount of data is impractical. we reserve more articles in the test set to ensure the evaluation is reliable. we followed cao and daumé iii (2020) to use lea (moosavi and strube, 2016) as the evaluation metric for coreference clusters. ## conclusion with policy decisions increasingly informed by computational analysis, it is imperative that methods used in these analyses be robust and accurate especially for marginalized groups. our contributions improved coreference resolution for lgbt individuals, a historically underrepresented and marginalized population at high risk for suicide; they may improve the identification of lgbt individuals in nvdrs and hence inform better policy aimed to reduce lgbt deaths. more generally, we show how to use augmentation rules to adapt nlp models to real-world application domains where it is not feasible to obtain annotated data from crowdworkers. finally, we introduced a novel dataset, vdn, which provide a challenging and consequential corpus for coreference resolution models. our studies demonstrate the challenges of applying nlp techniques to real-world data involving diverse individuals (including lgbt individuals and their families) and suggest ways to make these methods more accurate and robust-thus contributing to algorithmic equity.