The A ppos C orpus: a new multilingual, multi-domain dataset for factual appositive generation (2020)

Abstract: News articles, image captions, product reviews and many other texts mention people and organizations whose name recognition could vary for different audiences. In such cases, background information about the named entities could be provided in the form of an appositive noun phrase, either written by a human or generated automatically. We expand on the previous work in appositive generation with a new, more realistic, end-to-end definition of the task, instantiated by a dataset that spans four languages (English, Spanish, German and Polish), two entity types (person and organization) and two domains (Wikipedia and News). We carry out an extensive analysis of the data and the task, pointing to the various modeling challenges it poses. The results we obtain with standard language generation methods show that the task is indeed non-trivial, and leaves plenty of room for improvement.

## introduction news articles, image captions, product reviews and many other texts mention people and organizations, whose name recognition could vary for different audiences. a piece of news, for example, may concern people and organizations that are known locally, but are not necessarily well-recognized on a global level. in such cases, news pieces targeted at a wider audience would provide background information about the entity in focus, often in the form of an appositive. for example: in march 2017 , natalie jaresko, former minister of finance in ukraine, was appointed as the board's executive director. it is unlikely that many people outside of ukraine know the name natalie jaresko, so a foreign reader would likely benefit from the extra bit of information about her former occupation as justification for her new appointment. an appositive could also be less contextualized and provide information of more general importance, for example: the conservation unit is in the calhau bairro of são luís, the state capital. in general terms, appositives are phrases that appear next to a noun phrase and serve an explicative function @xcite . adding such explanations to text is a multi-step process. first, one has to decide whether an entity mention needs an appositive. that may not be the case for entities that are sufficiently well-known or that have been introduced earlier in the text. in case an appositive is indeed needed, the next step is to choose what information about the entity to disclose. if the information is to be of a factual nature, the writer needs to have prior knowledge of the entity, or access to an external knowledge resource- @xcite found appositives to be frequently based on facts of particular relevance to the context of the mention. lastly, the surface form of the appositive, well-fitted to the surrounding context, needs to be produced. viewed from the perspective of nlp, appositive generation is therefore an interesting and challenging natural language generation problem that involves reasoning over facts from an external knowledge source, with reference to a given context. the task of appositive generation, first introduced by @xcite , is still in its early stages and data resources are limited. we expand on previous work in appositive generation with a new, more realistic, end-to-end definition of the task, instantiated by a dataset, apposcorpus, foot_0 that spans four languages (english, spanish, german and polish), two entity types (person and organization) and two domains (wikipedia and news). while wikipedia as a domain is curated for a world-wide audience and as such may not benefit much from appositive generation, we posit that it is a valuable source of abundant cross-lingual data which could be used as the basis for transfer learning. in addition to a large training set automatically sourced from wikipedia, we therefore also introduce a gold standard test sourced from newswire, one of the true target domains for appositive generation @xcite . the next section of the paper outlines the theoretical framework behind the task of appositive generation. in section 3 we describe the automatic data collection procedure used to obtain training data, and in section 4 we detail the further manual validation performed to obtain quality data for cross-domain evaluation. the properties of the dataset are discussed in detail in section 5. sections 6, 7 and 8 describe the experiments we performed and the main findings from them. section 9 concludes the paper and outlines avenues for future research. ## dataset collection: wikipedia we used the march 2020 wikipedia dump foot_1 for english, spanish, german and polish, which we parsed with wikiextractor, foot_2 preserving internal links. 4 the choice of these particular languages was mostly based on the availability of good dependency parsers. since dependency parsing is an integral step of the data collection process @xcite , it has to be as precise as possible, to maximize the quality of the outcome. 5 below, we describe in detail the data collection procedure. ## dataset collection: news we sourced our data for cross-domain evaluation from the news domain, following previous work @xcite , using these news corpora: global voices (english, spanish, german, polish) @xcite , news commentary (english, spanish, german) @xcite and paralela @xcite . ## data analysis this section outlines some findings on the properties of our dataset, based on general statistics and wikidata cross-referencing. the procedure described above yielded less than four thousand samples of org appositives for polish, so this subset is omitted from the apposcorpus. ## experiments to show how the new task formulation can be used, we experiment with three established language generation methods: the main method of @xcite , which we refer to as base; an extension of base with external knowledge injected through embeddings with knowledge-base grounding (kb); and a model enhanced with an explicit copy mechanism (copynet, @xcite ). notice that our goal here is not to build the best model for this task, but to develop reasonable models which can serve as baselines for future work in this area. foot_6 lstm with external knowledge, kb external knowledge can be beneficial to a better understanding of the context and how it relates to the different facts known about an entity. here, we use the same architecture as above, but initialize the embedding matrix of the model with the ntee (neural text-entity encoder) word embeddings, trained on wikipedia with wikidata grounding @xcite . they aim to represent a text and its relevant entities close to each other. we deem these embeddings suitable for our modeling setup, where input text and facts are represented in a shared space. unfortunately, the ntee embeddings are available only for english. so we used word-level translation to "project" them to spanish, german and polish. see more details in appendix a.4. this approach is likely to introduce some noise, but we only use the projection to initialize the embedding matrix which is then further trained. so any signal coming from the embeddings can be used by the model and any noise can be filtered out during training. lstm with a copy mechanism, copynet motivated by the observation that there is an overlap of at least one token between wikidata facts and appositives for about 25% of the datapoints in our dataset, we experiment with a method that allows the decoder to copy tokens directly from the input: copynet @xcite . @xcite correctly point out that in their constrained data setting, where data points were selected based on word overlap with wikidata facts, using a copy mechanism would result in double-counting, i.e. artificially boosted results. in our data setting, however, this is not the case. all three approaches are end-to-end in the sense that we do not split up the classification task of whether or not to predict an appositive from the task of generating an appositive where it is due. as negative samples in the dataset have the @xmath0, the models are performing the classification task implicitly by choosing whether to predict @xmath1. preliminary experiments with the base architecture showed that the choice between providing the model with three sentences of preceding context, one or zero had little impact on its performance, so all results reported below use one sentence of preceding context, following @xcite . further details on the implementations and the hyperparameters we used can be found in appendix a.3. ## results we view the results of our experiments from two angles: one concerns the expansion of the task definition we achieve with apposcorpus, from a constrained scenario to an end-to-end one; the other concerns the increased coverage of the dataset, which allows us to compare and contrast appositive generation across different languages and named entity types. ## conclusion apposcorpus targets factual appositive generation, a phenomenon frequently occurring in a range of textual domains. it substantially extends the prior resources in the area by spanning four languages, two named entity types , and two domains. this resource also allows the burgeoning field to investigate end-to-end appositive generation. our manual and automatic evaluations with apposcorpus show that standard model architectures can approach the quality of human targets in specific cases but there is still room for improvement. with this dataset, appositive generation can be studied in much more depth than previously possible, ultimately paving the way for novel nlp applications in the generation and writing space. the focus in future research, we believe, should fall on explicit methods for cross-domain learning, on richer knowledge sources, and on the development of test sets for new domains.