CLSE : Corpus of Linguistically Significant Entities (2022)

Abstract: One of the biggest challenges of natural language generation (NLG) is the proper handling of named entities. Named entities are a common source of grammar mistakes such as wrong prepositions, wrong article handling, or incorrect entity inflection. Without factoring linguistic representation, such errors are often underrepresented when evaluating on a small set of arbitrarily picked argument values, or when translating a dataset from a linguistically simpler language, like English, to a linguistically complex language, like Russian. However, for some applications, broadly precise grammatical correctness is critical – native speakers may find entity-related grammar errors silly, jarring, or even offensive. To enable the creation of more linguistically diverse NLG datasets, we release a Corpus of Linguistically Significant Entities (CLSE) annotated by linguist experts. The corpus includes 34 languages and covers 74 different semantic types to support various applications from airline ticketing to video games. To demonstrate one possible use of CLSE, we produce an augmented version of the Schema-Guided Dialog Dataset, SGD-CLSE. Using the CLSE’s entities and a small number of human translations, we create a linguistically representative NLG evaluation benchmark in three languages: French (high-resource), Marathi (low-resource), and Russian (highly inflected language). We establish quality baselines for neural, template-based, and hybrid NLG systems and discuss the strengths and weaknesses of each approach.

## introduction natural language generation (nlg) @xcite is an umbrella term for the problem figure 1 : the corpus of linguistically significant entities (clse) is created by annotating a large number of entities with their linguistic properties. entities are grouped by (language, semantic_type, linguistic_signature) triples. this results in a corpus of entities that, for a given language and semantic type, is balanced across linguistic properties. note that some linguistic signatures have few or no annotated entities because they simply do not occur. unlike most natural language processing tasks, it is hard to produce reusable ground truth labeled data for nlg: there can be many different outputs that are grammatical and human-like. this is why it is important to invest in nlg resources, especially for non-english languages. the gem benchmark @xcite is such an initiative for nlg evaluation. its second version @xcite covers 40 tasks and 51 languages and provides an easy way for others to contribute. in parallel to adding new nlg resources, a framework for dataset augmentation has recently been introduced by @xcite . the main idea is to perform transformation and slicing of the existing datasets to boost certain properties as a case singular plural nominative книга @xcite книги (knigi) genitive книги (knigi) книг (knig) dative книге (knige) книгам (knigam) accusative книгу @xcite книги (knigi) instrumental книгой (knigoy) книгами (knigami) prepositional книге @xcite книгах (knigakh) cf. "я купил <книгу>" ("i bought @xmath1; dative case) and "моя <книга> потерялась" ("@xmath2; nominative case). the rest of the paper is organized as follows. section 2 introduces the clse corpus annotated by linguist experts. in section 3, we discuss the problem of nlg for task-oriented dialogs and how clse can be applied to it. in section 4 we show how to construct a linguistically diverse dataset for this task, a dataset we refer to as sgd-clse. in section 5, we introduce three different baseline nlg systems which we evaluate on this dataset. results are discussed in section 6. ## corpus of linguistically significant entities (clse) the google knowledge graph api foot_0 @xcite provides access to millions of entries that describe real-world entities like people, places, and things. each entity is a node in the graph and can be associated with any number of schema.org semantic types, such as person, administrativearea, or touristattraction. we first source lexical annotations from expert linguists for a large number of entities in the knowl-edge graph. lexical annotations are languagespecific and pertain to broader categories of linguistic properties like animacy, 2 case, 3 classifier, countability, definiteness, gender, and number. different languages consists of different linguistic properties. for example, the concept of animacy is not used in the english language. descriptions of each linguistic property class are included alongside the dataset release. linguistic annotations for an entity include those that are important to handle in a template-based language generation context. for instance in english, location entities have locative preposition annotations while people entities have gender annotations. 4 in other languages like french, all entities are annotated for grammatical gender, and entities with an article are marked depending on whether its article stays unchanged or gets merged with a preposition (like it would for common nouns). we use linguists who are native speakers in their corresponding language to source linguistic annotations for popular entities. except for the following eight low-resource languages-bengali (bn), gujarati (gu), kannada (kn), malayalam (ml), marathi (mr), tamil (ta), telugu (te), and urdu (ur)-all annotators possess at least a bachelors degree in some branch of linguistics. linguist annotators' median age ranges from 25 to 35, and they are roughly equally split between male and female. instead of expert linguists-or in addition to them-one may use data mining techniques @xcite . we introduce the concept of a linguistic signature, which is a linearized string representation of an entity's linguistic attributes for a specific language. the maximum hypothetical number of distinct linguistic signatures for a language is the cartesian product of all linguistic features and values for that language. however, not all linguistic signatures are naturally occurring or relevant. for example, mass nouns foot_4 that start with a vowel do not occur in spanish. them) to ensure linguistic diversity appropriate for each language. to obtain entities based on linguistic variation, we annotate a large number of entities for each semantic type to create a table of (language, semantic_type, entity_id, name, linguistic_signature). we group rows in the table by (language, semantic_type, linguistic_signature) triples. the complete corpus covers 34 languages, 74 semantic types, and 222 distinct linguistic signatures. the full corpus of linguistically significant entities is available at clse.page.link/data. ## clse case study: task-oriented dialogs to demonstrate how one may use this corpus, we consider the problem of nlg for task-oriented dialog systems @xcite . unlike opendomain chit-chat systems, the natural language interface of virtual assistants such as amazon alexa, apple siri, or google assistant is highly taskoriented. users often interact with their virtual assistants to accomplish a specific action, like finding flights, booking restaurants, buying tickets, etc. in a task-oriented dialogue, the conversation between the user and the assistant is tracked by a dialog manager that uses a dialog state, a summary of the entire conversation up to the current turn (see, e.g., @xcite ). the dialog state consists of slots and values related to the specific intents, services, and actions in question. the assistant uses the dialog state to 1) invoke external apis with appropriate parameter values, as specified by the user ## sgd-clse: generating linguistically diverse data using the clse we perform our experiments on the schema-guided dialog dataset from dct8 @xcite . to emulate a data-to-text setup, we pair each utterance (text) with that utterance's acts, services, and slots. an example of such a pair can be seen in owing to the lack of noun inflections in english, we can also produce a delexicalized form of the utterance by searching for and stubbing out the argument values within. e.g., for the utterance in in our case study, we assume that we already have a system that can produce english utterancesarguably a much easier task given the simplicity of english grammar-and instead focus on localization: generating fluent and grammatical output in a target language. in this setup we get the following as input: dialog state, structured data, english text, and target language. we focus on three languages: french, russian, and marathi. the motivation for choosing these languages is to have a widely studied high-resource language (french), a highly inflected language (russian), and a low-input (structured data) output (utterance) service_name: "restaurants_1" "how would you like bazille, which is situated in san jose?" actions: ["offer_restaurant", "offer_city"] slots: {restaurant: "bazille", city: "san jose"} the process of creating the sgd-clse dataset, leveraging the clse, is as follows. 1) for every (service_name, action, slot_names) triple, we generate argument values by randomly sampling slot values from the clse for a specific target language: french, russian, or marathi respectively. this requires us to establish a mapping from each entity-related slot name to a semantic type. for example, city slots in the sgd schemas would be mapped to the city schema.org type in the clse corpus. 2) we substitute these new argument values into the english delexicalized utterances (templates) from sgd. the goal of entity re-sampling using the clse is to generate as linguistically diverse data as possible, for maximum linguistic coverage in the target language. foot_6 3) finally, we use professional translators to translate those utterances into the corresponding languages. figure 3 illustrates the algorithm. we use the aforementioned process with three examples per (service, action, slot_names) triple to generate language-specific linguistically diverse examples in french, russian, and marathi foot_7 on the train partition of the sgd dataset. we then use two of these examples for train and one for what we call "within-triple test" split (wt-test). 9 unlike the sgd dev and test splits, the wt-test split has a particular focus on linguistics as opposed to generalization across dialog states: it contains states that were seen during training, but with a linguistically diverse set of slot values. in addition to train and wt-test sets, we also use a sample from the original sgd dev and test sets translated into the target languages. ## experimental setup below we describe our setup to evaluate different nlg models using the sgd-clse dataset. ## results we split this section into two. first, we look at the results on the unseen test set. these contain dialog situations, (service_name, action, slot_names), which the systems did not see during training. we then move to the wt-test split, where we expect the gap to the human-written responses to be smaller. ## conclusion and discussion building a natural language generation system that can handle a broad diversity of entities with varying linguistic phenomena remains an open challenge. with the clse, any schema-informed nlg datasets can use techniques described in sections 3 and 4 to produce better linguistically represented data, and measure metrics on that data to draw more linguistically thoughtful conclusions. the space of possible inputs that nlg systems will be expected to handle may be highly unconstrained, and designing solutions that are linguistically robust, and defensibly so, is an ambitious and worthwhile pursuit. our results in section 6 establish an evaluation procedure probing nlg systems for their linguistic capabilities. we also evaluate four baselines using this procedure and conclude that none of them is at the human level yet. improving upon these baselines can be approached from two sides: either 1) improving factual accuracy and reducing hallucinations of a purely neural data-to-text approach or 2) improving the quality of grammatical error correction applied to a template-based approach. beyond nlg for virtual assistants, the clse data may also be used for other nlg tasks. e.g., we can use it to augment machine translation datasets by swapping named entities to further probe the fluency of generated translations. the full clse dataset is openly available at clse.page.link/data.