Instilling Type Knowledge in Language Models via Multi-Task QA (2022)

Abstract: Understanding human language often necessitates understanding entities and their place in a taxonomy of knowledge—their types .Previous methods to learn entity types rely on training classifiers on datasets with coarse, noisy, and incomplete labels. We introduce a method to instill fine-grained type knowledge in language models with text-to-text pre-training on type-centric questions leveraging knowledge base documents and knowledge graphs.We create the WikiWiki dataset: entities and passages from 10M Wikipedia articles linked to the Wikidata knowledge graph with 41K types.Models trained on WikiWiki achieve state-of-the-art performance in zero-shot dialog state tracking benchmarks, accurately infer entity types in Wikipedia articles, and can discover new types deemed useful by human judges.

## introduction entities can be categorized by their types, which indicate where they belong in a taxonomy of knowledge. for example, venus is a planet and thus also an astronomical body. much like how knowledge acquisition in cognitive development progresses from recognizing concrete objects to gradually understanding their relations to one another @xcite , we aim to extend language models' existing rough understanding of entities @xcite to the types that govern how entities are related. instilling type knowledge in multi-purpose models can improve performance in tasks like entity linking @xcite , question-answering @xcite , and semantic parsing @xcite . while language models can memorize some facts @xcite , they frequently hallucinate false information @xcite . current attempts to learn to infer types for entities are hampered by 1) the difficulty of collecting diverse, large-scale typing datasets; and 2) how existing corpora assume independence between types @xcite , while in reality types sit at levels of granularity that are useful in different settings: a pizza store may care whether a user likes cheese pizza; a restaurant recommender might care if the user wants pizza; finally, a general dialog agent might only care if a user wants food. we address both issues by proposing a simple and effective approach for pre-training generative language models to answer questions about entities, types, and surface forms (mentions) in a large public knowledge graph (kg) consisting of wikipedia articles and wikidata nodes. we leverage high quality type labels in a large corpus of knowledge-rich text and an ordered, hierarchical type ontology. to summarize our main contributions: 1) we create the new wikiwiki dataset comprising 10m wikipedia articles linked to nodes from wikidata; 2) we propose a pre-training scheme for generative language models using type-centric questionanswering based on wikiwiki; 3) we achieve stateof-the-art (sota) performance in zero-shot domain adaptation for dialog state tracking using our type-instilled models, with average per-domain gains of 14.9% (49.4% relative) joint accuracy; and 4) we show that our models can precisely infer types for seen and unseen entities in new articles from wikiwiki, and propose novel types that humans judge to be accurate and appropriate. entity representation learning many sota systems for knowledge retrieval and qa rely on learned dense embeddings of individual entities or types to perform multi-class classification @xcite @xcite . several recent frameworks aim to learn entity knowledge during language model pretraining via entity masking @xcite or contrastive learning @xcite . systems for entity typing @xcite and disambiguation @xcite also learn dense vector encodings that are later matched via dot-product scoring. cao et al. (2021) aim to address some context: these included carbon dioxide by burning diamond, and mercuric oxide by heating mercury. this type of experiment contributed to the discovery of "dephlogisticated air" by priestley, which became better known as oxygen, following lavoisier's investigations. entity/type discovery (20%): list all concepts and types mentioned here. answer: priestley (chemist), lavoisier (chemist), mercuric oxide (chemical compound), mercury (chemical element), and dephlogisticated air (superseded scientific theory) entity typing (30%): what is dephlogisticated air an example of? answer: superseded scientific theory entity recognition (20%): what does priestley refer to? answer: joseph priestley (chemist) slot filling (30%): which chemists are mentioned here? answer: joseph priestley and antoine lavoisier ## type-centric multitask modeling wikiwiki corpus to train an entity-and typeaware language model, we build the wikiwiki dataset by combining wikipedia articles with the wikidata kg @xcite . wikipedia articles have been used to enrich corpora for dialog @xcite , coreference resolution @xcite , and qa @xcite . kgs have been used for entity typing and relation extraction @xcite . @xcite use wikipedia pages as context for relation triples mined from wikidata. we link articles, entities, and types as in figure 1: like @xcite , we take wikipedia hyperlinks as links between entities (target page) and their mentions (link text); we link pages to wikidata nodes via id; and for each node we extract types t from wikidata where @xmath0. 1 to address sparsity of hyperlinks, we follow @xcite and use spacy to identify additional entities. we sample 10m articles for training, with two disjoint 5k-article splits for evaluation, containing seen and unseen (new ent) entities respectively ( entities in wikidata on average are assigned 1.28 types; for entities with multiple types, not all types are necessarily relevant to a context. for example, take the following passage: "obama was elected to the illinois senate in 1996, succeeding democratic state senator alice palmer from illinois's 13th district, which, at that time, spanned chicago south side neighborhoods from hyde park-kenwood south to south shore and west to chicago lawn." while wikidata entities may have 5+ types, many are not directly relevant to a context. for example, while barack obama has types including politician, jurist, political writer, community organizer, and podcaster, the latter is not relevant to the context. to teach our models to infer types relevant to the context at hand, in pre-training data we take only types that are shared between barack obama and other entities in the document (e.g. alice palmer-politician). we have made the wiki-wiki dataset publicly available on github. 2 pre-training tasks to instill type-centric knowledge from wikiwiki, we train our models to answer four types of knowledge-based questions conditioned on a passage from wikipedia (examples in user: i'm looking for a place to stay during my upcoming trip to cambridge. system: i can definitely help you with that! what area are you staying in, and what is the price range you are looking for? user: it should be located in the west and it should be cheap. belief state: [hotel price range]: cheap; [hotel area]: west ## experiments we demonstrate the effectiveness of our pretraining on two tasks that require type understanding: zero-shot domain generalization in dialog state tracking (dst), and fine-grained entity typing. zero-shot dst the goal of dialog state tracking (dst) is to infer user intent and goals from conversations by filling in belief slots @xcite . in many real-world settings, dst models must be able to predict new slot values (i.e. new entities that are not present in the training corpus) and new slot types (e.g. requirements for applications in new domains). this problem setting is known as zero-shot dst ( our method is complementary to systems for creating synthetic in-domain dialogs @xcite . as seen in ## conclusion in this paper, we 1) propose a text-to-text pretraining scheme to instill type knowledge in language models via qa and 2) release the wikiwiki dataset built from wikipedia articles and the wikidata kg. we show that wikiwiki is larger-scale and more accurate than existing fine-grained type recognition datasets. we demonstrate that our typecentric pre-training framework allows us to train language models that can better generalize to unseen domains, entities, and types-which in turn lead to improved model performance on downstream tasks like dialog state tracking (achieving sota results on zero-shot dst with average gains of 14.9% joint accuracy). our models can extrapolate type knowledge and infer novel types that humans judge to be useful and precise. as the body of human knowledge grows, we see an opportunity to use life-long learning @xcite on news and publications to expand and model the taxonomy of knowledge.