Unified Semantic Typing with Meaningful Label Inference (2022)

Abstract: Semantic typing aims at classifying tokens or spans of interest in a textual context into semantic categories such as relations, entity types, and event types. The inferred labels of semantic categories meaningfully interpret how machines understand components of text. In this paper, we present UniST, a unified framework for semantic typing that captures label semantics by projecting both inputs and labels into a joint semantic embedding space. To formulate different lexical and relational semantic typing tasks as a unified task, we incorporate task descriptions to be jointly encoded with the input, allowing UniST to be adapted to different tasks without introducing task-specific model components. UniST optimizes a margin ranking loss such that the semantic relatedness of the input and labels is reflected from their embedding similarity. Our experiments demonstrate that UniST achieves strong performance across three semantic typing tasks: entity typing, relation classification and event typing. Meanwhile, UniST effectively transfers semantic knowledge of labels and substantially improves generalizability on inferring rarely seen and unseen types. In addition, multiple semantic typing tasks can be jointly trained within the unified framework, leading to a single compact multi-tasking model that performs comparably to dedicated single-task models, while offering even better transferability.

## introduction semantic typing is a group of fundamental natural language understanding problems that aim at classifying tokens (or spans) of interest into semantic categories. this includes a wide range of long-standing nlp problems such as entity typing, relation classification, and event typing. inferring the types of entities, relations or events mentioned is not only crucial to the structural perception of human language, but also plays an important role in many downstream tasks such as entity linking @xcite , information extraction @xcite ) and question answering @xcite . most traditional methods tackle semantic typing problems by training task-specific multi-class classifiers with token or sentence representations from language models to predict a probability distribution over a pre-defined set of classes @xcite . however, this approach comes with several limitations. first, these models simply convert labels into indices, thus completely ignoring the rich semantics carried by the label text itself. for example, given "currently ritek is the largest producer of oleds.", knowing what the entity type company means would naturally simplify the inference of "ritek" is a company in this context. second, models trained as classifiers do not generalize well to class labels that are rarely seen or unseen in the training data, as these models rely on the abundance of annotated examples to associate semantics to label indices. in particular, since these classifiers are limited by the pre-defined label set, they cannot infer any unseen labels unless being re-trained or incorporated with label mapping rules. as a result, these models struggle to handle more fine-grained semantic typing tasks in real-world scenarios @xcite where any free-form textual labels may be used to represent the types, many of which may also be unseen during training. in contrast to the aforementioned traditional paradigm for semantic typing, several studies have explored alternative approaches such as promptbased learning @xcite and indirect supervision from nli models @xcite to make more efficient use of label semantics. however, these methods usually require hand-crafted ## method in this section, we present the technical details of unist, our unified framework for semantic typing. we first provide a general definition of a semantic typing problem ( §2.1), followed by a detailed description of our model ( §2.2), training objective( §2.3), and inference ( §2.4). ## experiments in this section, we evaluate unist on single-task experiments on three semantic typing tasks: entity typing ( §3.1), relation classification ( §3.2) and event typing ( §3.3). we then assess the generalizability of unist by conducting zero-shot and few-shot prediction, and study the effects of task description ( §3.4). finally, we train unist under multi-task setting to solve all three tasks simultaneously ( §3.5). ## related works we present two lines of relevant research topics. each has a large body of work which we can only provide as a highly selected summary. semantic typing. semantic typing tasks can be generally categorized into lexical typing (e.g., entity typing, event typing) and relational typing (or classification). a large number of specialized approaches have been developed for individual semantic typing tasks. for example, prior studies on entity typing have exploited label dependencies and hierarchies @xcite , capturing label relations with knowledge bases @xcite , as well as automatic data augmentation and denoising techniques @xcite to deal with fine-grained type vocabularies. relation classification has been tackled by modeling dependency structures @xcite , learning span representations @xcite , entity representations @xcite , and injecting external knowledge into pre-trained language models @xcite @xcite . nevertheless, most previous methods have formulated semantic typing as a multi-class classification problem without capturing label semantics. ## conclusion we propose unist, a unified framework for semantic typing that exploits label semantics to learn a joint semantic embedding space for both inputs and labels. by incorporating model-agnostic task descriptions, unist can be easily adapted to different semantic typing tasks without introducing taskspecific model components. experimental results show that unist offers both strong performance and generalizability on entity typing, relation classification, and event typing. our unified framework also facilitates learning a single model to solve different semantic typing tasks simultaneously, with performance on par with dedicated models trained on individual tasks.