Knowledge-Augmented Methods for Natural Language Processing (2022)

Abstract: Knowledge in natural language processing (NLP) has been a rising trend especially after the advent of large scale pre-trained models. NLP models with attention to knowledge can i) access unlimited amount of external information; ii) delegate the task of storing knowledge from its parameter space to knowledge sources; iii) obtain up-to-date information; iv) make prediction results more explainable via selected knowledge. In this tutorial, we will introduce the key steps in integrating knowledge into NLP, including knowledge grounding from text, knowledge representation and fusing. In addition, we will introduce recent state-of-the-art applications in fusing knowledge into language understanding, language generation and commonsense reasoning.

## brief tutorial outline in recent years, the field of natural language processing has considerably benefited from largerscale models, better training strategies, and greater availability of data, exemplified by bert * @xcite , roberta * @xcite , and gpt models @xcite @xcite . it has been shown that these pretrained language models can effectively characterize linguistic patterns in text and generate highquality context-aware representations @xcite . however, these models are trained in a way where the only input is the source text. as a result, these models struggle to grasp external world knowledge about concepts, relations, and common sense @xcite . in this tutorial, we use knowledge to refer to this external information which is absent from model input yet useful for the model to produce target output. knowledge is important for language representation and should be included into the training and inference of language models. knowledge is also an indispensable component to enable higher levels of intelligence which is unattainable from statistical learning on input text patterns.