Deep Learning on Graphs for Natural Language Processing (2021)

Abstract: Due to its great power in modeling non-Euclidean data like graphs or manifolds, deep learning on graph techniques (i.e., Graph Neural Networks (GNNs)) have opened a new door to solving challenging graph-related NLP problems. There has seen a surge of interests in applying deep learning on graph techniques to NLP, and has achieved considerable success in many NLP tasks, ranging from classification tasks like sentence classification, semantic role labeling and relation extraction, to generation tasks like machine translation, question generation and summarization. Despite these successes, deep learning on graphs for NLP still face many challenges, including automatically transforming original text sequence data into highly graph-structured data, and effectively modeling complex data that involves mapping between graph-based inputs and other highly structured output data such as sequences, trees, and graph data with multi-types in both nodes and edges. This tutorial will cover relevant and interesting topics on applying deep learning on graph techniques to NLP, including automatic graph construction for NLP, graph representation learning for NLP, advanced GNN based models (e.g., graph2seq, graph2tree, and graph2graph) for NLP, and the applications of GNNs in various NLP tasks (e.g., machine translation, natural language generation, information extraction and semantic parsing). In addition, hands-on demonstration sessions will be included to help the audience gain practical experience on applying GNNs to solve challenging NLP problems using our recently developed open source library – Graph4NLP, the first library for researchers and practitioners for easy use of GNNs for various NLP tasks.

## description deep learning has become the dominant approach in natural language processing (nlp) research today, especially when applied on large scale corpora. conventionally, sentences are typically considered as a sequence of tokens in nlp tasks. hence, popular deep learning techniques such as recurrent neural networks (rnn) and convolutional neural networks (cnn) have been widely applied for modeling text sequence. however, there is a rich variety of nlp problems that can be best expressed with a graph structure. for instance, the structural and semantic information in sequence data (e.g., various syntactic parsing trees like dependency parsing trees and semantic parsing graphs like abstract meaning representation (amr) graphs) can be exploited to augment original sequence data by incorporating the task-specific knowledge. as a result, these graphstructured data can encode complicated pairwise relationships between entity tokens for learning more informative representations. however, it is well-known that deep learning techniques that were disruptive for euclidean data such as images or sequence data such as text are not immediately applicable to graph-structured data. therefore, this gap has driven a tide in research for deep learning on graphs, especially in development of graph neural networks (gnn). this wave of research at the intersection of deep learning on graphs and nlp has influenced a variety of nlp tasks. there has seen a surge of interests in applying/developing various types of gnns and achieved considerable success in many nlp tasks, ranging from classification tasks like sentence classification, semantic role labeling and relation extraction, to generation tasks like machine translation, question generation and summarization. despite these successes, deep learning on graphs for nlp still face many challenges, namely, this tutorial of deep learning on graphs for natural language processing (dlg4nlp) is timely for the computational linguistics community, and covers relevant and interesting topics, including automatic graph construction for nlp, graph representation learning for nlp, various advanced gnn based models (e.g., graph2seq, graph2tree, and graph2graph) for nlp, and the applications of gnns in various nlp tasks (e.g., machine translation, natural language generation, information extraction and semantic parsing). the intended audiences for this tutorial mainly include graduate students and researchers in the field of natural language processing and industry professionals who want to know how the state-of-the-art deep learning on graphs techniques can help solve important yet challenging natural language processing problems. in addition, hands-on demonstration sessions will be included to help the audience gain practical experience on applying gnns to solve challenging nlp problems using our recently developed open source library -graph4nlp, the first library for researchers and practitioners for easy use of graph neural networks for various nlp tasks. after attending the tutorial, the audience are expected to 1) have a comprehensive understanding of basic concepts of deep learning on graphs for nlp; 2) learn major recent advances of research in the intersection of nlp and gnns; and 3) explore novel research opportunities of gnns for nlp, and learn how to use or even design novel algorithms with gnns for effectively coping with various nlp tasks. we will start with a broad overview of various nlp problems that deal with graph structured data, and highlight some challenges of modeling graphstructured data in the field of nlp with traditional graph-based algorithms (e.g., random walk methods, spectral graph clustering, graph kernels). we will then introduce the general idea as well as some commonly used models of gnns, which have been an emerging popular tool to deal with graph structured data. after the introduction of nlp tasks on graph data and graph neural networks, we will describe some important yet challenging techniques for deep learning on graphs for nlp, including automatic graph construction from text, graph representation learning for nlp and various advanced gnn based models (e.g., graph2seq, graph2tree, and graph2graph) for nlp. some representative nlp applications are introduced following the methods. we also include a hands-on demonstration session on how to quickly build gnn-based models for solving nlp tasks using our recently developed open source library graph4nlp, which was designed for the easy use of gnns for nlp. we will summarize the tutorial and highlight some open directions in the end of this tutorial. the introduction, methodologies, applications, hands-on demonstration, and conclusion and open directions form the five segments of this tutorial. ## prerequisites the audience is expected to have some basic understanding of natural language processing and deep learning. however, the tutorial will be presented at college junior/senior level and should be comfortably followed by academic researchers and industrial practitioners. ## tutorial outline the intended duration of this tutorial is 3.5 hours, including a half hour break. i. ( 20 minutes ) introduction 1. natural language processing: a graph perspective 2. graph based algorithms for natural language processing 3. deep learning on graphs: graph neural networks i. foundations ii. methodologies iii. applications in natural language processing: an overview iv. high-level dlg4nlp roadmap ii. (70 minutes) methodologies 1. automatic graph construction from text i. static graph construction ii. dynamic graph construction 2. graph representation learning for nlp i. graph neural networks for improved text representation ii. graph neural networks for joint text & knowledge representation iii. graph neural networks for various graph types 3. gnn based encoder-decoder models i. graph-to-sequence models ii. graph-to-tree models iii. (60 minutes) applications 1. semantic parsing 2. machine reading comprehension 3. information extraction 4. natural language generation 5. machine translation iv. (20 minutes) hands-on demonstration 1. a brief overview of the graph4nlp library 2. live demo v. (10 minutes) conclusion and open directions ## reading list we aim to make the tutorial self-contained. for trainees interested in reading important studies before the tutorial, we recommend the following papers regarding gnns @xcite @xcite , automatic graph construction for nlp @xcite , joint text and knowledge representation learning @xcite , modeling directed graphs @xcite and heterogeneous graphs @xcite , and gnn based encoder-decoder models @xcite @xcite . ## diversity beside the state-of-the-art deep learning on graphs techniques we are planning to cover, we will discuss how these graph-based deep learning techniques can be used in several nlp applications that exploit multilingual data, including but not limited to machine translation and information extraction. our tutorial lectures are full of diversities from many perspectives. our team have male and female researchers(two female tutors and two male tutors); we are from three different institutions including ibm research, uiuc, and facebook ai; we are senior, middle-career, and junior-career researchers. we are researchers and professors from academic institutions and industrial labs.