Meaning Representations for Natural Languages: Design, Models and Applications (2022)

Abstract: This tutorial reviews the design of common meaning representations, SoTA models for predicting meaning representations, and the applications of meaning representations in a wide range of downstream NLP tasks and real-world applications. Reporting by a diverse team of NLP researchers from academia and industry with extensive experience in designing, building and using meaning representations, our tutorial has three components: (1) an introduction to common meaning representations, including basic concepts and design challenges; (2) a review of SoTA methods on building models for meaning representations; and (3) an overview of applications of meaning representations in downstream NLP tasks and real-world applications. We will also present qualitative comparisons of common meaning representations and a quantitative study on how their differences impact model performance. Finally, we will share best practices in choosing the right meaning representation for downstream tasks.

## background in this tutorial, we primarily discuss one thread of meaning representations encompassing the proposition bank (propbank) @xcite , abstract meaning representations (amr) as well as uniform meaning representations (umr), a recent extension to amr. we will discuss the representations themselves, discuss the latest semantic role labeling (srl) and amr parsing techniques using these representations, and overview applications of these meaning representations to practical natural language applications. these approaches all share the use of the predicate-specific semantic roles defined in the proposition bank (propbank) @xcite . in such an approach, the particular sense of "afford" in "the public was afforded a preview of the show", is sense-tagged as "afford.02" in propbank, and it requires three semantic roles, arg0 the provider, arg1 the thing that is provided, and arg2 the recipient of arg1. we will seek to provide attendees with good intuitions about the behavior and advantages of how such predicate-specific roles work across these different meaning representations. we will also contextualize how such an approach to semantics compares to other approaches such as framenet @xcite . amr can be viewed as an extension of prop-bank to handle wide-coverage sentence representation. whereas propbank is annotated on a predicate-by-predicate basis and predicates are can be viewed as independent, abstract meaning representation (amr) @xcite adopts propbank-style semantic roles but also connects the different predicates in a sentence in a graph. such an amr graph seeks to represent the meaning of sentences as a single-rooted directed acyclic graph, where the nodes are labeled with entity or predicate types, and edges are labeled with semantic roles (e.g., arg0, arg1) or general semantic relations (e.g., time, location). amr captures the essential predicate-argument structure of a sentence that is applicable to a variety of applications as well as to languages such as chinese. extensions to amr attempt to increase coverage beyond the sentence, to add additional semantic phenomena, and to increase cross-linguistic applicability @xcite . we discuss these extensions with a focus on the new uniform meaning representation(umr) approach, which extends amr to add coverage of aspect, scope, person and number annotation to the sentence level representation, adds a document-level representation that captures temporal and modal dependencies as well as coreference relations that can go beyond sentence boundaries, and which defines conventions for amr-style annotation of languages without existing propbank lexicons. the discussion of umr will provide attendees with an understanding of 1 which semantic phenomena are out of scope for amr and how projects like umr address them. in this tutorial we will provide an in-depth discussion of these meaning representations. when doing so, we will also discuss how they are similar to or different from other meaning representations such as semantic dependencies @xcite , minimal recursion semantics (mrs) @xcite , discourse representation theory (drt) @xcite , and ucca @xcite . the increasing availability of meaning representation datasets such as propbank as well as significant advances in modeling techniques have led to increased interest and progress in computational models for meaning representation parsers. in this tutorial, we will discuss models for srl and amr tasks. we will start with the traditional srl models that rely heavily on syntactic feature templates @xcite @xcite , go on to advanced neural srl models @xcite , and include more recent work @xcite . for amr parsing, we will cover early approaches and sota methods for graph-based methods @xcite @xcite @xcite , transition-based methods @xcite @xcite @xcite , grammar-based methods @xcite @xcite sequence-tosequence methods @xcite , and other methods @xcite @xcite @xcite . we will discuss whole-document amr parsing @xcite . there is a wide range of nlp tasks that leverage meaning representations as an effective way to infuse knowledge into their models for better performance and interpretability. for instance, srl has been widely used to build better models for information extraction, such as open information extraction @xcite and event extraction @xcite , opinion mining @xcite , machine translation @xcite , natural language inference @xcite , and reading comprehension @xcite . similarly, amr has been adopted for a variety of downstream nlp tasks such as information extraction @xcite @xcite , summarization @xcite , machine translation @xcite , question answering @xcite @xcite , and dialog @xcite . with the increasing availability of high-quality meaning representation parsers, we also see increasing adoption of meaning representation in wide-range of real-world applications, from an enterprise-grade contract understanding system @xcite to customizable targeted sentiment analysis. ## tutorial type we are proposing a 6-hour cutting edge tutorial to cover in depth on the design, modeling, and application of meaning representations. ## outline of the tutorial the proposed tutorial is organized as follows: i. introduction (15 minutes). we will provide a high-level overview and evolution of common meaning representation, discussing key concepts, unique challenges and examples of applications. ii. common meaning representations (150 minutes) in this section, we will provide an indepth review of three common meaning representation -propbank and framenet that have been widely used to train semantic role labeling systems, abstract meaning representation, a sentence-level meaning representation that inherits propbank-style semantic roles, and uniform meaning representation, a cross-lingual document-level meaning representation that to a large extent inherits the sentence-level representation of amr. we also provide a brief overview of other common meaning representations as a brief background. we will also discuss the unique challenges around designing meaning representation. concretely, we will organize this section as follows: we next discuss different aspects of amr, and cover how amr represents word senses, semantic roles, named entity types, date entity types, and relations. iii. modeling meaning representation (100 minutes) we will next discuss computational models for srl and amr parsing, from early approaches to current end-to-end sota methods. we will discuss gaps and challenges in building and evaluating such models. we will also share a quantitative comparison study based on sota models and demonstrates how the differences of the meaning representations lead to differences in model performance on various examples. iv. applying meaning representation (75 minutes) we will share applications of the meaning representations for a wide range of tasks from information extraction to question answering. we will discuss how the differences in these meaning representations discussed earlier impact the choice of which one(s) to use for which downstream tasks. v. open questions and future work (15 minutes) we will conclude the tutorial by raising several open research questions in this space (e.g., creating datasets for training and evaluation at scale) and ways we as a community might work forward on these issues. ## breadth of the tutorial this tutorial will have three components. the first component (45%) will introduce core concepts related to meaning representations, common meaning representations and key challenges in designing (including scaling to different languages) and developing those meaning representations. the second component (30%) will review the state-of-the-art models for two common meaning representations: srl and amr. it will also provide a quantitative comparison study of how the differences in meaning representations impact model performance. finally, the last component (25%) will show how real-world applications as well as research projects leverage meaning representations for better performance and more transparency and how to decide which meaning representation to use based on downstream tasks. ## diversity of the team this tutorial is to be given a team of researchers from five different institutions across academia and industry, both junior instructors (including 1 assistant professor, and 2 junior industry researcher) and researchers with extensive experience in academic and corporate research settings. the team includes creators, modelers, and users of common meaning representations. the team also has a good gender balance (two female and four male instructors). ## target audience and objectives this tutorial welcomes all stakeholders in the nlp community, including nlp researchers, domainspecific practitioners, and students. in this tutorial, attendees will ## prerequisites as stated before, our tutorial presumes no prior knowledge on the core concepts of meaning representation. however, a basic understanding of nlp, machine learning (especially, deep learning) concepts may be helpful. we intend to introduce the necessary concepts related to meaning representation during the introductory section of the tutorial. ## reading list we aim to make the tutorial self-contained, but it will be helpful if the attendees can get some basic understanding of this field by going through the following reading list: propbank: @xcite , amr: @xcite , umr: (gysel et al., 2021), srl models: @xcite , and amr models: @xcite @xcite . ## audience size estimation we are proposing a cutting edge tutorial on meaning representation. no similar tutorial has been given in acl/emnlp/naacl/coling in the past five years. since meaning representation is an important topic in nlp, we expect that this tutorial will be popular with 50 -100 attendees. ## open access we agree to allow the publication of our slides and video recording of our tutorial in the acl anthology. ## technique equipment to give this tutorial, we need to have internet access and a projector or large screen. no special requirements needed. ## preferred venue due to travel restrictions of our instructors, we prefer naacl and acl over the other venues. ## ethics statement infusing meaning representations into nlp models are shown to be effective in injecting knowledge into such models. as such, meaning representations allow deep understanding of languages and identify more nuanced instances of ethics concerns (e.g. biases). furthermore, meaning representations allow the building of fully interpretable yet effective models. we hope that this tutorial helps the audience to develop a deeper appreciation for such topics and equips them with powerful tools to mitigate recent concerns that have arisen with nlp models with regard to explainability and bias. martha palmer is the helen & hubert croft professor of engineering in the computer science department, and arts & sciences professor of distinction for linguistics, at the university of colorado, with over 300 peer-reviewed publications. her research is focused on capturing elements of the meanings of words that can comprise automatic representations of complex sentences and documents in many languages. she is a co-director of clear, an acl fellow, and an aaai fellow. nianwen xue is a professor in the computer science department and the language & linguistics program at brandeis university. his core research interests include developing linguistic corpora annotated with syntactic, semantic, and discourse structures, as well as machine learning approaches to syntactic, semantic and discourse parsing. he is an action editor for computational linguistics. ishan jindal is a research staff member with ibm research -almaden. his research interest lies at the intersection of machine learning and nlp, primarily in semantic parsing and model analysis for enterprise use cases. he regularly publishes papers at ml and nlp conferences. jeffrey flanigan is an assistant professor in the computer science and engineering department at university of california santa cruz. he research interests are in semantic parsing and generation, with a focus on amr, and using semantic representations in downstream applications such as summarization and machine translation. previously he has given a tutorial in amr at naacl 2015. tim o'gorman is a senior research scientist at thorn. he was involved in amr 2.0 and 3.0 annotations, the multi-sentence amr corpus, and updates to propbank. he co-organized the conll'19 and '20 meaning representation parsing shared task. his interests are in the extensions of meaning representations to cross-sentence phenomena. yunyao li is a distinguished research staff member and senior research manager with ibm research -almaden. her expertise is at the intersection of nlp, databases, hci, and information retrieval. her work has resulted in 80+ peer-reviewed publications and transferred into 20+ commercial products. she regularly gives talks and tutorials, such as explainability for nlp (aacl'20, kdd'21), and deep learning on graphs for nlp (naacl'21, kdd'21, ijcai'21). she is an acm distinguished member.