Contrastive Data and Learning for Natural Language Processing (2022)

Abstract: Current NLP models heavily rely on effective representation learning algorithms. Contrastive learning is one such technique to learn an embedding space such that similar data sample pairs have close representations while dissimilar samples stay far apart from each other. It can be used in supervised or unsupervised settings using different loss functions to produce task-specific or general-purpose representations. While it has originally enabled the success for vision tasks, recent years have seen a growing number of publications in contrastive NLP. This first line of works not only delivers promising performance improvements in various NLP tasks, but also provides desired characteristics such as task-agnostic sentence representation, faithful text generation, data-efficient learning in zero-shot and few-shot settings, interpretability and explainability. In this tutorial, we aim to provide a gentle introduction to the fundamentals of contrastive learning approaches and the theory behind them. We then survey the benefits and the best practices of contrastive learning for various downstream NLP applications including Text Classification, Question Answering, Summarization, Text Generation, Interpretability and Explainability, Commonsense Knowledge and Reasoning, Vision-and-Language.This tutorial intends to help researchers in the NLP and computational linguistics community to understand this emerging topic and promote future research directions of using contrastive learning for NLP applications.

## tutorial structure and content this tutorial first gives an introduction to the foundation of contrastive learning and then reviews the nlp application of contrastive learning. our tutorial covers both contrastive data augmentation for nlp and contrastive representation learning for nlp. the former focuses on the data side: how we can create contrastive data examples. this is useful not only for contrastive learning signals, but also for many other reasons such as evaluating model behaviors, augmenting data for low-resource training, producing contrastive explanation, promoting faithful text generation. the latter focuses on the learning algorithm side: how we can use contrastive learning broadly in different nlp tasks. here is the outline with an estimated schedule. part 1: foundations of contrastive learning (60 min) • contrastive learning objectives (15 min) • contrastive data sampling and augmentation strategies (15 min) • successful applications (15 min) • analysis of contrastive learning (15 min) part 2: contrastive learning for nlp (90 min) • contrastive learning in nlp tasks (30 min) • task-agnostics representation (15 min) • faithful text generation (15 min) • data-efficient learning (15 min) • interpretability and explainability (15 min) part 3: lessons learned, practical advice, and future directions (30 min) the following subsections give more details with reference papers for each part. ## reading list we compile the a light reading list for the audience learning before coming to the tutorial: our presenters come from 3 institutions based in the u.s. and china including 3 male and 1 female researchers on different levels of academic seniority. as contrastive learning can be applied broadly, our tutorial spans many different nlp tasks and domains covering text classification and sentence embeddings, information extraction, machine translation, question answering, summarization, text generation, few-shot learning, interpretability and explainability, commonsense knowledge and reasoning, vision-and-language, distillation and model compression. therefore, the audience will come from diverse backgrounds. rui zhang is an assistant professor in the computer science and engineering department of penn state university and a co-director of the psu nlp lab. he is one of the recipients of 2020 amazon research awards. he serves as an area chair at naacl 2021, emnlp 2021, and nlpcc 2021. he co-organizes the interactive and executable semantic parsing workshop at emnlp 2020 which attracted an international audience with 100+ researchers from diverse academic and demographic backgrounds. he has been working on contrastive learning for fewshot named entity recognition (das et al., 2021) and text generation (shu et al., 2021). https: //ryanzhumich.github.io/ yangfeng ji is the william wulf assistant professor in the department of computer science at the university of virginia, where he leads the natural language processing group. his research interests include building machine learning models for text understanding and generation. his work on entity-driven story generation won an outstanding paper award at naacl 2018. he is a co-author of an emnlp 2020 tutorial on the amazing world of neural language generation. https://yangfengji.net/ yue zhang is an associate professor at westlake university. his research interests include nlp and its underlying machine learning algorithms and downstream applications. he was the area chairs of acl (2017/18/19/20/21), col-ing (2014/18), naacl (2015/19/21), emnlp (2015/17/19/20), eacl (2021) and ijcai (2021). he won the best paper awards of ialp (2017), coling (2018) and best paper honorable mention of semeval (2020). he is the author of emnlp 2018 tutorial on joint models for nlp. https://frcchang.github.io/ rebecca j. passonneau is a professor in the computer science and engineering department of penn state university and a co-director of the psu nlp lab. her area of research is natural language processing, with a focus on semantics and pragmatics. her work is reported in over 130 journal and refereed conference publications. she won a best paper runner up at naacl 2010. she is a tutorial co-chair for naacl 2018. https: //sites.psu.edu/becky/ ## ethics statement as contrastive learning often involves data augmentation and manipulation, our ethical consideration mainly focuses on properly dealing with bias in the dataset. as bias and fairness created by contrastive learning algorithms are still under-explored, we will also discuss such relevant topics in the section on future directions.