Indirectly Supervised Natural Language Processing (2023)

Abstract: This tutorial targets researchers and practitioners who are interested in ML technologies for NLP from indirect supervision. In particular, we will present a diverse thread of indirect supervision studies that try to answer the following questions: (i) when and how can we provide supervision for a target task T, if all we have is data that corresponds to a “related” task T′? (ii) humans do not use exhaustive supervision; they rely on occasional feedback, and learn from incidental signals from various sources; how can we effectively incorporate such supervision in machine learning? (iii) how can we leverage multi-modal supervision to help NLP? To the end, we will discuss several lines of research that address those challenges, including (i) indirect supervision from T ′ that handles T with outputs spanning from a moderate size to an open space, (ii) the use of sparsely occurring and incidental signals, such as partial labels, noisy labels, knowledge-based constraints, and cross-domain or cross-task annotations—all having statistical associations with the task, (iii) principled ways to measure and understand why these incidental signals can contribute to our target tasks, and (iv) indirect supervision from vision-language signals. We will conclude the tutorial by outlining directions for further investigation.

## introduction conventional approaches to nlp rely on taskspecific labeled examples of a large volume. this does not apply to scenarios where tasks may be too complicated or costly to annotate, or the system is required to handle a new task immediately. many people increasingly perceive that pretrained language models (plms) use self-supervision, and therefore there is no need for supervision anymore. while this is probably true for encoder-only models (e.g., bert @xcite ), this does not hold for decoder models, where people nowadays use vast amounts of supervision and reinforcement learning signals. therefore, it is still desirable to gather supervision that has already existed in related tasks or is pretty cheap, which is termed "indirect supervision" in this tutorial. recently, there have been increasing works that study indirect supervision for a wide range of nlp tasks. for example, @xcite and @xcite respectively leveraged the rich annotation of a source task (natural language inference or summarization) to address the poorly-annotated target tasks. to make better use of the natural texts, some literature @xcite @xcite proposed to explore incidental supervision, e.g., phonetic similarity and similar temporal distribution for named entity transliteration, to help downstream tasks. that sort of incidental supervision is often weak signals that exist in the data and the environment independently of the tasks at hand, and is hard to be encoded by plms. furthermore, when accessing supervision from pure text is challenging, researchers turned to other modalities for indirect supervision @xcite . this tutorial presents a comprehensive introduction of those lines of frontier research on indirectly supervised nlp. in particular, it tries to answer the following questions: (i) which source task is easier to be adapted to solve various target tasks and any constraints there? (ii) what are the limitations of pretrained language models in discovering supervision from natural texts, and how can we alleviate them with incidental signals? (iii) are there any theoretical measures that can indicate the benefits of the incidental signals to a given downstream task? (iv) how to mitigate the gap between different modalities if we want to utilize image/video knowledge to guide nlp? by addressing those critical questions, we believe it is necessary to present a timely tutorial to comprehensively summarize the new frontiers in indirectly supervised nlp research and point out the emerging challenges that deserve further investigation. participants will learn about recent trends and emerging challenges in this topic, representative tools and learning resources to obtain ready-to-use models, and how related technologies benefit end-user nlp applications. ## outline of tutorial content this half-day tutorial presents a systematic overview of recent advancements in indirect supervision methods for nlp. the detailed contents are outlined below. ## specification of the tutorial the proposed tutorial is considered a cutting-edge tutorial that introduces new frontiers in indirectly supervised nlp. the presented topic has not been covered by any * cl tutorials in the past 4 years.