Alfred: A System for Prompted Weak Supervision (2023)

Abstract: Alfred is the first system for programmatic weak supervision (PWS) that creates training data for machine learning by prompting. In contrast to typical PWS systems where weak supervision sources are programs coded by experts, Alfred enables users to encode their subject matter expertise via natural language prompts for language and vision-language models. Alfred provides a simple Python interface for the key steps of this emerging paradigm, with a high-throughput backend for large-scale data labeling. Users can quickly create, evaluate, and refine their prompt-based weak supervision sources; map the results to weak labels; and resolve their disagreements with a label model. Alfred enables a seamless local development experience backed by models served from self-managed computing clusters. It automatically optimizes the execution of prompts with optimized batching mechanisms. We find that this optimization improves query throughput by 2.9x versus a naive approach. We present two example use cases demonstrating Alfred on YouTube comment spam detection and pet breeds classification. Alfred is open source, available at https://github.com/BatsResearch/alfred .

## introduction acquiring labeled data is a significant challenge for machine learning for its time-consuming and expensive nature. programmatic weak supervision (pws) provides a more efficient method of data annotation by using noisy heuristics to label data. in a typical pws setup, domain experts design labeling functions (lfs) as programs that vote for a label or abstain. @xcite recently, there has been a growing interest in creating lfs from large, pre-trained models through prompting @xcite @xcite . in the shift to this new setting, executing lfs goes from the least to the most computationally expensive part of the process, highlighting the importance of providing a software infrastructure that facilitates efficient development. however, existing toolkits for large language models mainly prioritize prompt templating and tuning, leaving an unmet need for a system that that connects prompting with the creation of training data. prompted models offer a unique opportunity to enhance existing pws systems. traditional pws systems require programming lfs with code that specifies heuristic domain knowledge. with large pre-trained models, natural language-based prompts can be used as lfs, also known as prompted lfs @xcite . this approach allows the easy expression of complex rules that were previously difficult to specify using code, as the example in figure 1 shows. the ability to use prompts to define labeling functions simplifies and streamlines the weak supervision process, as well as potentially elevating the quality of the annotations. this benefit is particularly helpful for tasks involving computer vision, where previously pws has been limited to tasks for which models can identify key features (such as objects) over which to program rules. whether the domain is language or multi-modal, prompts let users experiment with different heuristics (and phrasings of those heuristics). therefore, enabling an iterative development experience is essential for the success of a promptbased pws system. the switch to prompted models for weak supervision also presents significant challenges. it first requires rethinking the abstractions and workflow of first-generation pws systems. instead of editing code and managing libraries of functions, users must manage libraries of prompts, track their outputs on multiple datasets, and develop strategies for mapping those outputs to labels. this change is complicated by large models' demand for computational resources. the throughput of the models is a new development bottleneck. the extra overhead of remotely hosted models is a further hindrance to def biden_mention(instance): if re.match( r"(president\s)?(jo(seph)|(e)\s)?(biden)/i", instance): return politics else: return abstention "text: [[instance]] does this text mention president biden?" "yes" ➡ politics "no" ➡ abstention labeling function prompted labeling function def positive_sentiment(instance): for token in tokenize(instance): if token in positive_words: return positive return abstention "text: [[instance]] does this comment express positive sentiment?" "yes" ➡ positive "no" ➡ abstention def stripe_detect(instance): if stripe_detector.infer(instance) >= 0.5: return [tiger, zebra] else: return abstention ] is replaced by the news article to be labeled. the response is mapped to a vote on the true label. the second example demonstrates how heuristics about positive sentiment that were previously hard to define can be flexibly expressed as a natural language question. instead of defining a set of keywords for fuzzy sentiment matching, we can simply ask large pretrained models for answers about the sentiment. for the third example, we consider an animal labeling task where we use the visual attributes "stripes" to vote for the classes tiger and zebra. previously, we would need to first collect supervised training data for attributes like stripes and then train classifiers to make the decisions. with modern vision-language models (e.g. clip), we can simply express the attribute detection task as a set of candidate prompts. existing open-source software for prompting concentrates on prompt engineering @xcite , prompt chains @xcite or continuous (i.e., soft) prompt tuning @xcite ; placing less emphasis on throughput for a large-model-in-the-loop workflow. additionally, many existing open-source systems have not developed support for vision-language models, despite their benefits for data labeling. incorporating large pre-trained models into a pws system remains an open problem in the open-source software space, requiring innovative solutions to address these challenges and complement existing software focused on other aspects of prompting. we present alfred, a versatile pws system that leverages large pre-trained models for image and text annotations. alfred aims to provide an environment for the rapid development of prompt-based supervision, while maintaining a consistent development experience similar to established pws systems. we designed alfred with usability and efficiency in mind, aiming to provide a rapid and smooth experience for developing prompt-based supervision. alfred supports popular large language models from hugging face's transformer package @xcite , including the gpt family @xcite , the t5 family @xcite , etc., and vision-language models like clip @xcite , etc. alfred also supports local onnx models, or api-based models from ope-nai, ai21, and cohere. moreover, alfred provides easy templating tools to help users quickly create, evaluate, and refine prompted lfs. alfred offers easy ways to deploy inference servers remotely, in addition to local model hosting. alfred also optimizes model inference throughput with improved batching techniques and provides utilities for efficient llm deployment and interaction. finally, alfred contains a library of label models to distill the outputs of prompted labeling functions into the final training datasets for downstream end models. alfred is a prototype for a second generation of pws systems with prompting at their core. to this end, we highlight three key feature of alfred: ## related work and background alfred sits at the intersection of programmatic weak supervision and large pretrained models. in this section, we overview the most related work. programmatic weak supervision. traditionally, supervised learning relied on manually labeled data, and data labeling has been seen as a key bottleneck for many applications. recently, a family of programmatic weak supervision (pws) methods have offered an alternative to costly manual annotations by incorporating multiple sources of noisy labels to create training datasets @xcite ). prompting for pre-trained models. with the emergence of large pre-trained langugae and visionlanguage models, prompting has become a popular approach to many few-shot and zero-shot tasks @xcite @xcite . prompting can create training examples, generate data, modify datasets, and improve model reasoning @xcite @xcite @xcite @xcite . this presents a unique opportunity for combining prompting for large pretrained models and weak supervision. recent studies have investigated strategies to combine large language models into weak supervision frameworks @xcite @xcite . alfred aims to provide a platform for the rapid development of weak supervision applications that rely on large pre-trained language and vision-language models, as well as enable experimentation with new ways of using those models. ## prompted lf development alfred is designed to enable the development and application of prompted labeling functions (lfs) compared to the typical workflow of pws systems, where developing lfs is not computationally demanding, developing prompted lfs has model inference as a bottleneck. these large models are often hosted remotely on virtual instances or computing clusters, which can add to the challenge of iterative prompt development. in an iterative development environment, creating prompted labeling functions requires a platform that provides optimal throughput and low latency for a rapid local development experience. to illustrate alfred's key focuses, we illustrate a typical workflow (figure 2 ) for using alfred to create a training dataset: step 1: task setup for a large model to be used in the development loop, developers can elect to use either self-hosted models or api-based models. for self-hosted models, alfred provides an alfredserver to host the model on cloud or cluster nodes. as the main development interface, the user can simply start a client by specifying the type of model. before creating prompted lfs, users need to familiarize themselves with the task by exploring the raw, unlabeled dataset. if there is no development subset available, the developer can annotate a small portion of the data as a held-out evaluation benchmark. alfred implements a dataset class based on apache arrow for fast data access. user may load a dataset from csv, json or dataframe objects. it also offers direct support for datastes from the hugging face 'dataset' package @xcite . during the exploration process, developers may gain insights into the data and the label space, and identify potentially useful heuristics. moreover, users can freely experiment with prompts with a few unlabeled instances by directly interacting with the client . ## system design in this section, we describe and highlight the key design decisions for alfred. ## example use cases in this section, we present two example use cases for how alfred can be used to create training data for specific machine learning tasks using natural language prompts in both text and image domains. we measure the labeling quality by taking the top-1 accuracy of the estimated probabilistic labels given by the label model. the notebooks to reproduce these examples are in the alfred repository. ## discussion and future work this paper introduces alfred, a prototype for the next generation of programmatic weak supervision systems that leverage the potential of large pretrained models. alfred complements the existing ecosystem of large-pretrained-model toolkits, offering optimized inference throughput, a smooth local development experience, and compatibility with vision-language models to support image annotation tasks. alfred represents a notable advancement in the domain of programmatic weak supervision, as it enables users to express their domain-specific knowledge and heuristics with flexible natural language prompts for language and vision-language models. this approach can be more user-friendly than conventional pws systems, which requires expert programming of weak supervision sources. our objective is for alfred to serve as the infrastructure and experimentation platform for many future weak supervision research projects and applications. furthermore, we plan to extend alfred's capabilities to accommodate a wider range of multimodal large pre-trained models, such as whisper @xcite and layoutlms @xcite .