LMT urk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Framework (2022)

Abstract: Vast efforts have been devoted to creating high-performance few-shot learners, i.e., large-scale pretrained language models (PLMs) that perform well with little downstream task training data. Training PLMs has incurred significant cost, but utilizing the few-shot learners is still challenging due to their enormous size. This work focuses on a crucial question: How to make effective use of these few-shot learners? We propose LMTurk, a novel approach that treats few-shotlearners as crowdsourcing workers. The rationale is that crowdsourcing workers are in fact few-shot learners: They are shown a few illustrative examples to learn about a task and then start annotating. LMTurk employs few-shot learners built upon PLMs as workers. We show that the resulting annotations can be utilized to train models that solve the task well and are small enough to be deployable in practical scenarios. Active learning is integrated into LMTurk to reduce the amount of queries made to PLMs, minimizing the computational cost of running PLM inference passes. Altogether, LMTurk is an important step towards making effective use of current PLMs.

## introduction equipped with prolific linguistic features @xcite @xcite and rich world knowledge @xcite @xcite , large-scale pretrained language models (plms) have been shown to be versatile: they are now basic building blocks @xcite of systems solving diverse nlp tasks in many languages @xcite @xcite @xcite @xcite . recent work shows that plms are effective few-shot learners @xcite @xcite 1 resources are available at: github.com/lmturk ## conclusion in this work, our focus is the research question: how to make effective use of current few-shot learners? we propose lmturk, a simple yet effective method that considers plm-based few-shot learners as non-expert annotators in crowdsourcing; active learning strategies are incorporated to reduce the cost of annotation. we further show that processing the annotations from lmturkers can be beneficial. future work may combine lmturker annotations with human annotators in a human-in-theloop setup (monarch, 2021) to increase the overall utility of invested resources @xcite . scaling up to even larger plms likely to further boost model performances @xcite applying lmturk to multilingual fewshot learners @xcite @xcite is also promising.