A dapter H ub Playground: Simple and Flexible Few-Shot Learning with Adapters (2022)

Abstract: The open-access dissemination of pretrained language models through online repositories has led to a democratization of state-of-the-art natural language processing (NLP) research. This also allows people outside of NLP to use such models and adapt them to specific use-cases. However, a certain amount of technical proficiency is still required which is an entry barrier for users who want to apply these models to a certain task but lack the necessary knowledge or resources. In this work, we aim to overcome this gap by providing a tool which allows researchers to leverage pretrained models without writing a single line of code. Built upon the parameter-efficient adapter modules for transfer learning, our AdapterHub Playground provides an intuitive interface, allowing the usage of adapters for prediction, training and analysis of textual data for a variety of NLP tasks. We present the tool’s architecture and demonstrate its advantages with prototypical use-cases, where we show that predictive performance can easily be increased in a few-shot learning scenario. Finally, we evaluate its usability in a user study. We provide the code and a live interface at https://adapter-hub.github.io/playground .

## introduction the success of transformer-based pretrained language models @xcite was quickly followed by their dissemination, gaining popularity through open-access python libraries like huggingface @xcite , adapter-hub @xcite or sbert @xcite . researchers and practitioners with a background in computer science are able to download models and fine tune them to their needs. they can then upload their fine-tuned model and contribute to an open-access community of state-of-the-art (sota) language models for various tasks and in different languages. this has significantly contributed to the democratization of access to the latest nlp research as the individual implementation process has been simplified through the provision of easy-to-use and actively managed code packages. however, one still needs a certain level of technical proficiency to access these repositories, train models, and predict on new data. this is a limiting factor for researchers in disciplines who could benefit from applying sota nlp models in their field, but lack the technical ability. furthermore, there is growing interest for text classification models in interdisciplinary research @xcite , although often the methods are not sota in nlp. in this work, we hope to bridge this gap by providing an application which makes the power of pretrained language models available without writing a single line of code. inspired by the recent progress on parameter-efficient transfer learning @xcite , our application is based on adapters which introduce small and learnable task-specific layers into a pretrained language model. during training, only the newly introduced weights are updated, while the pre-trained parameters are frozen. adapters have been successfully applied in machine translation @xcite ; @xcite , cross-lingual transfer @xcite @xcite , community qa @xcite , task composition for transfer learning @xcite @xcite ) and text generation @xcite . adapters are additionally computationally more efficient @xcite and more robust to train @xcite . in our work, we build our application on top of the adapterhub @xcite ) library which stores task-specific adapters with a large variety of architectures and offers upload functionalities for community-developed adapter weights. we leverage this library to allow no-code access to pretrained adapters for a many text classification tasks using dynamic code generation. finally, our application enables the analysis of multi-dimensional annotations to further investigate model performance. our application supports both nlp and interdisciplinary researchers who want to evaluate the transferability of existing pretrained adapters to their specific domains and use cases. we intend this application for both zero-shot as well as few-shot scenarios where a user annotates a small number of data points and monitors model improvements. this is especially interesting for intermediate task training @xcite where models trained on a compatible task are utilized and fine-tuned on the target task. some efforts are being made to abstract away engineering requirements to use sota nlp @xcite , but their usage still requires certain technical skills. existing no-code (or automl) applications like akkio foot_1 , lobe foot_2 , or teachable ma-chines foot_3 allow users to upload data, annotate it using self-defined labels, and train a model for prediction. most approaches focus on vision tasks and follow commercial goals. to the best of our knowledge, we are the first to provide a non-commercial, no-code application for text classification. our application is transparent (i.e. details about usable pretrained adapters are traceable), and extendable via the community-supported adapterhub library. finally, we enable execution on third party computational servers for users without access to the required gpu hardware for efficient training and prediction @xcite , while also providing the necessary scripts to setup a self-hosted computing instance, mitigating technical dependencies. our contributions are: 1) the adapterhub playground application which enables no-code inference and training by utilizing pretrained adapters; 2) prototypical showcase scenarios from social sciences using our application for few-shot learning; ## adapterhub playground the adapterhub playground is a lightweight web application offering no-code usage of pretrained adapters from the adapterhub library. a user interface accompanied by dynamic code generation allows the utilization of adapters for inference and training of text classification tasks on novel data. below, we describe the application workflow foot_4 , provide details on the specific functionalities and highlight the technical architecture. ## few-shot scenario several prominent tasks in nlp such as sentiment analysis @xcite , stance detection @xcite or identifying semantically similar texts @xcite are of great interest in social science research @xcite @xcite . we therefore replicated two scenarios, namely sentiment analysis and semantic textual similarity. we envision a situation where a user has collected textual data (e.g. sentence-level) for a given task and wishes to perform analysis using a text classification pipeline. a labeled test set to evaluate the performance of the classifier, and further training data is available. ## usability study adapterhub playground is designed to be simple to use, requiring minimal training effort and technical knowledge. while we followed these principles throughout the conception and implementation of the application, we also evaluated the usability with users from our target group. therefore, we followed the approach by @xcite and let study participants conduct a series of tasks which were designed to reflect a use-case scenario as described in §3. afterwards, we used a questionnaire to capture their experiences. participants. we recruited study participants (@xmath0, the majority of whom were (post)graduate-level researchers at a university (two professors, two post-docs, six phds, one b.sc.). our data suggests that the participants have limited or no understanding of the technical computer science concepts but can envision themselves using the adapterhub playground (for details see appendix d). thus, our participants belong to one of the target groups we aim to aid with this application. procedure. the participants were provided a textual description of several tasks to be completed. 22 users were asked to complete a training and prediction action in a sentiment analysis scenario. we provided both labeled test data and unlabeled training data again using the dataset by @xcite . 23 after completing the tasks, we asked the participants to complete a questionnaire targeting their experience with the tool. ## conclusion and future work open-access dissemination of sota nlp models has accelerated their use and popularity, yet the required technical proficiency to apply them remains a limiting factor for their democratisation. to mitigate this, we introduced the adapterhub playground application which provides an easy-to-use web interface for no-code access to light-weight adapters for text classification tasks. our system is built on-top of the open-source adapterhub library and uses a dynamic code generation approach. we demonstrated the features of the application using two exemplary use-case scenarios and evaluated its usability in a user study with researchers from communication sciences. in addition to providing execution on third-party hardware, we also enabled a self-hosted computational instance. as future work, we plan to extend the application with dynamic user control over all hyperparameter specifications in the expert mode. to support users in efficient sampling of profitable training instances, we plan to investigate the integration of active learning methods @xcite . a running instance of our tool can be found under https://adapter-hub. github.io/playground .