E mpowering A ctive L earning to J ointly O ptimize S ystem and U ser D emands (2020)

Abstract: Existing approaches to active learning maximize the system performance by sampling unlabeled instances for annotation that yield the most efficient training. However, when active learning is integrated with an end-user application, this can lead to frustration for participating users, as they spend time labeling instances that they would not otherwise be interested in reading. In this paper, we propose a new active learning approach that jointly optimizes the seemingly counteracting objectives of the active learning system (training efficiently) and the user (receiving useful instances). We study our approach in an educational application, which particularly benefits from this technique as the system needs to rapidly learn to predict the appropriateness of an exercise to a particular user, while the users should receive only exercises that match their skills. We evaluate multiple learning strategies and user types with data from real users and find that our joint approach better satisfies both objectives when alternative methods lead to many unsuitable exercises for end users.

## introduction state-of-the-art machine learning approaches require huge amounts of training data. but for many nlp applications, there is little to no training data available. interactive nlp systems are a viable solution to alleviate the cost of creating large training datasets before a new application can be used. such systems start with no or few labeled instances and acquire additional training data based on user feedback for their predictions. active learning @xcite is a frequently used technique to quickly maximize the prediction performance, as the system acquires user feedback in each iteration for 1 our code and simulated learner models are available on github: https://github.com/ukplab/ acl2020-empowering-active-learning those instances that likely yield the highest performance improvement (e.g., because the system is yet uncertain about them). active learning has been shown to reduce the amount of user feedback required while improving system performance for interactive nlp systems (p.v.s and meyer, 2017; @xcite and to reduce the annotation costs in crowdsourcing scenarios @xcite . however, outside the typical annotation setup, it can be boring or frustrating for users to provide feedback on ill-predicted instances that hardly solve their needs. consider a newly launched web application for learning a foreign language, which aims at suggesting exercises that match the user's proficiency according to vygotsky's zone of proximal development @xcite . the underlying machine learning system starts without any data, but employs active learning to select an exercise the system cannot confidently predict. then, it adjusts its model interactively based on the user's feedback. while the system is still uncertain, the users often receive inappropriate (e.g., too hard or too easy) exercises. thus, they get the impression that the system does not work properly, which is especially harmful during the inception phase of an application, as the community opinion largely defines its success. in this paper, we distinguish the system objective of maximizing the prediction performance with minimal labeled instances and the user objective of providing useful instances for the user's current needs. for the first time, we propose an active learning approach that jointly optimizes these seemingly counteracting objectives and thus trades off the demands of system and user. the users of educational applications can particularly benefit from this, as they can learn most if they receive appropriate learning material while the underlying system requires considerable training to reach acceptable performance. we employ our new approach in a language learning platform for c-tests (i.e., cloze tests, in which the second half of every second word is replaced by a gap). our system successfully learns how to predict the difficulty of a c-test gap (system objective) and how to provide a c-test that is neither too easy for the current user, which would cause boredom, nor too hard, which would create frustration (user objective). predicting the difficulty of an exercise and correspondingly selecting exercises that match a user's proficiency are important steps towards selfdirected language learning and massive open online courses (moocs) on language learning. though we focus on this educational use case in this paper, our approach may also yield new insights for other problems that suffer from seemingly counteracting system and user objectives, for example, interactively trained recommender systems for books, movies, or restaurants. adaptive learning. many systems provide user adaptation, and research has shifted from predefined sets of rules for adaptation to data-driven approaches. several works investigate adaptive methods to provide exercises which are neither too hard nor too boring. for instance, @xcite model learning in a game-theoretic sense where the goal is to adjust the difficulty to neither being too easy nor too hard. other works investigate adaptation in the context of testing @xcite @xcite and propose methods for an adaptive selection of appropriate tests for better assessing a student's proficiency. in a large survey, truong (2016) discusses how to integrate different learning styles, modeling categorical student behavior, into an adaptive learning environment and emphasizes the need for more sophisticated methods. despite much research in adaptive and active learning, none of the previous works consider jointly modeling and optimizing both the system and user objectives which may retain a user's motivation and keep them from leaving the platform due to boredom or frustration. ## approach figure 1 shows our proposed interactive learning setup. the active learning component iteratively samples instances from a pool of unlabeled data and asks the user for a label that can be used to train the machine learning system. previous work on active learning focused on optimizing the system objective (blue). that is, only the system provides feedback to the active learning component (e.g., how certain it is about the predicted label of an instance). in our work, we first model the user objective (green) and propose sampling strategies that maximize the user satisfaction based on the user's feedback (e.g., the user's label for an instance). finally, we study our novel joint optimization strategies (gold) that trade off the demands of the system and the users. whereas we distinguish between the user's feedback (exercise-level) and labeled instances (gap-level) in our work, our proposed approach can easily be adapted to more specific cases where the (implicit) user feedback and the provided label are the same. 3 in the remainder of this section, we introduce sampling strategies that select which instance should be presented to the user next. we use the following notation: let x be the pool of unlabeled instances. in every iteration of the application (e.g., when a user requests a new exercise), the sampling strategy s(v) returns an instance @xmath0. the user then provides a label y for instance x, potentially with additional feedback on the user's satisfaction. the active learning component finally removes x from its pool x and adds (x, y) to the set of labeled instances, before the system is retrained with the increased labeled training set. the simplest sampling strategy that we use as a baseline is random sampling s rand (v), which selects an @xmath1, regardless of the user. in the following subsections, we discuss more advanced strategies that optimize the system or user objective as well as our new joint optimization strategies. ## instantiation we consider our jointly optimized active learning particularly beneficial for educational applications, since (1) the users of such a system may fail to achieve their learning goals with inappropriate exercises. additionally, (2) it is difficult to acquire large difficulty-annotated datasets for training, as actual users are required for producing realistic training data and existing learner datasets can hardly be shared due to privacy concerns. we therefore instantiate our approach for a language learning platform that predicts the difficulty of exercises and learns to provide appropriate (neither too easy nor too hard) exercises to its users. ## conclusion in this work, we investigated how we can incorporate user feedback into existing active learning approaches without hurting the user's actual needs. we formalize both system (active learning) and user objectives and propose two novel sampling strategies which aim to maximize both objectives jointly. we evaluate our sampling strategies for the task of selecting suited c-tests, a type of fill-the-gap exercise, which fit the current proficiency of a human learner. we create simulated learners for five different proficiency levels from real-world data and use them to define different learning behaviors. our experiments show that both our novel sampling strategies are successfully selecting instances which lead to a better model training while not hurting a learner's progress by selecting too easy or too difficult c-tests. although system and user objective at first seem counteracting, our experiments indicate that they complement each other as jointly optimizing them outperforms optimizing only one of the goals. additional experiments with an adaptive λ for our trade-off sampling strategy show that properly balancing system and user objective can lead to considerable improvements in performance for both objectives. our findings open up new opportunities for training models on low-resource scenarios with implicitly collected user feedback while jointly serving the user's actual needs. additional use cases like the training of personalized recommendation models as well as the use of reinforcement learning to find a good trade-off between system and user objective remain to be investigated in future work.