Offline Reinforcement Learning from Human Feedback in Real-World Sequence-to-Sequence Tasks (2021)

Abstract: Large volumes of interaction logs can be collected from NLP systems that are deployed in the real world. How can this wealth of information be leveraged? Using such interaction logs in an offline reinforcement learning (RL) setting is a promising approach. However, due to the nature of NLP tasks and the constraints of production systems, a series of challenges arise. We present a concise overview of these challenges and discuss possible solutions.

## introduction when natural language processing (nlp) systems are deployed in production, and interact with users ("the real world"), there are many potential ways of collecting feedback data or rich interaction logs. for example, one can ask for explicit user ratings @xcite , or collect user clicks @xcite , or elicit user revisions @xcite to get an estimate of how well the deployed system is doing. however, such user interaction logs are primarily used for an one-off assessment of the system, e.g., for spotting critical errors, detecting domain shifts, or identifying the most successful use cases of the system in production. this assessment can then be used to support the decision of keeping or replacing this system in production. from a machine learning perspective, using interaction logs only for evaluation purposes is a lost opportunity for offline reinforcement learning (rl). logs of user interactions are gold mines for off-policy learning, and they should be put to use, rather than being forgotten after a one-off evaluation purpose. to move towards the goal of using user interaction logs for learning, we will discuss which challenges have hindered rl from being employed in real-world interaction with users of nlp systems so far. concretely, our focus is on sequence-tosequence learning for nlp applications (see § 2 for an overview). for example, many machine translation services provide the option for users to give feedback on the quality of the translation, e.g., by collecting post-edits. similarly, industrial chatbots can easily collect vast amounts of interaction logs, which can be utilized with offline rl methods @xcite @xcite . in the following, we will thus present challenges that are encountered in userinteractive rl for nlp systems. with this discussion, we aim to (1) encourage nlp practitioners to leverage their interaction logs through offline rl, and (2) inspire rl researchers to steel their algorithms for the challenging applications in nlp. ## offline feedback for seq2seq in nlp in sequence-to-sequence (seq2seq) learning, the task is to map an input sequence @xmath0, x 2 , . . . , x |x| , ∀x @xmath1, y 2 , . . . , y |y| , ∀y @xmath2, where x , y denote the sets of input and output vocabularies, respectively. the conditional distribution of the output sequence given the input can be modeled with a policy π θ with learnable parameters θ. assuming a left-to-right generation order, the output sequence y is generated by conditioning on previous output elements @xmath3: mapping the sequence-to-sequence problem formulation to nlp tasks, we have for example: the most distinctive feature of seq2seq nlp tasks for rl are the extremely large, structured output spaces: given the output vocabulary of size |y| and a maximum sequence length m , there are |y| m possible combinations of output sequences. for instance, in machine translation there might be as many as 30 000 output tokens in the vocabulary and the output sequence length could easily be 100, leading to a total of 30 000 100 possible outputs. a successful policy identifies the few combination of tokens that form valid output sequences. in the most extreme case only one output sequence exists that will be correct. , e.g., in a semantic parsing setup, where potentially only one specific sql query will return the correct answer when executed. to train a policy, supervised data can be used. there we assume a given dataset d @xmath4, y t )} t @xmath5, aiming to maximize the model score for the given reference output. in practice, it may be too expensive to collect correct, i.e., supervised, output sequences, since they require skilled annotators, e.g., trained translators for a machine translation task. therefore, one option is to pre-train the policy on some available supervised data, which will allow the model to concentrate on reasonable areas in the output space @xcite . the model can then be used to produce potentially imperfect output sequences and humans can judge an output ỹ and a reward δ @xmath6, 1] is assigned. model parameters may be optimized by pairing the model outputs with their reward estimates. depending on the use case, quality judgments may also exist for single elements in the structure, adding δ (t,j) for every step in the output sequence. the core idea is that the weighting by δ enables learning from imperfect outputs while respecting their faults. in rl, these quality assessments are used to reward desirable model actions, here desirable sequence outputs. when collecting quality judgments from human users in production systems, it would be risky to directly update the model online according to their feedback. 1 some user feedback might be adversarial, inappropriate, or not representative when used for training without prior treatment @xcite @xcite . 2 furthermore, interpreting feedback wrongly (e.g., through incorrect credit assignment @xcite ), or receiving misleading feedback @xcite , could easily push the policy into less favorable conditions. because updating systems online is too risky, quality judgments are instead stored in interaction logs, i.e., d @xmath7, ỹt , δ t )} t @xmath8, and the system is updated offline. as a result, the imperfect output sequences are produced by a possibly different policy, the logging policy µ, and updates to our learning policy are conducted offline, which is a classic off-policy rl scenario. due to the logging setup, the collected dataset is biased towards the choices of the deployed model, the logging policy µ. this results in a counterfactual learning scenario @xcite . the bias may be corrected via importance sampling. if the logging policy is known and µ(ŷ | x) is logged as well, the policy can then be optimized for the inverse propensity scoring (ips) objective @xcite : ## conclusion there is large potential in nlp to leverage user interaction logs for system improvement. we discussed how algorithms for offline rl can offer promising solutions for this learning problem. however, specific challenges in offline rl arise due to the particular nature of nlp systems that collect human feedback in real-world applications. we presented cases where such challenges have been found and offered solutions that have helped. so far, the solutions have mainly been explored in the context of machine translation and semantic parsing. in the future, it will be interesting to explore further tasks and additional real-world use cases to find out how to best learn from human feedback.