C o G en: Learning from Feedback with Coupled Comprehension and Generation (2024)

Abstract: Systems with both language comprehension and generation capabilities can benefit from the tight connection between the two. This work studies coupling comprehension and generation with focus on continually learning from interaction with users. We propose techniques to tightly integrate the two capabilities for both learning and inference. We situate our studies in two-player reference games, and deploy various models for thousands of interactions with human users, while learning from interaction feedback signals. We show dramatic improvements in performance over time, with comprehension-generation coupling leading to performance improvements up to 26% in absolute terms and up to 17% higher accuracies compared to a non-coupled system. Our analysis also shows coupling has substantial qualitative impact on the system’s language, making it significantly more human-like.

## introduction language comprehension and generation are closely related processes. indeed, observations such as the ability to finish incomplete partner utterances in dialogue @xcite , as well as neuroscientific evidence @xcite @xcite have led to integrated accounts of comprehension and generation in cognitive science @xcite , where processes related to generation are active during comprehension and vice versa. this suggests a potential for coupling the two in computational systems, and creating a virtuous cycle, where the improvement of one capability drives learning and performance in the other. this is particularly compelling in systems that continually learn and improve through interaction with users, where the dynamics between the two capabilities play out over time. we study the dynamics of this coupling in a continual learning foot_0 setting, where trends in learning and behavior can be observed over time. we design an interaction scenario where models can take both listener (comprehension) and speaker (generation) roles, and receive feedback while interacting with human partners. we couple comprehension and generation through several mechanisms, and observe the impact this coupling has on the long-term dynamics of performance and language. we instantiate comprehension and generation as the listener and speaker roles of a two-player reference game @xcite ) involving abstract visual stimuli @xcite , which remain challenging for state-of-the-art vision-language models (figure 1 ). we deploy a single model that can take both roles. the process alternates between the model interacting with human partners, and training to improve both comprehension and generation capabilities based on feedback from the interactions. we couple comprehension and generation through two strategies: (a) at inference-time via a joint inference process that incorporates the opposing role, and (b) at training time by generating examples and rewards for each role from feedback on performance in that role as well as the opposing role. figure 2 illustrates the deployment and coupling mechanisms. the combination of these strategies creates a virtuous cycle that evolves over time, as the system continually trains and improves. as one capability improves (e.g., comprehension), the model's performance on the opposing capability (e.g., generation) also improves via the joint inference procedure. this, in turn, leads to better interactions, and the feedback the model receives changes as its capabilities advance and its failure modes change. the coupling of feedback signals via the training data qualitatively changes the training beyond a simple increase in the amount of data. whereas a generation system that trains on feedback is only exposed to its own language, the coupled system is continually exposed to a stream of new human language. this can enable the system to expand its generation abilities and make the language more similar to humans, beyond simply refining it to maximize interaction performance. we conduct extensive experiments, concurrently deploying a baseline and multiple model variants for thousands of interactions with human partners in a controlled study. our focus is to observe both performance and language trends over time. our coupled approach shows dramatic and fast performance gains, overall improving by 19.48% for comprehension and 26.07% for generation, in absolute terms. at the conclusion of our deployment, the coupled approach outperforms the non-coupled baseline by 14.80% for comprehension and 17.10% for generation. furthermore, coupling results in greater data efficiency, with the full system still outperforming this baseline with less than one-third the number of human interactions. we observe coupling dramatically influences the generated language, with the coupled approach exhibiting a larger effective vocabulary and greater alignment with human language according to both linguistic measures such as utterance length and automated metrics such as mauve @xcite . our code, data, and experiment logs are available at https://github.com/lil-lab/cogen . ## interaction scenario and overview we study the coupling of comprehension and generation by training and deploying an agent that interacts with human users, and continually learns from these interactions. this allows us to observe how the interplay between comprehension and generation evolves over time, and what the long-term effects of coupling the two processes are. interaction scenario we use a reference game as our interaction scenario (figure 1 ). each game involves two players: a speaker and listener. both participants are presented a set of abstract tangram images as context @xmath0, . . . , i n }. each participant observes the images in a different order. the speaker is given a target i @xmath1, and generates an utterance u, with the goal of allowing the listener to pick the target i t from the set of the images. the listener then makes a choice. an interaction succeeds if the listener picks the intended target. reference games have been extensively used in research, including in nlp (e.g., @xcite and cognitive science (e.g., @xcite @xcite , and provide a balance between complexity and research feasibility: (a) the interaction includes both generation and comprehension; (b) they are relatively accessible for crowdsourcing workers; (c) they are well scoped so learning is feasible without excessive data requirements; and (d) success is easy to measure. the tangram shapes we use have been shown to elicit rich linguistic behavior, both from listeners and speakers @xcite . they also remain challenging for contemporary models. our models' initial performance is at least 33.1% below human accuracy (section 6). deployment we deploy our model to interact with humans in rounds. each round includes a predetermined number of interactions, each with the model taking one of the roles (speaker or listener) and the human participant taking the other. we derive feedback signals from the interactions, construct training examples, and train our model. following training, we re-deploy for the next round. inference and learning we use idefics2-8b @xcite as our model, an autoregressive llm that can also take images as part of its input. the model is parameterized by θ. as a speaker (generation), the model computes a probability distribution p s (u|i, t; θ) over descriptions u of the target @xmath2, . . . , n } in the given context i. as a listener (comprehension), it computes a distribution p l (t|i, u; θ) over target t selections, given the context i and a description utterance u. both utterances and target selections are generated via a conventional auto-regressive process. following each deployment round, we train the model using all the feedback data collected so far by treating the feedback as rewards for contextual bandit learning. evaluation our main evaluation is conducted through interaction with humans, where each round forms the evaluation of the model so far. we evaluate the comprehension performance of the model from its target selection accuracy as a listener. we evaluate generation as the accuracy of the human listener in selecting the target given the model's generated description when in the speaker role. we also study the linguistic trends of the model's generations over time to better understand the dynamics created by coupling comprehension and generation. this includes analyzing its similarity to human language and its linguistic properties. ## continual learning we combine the continual learning approaches of @xcite (generation) and @xcite . both approaches map feedback to rewards, treat learning as a contextual bandit problem, and use @xcite , a relatively simple policy gradient algorithm. we adopt these design choices. a key difference of our process is that we combine the comprehension and generation objectives to train a single model. deployment and learning are interleaved. each round ρ starts with deploying the model parameterized by θ ρ to collect interactions with humans. we record feedback signals from these interactions. upon collecting a set of interactions, we re-train the model given all data collected so far to estimate new parameters θ ρ+1 . the model is then deployed for the next round, and the process continues. ## coupling comprehension and generation we couple comprehension and production during both learning and inference. we also use one model for both tasks, creating a coupling at the parameter level, which is common in contemporary methods, partially due to high memory needs. ## experimental setup game construction we construct reference game contexts using the kilogram dataset @xcite of 1,016 abstract tangram shapes. each context comprises 10 images drawn from this dataset. we use a clip model @xcite finetuned on kilogram annotations from @xcite to ensure visual similarity between images in each context and increase task difficulty. appendix b provides further details. ## results and analysis we focus on two broad questions: (a) does coupling influence the rate of improvement on task performance (section 6.1) and (b) does it lead to quantifiable differences in the generated language over time (section 6.2). overall, we find the answer to both questions is positive, with strong effects. ## related work our joint inference strategy (section 4.2) is technically based on approximations @xcite of the rational speech acts framework (rsa; @xcite , which frames pragmatic reasoning as a recursive process between listener and speaker models. rsa has been studied extensively with the focus of developing models that reason pragmatically (e.g., @xcite , including through incorporation in learn10 we cannot compute snd for human participants because of insufficient data per round. continually learning from interactions with human users has been studied in the context of instruction generation @xcite and following @xcite , question answering @xcite , and ad-hoc adaptation @xcite . in our work, continual learning enables us to study long-term dynamics that arise from coupling comprehension and generation. our continual learning setup is different from the reinforcement learning from human feedback framework (rlhf; ziegler et al., 2019) in relying on binary signals derived from interactions with users, while rlhf requires external annotators that compare output pairs. the reference game scenario has been extensively used in cognitive studies as a prototypical, but simple interaction design @xcite . it has been used to study convention formation at dyadic @xcite and populationlevels @xcite , and demonstrate com-putational theories of pragmatic reasoning @xcite , among other behaviors. it has also been used to develop computational methods, such as to evaluate contrastive captioning @xcite and abstract reasoning of visionlanguage models @xcite . the tangram images we use, abstract shapes composed of the same set of seven primitives, likewise have extensive use as stimuli in cognitive science @xcite @xcite . they also remain challenging for contemporary models @xcite , making them well suited to demonstrate model improvement. ## conclusion we study the dynamics of coupling language comprehension and generation at inference-and training-time through a continual learning setting where an agent learns from interactions with humans. coupling has significant impact over time, leading to improved agent performance, sample efficiency, and similarity to human language. our work points to multiple directions for future work, including coupling the processes through the training objective in addition to data at trainingtime, developing more efficient alternatives to sampling utterances during joint inference for generation, and the study of alternative interaction scenarios, including multi-turn settings where dynamics between comprehension and generation can affect an interaction throughout its duration. scaling up our approach and experimental setting to a realworld deployment featuring a wider range of tasks and a broader set of feedback signals, such as natural language feedback, constitutes a particularly important direction.