Yes, this Way! Learning to Ground Referring Expressions into Actions with Intra-episodic Feedback from Supportive Teachers (2023)

Abstract: The ability to pick up on language signals in an ongoing interaction is crucial for future machine learning models to collaborate and interact with humans naturally. In this paper, we present an initial study that evaluates intra-episodic feedback given in a collaborative setting. We use a referential language game as a controllable example of a task-oriented collaborative joint activity. A teacher utters a referring expression generated by a well-known symbolic algorithm (the “Incremental Algorithm”) as an initial instruction and then monitors the follower’s actions to possibly intervene with intra-episodic feedback (which does not explicitly have to be requested). We frame this task as a reinforcement learning problem with sparse rewards and learn a follower policy for a heuristic teacher. Our results show that intra-episodic feedback allows the follower to generalize on aspects of scene complexity and performs better than providing only the initial statement.

## introduction the communicative acts of humans in collaborative situations can be described as two parts of a joint act: signalling and recognizing. in such joint activities, these signals work as coordination devices to increment on the current common ground of the participants @xcite . the ability to act on these language signals is crucial for future machine learning models to naturally collaborate and interact with humans @xcite . such a collaborative interaction with humans usually happens fluently, where one communicative act is performed after the other. the framework of reinforcement learning (rl) @xcite describes such mechanics where an agent is exposed in steps to observations of an environment with dynamic factors such as the position of objects or language expressions. the goal is that the agent learns to behave generally well in figure 1 : an exemplary interaction between a teacher and a follower that controls the gripper (the grey square). after an initial referring expression l re at t 0 , the teacher provides feedback l fbt based on the follower's actions until the correct piece is selected at time step t . a key challenge here is the variability of expressions in language that can be said to the agent during an interaction. even in relatively simple environments, there might arise an overwhelming amount of situations for an agent to handle @xcite . recent work on collaborative agents focuses on large precollected datasets for imitation learning to learn agents in complex simulated visual environments @xcite @xcite or frames the learning as a contextual bandit problem @xcite . nevertheless, other work has shown that intermediate language inputs are a valuable signal to improve the agent's learning performance in task-oriented visual environments @xcite . in this paper, we present an initial study that evaluates a follower's learning success given a teacher's intra-episodic feedback in a collaborative setting. we use a referential language game (in english) as a controllable example of a task-oriented collaborative joint activity (see figure 1 ). in this game one player (the follower) is supposed to select a piece based on the another player's directives (the teacher). we assume a teacher that utters referring expressions as initial instructions and then responds to the follower's actions with intra-episodic feedback. we frame this as a rl problem with sparse rewards where the intermediate feedback is not part of the reward function but its potential usefulness is learnt by the follower alone. foot_0 2 related work vision and language navigation. in vision and language navigation, an agent is given a natural language instruction which is to be understood to navigate to the correct goal location in a visually observed environment @xcite . the follower can usually ask an oracle for further information, if necessary @xcite @xcite . we extend on this idea and aim for an ongoing interaction with corrections that loosens the turn-based paradigm by letting the oracle choose when to speak as part of the environment. hence, in our reference game, the language back-channel for the follower is cut, so that we force the follower to rely more on the visual observations for task success. continual learning from human feedback. @xcite let humans instruct the follower and then ask them to rate the agent's behaviour (thumbs up or down). this binary feedback is used for further training as the reward signal in a contextual bandit framework. they show that the agent improves over several interactions with humans. similarly we evaluate the learning process in the context of rl because it imposes "weaker constraints on the regularity of the solution" @xcite , but take a broadly available, off-theshelf learning algorithm @xcite to directly study the effects of different kinds of feedback. the feedback given to our agent is of natural language and not directly bound to the re-ward; the follower needs to learn the meaning of the language feedback itself. language-guided policy learning. chevalier-boisvert et al. (2019) compared the sampling complexity of rl and imitation learning (il) agents on various language-conditioned tasks. they proposed a 2-dimensional visual environment called minigrid in which an agent is given a single mission statement that instructs the agent to achieve a specific state, e.g. "take the red ball". in contrast to them we intentionally do not use il approaches, because then the agent would have already learnt how to ground the language signals. we want to test if the agent can pick-up on the language from the interaction alone. for this, we similarly propose a diagnostic environment to directly control for the distributions of target objects (cf. skewed distribution of target objects in cvdn @xcite ) and feedback signals. other work uses the minigrid environment to propose a meta-training approach that improves the learning via natural language corrections, e.g. "pick up the green ball" @xcite . the agent is given an episodic correction if a specific task cannot be solved. in this way, the agent must not only ground the mission statement but also ground the corrections into actions. @xcite improve policy learning with intra-episodic natural language sub-goals e.g. "pick up the ball". these sub-goals are provided by a trained teacher policy when a previous sub-goal has been reached. in contrast, we rather follow earlier work @xcite on monitoring execution and use a heuristic teacher which provides intraepisodic language feedback whenever it appears feasible. the agent has to learn that certain pairs of feedback and behaviour at a specific time-step lead to the task's success and others to failure. ## the cogrip environment we use a collaborative game of referential and interactive language with pentomino pieces as a controllable setting. a teacher instructs a follower to select a specific piece using a gripper. both are constrained as follows: the teacher can provide utterances but cannot move the gripper. the follower can move the gripper but is not allowed to provide an utterance. this asymmetry in knowledge and skill forces them to work together and coordinate. @xcite found that this settings leads to diverse language use on the teacher's side. ## experiments in this section we explore the effects of the teacher's language and intra-episodic feedback on the follower's success and ask whether the follower generalizes on aspects of scene complexity. as suggested by madureira and schlangen (2020) we explore the question of which language is most effective. the ia constructs the initial reference by following a preference order over object properties @xcite . we hypothesize that a particular order might be more or less suitable depending on the task. thus we conduct a series of experiments without the feedback signal where the preference order is varied as the permutation of color, shape and position. our results indicate that such orders perform better that prioritize to mention positional attributes as distinguishing factors of the target piece (see ## conclusion in this work, we studied the effects of a teacher's language and intermediate interventions (the feedback) towards a learner's success and whether the learner generalizes on aspects of scene complexity. our results show that there is a most beneficial language for the teacher. its intra-episodic feedback allows the learner to learn faster and generalize better than without intermediate help. an exciting direction for further work is to show the benefits of language feedback for other reinforcement learning problems, to overcome the limits of the heuristic teacher strategy and to reduce the need for feedback after successful training.