Symbol Grounding and Task Learning from Imperfect Corrections (2021)

Abstract: This paper describes a method for learning from a teacher’s potentially unreliable corrective feedback in an interactive task learning setting. The graphical model uses discourse coherence to jointly learn symbol grounding, domain concepts and valid plans. Our experiments show that the agent learns its domain-level task in spite of the teacher’s mistakes.

## introduction interactive task learning (itl) aims to develop agents that can learn arbitrary new tasks through a combination of their own actions in the environment and an ongoing interaction with a teacher (see @xcite for a survey). because the agent continues to learn after deployment, itl allows an agent to learn in an ever changing environment in a natural manner. one goal of itl is to have the interactions be as natural as possible for a human teacher, and many different modes of interaction have been studied: non-verbal through demonstration or teleoperation @xcite , or natural language: an embodied extended dialogue between teacher and agent, like between a teacher and apprentice. our interest lies in natural language interactions where teachers can provide instructions @xcite , describe current states @xcite and define concepts @xcite , goals @xcite , and actions @xcite , while the agent asks clarifying questions @xcite and executes instructed commands. teachers can also use corrective feedback @xcite . these approaches all assume that the teacher offers information that is both correct and timely. however, humans are error prone, and so in this paper we study how agents can learn successfully from corrective feedback even when the teacher makes mistakes. appelgren and lascarides' model exploits discourse coherence @xcite @xcite : that is, constraints on how a current move relates to its context. but their models assume that the teacher follows perfectly a specific dialogue strategy: she corrects a mistake as and when the agent makes it. however, humans may fail to perceive mistakes when they occur,. they also may, as a result, utter a correction much later than when the agent made the mistake, and thanks to the teacher being confident, but wrong, about the agent's capacity to ground nl descriptions to their referents, the agent may miscalculate which salient part of the context the teacher is correcting. in this paper, we present and evaluate an itl model that copes with such errors. in §2, we use prior work to motivate the task we tackle, as described in §3. we present our itl model in §4 and §5, focusing on coping with situations where the teacher makes mistakes of the type we just described. we show in §6 that by making the model separate the appearance that the teacher's utterance coherently connects to the agent's latest action with the chance that it is not so connected, our agent can still learn its domain-level task effectively. ## background interactive task learning (itl) exploits interaction to support autonomous decision making during planning @xcite . similar to @xcite , our aim is to provide the agent with information about goals, actions, and concepts that allow it to construct a formal representation of the decision problem, which can thereafter be solved with standard decision making algorithms. rather than teaching a specific sequence of actions (as in e.g., @xcite ; @xcite ), the teacher provides the infor-mation needed to infer a valid plan for a goal in a range of specific situations. in this work we focus on learning goals, which express constraints on a final state. the agent learns these goals by receiving corrective dialogue moves that highlight an aspect of the goal which the agent has violated @xcite . natural language (nl) can make itl more data efficient than non-verbal demonstration alone: even simple yes/no feedback can be used to learn a reward function @xcite or to trigger specific algorithms for improving behaviour @xcite . more extended nl phrases must map to semantic representations or logical forms that support inference @xcite . like prior itl systems, @xcite we assume our agent can analyse sentential syntax, restricting the possible logical forms to a finite set. but disambiguated syntax does not resolve semantic scope ambiguities or lexical senses @xcite , and so the agent must use context to identify which logical form matches the speaker's intended meaning. recovering from misunderstandings has been addressed in dialogue systems (eg, @xcite , and itl systems cope with incorrect estimates of denotations of nl descriptions (eg, @xcite . here, we address new sources of misunderstanding that stem quite naturally from the teacher attempting, but failing, to abide by a particular dialogue strategy: ie, to correct the agent's mistakes as and when they're made. this can lead to the learner misinterpreting the teacher's silence (silence might not mean the latest action was correct) or misinterpreting which action is being corrected (it might be an earlier action than the agent's latest one). we propose a model that copes with this uncertainty. ## task in our task an agent must build towers in a blocks world. the agent begins knowing two pddl action descriptions: put(x, y) for putting an object x on another y; and unstack(x, y) for removing an object x from another object y and placing x back on the table. further, it knows the initial state consists of 10 individual blocks that are clear (i.e., nothing on them) and on the table, and that the goal g contains the fact that all the 10 blocks must be in a tower. figure 1 : the colours of objects fit into different colour terms. each individual hue is generated from a gaussian distribution, with mean and variance selected to produce hues described by the chosen colour term. there are high level categories like "red" and "green" and more specific ones like "maroon". this figure shows examples of hues generated in each category, including one that is both red and maroon. however, putting the blocks in a tower is only a partial description of the true planning problem, and the agent lacks vital knowledge about the problem in the following ways. first, the true goal g includes further constraints (e.g., that each red block must be on a blue block) and the agent is unaware of which such constraints are truly in the goal. further, and perhaps more fundamentally, the agent is also unaware of the colour terms used to define the constraints. i.e. the word "red" is not a part of the agent's natural language vocabulary, and so the agent does not know what "red" means or what particular set of rgb values the word denotes. instead, the agent can only observe the rgb value of an object and must learn to recognise the colour through interaction with the teacher, and in particular the corrective dialogue moves that the teacher utters. the possible goal constraints are represented in equations (1-2), where c 1 and c 2 are colour terms; e.g., "red" (r for short) and "blue" (b). in our experiments (see §6), a simulated teacher observes the agent attempting to build the tower, and when the agent executes an action that breaks one or more of the rules in the goal g, the teacher provides nl feedback-e.g., "no, red blocks should be on blue blocks". the feedback corrects the agent's action and provides an explanation as to why it was incorrect. however, linguistic syntax makes the sentence ambiguous between two rules-"red blocks should be on blue blocks" could mean r r,b 1 or r r,b 2 . thus, the agent must disambiguate the teacher's message while simultaneously learning to ground new terms in the embodied environment, in this example the terms "red" and "blue". this latter task amounts to learning which rgb values are members of which colour concepts (see figure 1 ). ## coherence to learn from the teacher's feedback the agent reasons about how an utterance is coherent. in discourse each utterance must connect to a previous part of the discourse through a coherence relation, and the discourse relation which connects the two informs us what the contribution adds to the discourse. in our multimodal discourse each of the teacher's utterances u connect to one of the agent's actions a through the discourse relation "correction". the semantics of correction stipulate that the content of the correction is true and negates some part of the corrected action @xcite . in our domain, this means that the teacher will utter u if the agent's latest action a violates the rule that she intended u to express. if @xmath0, red blocks should be on blue blocks" then, as previously stated, this is ambiguous between is r r,b 1 and r r,b 2 . so, a must violate one of these two rules: the rule r r,b 2 requires all blue blocks to have red blocks on them, meaning that s 1 in figure 2 does not directly violate the rule, but s 2 does because it is only violated when a blue block does not have a red block on it: so r r,b 1 is not directly violated in s 2 and r r,b 2 is not directly violated in s 1 but it would still be impossible to complete a rule compliant tower without undoing the progress that has been made on tower building. this is because the block which is currently not in the tower cannot be placed into the current tower in a rule compliant manner. for r r,b 1 in s 2 the red block needs a blue block to be placed on, but no such blue block exists. similarly, for r r,b 2 in s 1 the blue block needs a red block to place on it, but no additional red blocks are available. in this way the rules are indirectly violated in these states, which occurs when the number of available blocks of each colour makes it impossible to place all of those blocks: our teacher signals if the error is due to a direct violation v d by pointing at the tower or an indirect violation v i by pointing at a block which cannot be placed in the tower any more (e.g., the blue block in s 1 or the red block in s 2 ). when the agent observes the teacher say @xmath1, put red blocks on blue blocks" it can make inferences about the world, with confidence in those inferences depending on its current knowledge. for example, if it knows with confidence which blocks are "red" or "blue", then it can infer via equations (4-7) which of the rules the teacher intended to convey. alternatively, if the agent knows which rule was violated then the agent can infer the colour of the blocks. we use this in §5 to learn the task. however, if the agent is completely ignorant about the referents of the colour terms, it may not be able to make an inference at all. in this case it will ask for help by asking the colour of one of the blocks: "is the top block red?". either answer to this question is suffiicent for the agent to disambiguate the intended message and also gain training exemplars (both positive and negative) for grounding the relevant colour terms. if the teacher's dialogue strategy is to always correct an action which violates a rule (either directly or indirectly) directly after this incorrect action is executed, then the teacher's silence implies that the latest executed action does not violate a rule. this means that if the agent knows, for example, that if a green block is placed on a blue block then either green blocks must always be placed on blue blocks (r g,b 1 ) or no rule constraining green blocks exists. in this way the teacher's silence implies assent. ## system description an agent for learning from correction to perform the task described in §3 must be able to update its knowledge given the corrective feedback and then use that updated knowledge to select and execute a plan. the system we have built consists of two main parts: action selection and correction handling. ## experiments in §4 we mentioned four types of teacher error and in our experiments we vary the level of the teacher's error in these different types. we believe the most likely is missing indirect errors (mi) since spotting these requires search on all possible future actions. so our first faulty teacher varies p m @xmath2.0, 0.1, 0.25, 0.5, 1.0}: ie, from no errors to never correcting any indirect violations at all. our second teacher makes mistakes with direct violations. we believe missing and adding direct violations will be linked, so we experiment with: p @xmath3.0, 0.1, 0.25}. we study two types of agents in our experiments. the first, baseline agent, ba, ignores the fact that the teacher may be faulty. it simply uses the model described in @xcite . the only difference is that since the teacher is actually making mistakes, sometimes the agent may be given contradictory evidence which would cause the inference to fail. in such a situation the agent would simply ignore everything that was said in the current scenario and move on to the next planning problem instance. the second agent is a mistakeaware agent, ma, which makes inferences using the model from §5.3, matching its belief about the teacher's faultiness to the true probability. in our experiments each agent is given 50 planning problems. each planning problem has a different goal and a different set of 50 planning problem instances. the agent is reset between each planning problem, but retains knowledge between the 50 problem instances. we measure the number of mistakes the agent makes, which we call regret. a mistake is counted when an action takes a tower from a state where it is possible to complete it in a rule compliant way to one where it isn't without unstacking blocks. in figures 6 and 7 we present the mean performance over the 50 planning problems, and we use paired t-tests to establish significance. let's begin by looking at the results for agents learning from teachers that fail to make corrections for indirect violations, shown in figure 7 . clearly when the teacher is faulty the agent performs worse (a result which is shown significant through a pairwise t-test and significance threshold @xmath4.01). however, two interesting things can be observed. first, the slope of the curves are about the same for the agents learning from the faulty teacher and those learning from the faultless teacher. what this implies is that although the agent takes longer to learn the task when the teacher misses indirect violations it does seem to reach an equal proficiency by the end. we can explain the fact that the agent makes more mistakes by the fact that it is unaware of several mistakes it is making, however, when it is made aware of a mistake it still manages to learn. the second point is that the ba and ma agents are equally good at learning at all levels of teacher error. there is a good reason for this. when the teacher misses indirect violations the agent can actually trust all other information it receives. if it is given a direct correction then it knows for certain that the teacher give a coherent correction. this is true for all the feedback the agent receives when the only error the teacher makes is missing indirect violations. for this reason there isn't actually any need to change the way in which the agent learns, which is reassuring given that we believe the indirect violations to be more likely to happen in practice. looking at the results when the teacher will both miss and add corrections for direct violations, shown in figure 6 , we see that the agent's performance is much worse, both compared to the faultless agent and to the agents learning from the teachers making direct violations (these results are also significant given a pairwise t-test and significance threshold @xmath5.01). the big difference in this case is that the agent ba performs much worse than ma, especially when the likelihood of failure is higher. this is true both if we look at the final number of mistakes, but also at the slope of the curve, indicating that the agent is still making more mistakes by the end of training. figure 8 shows why: it compares the difference between the terminal regret for the faultless teacher vs. the faulty one. for ba there is a much larger spread of outcomes, with a long tail of very high regrets. the results for ma reside in a much narrower region. this implies that in contrast to ma, ba performs extremely badly in a significant number of cases. the high regret scenarios can be explained by situations where the agent has failed to learn the task successfully and is therefore acting almost randomly. so, making the agent mistake-aware stabilises the learning process, allowing the agent to recover from the teacher's mistakes without completely failing to learn the task, as seen in the baseline. ## conclusion in this paper we present an itl model where the agent learns constraints and concepts in a tower building task from a teacher uttering corrections to its actions. the teacher can make mistakes, and to handle this we introduce a separation between the teacher uttering a correction (observable) vs. whether that correction coherently relates to the latest action (latent). our experiments showed that this separation significantly reduces the proportion of situations where the agent fails to learn; without the separation, learning can go catastrophically wrong when the teacher's mistakes involve direct violations.