Interactive Symbol Grounding with Complex Referential Expressions (2022)

Abstract: We present a procedure for learning to ground symbols from a sequence of stimuli consisting of an arbitrarily complex noun phrase (e.g. “all but one green square above both red circles.”) and its designation in the visual scene. Our distinctive approach combines: a) lazy few-shot learning to relate open-class words like green and above to their visual percepts; and b) symbolic reasoning with closed-class word categories like quantifiers and negation. We use this combination to estimate new training examples for grounding symbols that occur within a noun phrase but aren’t designated by that noun phase (e.g, red in the above example), thereby potentially gaining data efficiency. We evaluate the approach in a visual reference resolution task, in which the learner starts out unaware of concepts that are part of the domain model and how they relate to visual percepts.

## introduction the subfield of robotics known as interactive task learning (itl, see @xcite for a survey) addresses scenarios where a robot must learn to adapt its behaviour to novel and unforeseen objects, relations, and attributes that are introduced into the environment after deployment. the itl agent learns its novel task via evidence from its own actions and reactive guidance from a teacher. this paper focuses on symbol grounding @xcite in the context of itl @xcite : the learner must use the teacher's embodied natural language utterance and its context to learn a mapping from natural language expressions to their denotations, given the visual percepts. there are two challenges in learning symbol grounding models (grounders) in itl. firstly, in contrast to many grounders @xcite , itl requires incremental learning: knowledge is acquired piecemeal via an extended interaction, and it must influence planning as and when it occurs. secondly, previous work limits the teacher's language to bare nouns (e.g., square) or very short phrases (e.g., blue square, square above circle) @xcite . but there's evidence from @xcite that speakers use complex referring expressions even when simpler ones would successfully refer. such language creates the possibility that novel symbolsneologisms-are introduced in a context where their denotation is not designated by the teacher. in this work we study the natural language of complex referential expressions (res) like "a blue square behind both red circles" which teachers can use when designating an object. our aim is for the learner to extract knowledge that improves their domain representation and state estimates-a necessary condition for successful planning. contemporary grounders miss learning opportunities that complex res afford: the re example above not only entails that its referents are blue and square, but also that there exists two objects that are both red and circle and that they are above the designated objects, and everything else in the domain is either not red or not a circle (thanks to the meaning of both). thus, a complex re and its designation can be used to gather multiple (noisy) training exemplars (both positive and negative) for several symbols at once, even if they have not been designated. in this work, we develop a method to integrate knowledge from interactively gathered evidence in the form of complex re-designation pairs to aid data acquisition for a (neural) few-shot grounder. we explore the effect of such a method on data efficiency and the overall grounder's performance. a major novel component to our procedure is that we exploit the formal semantics of closed class word categories (e.g., quantifiers and negation) to boost the data efficiency of few-shot neural grounding models. our experiments show these symbolic inductive biases are successful. representation learning. models for jointly learning a representation for vision and language utilize either explicit alignment via bounding boxes or instance segmentation @xcite @xcite @xcite , or a large-volume of weakly labeled data in the form of image-caption pairs @xcite . these models rely on offline learning with large datasets. this work, on the other hand, explores how to incrementally extract knowledge from few-shot learning, using sequentially observed evidence that includes neologisms. ## methodology below we present the procedure of interactive grounding with referential expressions (igre). the overall framework is given in figure 1 . ## results and discussion figure 2 shows the evolution of the performance of the igre grounder and the two baselines on the test set, as it gets exposed to more information (i.e., re-designation pairs) over time. in the intrinsic evaluation (domain model prediction), there is no significant difference between the three grounders considered. yet, for extrinsic evaluation (reference resolution), we observe that igre outperforms the head and exists baselines over time (both a steeper and a smoother curve). by the end of the interaction, a t-test shows significant differences in igre's performance compared with both baselines (p-value of 0.01). we suspect that the reason why the three models performed differently in extrinsic evaluation even though they don't with intrinsic evaluation is down to the fact that igre uses its complete and accurate knowledge of the meanings of closed class words like quantifiers and negation at test time as well as training time in the extrinsic evaluation, but not in the intrinsic evaluation. the igre model can use these meanings to constrain and correct error-prone estimates of referents for open class words at test time in the reference task (as well as using their meanings to boost the training sets). for example, the re "both squares" implies there exist exactly two squares; if the symbol grounding model has an uncertain belief that there are more (or fewer) squares than this, it will select the two most probably candidates (and infer that all other entities are c1 s1 r2 reference p r f1 p r f1 p r f1 p r f1 head 0.17 0.54 0.25 0.16 0.52 0.23 0.16 0.50 0.25 0.14 0.04 0.06 exists 0.15 0.49 0.21 0.16 0.48 0.22 0.33 1.00 0.49 0.21 0.06 0.10 igre 0.17 0.51 0.23 0.17 0.56 0.25 0.33 1.00 0.50 0.42 0.10 0.16 ## conclusions in this work, we presented igre-a grounder that supports incremental learning of the mapping from symbols to visual features whenever the teacher presents a linguistically complex re and its desig-nation(s) in the visual scene. the grounder starts the learning process with no conceptualisation of the domain model, and so the learner must revise its hypothesis space of possible domain models as and when the teacher introduces new and unforeseen concepts via neologisms. we showed how exploiting the model-theoretic interpretation of the formal semantic representations of res, and in particular the truth conditions of 'logical' words like quantifiers and negation, can inform the acquisition of noisy training exemplars that in turn guide learning-igre reasons about the likely denotations of symbols within an re that aren't designated by that re, and when sufficiently confident it exploits them to update its grounding model. we showed that: 1) this grounding approach is more data efficient then a model that omits such observations and reasoning, using only the designated symbols; and 2) it is beneficial to exploit the logical consequences of the logical symbols, to gain even more data efficiency and training stability. in both cases, there was much to be gained from such reasoning because in contrast to the baselines, it contributes to acquiring negative exemplars: in other words, objects that get associated with not being red, for example.