Fast visual grounding in interaction: bringing few-shot learning with neural networks to an interactive robot (2020)

Abstract: The major shortcomings of using neural networks with situated agents are that in incremental interaction very few learning examples are available and that their visual sensory representations are quite different from image caption datasets. In this work we adapt and evaluate a few-shot learning approach, Matching Networks (Vinyals et al., 2016), to conversational strategies of a robot interacting with a human tutor in order to efficiently learn to categorise objects that are presented to it and also investigate to what degree transfer learning from pre-trained models on images from different contexts can improve its performance. We discuss the implications of such learning on the nature of semantic representations the system has learned.

## introduction robots need to ground real world objects and entities that they see with their cameras to natural language that they hear or generate when interacting with a human tutor. there are four properties that distinguish this kind of machine learning from the approaches that are based on a corpus: (i) agents have to make reliable classifications already after being exposed to a few examples; (ii) they may utilise background knowledge; (iii) learning is not done in large batches of examples but in small increments as the interaction unfolds; and (iv) the mechanisms of interaction are used to control the rate of learning. matching networks @xcite is a neural network algorithm designed for one-shot and few-shot learning of a classifier in a standard dataset. for object classification, their principal advantage is their capability to learn objects from few labelled instances rapidly. this property makes them a possible candidate for modelling meaning representations for robot interactions. the learning algorithm for matching networks fits in the supervised learning paradigm. its central idea is influenced from the meta-learning paradigm with memory-augmentation @xcite which has not been evaluated for interactive scenarios. in this work, we implement matching networks and evaluate its performance in critical interactive scenarios which includes both offline and online learning with a simulated interactive agent. 1 the contributions of this paper are as follows: (i) we create our own implementation of the matching network model from @xcite and (ii) integrate it with interactive strategies of a situated agent; (iii) we test the performance of such a simulated agent (a) on baseline recognition of objects presented to the agent, (b) where the support set of image categories is from another context, and (c) where new object categories must be learned; (iv) we discuss implications of interactive learning using this model for representations of meaning and semantics of natural language and outline several future experiments in which the model could be exploited and improved. as shown in figure 1 , the matching networks compare an unlabelled target image (t) with the labelled images of a support set (s) to determine how similar t is to different categories in s. after encoding images of the support set s with convnet θ , the support set embedding module (g θ ) is responsible to implicitly learn the interdependency between the visual features and categories by creating an embedding space for images where their similarity must correspond to their shared category. after encoding the target image (t) with convnet θ , the target embedding module (f θ ) is expected to learn the projection of convolutional visual features onto the embedding space of categories constructed by (g θ ). after obtaining the output from the embedding functions, the matching layer computes the similarities between the target embedding and the support set embeddings as if it attends on the categories of similar images in the support set: training algorithm notably, the labelled images in the support set are the augmented memory of the neural networks. the training objective is to minimise the error in categorisation by learning prototype representations of the images of each category in the support set. in principle, components of the model can be trained in an end-to-end fashion. with categorical cross entropy loss, in each supervised learning step, the predicted loglikelihood of the correct labels is back-propagated to all parameters of the modules: loss(y t , ŷt ) = -log(y t • ŷt ) implementation we have implemented a variation of matching networks in tensorflow @xcite with a small number of parameters. in our implementation, g and f are one-layer feed forward networks with relu activation and l2 normalisation, convnet is vgg16 @xcite and the matching layer is the cosine similarity of two vectors with softmax normalisation with no trainable parameters. in the normalisation of f and g in att, their dot product is equivalent to their cosine similarity. the code of this implementation is openly available. 2 differences with @xcite unlike @xcite , in the implementation of g and f we only process one image at a time instead of using bigger networks to process the support set in one go with bilstm (for g) and attlstm (for f) @xcite . while the incentive behind using lstms @xcite is their ability to encode dependencies between items, they also encode sequential dependencies which are not beneficial for sets of labelled images in s. this was why we do not use lstms, despite its promising results in @xcite . our goal is to use the model in an interactive scenario where the support set grows incrementally. since the implemented design requires retraining the parameters, smaller models are preferable. therefore, we chose the simplest solution that is sufficient to give reliable results. we simply concatenate the images in the support set vector to retrain the re-parameterised embeddings with a random order of categories. ## interactive grounding of visual objects robotic systems have to learn constantly new knowledge about their dynamic environment, for example when they encounter new objects. one of the major differences between learning from datasets and learning interactively is that the system must be able to use the knowledge that it has learned very quickly, after seeing only a couple of examples. therefore, in this respect few-shot learning is ideally suited for this domain. however, an interacting agent with a human tutor can exploit mechanisms of human interaction (interaction strategies) to learn faster, for example by being able to control how information is presented to the human or by requesting new information that it is missing @xcite . our situated agent setup is based on the kille framework @xcite . it consists of a stationary microsoft's kinect v1 rgb-d sensor connected to an ubuntu 16.04 system. the sensor is supported by the freenect drivers foot_3 integrated in the robot operating system (ros) framework @xcite . our system could use any rgb camera supported by ros. its intended scenario of usage is grounding of (small) objects on a table top that a human tutor presents to it. several interaction strategies can be defined in this domain @xcite . because in this work we explore the integration of the few-shot learning architecture with a focus on data sparseness and its contextual dependence, we investigate two strategies to learn objects. firstly, the human tutor can present the object to the agent and describe what it is (e.g. this is an apple). the image of the object is saved to the dataset. then, depending on the number of examples that the system has seen of this object category, it either waits to see more examples (e.g. please, show me more examples of apple), learns a new category (e.g. i am learning apple) if it has five images of this category, or updates its knowledge (e.g. i am updating my systems on apple) if it has seen more than five images. the second interaction strategy allows the robot to understand and respond to questions about objects that are presented to it (e.g. what is this?). the robot first attempts to classify the object presented and includes the highest scoring label in its answer and the certainty of the guess based on the score of the label (e.g. this is an apple; i think this is an apple; i am not sure. is this an apple?; i don't know what is this. please, tell me.). the user can then provide feedback and affirm the guess or tell the system the correct label. the application of the matching networks has two stages: training and prediction. when training, we build the support set s with a few (k) images of each labelled class (n). then, we build a target set t from the images from s to train our model. hence, we use s as both the support set and the target images in the training phase. when a user asks the system to identify an object, the same support set for training is used (s) but the target t is the new input taken with the camera. the target must belong to one of the categories of s. the online training of the matching network is performed every time the robot needs to learn a new category or update its knowledge of one of the existing categories. to learn a new category, the system collects five images of this category and adds the images as a new category of objects to the support set s which now has an extra category. to update the knowledge of an existing category, the system includes in the support set s the new image taken by the camera and trains the model with the new s. this update of s is performed (i) when a user presents an object to the robot and (ii) when the robot incorporates feedback from the user after incorrectly classifying an object or after clarifying a category of an object that it was not sufficiently confident in. with few images and categories the training is fast and the learning is reliable for our robot interaction. however, as s grows, so does the training time. as it is expected that the system would need to constantly learn new objects and categories, this issue will have to be addressed in the future. having to wait for a long time for retraining the model every time we change the size of the support set is not so good from the perspective of user experience and scalability of the system. one strategy we could take is to implement an additional module of a short-term memory for the current objects and wait with the learning updates until when the robot is not interacting with a user and is resting. in order to measure the success of learning under different conditions we automatised the testing procedure. to test each configuration with identical data, we created a small objects dataset (sota), a collection of 400 images with equal distribution of images over 20 categories. 4 each image depicts a single small everyday object (such as apple, pen or shoe), which was placed in front of the robot's camera while interacting with our system in real time. the matching networks are trained as outlined in section 3: we build a support set (s) with n categories or labels (5 or 20) and k images per label (1, 5 or 10-shot). during training, the images from s are also used as target images, making the number of training instances of k × n. for instance, when training a model with 20 labels (n) and 5-shot (k), there are 100 training instances of target images. during evaluation, the images that we use as targets are the remaining 10 images per each category from sota that have not been used to train the model and so they are unknown to the system. results in in this experiment we have not applied any selection on the support dataset. that is, we have taken images from sota at random to build s with them (the selection is saved so we can reproduce the same s). different selections of the support set might improve or worsen the performance of the system. the selection of the categories could be derived by another process modelling how humans categorise and discriminate objects in particular contexts. ## discussion and conclusions we have explored how a neural network algorithm for one-shot learning, matching networks @xcite , can be used in an interactive scenario between a robot agent and a human tutor. our evaluation demonstrates that fast learning with neural networks is possible for this task and it is effective with a few categories. the matching networks do not learn how to directly ground words describing objects in visual features @xcite . instead, it relies on a small support set, contextually introduced objects that it stores, and then learns how to discriminate their categories from one another. this is a particularly appropriate meaning representation for situated agents: (i) the system is able to express gradient beliefs: the outputs of the network are probabilities of the target belonging to each category; (ii) contextual priming information can be integrated by controlling the categories of objects in the support set which demonstrates a potential for modelling top-down attention of an agent @xcite ; (iii) it models the saussurean notion of lexical meaning, namely that language is a system of differences without positive terms @xcite ; (iv) as a result the system is very robust and discrimination of categories can already be achieved with a small number of examples. the system could be extended in several ways, which will be the focus of our future work. first, additional experiments will involve a more distant type of knowledge transfer, to offline pre-train the matching networks also on datasets of images with large number of categories and then fine-tune the pre-trained model in the local domain. another extension is to have a process in place to control the size of the support set over time and transfer the matching knowledge to the memory as required. the update procedure that trains the model from scratch every time affects the user experience negatively, especially when the support set becomes large. additional procedures could be added to deal with this issue: (1) a training process scheduled when the user is not interacting; (2) a training process of a new model during the interaction when this is still using the older model; (3) implementing methods in the framework of continual learning to prevent catastrophic forgetting when updating our models @xcite . we will also investigate the influence of using different support sets in terms of variation of objects within a category, variation of objects sampled from different views and selection of object categories as referred to in the current conversation. new interactive strategies with the robot will be investigated and implemented as a way to make the interactive learning of our framework more efficient, for example to unlearn efficiently incorrectly learned labels @xcite . as another direction of future work, we noted that a uniform image background in our experiments negatively influences both performance and scalability. this could be countered by automatic localisation and segmentation of relevant regions of images either with bounding boxes around objects @xcite or using soft-attention over regions of the image @xcite to remove the background of the objects. finally, further studies of this framework should go beyond learning of visual grounding of objects. one direction of our future work is to learn grounding of relations including spatial relations with few-shot learning and matching knowledge. another direction is to consider linguistic and distributional knowledge that could be transferred from cross-modal resources @xcite as an external matching knowledge. in in