Symbol and Communicative Grounding through Object Permanence with a Mobile Robot (2022)

Abstract: Object permanence is the ability to form and recall mental representations of objects even when they are not in view. Despite being a crucial developmental step for children, object permanence has had only some exploration as it relates to symbol and communicative grounding in spoken dialogue systems. In this paper, we leverage SLAM as a module for tracking object permanence and use a robot platform to move around a scene where it discovers objects and learns how they are denoted. We evaluated by comparing our system’s effectiveness at learning words from human dialogue partners both with and without object permanence. We found that with object permanence, human dialogue partners spoke with the robot and the robot correctly identified objects it had learned about significantly more than without object permanence, which suggests that object permanence helped facilitate communicative and symbol grounding.

## introduction communicative grounding is the process of mediating what words mean @xcite and symbol grounding is the establishment of connections between language and the perceptual, physical world @xcite . following @xcite that explained how symbol grounding is a side effect of communicative grounding, children who are learning their first language cannot learn symbol grounding without simultaneously being engaged in communicative grounding. consider the following example, within the physical space of a room. a child (c) picks up a ball (b) and a caregiver (p) engages in dialogue with the child about the ball: in the following section, we compare our work to others then explain our method for tracking object permanence using a simultaneous localization and mapping (slam) module and the the robot-ready sds system that we used. we then explain our experiment and conclude. ## background & related work object permanence is a crucial milestone in cognitive development, and it has been suggested by moore and @xcite that as early as four months this milestone is reached. @xcite shows that as infants enter the sixth stage of object permanence development (where children understand that objects completely removed from their view still exist) they start to learn relational words. a more recent study explores the development of search behavior in 7 month old infants after they guide them in understanding the effects of their actions upon hidden objects. this indicates that object permanence is crucial in jects depicted in images only offer a single visual experience (though multiple referring expressions) to the objects. of similar importance in cognitive development is communicative grounding. researchers, notably @xcite , have investigated how the collaborative efforts of a robot in situated human-robot dialogue affects both perceived and true grounding which involved a situated setup of objects similar to our experiment. this notion of common ground and communicative grounding has also been explored in other human-robot interaction work @xcite @xcite @xcite and work involving human interactions with virtual agents @xcite . our work extends and builds on prior work as we focus on using object permanence in a robot to improve its language learning abilities. ## proposed system in this section we explain how we modeled the dialogue for language learning, integrated with robot modules. we first explain the choice of robot: digital dream lab's cozmo robot. @xcite showed that participants perceived cozmo as young and with potential to learn, which is precisely the setting and perception that we want dialogue partners to have when interacting with cozmo. cozmo is small, has a track for movement, a lift and a head with an oled display which allow it to display its eyes. within the head is a small camera and a speech synthesizer (with a "young" sounding voice). for this study we make use of cozmo's camera for object detection, track for navigation and most importantly cozmo's built-in slam (simultaneous localization and mapping) functionality for object permanence. cozmo has no microphone, so we use an external microphone. the system outlined in this paper uses the incremental framework retico @xcite extended for multimodal use with cozmo @xcite , leveraging existing modules as well as the newly developed object permanence module. the full sds is depicted in figure 3 . the modules include: object detection, feature extraction, automatic speech recognition, natural language understanding, grounded semantics, action management (navigation & speaking), and object permanence. object detection the object detection module uses yolo object detection @xcite . the model we used was pre-trained on the mscoco dataset @xcite containing 91 object types with a total of 2.5 million labeled objects in 328 thousand images. we apply this model as a means for object region classification in order to draw bounding boxes around objects in images received from cozmo's camera. we discard the labels and only use the bounding box information as to avoid the use of a pretrained vocabulary since children are born without linguistic knowledge. the output of this module is the bounding box information of the objects in view to cozmo. ## evaluation in this section, we explain how we evaluated our model with human participants to determine if cozmo performs communicative and symbol grounding more effectively with object permanence. we compare two versions of our system: one that did not have an access to the object permanence module and one that did. our evaluation included objective measures logged by the system and by the participants used to measure symbol grounding by tracking correctly "learned" words, as well as subjective measures collected using participant questionnaires used to measure communicative grounding. procedure study participants met in our lab located near boise state university's computer science building. the lab is setup for the participant interaction as follows. a large table is setup with 7 objects on the table as shown in figure 4 . we chose the 7 objects to vary shape and color, but wanted to have a degree of overlap for words that might be used to describe them (e.g., can or blue). in front of each object is a straight line drawn on the table and a box. 2 cozmo is placed in front of the leftmost object. the microphone that feeds into the asr module is positioned in front and to the left of the table with the objects and cozmo. the participant stands or sits (as they prefer) at the front of the table. cozmo is not introduced to the participant until the participant has signed a consent form and the task has been explained to them. the experimenter was present to examine the state of the robot and the microphone, answer any questions the participant may have, and troubleshoot any problems that arose. the experimenter was permitted to offer a constrained set of coaching tips to the participant during the experiment, given the participant needed a reminder of their task or the instructions. following each interaction with cozmo, the participant was instructed to complete a questionnaire. following the completion of the experiment and surveys, the participant was paid $10. we recruited 24 participants to interact with cozmo for two twenty-minute periods over the course of a single session. most study participants recruited were from the boise state university department of computer science. 18 of the participants were male; 6 were female. the entire time for each participant was approximately one-hour. after signing the informed consent, cozmo was introduced to the participant, with the following explanation; ( 1 ) cozmo has a camera that can see the world; (2) cozmo has a microphone and can hear them; (3) cozmo doesn't know anything, but would like to know more about the world; (4) for the next 20 minutes, it is your job to teach cozmo as many words as they can, about the seven objects in front of cozmo; (5) cozmo will move in front of an object. if cozmo does not hear you speak he will move on to a new object. if cozmo does hear you speak, then it will observe the object and repeat the word he learned. the word cozmo learned will always be the last word you spoke. do not teach cozmo any more words until it repeats this word. for every word you teach it, you can write it down so you can keep track; (6) if cozmo is not on the square in front of the object when he moves to an object you must readjust it to that square; (7) after teaching cozmo about two different words for two different objects you can and should ask it what and where questions to check his knowledge; ( 8 ) for every question he gets right answer "yes" and put down a tally mark to record a correct answer for every question he gets wrong answer "no" only; (9) only speak to cozmo when he is in front of an object. we used an a/b design, meaning that each participant went through the same procedure twice, once with cozmo having access to object permanence and once without access. to mitigate priming effects, the order in which the test condition was presented was alternated. two system versions the test condition is the system version that had access to object permanence and is explained in section 3. the baseline point of comparison for this study was a system that did not have access to the object permanence module. the overall functionality of the baseline system was the same as the system with access to object permanence, except that the system could not track and locate objects when participants asked them to. the grounded semantics module in this case only performs traditional symbol grounding between words and visual representations-not specific objects. this meant that the robot behavior when a find dialogue state was entered (i.e., after a where dialogue act from the participant) was different: instead of moving directly to the identified object, the robot would move towards a random object one at a time and check each object to determine if they matched the description. if an object did match, the robot would repeat the description to the participant, who then in turn offered positive or negative feedback. under the best circumstances for the baseline system, the robot would randomly move towards an object that fit the description on the first attempt. but if the first object did not fit the description, then the robot moved towards a different object and repeated until an object matched the description. to give the baseline version a higher chance of the robot actually finding the objects, the objects were placed in a line and the robot systematically drove directly to a randomly selected object. this was designed to give the baseline system version some degree of the object permanence functionality as a stronger point of comparison. metrics all module communication is logged using the platform for situated intelligence @xcite . we specifically track the number of utterances made by the participants, including positive and negative feedback, and the number of questions asked. the participant themselves keep track of the number of questions cozmo correctly answers (i.e., if cozmo correctly identified an object). these metrics act as a way to measure symbol and communicative grounding, as well as engagement (i.e., more utterances means more engagement). we evaluate the robot based on questionnaire responses filled out by the participants following each interaction to establish that communicative grounding took place. we used the godspeed questionnaire @xcite , a 5-point likert-scaled questionnaire with 24 questions using negative (left side) to positive (right side) ratings of a robot's anthropomorphism, animacy, likeability, and perceived intelligence. in addition to the godspeed questions, we asked the participants the following questions to further ascertain their perceptions of our system and robot (some items have boldface text to link them with results): results participants on average estimated cozmo's age with the test condition at 3.5 years of age compared to 2.6 years of age with the baseline, suggesting that cozmo was perceived to be more intellectually advanced with the test condition, but still an early language learning child, which also tells us that the robot did not exhibit behaviors that participants perceived as too advanced for our task. we also observe higher responsiveness in the object permanence version which likely results from participants observing that cozmo answered questions quickly and with high accuracy, suggesting that communicative grounding was better with the object permanence version (see appendix for more results comparing perceived intelligence and responsiveness). finally, ratings for interest and desire to spend more time with the robot are significantly higher with object permanence than without. this is especially evident when observing that the mean value for interest at the end of the interaction is 4.7 with the test condition and 4.2 without; the average increase in interest from the beginning of the interaction for the test condition is 0.54 as compared to 0.83 without. furthermore, using questions asked as a measurement for engagement (since it shows active interest in what cozmo is learning) we observe that with object permanence, cozmo is asked approximately 9 more questions than without showing that object permanence has a significant effect on engagement (see ## conclusion we conducted an experiment with twenty-four participants who performed a language acquisition task with cozmo both with and without object permanence. we analyzed our results by comparing the participants' survey responses to measure communicative grounding and number of words heard, questions asked, and percent of questions answered correctly to measure symbol grounding between the experimental and control interactions. we found that a robot with object permanence resulted in improved communicative and symbol grounding due to stronger engagement from the participant and a higher percentage of correct answers from cozmo. user perceptions of cozmo with object permanence also greatly improved overall. this indicates that object permanence does in fact have a positive affect on communicative and symbol grounding. our findings suggest that an understanding of object permanence is a necessary component of any spoken dialogue system built to reach the potential of natural dialogue between humans. figure 5: engagement ratings for cozmo with and without object permanence