Interactive Acquisition of Fine-grained Visual Concepts by Exploiting Semantics of Generic Characterizations in Discourse (2023)

Abstract: Interactive Task Learning (ITL) concerns learning about unforeseen domain concepts via natural interactions with human users. The learner faces a number of significant constraints: learning should be online, incremental and few-shot, as it is expected to perform tangible belief updates right after novel words denoting unforeseen concepts are introduced. In this work, we explore a challenging symbol grounding task—discriminating among object classes that look very similar—within the constraints imposed by ITL. We demonstrate empirically that more data-efficient grounding results from exploiting the truth-conditions of the teacher’s generic statements (e.g., “Xs have attribute Z.”) and their implicatures in context (e.g., as an answer to “How are Xs and Ys different?”, one infers Y lacks attribute Z).

## introduction consider a general-purpose robot assistant purchased by a restaurant, which must acquire novel domain knowledge to operate in this particular venue. for example, the agent must learn to distinguish brandy glasses from burgundy glasses (fig. 1a ), but these subcategories of glasses are entirely absent from the agent's domain model in its factory setting. learning to distinguish among fine-grained visual subcategories is a nontrivial feat @xcite ; most current approaches require careful engineering by ml practitioners, making them unsuitable for lay users to readily inspect and update the robot's domain knowledge. there are also challenges regarding data efficiency, which using natural language can potentially address @xcite . a single generic statement-e.g., "brandy glasses have short stems"-expresses content that would take many visual examples to infer. such statements, given their dialogue context, may also carry additional meaning that is linguistically implicit. for instance, if the statement "brandy glasses have short stems" is given as an answer to the contrastive question "how are brandy glasses and burgundy glasses different?", then it implies that burgundy glasses don't have short stems, and also, defeasibly, that these two types of glasses are similar in other conceivable respects @xcite . vision processing models that exploit natural language data exist @xcite @xcite ), but they generally treat language as supplementary signals for augmenting training examples, rather than leveraging a range of symbolic inferences licensed by purposeful utterances in dialogue. in this work, we develop an interactive symbol grounding framework, in which the teacher presents to the learner evidence for grounding during embodied dialogues like those illustrated in fig. 1b . the framework is based on a highly modular neurosymbolic architecture, in which subsym-bolic perceptual inputs and symbolic conceptual knowledge obtained during dialogues gracefully combine. we run proof-of-concept experiments to show that agents that exploit semantic and pragmatic inferences from generic statements in discourse outperform baselines that don't exploit semantics and pragmatics, or don't exploit symbolic inference at all. ## related work in fine-grained image analysis (fgia), a model learns to distinguish (patches of) images of subcategories that belong to the same basic category. fgia is challenging because images exhibit small inter-class variance and large intra-class variance, and labeling often requires specific domain expertise, hence high annotation costs @xcite . a natural approach to fgia is to utilize information of different modalities, including unstructured text descriptions @xcite , structured knowledge bases @xcite and human-edited attention maps @xcite . however, to our knowledge, no existing fgia models exploit nl generic statements provided in vivo during natural dialogues. existing interactive fgia methods @xcite @xcite query humans to refine predictions from off-the-shelf vision models at inference time but do not update the grounding models. in contrast, our framework supports continuous learning, updating the grounding model as and when the teacher says something noteworthy. our use case, described in §1, can be subsumed under the framework of interactive task learning (itl; @xcite . motivated by scenarios where unforeseen changes may happen to the domain after deployment, the core goal of itl is to acquire novel concepts that the learner is unaware of but are critical to task success. itl systems gather evidence from natural embodied interactions with a teacher that take place while the learner tries to solve its task. thus a key desideratum in itl is that learning should be online and incremental: the learner should change its beliefs and behaviours whenever the teacher provides guidance. natural language is a common mode of teacherlearner interaction in itl @xcite . accordingly, several itl works draw inspiration from linguistic theories to make learning more effective and efficient. while the for- this is a bordeaux glass. 𝑜 1 𝑜 3 𝑜 2 @xmath0. 𝑃 𝑜1 . @xmath1. vision processing module language processing module symbolic reasoning module exemplar base episodic memory lexicon knowledge base long-term memory scene graph semantic representation final estimate preliminary scene graph dialogue record updated mal semantics of quantifiers and negation (rubavicius and lascarides, 2022) and of discourse coherence (appelgren and lascarides, 2020) has been explored in itl settings, none of the works in the itl literature have investigated the utility of exploiting the logical inferences licenced by the semantics and pragmatics of contrastive questions and their generic statement answers. ## agent architecture fig. 2 illustrates our neurosymbolic architecture for situated itl agents that can engage in extended dialogues with a teacher. its design enables both subsymbolic-level learning of visual concepts from perceptual inputs ("this looks like a x") and symbolic-level learning and exploitation of relational knowledge between concepts ("xs generally have attribute z") during task execution. here we stress that our proposed approach is not in direct competition with wide-coverage neural visionlanguage models, but actually complements them. as an itl framework, we offer a coping mechanism, to be employed when an existing pre-trained model is deployed in a domain where concepts are frequently introduced and changed, requiring the model to quickly adapt with only a few exemplars exemplar base few-shot concept prediction 𝐼, perceived scene image 𝑆𝐺, scene graph estimate ෪ 𝑆𝐺, preliminary scene graph template vision processing module 𝜒 𝛾 + , 𝜒 𝛾 -, positive/negative exemplars (for each concept 𝛾) (updated throughout episodes) 𝐵𝑖𝑛𝐶𝑙𝑓 𝛾 , induced binary classifiers (for each concept 𝛾) pre-trained neural model for few-shot object localization and feature extraction (weights stay frozen throughout episodes) boxes with feature vectors of unforeseen concepts. ## conclusion and future directions in this research, we have proposed an interactive symbol grounding framework for itl, along with a neurosymbolic architecture for the learner agent. we empirically showed that learners who can comprehend and exploit valid inferences from generic statements, including pragmatic content given their context of use, can learn to ground novel visual concepts more data-efficiently. our findings confirm it pays to study human-ai natural language interactions through the lens of discourse semantics, not only the truth conditions of isolated sentences but also their coherent connections to their context. in future, we plan to relax some of many simplifying assumptions we made for controlled experiments, possibly exploring other domains. for instance, the ideal assumption that teachers are infallible and communication is noise-free does not hold in most real-world scenarios @xcite . further, the set of linguistic constructions we have studied in this work is very constrained (as intended), and a natural next step is to accommodate a wider range of diverse and free-form nl constructions. it is also a strong assumption that the learner agent already has relatively reliable beliefs about object part and concept attributes. for example, if the learner does not know what the "stem" of a wine glass means, the absence of the concept must be resolved before communicating any generic characterizations involving stems. finally, our approach does not fully exploit the semantics of generic statements, which express qualitative rules that admit exceptions @xcite . the generic quantifier g did not play any significant role in this work. one major strength of asp is that it is well suited for modeling non-monotonic inferences, and it would be interesting to study how to model itl scenarios that can robustly address exceptions to generic rules. nick zangwill. 2011. negative properties. noûs, 45(3):528-556. xizhou zhu, weijie su, lewei lu, bin li, xiaogang wang, and jifeng dai. 2021. deformable {detr}: deformable transformers for end-to-end object detection. in international conference on learning representations.