Human Inspired Progressive Alignment and Comparative Learning for Grounded Word Acquisition (2023)

Abstract: Human language acquisition is an efficient, supervised, and continual process. In this work, we took inspiration from how human babies acquire their first language, and developed a computational process for word acquisition through comparative learning. Motivated by cognitive findings, we generated a small dataset that enables the computation models to compare the similarities and differences of various attributes, learn to filter out and extract the common information for each shared linguistic label. We frame the acquisition of words as not only the information filtration process, but also as representation-symbol mapping. This procedure does not involve a fixed vocabulary size, nor a discriminative objective, and allows the models to continually learn more concepts efficiently. Our results in controlled experiments have shown the potential of this approach for efficient continual learning of grounded words.

## introduction two of the important word acquisition problems are: 1) what must be learned to acquire a word, and 2) how to learn the word? to the first question, cognitive studies have shown that several critical steps in learning language naturally comes from joint attention establishment @xcite , and symbol grounding @xcite ). children's attention are usually redirected through a mother or teacher's guidance, and they learn to map these attended sensory inputs (e.g. color, sound, heat) with their corresponding words or sentences. living in a rich and diverse world enabled by our multiple body sensors, we learned to filter out the noise and pay attention to specific aspects of an input which we assign linguistic labels to. this attention establishment and information filtration process is the first step of word acquisition. after filtering out the noise, we are left with a mental representation of what a word entails. just as the word "car" triggers certain impressions of a common transportation in head, we store these representations as they could come in handy later when we use them to reason, imagine, and express ourselves. to acquire a word, humans learn to filter out noise to focus on key information from sensory inputs that contributes to its meaning @xcite , and store that meaning representation for future use @xcite . as for the second question, one of the common but under-explored methods is implicit or explicit comparisons. caretakers may lay out stuffed animals around a baby and name them one by one to differentiate them. in school, teachers may compare different components of the learning material, e.g. "today we learn 'colors'. this is red/blue/yellow...". comparison is the process of finding commonalities and highlighting differences @xcite . it allows children to attend to matching relational structures of inputs @xcite , filter out background noise, and learn to generalize as well as abstract. with comparisons, especially clean well-structured comparisons, children can learn a lot of condensed knowledge efficiently and cultivate their capabilities to tackle noisier challenges outside of the classroom @xcite @xcite . from these findings, we propose a new method of word acquisition for artificial intelligent (ai) agents. we mimic the classroom learning setting and constructed a small clean dataset named sola -simulated objects for language acquisition. this dataset allows the model to draw efficient similarity and difference comparisons, learn to filter out noise, pay attention only to key information that contributes to a word meaning, and store these word-representation mappings continually as new words are introduced. while a larger scale evaluation is needed in the future, through controlled the contributions of this work include: 1. constructed a small, clean dataset sola for studying efficient comparisons. ## dataset inspired by the classroom teaching setting and the progressive alignment theory @xcite , we created a new dataset sola (simulated objects for language acquisition). sola has little noise and clearly defined attributes to isolate different concepts for efficient sample comparisons and grounded language-feature mapping. we generated sola using the open-source simulation software kubric @xcite designed for semi-realistic image/video synthesis. sola (figure 1 ) contains images of individual simulated objects with three associated learning attributes: color, material, and shape. each object is a composition of one of 8 colors, 11 shapes, and 4 materials. we also diversify the images by capturing each object at 3 different light settings and 6 different camera angles. a total of 6336 red green blue alpha (rgba) images were generated. to evaluate the generalizability and robustness of the models on nosier inputs, we also composed a variation test set (d test_v ) of 989 rgba images by applying a stretch, shade change, or size transformation. an object in this test set is either stretched along one of the x, y, and z axis, colored with a darker or lighter shade, or shrunk to a medium or small size. although not used in this work, we rendered the depth, surface normal, segmentation map, and object coordinates images for each corresponding rgba image for future research. to evaluate the novel composition capability of the methods, we reserved 9 learning attribute pairs exclusively in the novel composition test set (d test_nc ). the rest were assembled into the train set (d train ) for word acquisition training. to evaluate models' abilities to continual learning, we split the vocabulary into two sets: a known vocabulary and an unknown vocabulary set, which leads to two datasets d known and d unknown . the d unknown dataset includes images describable by at least one of the three attributes: [yellow, glass, torus_knot], and the rest of the images are in d known . each training and testing dataset is broken down into known and unknown versions accordingly. more details about sola can be found in the appendix. several existing datasets offer dense compositional attribute annotations that can be helpful for language grounding, such as mit-states @xcite , ut-zappos @xcite , cub @xcite , visual genome @xcite , and @xcite . these datasets are great resources for scaling attribute concept learning, especially from noisy real world images, but are not designed to form clean structural alignment for comparative language acquisition. our work took the baby step of progressive alignment @xcite @xcite by offering the model structured and denoised sets of inputs for easier structural comparison and efficient feature extraction. following these works, we believe that equipping the model with a set of clean base knowledge can help it extend to messier inputs in the future. other abstract datasets such as clevr @xcite focus on diagnosing and probing the reasoning or interpretability of models through visual question and answering, and are not designed for language acquisition. additionally, our dataset includes 2 more materials and 8 more shapes than clevr, providing a lot more variance and opportunities for vocabulary learning. we also diversify lighting, camera angles, and further object transformations in the variation test set for generalization and composition analysis. we introduce sola as it offers clean, grouped images for structured comparative learning. more detailed dataset comparisons can be found in ## conclusion in this work, we took a human inspired approach to acquire multi-attribute concepts. we define the acquisition of word as learning both an information filtration process, and a representation-symbol mapping. we mimic the classroom setting, constructed a small clean dataset sola for efficient comparative and continual learning. we evaluated the learned representations in multi-attribute recognition, compositional simulation and reasoning tasks. our experiment results outperformed clip variations in controlled settings, and showed early signs of a promising new method for continual grounded word acquisition through comparative learning.