A Computational Acquisition Model for Multimodal Word Categorization (2022)

Abstract: Recent advances in self-supervised modeling of text and images open new opportunities for computational models of child language acquisition, which is believed to rely heavily on cross-modal signals. However, prior studies has been limited by their reliance on vision models trained on large image datasets annotated with a pre-defined set of depicted object categories. This is (a) not faithful to the information children receive and (b) prohibits the evaluation of such models with respect to category learning tasks, due to the pre-imposed category structure. We address this gap, and present a cognitively-inspired, multimodal acquisition model, trained from image-caption pairs on naturalistic data using cross-modal self-supervision. We show that the model learns word categories and object recognition abilities, and presents trends reminiscent of ones reported in the developmental literature.

## introduction to date, the mechanisms underlying the efficiency with which infants learn to speak and understand natural language remain an open research question. research suggests that children leverage contextual, inter-personal and non-linguistic information. visual input is a case in point: when spoken to, infants visually perceive their environment, and paired with the input speech, the visual environment could help bootstrap linguistic knowledge @xcite . unlike social cues, visual input has a natural physical representation, in the form of pixel maps or videos. previous multimodal language acquisition studies either considered toy scenarios with small vocabularies @xcite , or used visual encoders that were pretrained 1 github.com/slab-nlp/multimodal_clustering on large labeled data bases such as imagenet @xcite or visual genome @xcite . this has two drawbacks: first, systematic access to labeled data is a cognitively implausible assumption in a language acquisition setting; second, imposing a pre-defined categorization system precludes studying categories that emerge when learning from unlabeled multimodal data. this type of setting more closely resembles the data underlying early language learning at a time when the child has only acquired little conceptual information. although the subject of much psycholinguistic work, the computational study of multimodal word categories, formed without recourse to manual supervision has been scarcely addressed in previous work. model parameters token (img, capt) train time bert 110m 3300m -64 tpu days word2vec 1b 100b -60 cpu days clip 63m -400m 10k gpu days ours 3m 3m 280k 4 gpu days children -13k/day -- for the multimodal models (clip and ours) we only mention the size of the text encoder. in a self-supervised setup, a neural image classifier and a co-occurrence based word clustering module provide mutual supervision through joint training with an expectation-maximization (em) style algorithm. figure 1 illustrates our model. our input representation is cognitively plausible in that we use raw image data (pixels), and train our model on a comparatively small data set ( we show that semantic and visual knowledge emerges when training on the self-supervised categorization task. in a zero-shot setup, we evaluate our model on (1) word concreteness prediction as a proxy for noun identification, one of the first cues for syntax in infant language acquisition @xcite ; (2) visual classification and object segmentation. we also study the emerging latent word clusters and show that words are clustered syntagmatically @xcite : words representing entities that are likely to occur together are more likely clustered together (e.g., dog-bark), than words that share taxonomic categories (e.g., dogcat). this concurs with findings that young children acquire syntagmatic categories more readily than taxonomic categories @xcite . ## background and related work we briefly review previous studies of unimodal learning without pretraining (for both text and vision) and multimodal learning (studied mainly in acquisition implausible settings) to highlight the gap that this work addresses. unimodal learning. self-supervised learning without pre-training has been extensively studied, but predominantly in a unimodal scenario or under cognitively implausible assumptions. for the text modality, large language models have been developed in recent years (e.g., bert; devlin et al., 2019), trained on large unlabeled text corpora ( ## model our goal is to learn the meaning of words and raw images through mutual supervision by mapping both modalities to a joint representation. intuitively, given a visual input paired with relevant text (approximating a typical learning scenario), the output for each of the modalities is a binary vector in {0, 1} n , with non-zero dimensions indicating the clusters foot_0 to which the input is assigned and n is the total number of clusters (a predefined hyper-parameter). the clusters are unknown a priori and are formed during training. the goal is to assign matching text and image to the same clusters. in order to minimize assumptions on innate knowledge or pre-imposed categories available to the language learner, and enable the study of emerging categories from large-scale multi-modal input data, we deliberately avoid any pre-training of our models. ## training at each training step, the model observes a batch of (image, caption)-pairs. we first perform inference with both encoders, and then use the results of each encoder's inference to supervise the other encoder. text encoder. given the list of clusters predicted by the visual encoder and a tokenized caption @xmath0, ..., w n }, for each w @xmath1, we increment count(c j ) and count(w i , c j ). these are needed to compute the probabilities in equation ( 1 ). visual encoder. for each input image and corresponding cluster vector predicted by the text encoder, we use binary cross entropy loss comparing the output of the sigmoid layer with the predicted cluster vector and use backpropagation to update the parameters of the resnet model. ## experiments we trained our model on the 2014 split of mscoco @xcite , a dataset of naturalistic images with one or more corresponding captions, where each image is labeled with a list of object classes it depicts. mscoco has 80 object classes, 123k images and 616k captions (split into 67% train, 33% test). we filtered out images that did not contain any labeled objects, and images that contained objects with a multi-token label (e.g., "fire hydrant"). 5 after filtering, we are left with 65 ground-truth classes. the filtered training (test) set contains 56k (27k) images and 279k (137k) captions. we set apart 20% of the training set for hyper-parameter tuning. we trained our model with a batch size of 50 until we observed no improvement in the f-score measure from section 5.1 (40 epochs). training took 4 days on a single gm204gl gpu. we used @xmath2, θ @xmath3.08, and θ @xmath4.5. the visual threshold θ v was first set heuristically to 0.5 to avoid degenerate solutions (images being assigned to all or no clusters initially). then, n and θ t were determined in a grid search, optimizing the f-score measure from section 5.1. we used spacy @xcite for tokenization. ## discussion and conclusion we proposed a model for unsupervised multimodal lagnguage acquisition, trained to jointly cluster text and images. many of our design choices were guided by findings from cognitive studies of infant language acquisition: the joint learning of multiple modalities; learning word-level semantics (e.g., @xcite , suggest that children first learn to identify nouns and use this information to learn sentence-level semantics); and crosssituational learning (counting how many times each word co-occurred with each cluster, see @xcite . after training, our model demonstrates capabilities typical of infant language acquisition: word concreteness prediction and identification and segmentation of objects in a visual scene. however, we do not stipulate that infants begin their acquisition of language by clustering words. it would be interesting to design experiments to test this hypothesis, e.g., by connecting our work with laboratory work on joint word and category learning @xcite , or work on the emergence of syntagmatic vs. taxonomic categories in young children @xcite . while our model is cognitively more plausible compared to previous studies, the gap from a realistic setting of language acquisition is still large: (1) we assume the language input is segmented into words; (2) the input data, while naturalistic, is not typical of infants at the stage of language acquisi-tion; (3) the input only includes the visual and textual modality, but not, e.g., pragmatic cues like gesture; and (4) the model learns in a non-interactive setting, whereas physical and social interactions are considered crucial for language learning, and learns (and is evaluated) in a batch fashion while human learning is typically incremental @xcite . in the semantic word categorization and concreteness prediction experiments, we compared our multimodal model to unimodal text-only baselines, which we chose to be as similar as possible to our model. the results suggest that multimodality improves performance on both text tasks. however, it is unclear which specific information is encoded in the visual modality that benefits these text tasks. we leave this question for future research. syntagmatic categories, although highly intuitive in the context of human memory, were not the subject of many previous computational studies. we propose to further investigate this type of categories and its use. one interesting direction is to combine syntagmatic categories with interactivity: given a relevant signal from the environment the model can cycle through concepts in the syntagmatic category triggered by the signal, speeding up the extraction of relevant concepts in real time. one possible application of this direction is modelling the construction of ad-hoc categories, described by @xcite . while all experiments are conducted in english, our setting supports future work on other languages. the small training set and the nature of the data (image-sentence pairs that might be, to some extent, collected from the internet) allow our model to be extended to low-resource languages, while the minimal structural assumptions of the text encoder may imply some degree of language-agnosticity. in future work, we plan to improve the cognitive plausibility of our model by (1) incorporating typical input observed by children (by using videos taken in real scenes of child language acquisition, see @xcite ; and (2) changing the setting to an interactive one, where the model is transformed into a goal-driven agent that uses interactivity to learn and produce language. cluster 23: flamingo, hoe, racquet, parka cluster 24: violin, accordion, piano, cello, guitar, trombone, harmonica, trumpet, saxophone cluster 25: chandelier cluster 26: buzzard cluster 27: hamster cluster 28: gown, robe, bra, scarf, sweater, shirt, jacket, skirt, coat, dress, necklace, bouquet, blouse cluster 29: parsley, biscuit, celery cluster 30: goose, duck, falcon, rooster, canary, turkey, swan, gorilla, penguin, tiger, zebra, fan, pajamas, pants, jeans, van, pumpkin, tuna, chicken, rocker, lamb, ox, pony, ant, flea, beetle, cape, alligator cluster 31: thermometer cluster 32: machete cluster 33: cheetah, giraffe cluster 34: wrench, corkscrew, screwdriver, escalator, tongs cluster 35: thimble, tricycle, unicycle, tripod cluster 36: avocado, lettuce cluster 37: robin, hammer, pin, pen, bolts, scissors, brush, microwave, fridge, oven, drum, bedroom, curtains, peg, football, wand, mug, pot, shoes, trousers, vest, cloak, socks, boots, tent, inn, gloves, razor, bracelet, crown, buckle, shack, bottle, sack, plate, broom, candle, cork, dresser, couch, bureau, seal, whip, cup, clock, radio, closet, jar, shelves, kettle, pan, stick, spoon, veil, whistle cluster 38: crowbar, emu, clam, drapes, zucchini, radish, turnip, clamp, projector, bookcase, spatula, grater, spinach, pickle cluster 39: paintbrush, typewriter cluster 40: cantaloupe, leotards cluster 41: dishwasher, faucet, slippers, ashtray, bathtub a.2.4 clip cluster 1: mittens, doll, rabbit, mouse, squirrel, rat, cat, hamster cluster 2: plug, lantern, kettle, mug, thimble, urn, cup, candle, pot, blender, jar, book, bucket, toaster, bin, bottle, cage, lamp cluster 3: mirror, ashtray, table, tray, mat cluster 4: chandelier, bracelet, basket, unicycle, bolts, cap, barrel, tape, drum, umbrella, shell, necklace, stool, bouquet, ring, fan, tack, drill, telephone, wheel, saddle, microscope, clock, whip, chain, rope, crown, hose cluster 5: bow, broom, shovel, spatula, spoon, ladle, tongs, crowbar, spear, fork cluster 6: keyboard, typewriter, raft, piano, escalator, comb, sink, drain, accordion cluster 7: pumpkin, bread, biscuit, worm, carrot, cheese, orange, tangerine cluster 8: trailer, rocker, bike, gun, box, train, motorcycle, projector, van, tractor, radio, bus, truck, mixer, taxi, tank, car, ambulance, jeep cluster 9: jeans, leotards, bag, blouse, skirt, sack, tie, shirt, swimsuit, trousers, pajamas, socks, dress, carpet, veil, gown, pants, sweater, curtains, apron cluster 10: peacock, raven, robin, crow, blackbird cluster 11: elk, fawn, bison, cheetah, leopard, deer, emu, hyena, moose, zebra, giraffe cluster 12: donkey, ox, cow, pony, horse, camel, sheep, goat, lamb, calf, buffalo, bull cluster 13: thermometer, bullet, stick, tripod, pencil, ruler, peg, brush, paintbrush, hoe, pin, screwdriver, baton, wand, pen cluster 14: pepper, beans cluster 15: racquet, skateboard, skis, scooter, doorknob, surfboard, guitar, board cluster 16: plum, vine, grape, raspberry, cherry, olive, cranberry, raisin, blueberry cluster 17: pig, groundhog, porcupine, fox, seal, gorilla, beaver, whale, dog, bear, elephant, raccoon, salmon, lion, tiger cluster 18: violin, trumpet, saxophone, cello, trombone, harmonica cluster 19: parakeet, bluejay, finch, mink, dove, perch, birch, canary, bat, chickadee, sparrow, woodpecker cluster 20: sleigh, cannon, buggy, sled, canoe, wheelbarrow, limousine, wagon, tricycle, cart, trolley cluster 21: shed, elevator, garage, basement, barn cluster 22: avocado, walnut, pineapple, grapefruit, coconut, marble, strawberry, pear, apple, lime, nectarine, cantaloupe, peach, willow, tomato, lemon, potato cluster 23: crane, ostrich, pelican, stork, flamingo cluster 24: rock, boots, brick, slippers, shoes, helmet, stone, ball, bomb, balloon, football, bra cluster 25: flea cluster 26: radish, turnip, parsley, beets cluster 27: bookcase, shelves, cabinet, bureau, closet, desk, dresser, fridge, freezer, stereo, cupboard cluster 28: cottage, tent, house, pyramid, door, skyscraper, card, hut, bedroom, building, inn, apartment, shack, cathedral, chapel, church, level, cabin, wall