Dreaming Out Loud: A Self-Synthesis Approach For Training Vision-Language Models With Developmentally Plausible Data (2024)

Abstract: While today’s large language models exhibit impressive abilities in generating human-like text, they require massive amounts of data during training. We here take inspiration from human cognitive development to train models in limited data conditions. Specifically we present a self-synthesis approach that iterates through four phases: Phase 1 sets up fundamental language abilities, training the model from scratch on a small corpus. Language is then associated with the visual environment in phase 2, integrating the model with a vision encoder to generate descriptive captions from labeled images. In the “self-synthesis” phase 3, the model generates captions for unlabeled images, that it then uses to further train its language component with a mix of synthetic, and previous real-world text. This phase is meant to expand the model’s linguistic repertoire, similar to humans self-annotating new experiences. Finally, phase 4 develops advanced cognitive skills, by training the model on specific tasks such as visual question answering and reasoning. Our approach offers a proof of concept for training a multimodal model using a developmentally plausible amount of data.

## introduction recent advances in machine learning have produced large language models (llms) that, after training on massive text corpora, are capable of generating human-like text. however, when comparing llm training to human development, the amount of data required for successful model training far exceeds the quantities that humans learn from during their development @xcite . the human brain is thus often seen as a more sampleefficient learning machine than contemporary artificial neural network approaches (marcus, 2020). ## dataset selection in line with the babylm challenge requirements @xcite , we restrict our training data to 100 million words, which approximates the maximum number of words a 13-year-old would encounter in their lifetime @xcite . in contrast, the latest llama-3-8b model was trained on 15 trillion tokens @xcite , which is 150,000 times larger than our training budget. we created our own dataset of 100 million words, emphasizing diversity and quality. this word budget is split evenly between a textonly corpus and a multimodal image-text corpus. text-only data our text corpus comprises 50 million words selected from the top-scoring sentences of fineweb-edu's october 2024 common-crawl snapshot @xcite , based on their educational quality. fineweb-edu is a subset of the fineweb dataset @xcite , which is created using scalable, automated annotations to assess educational value. the educational scores were assigned by llama-3-70b-instruct, which rated 500,000 samples on a scale from 0 to 5 for their educational quality @xcite . models trained on this dataset have surpassed all other publicly available web datasets on several educational benchmarks, including mmlu @xcite , arc @xcite , and openbookqa @xcite . image-text data our image-text corpus consists of two groups: (1) image-caption data used for visual experience training ("phase 3" section 5.3); ## benchmarks we evaluate our model across six benchmarks: three focused on language-only tasks and three on vision-language tasks. except for glue, where we fine-tune the model on each subtask using lora @xcite , all benchmarks are evaluated in a zero-shot setting. ## model details we use the same model architecture provided by the babylm challenge organizers, called babyllama, which consists of a reduced llama architecture, and we equip it with the dinov2 large vision encoder to be able to handle visual inputs. tokenizer we train a bpe tokenizer with a vocabulary size of 16,000 on the text data from the curated dataset described in section 2. language model the language models employs the same architectural components as the llama model @xcite , but with only 16 transformer layers and a reduced hidden dimension size of 512. the intermediate size in the mlp is 1,024, resulting in a total model size of 58 million parameters. the attention mechanism uses 8 attention heads in each layer. ## self-synthesis training phases our framework trains the model in four phases. in each phase, we record the model checkpoint with the lowest validation loss and use it as a starting point for the following phase. for all phases, we use the adamw optimizer combined with a cosine learning rate scheduler and a batch-size of 256. the learning rate begins with a linear warm-up phase and then gradually decreases to zero over the course of the training. ## results