On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning (2024)

Abstract: Recent advancements in language and vision assistants have showcased impressive capabilities but suffer from a lack of transparency, limiting broader research and reproducibility. While open-source models handle general image tasks effectively, they face challenges with the high computational demands of complex visually-situated text understanding. Such tasks often require increased token inputs and large vision modules to harness high-resolution information. Striking a balance between model size and data importance remains an open question. This study aims to redefine the design of vision-language models by identifying key components and creating efficient models with constrained inference costs. By strategically formulating datasets, optimizing vision modules, and enhancing supervision techniques, we achieve significant improvements in inference throughput while maintaining high performance. Extensive experiments across models ranging from 160M to 13B parameters offer insights into model optimization.We will fully open-source our codebase, models, and datasets at https://github.com/naver-ai/elva.

## introduction recent advancements in integrating large language models (llms) with computer vision have led to the creation of sophisticated language and vision assistants @xcite @xcite . these systems are capable of interpreting text within images, enabling them to excel in complex tasks requiring both visual and textual understanding. notably, models like gpt-4(v) (openai, 2023) demonstrate a range of sophisticated capabilities, including visually-situated natural language understanding (nlu) tasks, positioning them as powerful assistants. however, these models also face significant challenges related to transparency and accessibility, limiting broader utilization. @xcite , chartqa @xcite , infograph-icvqa @xcite , seed-img @xcite , seed-2-plus @xcite , @xcite , scienceqa @xcite , and hal-lusionbench @xcite . see section 4.2 for benchmark details. elva excels with high performance, reduced latency, and lower memory usage. right: performance improvements from llava to elva at the 7b scale, achieved through strategies in section 4. open-source alternatives such as llava @xcite have emerged to address these issues. however, as these models grow in complexity, concerns about their reproducibility and resource efficiency persist. some open-source models provide only the model weights without comprehensive specifications, making replication and use more challenging. in the fast-evolving realm of vision-language models (vlms), simply expanding model size and consuming more resources does not necessarily enhance practical utility. it is crucial to strike a balance between high performance and resource efficiency to democratize access to advanced vlms. particularly, inference costs are a significant concern for practitioners developing real-world applications. despite the importance of this balance, key elements contributing to vlm success are still not fully explored. traditionally, to enhance the performance, many vlms have increased their model resolution, often leading to larger and more resource-intensive models. in this work, we challenge this approach by introducing elva (efficient language and vision assistant), a suite of vlms designed to maintain high performance while reducing inference costs. while we do increase training costs to a manageable extent, the primary research target of elva is to create models capable of handling highresolution tasks with low inference costs. our key contributions are as follows: ## large language and vision assistants architecture. the llava framework (figure 2 ) employs a pre-trained vision transformer (vit) @xcite as its vision encoder. input images are resized and divided into patches of size n ˆpp h ˆpw ˆcq, where n " ph{p h q ˆpw{p w q, with h and w representing the resized image size, and p h and p w denoting the patch size, which are hyperparameters. here, c " 3 denotes the number of channels, typically for rgb images. these patches are processed by the encoder to generate embeddings tz i p r d u, where d is the embedding dimension of the encoder and also a hyperparameter. these embeddings are then mapped to the input space of the language model via a multi-layer perceptron (mlp) before being fed into the model. optionally, the anyres mechanism @xcite ) can be applied, allowing for processing of larger images by first segmenting images into m parts to better capture local features. simultaneously, the entire image is processed to extract global features. both global and local features are sequentially fed into the language model, allowing it to utilize comprehensive image information. therefore, the total token count becomes p1`mqˆn. see @xcite for more details. however, this approach poses a computational challenge due to the increased token count. training objectives and datasets. the model is trained to minimize cross-entropy (ce) loss. during pre-training (alignment phase), it generates captions for images, with ce loss computed on the text. in the instruct tuning stage, given an image, question, and answer, the loss is computed on the answer text. the llava-1.5 dataset @xcite is widely used and this study aims to further enhance the dataset. more details are in section 4. ## efficiency challenges in llava models this section addresses common overhead issues in llava models, identifying critical limitations and defining the problem space for future work @xcite . ## experimental assessment in this section, we rigorously test and evaluate our elva models under varying conditions. we aim to understand their capabilities and limitations by benchmarking them against baseline models across both text-based and image-based tasks. ## conclusion this study introduces elva, a robust and efficient model framework for diverse multimodal tasks, including visually-situated nlu. empirical results demonstrate that elva surpasses existing baselines, with notable memory and latency efficiency. comprehensive experiments and analyses identify key components driving elva's enhanced performance. additionally, our analysis highlights both strengths and limitations, offering insights for further development. we envision our approach extending to other domains and tasks, particularly those requiring high-resolution and visually-situated nlu, even in resource-constrained environments.