T ro L : Traversal of Layers for Large Language and Vision Models (2024)

Abstract: Large language and vision models (LLVMs) have been driven by the generalization power of large language models (LLMs) and the advent of visual instruction tuning. Along with scaling them up directly, these models enable LLVMs to showcase powerful vision language (VL) performances by covering diverse tasks via natural language instructions. However, existing open-source LLVMs that perform comparably to closed-source LLVMs such as GPT-4V are often considered too large (e.g., 26B, 34B, and 110B parameters), having a larger number of layers. These large models demand costly, high-end resources for both training and inference. To address this issue, we present a new efficient LLVM family with 1.8B, 3.8B, and 7B LLM model sizes, Traversal of Layers (TroL), which enables the reuse of layers in a token-wise manner. This layer traversing technique simulates the effect of looking back and retracing the answering stream while increasing the number of forward propagation layers without physically adding more layers. We demonstrate that TroL employs a simple layer traversing approach yet efficiently outperforms the open-source LLVMs with larger model sizes and rivals the performances of the closed-source LLVMs with substantial sizes.

## introduction the great success of closed-source large language and vision models (llvms) such as gpt-4v @xcite , gemini-pro @xcite , and qwen-vl-plus @xcite has prompted the world to build open-source llvms using open-source large language models (llms) and visual instruction tuning @xcite . these movements are expected to contribute significantly to both research and industry in order to bootstrap their vision language performance, several studies have curated high-quality visual instruction tuning datasets @xcite ) by leveraging the power of closed-source llvms. further, they have physically increased the model sizes @xcite @xcite to enhance their capability to understand complex question-answer pairs. following these efforts, there has been a recent rise in techniques for employing additional modules: mixed vision encoders @xcite @xcite @xcite , multiple computer vision models @xcite @xcite , and mixtures of experts (moe) for efficiently scaling llvms to fulfill specific purposes @xcite @xcite @xcite @xcite . in the end, llvms combined with these techniques have shown improved vision language performances, achieving results comparable to directly scaled-up llvms with 26b, 34b, and 110b model sizes. additionally, some approaches have demonstrated performances that surpass those of the closed-source llvms. however, directly scaling the model size up or using additional modules may not be considered a fundamental solution to enlarging learning capabilities regarding complex question-answer pairs. this is because they physically add a considerable number of training parameters or borrow richer knowledge from external modules. in other words, it remains unexplored how llvms with smaller model sizes can effectively enhance learning capabilities despite their inherent physical limitations. to address this, simply increasing the image resolution size @xcite @xcite @xcite and dynamically dividing images into sub-parts for hierarchical focus @xcite @xcite ) may be good candidates to achieve our purpose without employing any additional modules mentioned. nonetheless, these strategies are mostly intended to embed rich image information to further improve overall and finegrained image understanding. hence, we need to focus on how multimodal llms can be enhanced by themselves. once more efficient llms that perform comparably to closed-source models are released publicly, it will mitigate the necessity for high-end gpus and accelerate significant advancements in various downstream applications, including on-device processing. therefore, we present a new efficient llvm family with 1.8b, 3.8b, and 7b model sizes, traversal of layers ( trol), which enables the reuse of layers in a token-wise manner. to overcome the inherent limitations of smaller-sized llvms, we opt to increase the number of forward propagations rather than physically adding more layers, as is normally done in scaled-up llvms. this technique, which we call layer traversing, allows llvms to retrace and re-examine the answering stream, akin to human retrospection and careful thought before responding with an answer. figure 1 represents how the layer traversing technique is practically implemented in trol-layer, where trol-mixer serves as the token-wise mixing operation under lightweight additional parameters: 49k, 98k, and 131k in total layers. this is a significantly tiny number compared with the 1.8b, 3.8b, and 7b model sizes. trol. the first step involves training a vision projector and all trol-mixers for each trol-layer. this is a crucial step because it not only aligns vision and language information but also tunes the trol-mixers with the answering stream in backbone multimodal llms, thereby facilitating the use of layer traversing. the second training step includes further training of these components along with the backbone multimodal llms. to achieve efficient training, we use q-lora @xcite training for the backbone multimodal llms under 4/8-bit quantization. in conducting the two-step training, we demonstrate that trol is an efficient model, yet it outperforms open-source llvms with larger model sizes (e.g., 26b, 34b, 72b, and 110b) and closedsource llvms with a substantially vast amount of parameters, as illustrated in figures 2 and 3 . trol proves its superior effectiveness on various evaluation benchmarks compared with substantially sized open-and closed-source llvms without directly scaling up the model size and without any additional modules. additionally, recent research has explored more direct approaches to enhance llvm's image perception capabilities, either by modifying visual input and utilizing additional modules. for example, qwen-vl @xcite , cogvlm @xcite , and mplug family @xcite increase image resolution to enrich visual information in llvms. moreover, few approaches @xcite @xcite improve visual tokens through image partitioning. additionally, the integration of additional vision encoders @xcite @xcite @xcite and external computer vision modules @xcite @xcite ) augment llvms' image perception capability, thereby enhancing overall performance on multimodal tasks. while these lines of works expand the overall learning capabilities of llvms, they do not necessarily enhance the fundamental capacity of llvms. consequently, there remains a need for further research into the intrinsic mechanisms of llvms without scaling models or leveraging additional modules. thus, we introduce a new llvm family, trol, specifically tailored to enhance the learning capabilities of llvms, where layer traversing is presented to efficiently reuse layers. this approach simulates the effect of retracing the answering stream, offering a focused solution to propel the advancement of llvms. numerous efficient approaches. despite the remarkable achievements of llvms in a short period of time, closed-source llvms require a significant number of parameters and resources. for that reason, many llvms have been exploring methods to create more efficient models. these methods aim to reduce the number of parameters or improve computational efficiency without significantly compromising performance. numerous methods to reducing the number of parameters involve sharing specific weights within the model @xcite @xcite , eliminating weights that contribute less to the performance @xcite @xcite @xcite , and employing quantization techniques @xcite @xcite . these efforts primarily focus on reducing inference time or accelerating training speed while maintaining performance, rather than fundamentally improving it. in contrast, trol aims to efficiently enhance the learning capabilities of llvms in understanding complex question-answer pairs. ## discussion and conclusion a new llvm family, trol, has demonstrated significant advancements in vision language per-formance despite its inherent limitation of having smaller layers compared to larger model sizes in both open-and closed-source llvms.