LAVIS : A One-stop Library for Language-Vision Intelligence (2023)

Abstract: We introduce LAVIS, an open-source deep learning library for LAnguage-VISion research and applications. LAVIS aims to serve as a one-stop comprehensive library that brings recent advancements in the language-vision field accessible for researchers and practitioners, as well as fertilizing future research and development. It features a unified interface to easily access state-of-the-art image-language, video-language models and common datasets. LAVIS supports training, evaluation and benchmarking on a rich variety of tasks, including multimodal classification, retrieval, captioning, visual question answering, dialogue and pre-training. In the meantime, the library is also highly extensible and configurable, facilitating future development and customization. In this technical report, we describe design principles, key components and functionalities of the library, and also present benchmarking results across common language-vision tasks.

## introduction multimodal content, in particular language-vision data including texts, images and videos are ubiquitous for real-world applications, such as content recommendation, e-commerce and entertainment. there has been tremendous recent progress in developing powerful language-vision models @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite @xcite . however, training and evaluating these models across tasks and datasets require domain knowledge and are not always welcoming to incoming researchers and practitioners. this is mainly due to inconsistent interfaces across models, datasets and task evaluations, and also the duplicating yet albef blip blip2 clip lavis.models lavis.models lavis.models lavis.tasks pre-train retrieval captioning multimodal classification vqa/videoqa multimodal dialogue lavis.common optimization registry distribution utils configs logging build build build cli entry point instruct-blip blip-diffusion figure 1: overall architecture of the lavis library. figure 1 shows the overall design of lavis. important features of lavis include (i) unified interface and modular design. key components in the library are organized using a unified and modular design. this allows effortless off-the-shelf access to individual components, swift development and easy integration of new or external components. the modular design also eases model inferences, such as multimodal feature extraction. (ii) comprehensive support of image-text, video-text tasks and datasets. lavis supports a growing list of more than ten common language-vision tasks, across over 20 public datasets. these tasks and datasets provide a comprehensive and unified benchmark for evaluating language-vision models. (iii) state-of-the-art and reproducible languagevision models. the library enables access to over 30 pre-trained and task-specific fine-tuned model checkpoints of 5 foundation models: albef @xcite , blip @xcite , blip2 @xcite , clip @xcite and al-pro @xcite , as well as state-of-the-art language-vision methods such as pnp-vqa @xcite , img2prompt @xcite . these models achieve competitive performance across multiple tasks, representing the up-to-date development status of the language-vision research. we also provide training, evaluation scripts and configurations to facilitate reproducible language-vision research and adoption. (iv) resourceful and useful toolkit. in addition to the core library functionalities, we also provide useful resources to reduce the learning barriers for the language-vision research. this includes automatic dataset downloading tools to help prepare the supported datasets, a gui dataset browser to help preview downloaded datasets and dataset cards documenting sources, supported tasks and leaderboards. ## related work other open-source implementations of individual models exist @xcite @xcite @xcite , yet do not provide centralized access. in summary, in contrast to previous efforts, our library stands out by providing easier access to stronger models on comprehensively many tasks and datasets. with this effort, we hope to significantly reduce the cost and effort to leverage and benchmark existing multimodal models, as well as to develop new models. ## library design this section delineates the design of lavis as shown in figure 1 . our key design principle is to provide a simple and unified library to easily (i) train and evaluate the model; (ii) access supported models and datasets; (iii) extend with new models, tasks and datasets. ## benchmarks and library toolkit in this section, we benchmark model performance across tasks and datasets in lavis. then we take our web demo interface to show a few case studies on multimodal content understanding. we also present a gui dataset browser that helps to preview supported datasets. ## conclusion and future work we present lavis, an open-source deep learning library for language-vision research and applications. the library is designed to provide researchers and practitioners with easier and comprehensive access to state-of-the-art multimodal capabilities, the library also features a unified interface and extensible design to promote future development. besides, the library also features extensive access to pre-trained weights and useful resources to reduce duplicating replication efforts. with these features, we expect lavis to serve as a one-stop library in multimodal ai for a wider audience. we continue to actively develop and improve lavis. in future releases, our priorities are to include more language-vision models, tasks and datasets to the library. we also plan to add more parallelism support for scalable training and inference. while we will maintain lavis in the long term, we invite contributions from the open-source community to join this evolving effort.