Papers
arxiv:2405.17247

An Introduction to Vision-Language Modeling

Published on May 27
ยท Submitted by akhaliq on May 28
#1 Paper of the day
Authors:
,
,
,
,
,
,

Abstract

Following the recent popularity of Large Language Models (LLMs), several attempts have been made to extend them to the visual domain. From having a visual assistant that could guide us through unfamiliar environments to generative models that produce images using only a high-level text description, the vision-language model (VLM) applications will significantly impact our relationship with technology. However, there are many challenges that need to be addressed to improve the reliability of those models. While language is discrete, vision evolves in a much higher dimensional space in which concepts cannot always be easily discretized. To better understand the mechanics behind mapping vision to language, we present this introduction to VLMs which we hope will help anyone who would like to enter the field. First, we introduce what VLMs are, how they work, and how to train them. Then, we present and discuss approaches to evaluate VLMs. Although this work primarily focuses on mapping images to language, we also discuss extending VLMs to videos.

Community

Here's a simple english summary of the paper - feedback is welcome from the authors: https://www.aimodels.fyi/papers/arxiv/introduction-to-vision-language-modeling

ยท
Paper author

Thanks for the summary. Just want to highlight that we discuss biases issues in section 4.2. While the compute cost is mentioned in 3.2.2.

Vision and Text: The Future of Deep Learning Explained!

๐Ÿ‘‰ Subscribe: https://www.youtube.com/@Arxflix
๐Ÿ‘‰ Twitter: https://x.com/arxflix
๐Ÿ‘‰ LMNT (Partner): https://lmnt.com/

By Arxflix
9t4iCUHx_400x400-1.jpg

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite arxiv.org/abs/2405.17247 in a model README.md to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite arxiv.org/abs/2405.17247 in a dataset README.md to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite arxiv.org/abs/2405.17247 in a Space README.md to link it from this page.

Collections including this paper 33