Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language

Published on Jun 28, 2023
· Featured in Daily Papers on Jun 29, 2023


We propose LENS, a modular approach for tackling computer vision problems by leveraging the power of large language models (LLMs). Our system uses a language model to reason over outputs from a set of independent and highly descriptive vision modules that provide exhaustive information about an image. We evaluate the approach on pure computer vision settings such as zero- and few-shot object recognition, as well as on vision and language problems. LENS can be applied to any off-the-shelf LLM and we find that the LLMs with LENS perform highly competitively with much bigger and much more sophisticated systems, without any multimodal training whatsoever. We open-source our code at and provide an interactive demo.


Hey, Im reviewing deep learning papers on twitter daily in Hebrew via hashtag # So far I've shortly reviewed about deep learning papers. You are invited to follow and comment

This paper review can be found at:

There are actually a lot of large language vision models out there already.

NVM this is actually pretty novel.

No description provided.

Sign up or log in to comment

Models citing this paper 0

No model linking this paper

Cite in a model to link it from this page.

Datasets citing this paper 0

No dataset linking this paper

Cite in a dataset to link it from this page.

Spaces citing this paper 0

No Space linking this paper

Cite in a Space to link it from this page.

Collections including this paper 1