VIT -Pro: Visual Instruction Tuning for Product Images (2025)

Abstract: General vision-language models (VLMs) trained on web data struggle to understand and converse about real-world e-commerce product images. We propose a cost-efficient approach for collecting training data to train a generative VLM for e-commerce product images. The key idea is to leverage large-scale, loosely-coupled image-text pairs from e-commerce stores, use a pretrained LLM to generate multimodal instruction-following data, and fine-tune a general vision-language model using LoRA. Our instruction-finetuned model, VIT-Pro, can understand and respond to queries about product images, covering diverse concepts and tasks. VIT-Pro outperforms several general-purpose VLMs on multiple vision tasks in the e-commerce domain.

## introduction the e-commerce domain inherently operates at the intersection of visual and textual data. from highresolution product images and packaging photos to detailed customer feedbacks provided during return/refund claims, the interplay between these modalities is central to ensuring smooth operations and customer satisfaction. this multi-modal nature of data is pivotal in scenarios like verifying product authenticity, monitoring quality control, and resolving customer grievances effectively. however, the sheer volume of such data, generated across stages of the logistics chain-packaging, shipping, delivery, and post-delivery-poses a significant challenge. efficiently leveraging this wealth of multi-modal information is critical for scaling operations while maintaining accuracy and customer trust. currently, manual investigations to address multi-modal customer queries, such as verifying product quality and delivery issues, are the standard practice but lack scalability. automating these investigations requires robust multi-modal systems that can precisely analyze visual and textual data together. while general-purpose vision-language models (vlms) are proficient in handling diverse tasks, they often lack the nuanced understanding required for domain-specific applications in the e-commerce sector. these applications include accurately recognizing and differentiating between similar products among a vast collection, extracting specific product attributes from images and descriptions, understanding product compatibility and accessorizing requirements, and assessing product quality and detecting damages or defects based on images. an additional challenge arises from real-world ("inthe-wild") images, as most images (apart from catalog images) are non-standard, with varying viewpoints, partially visible regions, occluded parts, and poor quality. to address these challenges, ecommerce stores may need to develop specialized vlms tailored to their specific domains. however, the development of such systems is hindered by the unavailability of domain-specific multi-modal datasets. addressing this data bottleneck is crucial to enabling automation at scale. to bridge this gap, we propose a scalable framework for curating multi-modal instructionfollowing datasets tailored to the e-commerce domain (illustrated in figure 1 ). this approach leverages readily available customer feedbacks, product catalog and associated images to transform them into rich instruction-following dataset using a pretrained llm. to ensure quality, we employ robust cleaning techniques, including attention-guided data validation, to filter irrelevant or noisy samples. the curated dataset facilitates the fine-tuning of vision-language models, equipping them with e-commerce-specific capabilities. our work makes the following key contributions: following data: we introduce a novel data generation strategy that transforms weakly associated image-text pairs from existing sources into a high-quality, multi-modal instruction-following dataset. this dataset, comprising 1.4m unique samples across diverse e-commerce tasks, is generated without manual annotation efforts. ## related work vision-language modelling for e-commerce has been studied and experimented in several existing works @xcite @xcite . however, most of these works are targeted towards visual question answering tasks for attribute extraction, catalog quality improvement, etc. using high-quality product catalog images and texts. consequently, these datasets and models are not scalable to other challenging tasks in the e-commerce domain involving in-the-wild product images (as discussed in section 1). compared to these existing works, ours is a pioneering attempt towards building a e-commerce domain specific vlm that can answer open questions in the wild on real-world images and tasks applicable at various stages in the product order life cycle. more recently, visual instruction tuning has proven to be a promising approach to enable models to follow diverse user instructions involving visual content. several open-sourced instruction-tuned models, including instructblip @xcite , llava @xcite , idefics2 @xcite , qwen-vl @xcite and larger propriety models, such as claudev3 @xcite , gpt4v (openai, 2023), achieve competitive performance on real-world tasks (visit-bench @xcite ). however, their zero-shot performance on e-commerce domain remains limited, because the e-commerce image-text pairs and tasks are drastically different from public datasets. our work, inspired by these existing works, extends multi-modal instruction-tuning to the e-commerce domain by efficiently constructing an llm-assisted instruction-following data from disparate sources for building a multi-task vlm with extended domain-specific capabilities. ## vision-language instruction tuning foundational vlms for e-commerce require domain-specific capabilities such as fine-grained product recognition, product counting, localization, attribute extraction, damage/defect detection, comparison of visually similar products, visual reasoning, multi-modal decisioning, etc. to impart this knowledge and instruction-following ability, we require training on a rich multi-task dataset meticulously crafted with distinct instructions and responses. to achieve this, we present techniques from both data and model perspective in this section. ## conclusions we showcased the potential of leveraging largescale weakly-associated image-text pairs commonly available in any e-commerce stores to build a multi-task vision-language model for e-commerce domain. vit-pro, demonstrates superior performance over open-source and commercial baselines on an internal e-commerce vision-language benchmark. comprehensive analyses highlight vit-pro's robustness under varying input configurations like resolutions, ocr, multi-image scenarios, optimization strategies and lora adapters. in future, we want to incorporate other data sources (e.g. x-rays) and tasks (e.g. product grading). ramakrishna vedantam, c. lawrence zitnick, and devi parikh. 2015. cider: consensus-based image description evaluation. preprint, arxiv:1411.5726. jesse vig and yonatan belinkov. 2019. analyzing the structure of attention in a transformer language model. preprint, arxiv:1906.04284. xiaohua zhai, basil mustafa, alexander kolesnikov, and lucas beyer. 2023. sigmoid loss for language image pre-training. preprint, arxiv:2303.15343.