U ni F ashion: A Unified Vision-Language Model for Multimodal Fashion Retrieval and Generation (2024)

Abstract: The fashion domain encompasses a variety of real-world multimodal tasks, including multimodal retrieval and multimodal generation. The rapid advancements in artificial intelligence generated content, particularly in technologies like large language models for text generation and diffusion models for visual generation, have sparked widespread research interest in applying these multimodal models in the fashion domain. However, tasks that use embeddings, such as image-to-text or text-to-image retrieval, have been largely ignored from this perspective due to the diverse nature of the multimodal fashion domain. And current research on multi-task single models lack focus on image generation. In this work, we present UniFashion, a unified framework that simultaneously tackles the challenges of multimodal generation and retrieval tasks within the fashion domain, integrating image generation with retrieval tasks and text generation tasks. UniFashion unifies embedding and generative tasks by integrating a diffusion model and LLM, enabling controllable and high-fidelity generation. Our model significantly outperforms previous single-task state-of-the-art models across diverse fashion tasks, and can be readily adapted to manage complex vision-language tasks. This work demonstrates the potential learning synergy between multimodal generation and retrieval, offering a promising direction for future research in the fashion domain.

## introduction the fashion domain presents a range of real-world multimodal tasks, encompassing multimodal retrieval @xcite @xcite and multimodal generation @xcite tasks. such tasks have been utilized in diverse e-commerce scenarios to enhance product discoverability, seller-buyer interaction, and customer conversion rates after catalog browsing @xcite . the remarkable progress in the field of arti-ficial intelligence generated content (aigc), particularly in technologies like large language models (llms) @xcite @xcite ) for text generation and diffusion models @xcite @xcite for visual generation, yielding significant advancements in numerous downstream tasks @xcite and sparking widespread research interest in applying these multimodal models to the fashion domain. instruction-tuned multimodal large language models @xcite @xcite (mllms) have emerged as a promising direction for developing a single multi-task model @xcite . however, due to the heterogeneous nature of multimodal fashion tasks @xcite , most existing mllms struggle to be directly applicable in the fashion domain. for example, in the fashion domain, retrieval tasks that rely on embedding ability, such as imageto-text or text-to-image retrieval, have largely been overlooked. furthermore, existing mllms lack the ability to solve the composed image retrieval (cir) @xcite task, which composes the reference image and related caption in a joint embedding to calculate similarities with candidate images and is particularly relevant in recommender systems @xcite @xcite . drawing inspiration from grit @xcite , which successfully combined generative and embedding tasks into a unified model for text-centric applications and enhanced embedding performance by incorporating a generative objective, it is evident that exploring task correlations and integrating embedding with generative models in the fashion domain is promising. while previous works @xcite in the fashion domain have also proposed using a single model for solving multiple 1. a yellow t-shirt with a graphic design on the front. the t-shirt has short sleeves and a crew neckline. 2. a long-sleeved top in a soft pink or mauve color. the top features a ribbed texture throughout. a lace or embroidered detail across the chest area. ## proposed model: unifashion in this section, we introduce the unifashion to unify the fashion retrieval and generation tasks into a single model. by combining retrieval and generative modules, the proposed unifashion employs a two-stage training strategy to capture relatedness between image and language information. consequently, it can seamlessly switch between two operational modes for cross-modal tasks and composed modal tasks. ## conclusion we have introduced unifashion, a unified framework designed to tackle challenges in multimodal generation and retrieval within the fashion domain. by integrating embedding and generative tasks using a diffusion model and llm, unifashion enables controllable, high-fidelity generation, significantly outperforming previous single-task state-ofthe-art models across various fashion tasks. our model's adaptability in handling complex visionlanguage tasks demonstrates its potential to enhance e-commerce scenarios and fashion-related applications. this study highlights the importance of exploring the learning synergy between multimodal generation and retrieval, offering a promising direction for future research in the fashion domain.