Light Up the Shadows: Enhance Long-Tailed Entity Grounding with Concept-Guided Vision-Language Models (2024)

Abstract: Multi-Modal Knowledge Graphs (MMKGs) have proven valuable for various downstream tasks. However, scaling them up is challenging because building large-scale MMKGs often introduces mismatched images (i.e., noise). Most entities in KGs belong to the long tail, meaning there are few images of them available online. This scarcity makes it difficult to determine whether a found image matches the entity. To address this, we draw on the Triangle of Reference Theory and suggest enhancing vision-language models with concept guidance. Specifically, we introduce COG, a two-stage framework with COncept-Guided vision-language models. The framework comprises a Concept Integration module, which effectively identifies image-text pairs of long-tailed entities, and an Evidence Fusion module, which offers explainability and enables human verification. To demonstrate the effectiveness of COG, we create a dataset of 25k image-text pairs of long-tailed entities. Our comprehensive experiments show that COG not only improves the accuracy of recognizing long-tailed image-text pairs compared to baselines but also offers flexibility and explainability.

## introduction multi-modal knowledge graphs (mmkgs) are knowledge graphs that integrate and align information from diverse modalities (e.g., text and images) @xcite @xcite . due to the growing demand for multimodal intelligence and extensive knowledge in various applications @xcite , mmkgs have received increasing attention in recent years. although the number of images in current mmkgs has increased, their coverage and accuracy are limited @xcite ; wang @xcite and add human annotations. the blue line represents the changes in the entities' viewtimes, which indicates their click frequency. the red dots indicate the number of correctly matched images found in the top 20 search results for each entity, and the red line smooths out these data points. et al., 2020), especially for long-tailed entities (i.e., less common entities). figure 1 illustrates that the trends in an entity's click frequency and the number of matching images found are similar, both displaying a long-tailed pattern, indicating the scarcity of images of long-tailed entities. aligning long-tailed entities with appropriate images (i.e., entity grounding) is crucial in constructing mmkgs. first, it expands the scope and completeness of mmkgs, which traditionally focus on common entities @xcite . second, adding visual content for long-tailed entities boosts efficiency in downstream tasks @xcite . third, pairing images with long-tailed entities provide valuable training data for developing and refining vision-language models, particularly for rare or domain-specific entities. grounding long-tailed entities in mmkgs is challenging. current methods @xcite @xcite for entity grounding rely on web resources, especially search engines. they collect images by matching image captions to entity names. while effective for well-known entities @xcite , these approaches struggle with longtailed entities due to several limitations: 1) search engines used for text matching show suboptimal performance because entity grounding involves matching images and texts; 2) although pre-trained vision-language models (pvlms) have shown impressive performance in various cross-modal tasks, they encounter challenges in recognizing longtailed entities due to their infrequent appearance during pre-training; 3) current methods lack the ability to explain their image selection, which is important for ensuring quality. to address these challenges, we develop cog, a two-stage framework that uses concept-guided vision-language models to ground long-tailed entities. initially, as shown in figure 2 , the triangle of reference theory @xcite explains the connection between thought, symbol, and referent. this demonstrates how humans link an entity to a real-world object through concepts. inspired by this theory, we improve pvlms to accurately identify images of long-tailed entities using concepts. notably, our cog allows for the use of replaceable pvlms, meaning it can integrate any module for matching images and texts, thus offering flexibility in application. next, we investigate how the choice of concept selection strategy affects the performance of cog, considering the varying scope of concepts. lastly, cog not only enhances recognition accuracy but also ensures explainability, providing a basis for further quality control (i.e., human verification). to sum up, our contributions are as follows: ## conclusion we propose a two-stage framework cog, using pvlms with concept guidance to ground longtailed entities in mmkgs effectively. our experimental results demonstrate that cog greatly improves the ability of pvlms to recognize imagetext pairs of long-tailed entities. furthermore, recognizing unfamiliar entities through concepts is convincing and provides clear evidence for human verification, suggesting a future direction for better handling long-tailed entity grounding.