W iki- VEL : Visual Entity Linking for Structured Data on Wikimedia Commons (2024)

Abstract: Describing Wikimedia Commons images using Wikidata’s structured data enables a wide range of automation tasks, such as search and organization, as well as downstream tasks, such as labeling images or training machine learning models. However, there is currently a lack of structured data-labelled images on Wikimedia Commons.To close this gap, we propose the task of Visual Entity Linking (VEL) for Wikimedia Commons , in which we create new labels for Wikimedia Commons images from Wikidata items. VEL is a crucial tool for improving information retrieval, search, content understanding, cross-modal applications, and various machine-learning tasks. In this paper, we propose a method to create new labels for Wikimedia Commons images from Wikidata items. To this end, we create a novel dataset leveraging community-created structured data on Wikimedia Commons and fine-tuning pre-trained models based on the CLIP architecture. Although the best-performing models show promising results, the study also identifies key challenges of the data and the task.

## introduction wikimedia commons is a service that hosts around 100 million community-contributed, openly licensed images and media files, including metadata, multilingual textual descriptions, and categories similar to wikipedia categories. at the same time, wikimedia's knowledge graph (kg), wikidata, offers detailed structured knowledge descriptions of over 100 million entities. in 2017, the commons: structured data project was initiated to organize and search images by better connecting the two efforts. community members tag relevant wikidata items in images, adding them to commons as structured data via new depict statements, enabling machine-friendly association of images with universal, language-independent concepts. in wikimedia commons, structured data unlocks the full potential of its image repository, providing users with a more enriching and productive experience. yet, as of november 2023, only 15% of wikimedia commons images are accompanied by structured data, suggesting that a considerable portion of this vast resource remains unexplored. this lack of structured data poses a challenge for users seeking to extract meaningful information from the extensive collection. structured data is crucial for modern information retrieval systems, providing a systematic framework for describing entities and their attributes, and enhancing discoverability and interoperability across platforms and applications. this gap in the coverage of commons image annotations can be addressed by automatically suggesting depicted items using visual entity linking (vel), a multi-modal task of linking visual items in an image with corresponding entities in a kg. this paper proposes the wiki-vel framework, applying the task of vel to wikimedia commons using the structured data of wikidata. this allows users to perform targeted searches and explore images based on specific topics, events, or attributes, enhancing the usability and utility of wikimedia commons. further, integrating vel on wikimedia commons opens opportunities for automation and innovation in content management and analysis. images annotated with structured data can be used for visual question answering, search algorithms, image classification, object detection, semantic segmentation, and recommender systems (de @xcite @xcite . our contributions are as follows: ## dataset on wikimedia commons, the community contributes meta-data for the uploaded images. this includes descriptions and licensing information as well as structured data in the form of wikidata items. in our work, we focus on the structured data describing entities in the images. to express this relation, the property depicts is used on wikimedia commons. ## experimental setup in the following, we describe the clip variants used in the proposed wiki-vel framework to link the wikicommons images to wikidata entities. ## conclusion in this paper, we propose the wiki-vel framework, linking the items portrayed in images with structured knowledge stemming from wikidata. we create a dataset from community-contributed, openlicensed wikimedia commons images labeled with the depicted entities in the form of wikidata entities. in our vel experiments, we show that the naive clip model shows promising performance by outperforming the zero-shot model and simple baselines. the performance of the clip fusion model also improved with more input data. however, all setups reached a plateau in learning due to the noisy real-world data. in our human evaluation, we show that the data quality also affects the metrics to evaluate model performance -humans perceive the model to be correct more than the automated metrics. looking towards the future, our results are promising for automatically providing structured labels for wikimedia commons images. to realise this vision, the wikimedia community could participate in a large-scale human evaluation to assess the integration of the model into commons to support contributors on image uploads and achieve the desired benefits from the structured data project. further, the dataset can easily be extended to a multilingual dataset by extracting the image description and item names in different languages from structured data.