Ameli: Enhancing Multimodal Entity Linking with Fine-Grained Attributes (2024)

Abstract: We propose attribute-aware multimodal entity linking, where the input consists of a mention described with a text paragraph and images, and the goal is to predict the corresponding target entity from a multimodal knowledge base (KB) where each entity is also accompanied by a text description, visual images, and a collection of attributes that present the meta-information of the entity in a structured format. To facilitate this research endeavor, we construct Ameli, encompassing a new multimodal entity linking benchmark dataset that contains 16,735 mentions described in text and associated with 30,472 images, and a multimodal knowledge base that covers 34,690 entities along with 177,873 entity images and 798,216 attributes. To establish baseline performance on Ameli, we experiment with several state-of-the-art architectures for multimodal entity linking and further propose a new approach that incorporates attributes of entities into disambiguation. Experimental results and extensive qualitative analysis demonstrate that extracting and understanding the attributes of mentions from their text descriptions and visual images play a vital role in multimodal entity linking. To the best of our knowledge, we are the first to integrate attributes in the multimodal entity linking task. The programs, model checkpoints, and the dataset are publicly available at https://github.com/VT-NLP/Ameli.

## introduction entity linking aims to disambiguate and link entity mentions within a text to their corresponding entities in knowledge bases. while earlier research @xcite @xcite @xcite @xcite predominantly focus on linking entities based on text, recent studies have started to extend it to multi-modality where both mentions and entities in knowledge bases are described with text and visual images @xcite @xcite @xcite @xcite . however, all these studies view each entity in the knowledge base as an atomic symbol while ignoring the meta-information, such as various attributes of each entity, which, we argue, is especially important in disambiguating entities in a multimodal context. in this work, we focus on multimodal entity linking (mel) which requires understanding finegrained attributes of mentions from both text and images and linking them to the corresponding entities in a target multimodal knowledge base where each entity is also illustrated with text, images, and a set of attributes. figure 1 shows an example where each entity, such as asus rog laptop -white in the target knowledge base is described with a set of attributes, such as screen size, system memory, graphics, and to disambiguate and link a particular mention, e.g., asus laptop to the target entity, we need to carefully detect the attributes of the mention from its text and image descriptions and compare it against each entity. such attributeaware multimodal entity linking is critical to ecommerce domains, e.g., analyzing user opinions from social media posts about particular products. yet, it is relatively less studied in the entity linking literature. to support research toward attribute-aware multimodal entity linking, we introduce ameli, which consists of (1) a multimodal knowledge base that includes 34,690 product entities collected from the best buy 2 website and each entity is described with a product name, a product description, a set of attribute categories and values, e.g., "color: black", and several images; and (2) a multimodal entity linking benchmark dataset that contains 16,735 figure 1 : an example for our attribute-aware multimodal entity linking. left: review text and image; right: product title, image, description, and attributes. to link the mention asus laptop to the target entity, we need to be aware of the attributes, e.g., memory and ssd capacity, and image features, e.g., color. we conduct baseline experiments with several entity linking methods and propose a new framework consisting of a natural language inference (nli) based text disambiguation model to compare the mention description and attributes of candidate entities from the knowledge base and an image disambiguation model based on contrastive learning. though our proposed approach significantly outperforms all the strong baselines, the experimental results still show a large gap between machine (51.54% f-score) and human performance (74.0% f-score). the contributions of this work can be summarized as follows: ## related work previous research on textual entity linking has established various benchmark datasets @xcite @xcite @xcite and state-of-the-art neural models @xcite @xcite @xcite @xcite @xcite . however, these approaches cannot be directly adapted to multimodal entity linking due to the fundamental differences in input modalities and challenges. multimodal entity linking has recently been explored in various contexts such as social networks @xcite @xcite , domain-specific videos @xcite and general news domains @xcite . these studies focus on reducing noise in the abundant visual input of social networks @xcite , learning distinguishable entity representations by contrastive learning @xcite @xcite , or directly generating target entity names @xcite . compared to these studies, our research considers the unique attributes along with visual and textual inputs. many studies have been proposed to extract attribute values of products from their titles and descriptions by formalizing it as a sequence tagging task @xcite @xcite or a question-answer problem @xcite . several recent studies @xcite @xcite incorporate visual clues, such as product images or visual objects, into textual descriptions and extract attribute values based on their fused representations. in this study, we explore the potential of leveraging attribute values extracted from noisy user reviews to improve multimodal entity linking and achieve this by implicitly inferring attribute values through natural language inference (nli). ## dataset construction data source our goal is to build (1) a multimodal knowledge base where each entity is described with text, images, and attributes, and (2) an entity linking benchmark dataset where each mention in a given context is also associated with text and several images and can be linked to a specific entity in the multimodal knowledge base. to construct these two benchmark resources, we use best buy foot_2 , a popular retailer website for electronics such as computers, cell phones, appliances, toys, etc., given that it consists of both multimodal product descriptions organized in a standard format and user reviews in both text and/or images which can be further used to build the entity linking dataset. as shown in figure 1 , each product in best buy is described with a product name, a list of product categories, a product description, a set of attribute categories and values as well as several images foot_3 . additionally, users can post reviews in text and/or images under each product, while each review can be rated as helpful or unhelpful by other users. we develop scripts based on requests foot_4 to collect all the above information. each product webpage also requires a button click to display the attributes, so we further utilize selenium foot_5 to mimic the button click and collect all the attributes and values for each product. in this way, we collect 38,329 product entities and 6,500,078 corresponding reviews. data preprocessing many reviews are not suitable for the multimodal entity linking task for various reasons. considering this, we designed several rules to preprocess the collected reviews: (1) remove reviews and products without images; (2) remove reviews with more than 500 tokens, since most of the state-of-the-art pretrained language models can only deal with 512 tokens; (3) remove a review if it is only labeled as "unhelpful" by other users since we observe that these reviews usually do not provide much meaningful information; (4) validate the links between reviews and their corresponding products and remove the invalid links. there are invalid links because best buy links each review to all variants of the target product. for example, for the review of asus laptop shown in figure 1 , the target product asus rog laptop -white has several other variants in terms of color, memory size, processor model, etc., while best buy links the review to all variants of the target product. since we take each product variant as an entity in our multimodal knowledge base, we detect valid links between reviews and product variants based on a field named productdetails, which reveals the gold target product variant information of the review in best buy's search response. after obtaining the valid link for each review, we remove invalid links between this review and all other products. (5) remove truncated images uploaded by users as these images cause "truncated image error" during loading with standard image tools such as pillow 7 . (6) remove reviews containing profanity words based on the block word list provided by free web header 8 . (7) review images can also contain irrelevant objects or information; for example, a review image for a fridge can also contain much information on the kitchen. we apply the object detection model @xcite to detect the corresponding object using the entity name as prompt and save the detected image patch as the cleaned review image. we remove an image if the entity object can not be detected from it. both original images and cleaned images are included in our dataset. (8) we also notice that many reviews do not contain enough context information from the text and images to link the product mention to the target product entity correctly. for example, in figure 2 , the target product is a canon camera. however, the review image does not show the camera itself, and the review text does not contain any specific information about the camera. to ensure the quality of the entity linking dataset, we further design a validation approach (explained in appendix a) to filter out reviews that do not contain enough context information. ## remaining challenges we randomly sample 50 reviews linked to incorrect entities under the system attribute setting from the test and identify the following key challenges for the entity disambiguation task 16 . attribute extraction: user reviews often contain informal language, idiomatic expressions, and diverse writing styles. this linguistic variability makes it challenging to accurately extract specific attribute values as different users might use other terms to describe the same attribute. furthermore, our knowledge base encompasses over 30,000 attribute values. determining the attribute referenced within a given review poses a challenging inference task. for 10% of the errors, our method fails to extract some key attributes. for example, given review #1 "plus their are 10 programmable buttons and rated to 50 million clicks with omron swicthes now you can't beat that." in figure 7 in appendix g, we can distinguish the gold entity with 10 buttons from the candidate entity with 17 buttons after extracting the attribute "number of buttons (total): 10". recognizing brand logos or integrating a better ocr model to detect text within images will also increase the quality of system attribute, as shown in review #2. more analysis on attribute extraction module is detailed in appendix h. reasoning over attributes: 18% of the errors can be fixed if the model pays more attention to appropriate attributes or conducts reasoning based on the attribute. for example, the review #3 in figure 7 in appendix g claims " i bought this because you can use it on your phone too ". as a result, we can skip the candidate entity with the attribute "compatible platform(s): windows, mac, playstation 4, playstation 5" since it does not support phones. in some cases, system attribute contains the key review attributes to distinguish the gold entity from the candidate entity. however, the model is fed with abundant multimodal context and fails to focus on the distinguishable attribute. for example, in review #4, the model fails to take care of the attribute "carafe capacity". fine-grained image matching: in 32% of the errors, the gold entity and candidate entity can be distinguished based on fine-grained image texture. for example, in review #5 in figure 7 in appendix g, the delicate pattern in the computer case acts as the main hint to link to the gold entity. since these inconspicuous patterns can be pretty elusive to spot, visual attributes will be helpful to guide attention in some cases. for example, in review #6, the difference between the gold and candidate entities is whether there is an ice and water dispenser on the fridge surface. with the visual attribute "ice and water dispenser location: external," the model can focus on the image patch on which the ice and water dispenser is normally located. candidate retrieval: 26% of the disambiguation errors are due to the gold entity not being in the top 10 retrieved candidates, a.k.a. error propagation from the candidate retrieval phase. we notice the following retrieval error patterns by comparing the gold entity with the top 10 retrieved candidates. ## conclusion we propose attribute-aware multimodal entity linking, which requires features extracted from images, text descriptions, and structured attributes to disambiguate and link each mention to the corresponding entity in a target knowledge base. to support this research, we construct ameli, consisting of a multimodal knowledge base that contains 34,690 product entities described with text, images, and finegrained attributes, and a multimodal review dataset that contains 16,735 review instances while each review is also associated with a text description and an image. we experiment with several highperformance entity-linking approaches, including a new approach that incorporates attributes of entities for disambiguation. experimental results show that the attributes indeed significantly enhance the model performance, but still, there is a large gap between the machine and human performance.