Benchmarking Diverse-Modal Entity Linking with Generative Models (2023)

Abstract: Entities can be expressed in diverse formats, such as texts, images, or column names and cell values in tables. While existing entity linking (EL) models work well on per modality configuration, such as text-only EL, visual grounding or schema linking, it is more challenging to design a unified model for diverse modality configurations. To bring various modality configurations together, we constructed a benchmark for diverse-modal EL (DMEL) from existing EL datasets, covering all three modalities including text, image and table. To approach the DMEL task, we proposed a generative diverse-modal model (GDMM) following a multimodal-encoder-decoder paradigm. Pre-training GDMM with rich corpora builds a solid foundation for DMEL without storing the entire KB for inference. Fine-tuning GDMM builds a stronger DMEL baseline, outperforming state-of-the-art task-specific EL models by 8.51 F1 score on average. Additionally, extensive error analyses are conducted to highlight the challenge of DMEL, facilitating future researches on this task.

## introduction linking ambiguous mentions to unambiguous referent in a knowledge base (kb) such as wikipedia, known as entity linking (el) @xcite , is an essential component for applications like question answering @xcite @xcite and recommendation systems @xcite . diverse-modal entity linking (dmel) extends the scope of interest from textual entity linking to heterogeneous input formats, such as linking visual and textual expressions to kb @xcite @xcite @xcite @xcite and linking mentions in natural language to tables or database (db) schema @xcite @xcite @xcite . figure 1 demonstrate three examples of dmel, including (a) classical textual entity linking, (b) textual-visual entity linking in which the question or mentions are paired with image(s), and (c) tabular schema linking in which the mentions are linked to column names or cell values. retrieval-based contrastive learning or ranking mechanism is the mainstream for early visual entity linking by leveraging a matching score between the mention and the kb entities @xcite @xcite . however, these methods require storage of dense representations of all kb entities, and when the size of entities increases (e.g. wikipedia has 6m articles), it raises concerns for space complexity and also inference-time latency. meanwhile, linking mentions to tables or db schemes, known as schema linking, remains an important but under-explored task. for example, in text-to-sql generations, incorrect schema linking usually counts for a large portion of the errors @xcite @xcite @xcite . previous string matching heuristic @xcite or embedding matching methods @xcite @xcite lack semantic and schema understanding, and can hardly generalize well to new domains. last but not the least, previous endeavors of entity linking are limited to individual tasks including textual el, textual-visual el, or schema linking, and lack a general view for the dmel problem. to this end, we propose a unified dmel task that includes existing el datasets on all three modalities -text, image, and table. the unified dmel task is challenging because the model needs to handle a wide spectrum of modality configurations together. on the modeling side, because storing all entity information (e.g. all the images in the entire kb) is expensive at inference time, we propose to use a unified generative model that can take diverse-modal input and generate entity names in an autoregressive fashion. additionally, the mention diversity and ambiguity issue in schema linking can be addressed by pre-training the generative model. in this work, we build a generic diversemodal architecture for end-to-end dmel. the dmel dataset is constructed from five existing datasets, including gerbil benchmark, wikidiverse, melbench-wikipedia, squall, and slsql, covering diverse el tasks. the proposed generative diverse-modal model (gdmm) is first pretrained on large-scale text corpus blink and images corpus from wikipedia kb, offering profound prior knowledge. extensive experiments are then conducted on the dmel benchmark to compare our proposed generative model to previous stateof-the-art methods. experimental results show that gdmm achieves strong performance on the dmel dataset and outperforms state-of-the-art task-specific el models. our contributions include: ## problem formulation we assume to have a kb (e.g., wikipedia or a db schema) where each entity is a unique entry in the kb. we formulate the following dmel problem: given a multimodal input {x i , v i , u i } of textual (l), visual (v), and tabular (u) modality respectively, an entity mention m i within the input, and a candidate set the task is to link the mention m i to one entity in c i . we assume the entity span is given. sometimes the candidate set can be the entire entity collection e. particular instances of dmel problem include but are not limited to: textual entity disambiguation where a given mention m i in x i will be linked to one entity in c i ; textual-visual entity disambiguation where the m i and a given image v i will be linked to one entity in c i ; schema linking where a m i in a sql query will be linked to table schema, i.e., a column name within given tables u i . if the mention is not a valid entity or not in c i , the target label is "nil". ## dmel benchmark we build the dmel benchmark from five existing datasets, including gerbil benchmark @xcite , wikidiverse @xcite , melbench-wikidata (as melbench in the rest of the paper) @xcite , squall @xcite , and slsql @xcite . we evaluate textual-visual entity disambiguation capability on wikidiverse and melbench, and evaluate tabular text encoder image encoder johnson (right) playing for { manchester city } in 2010. fusion encoder text table image encoding constrained decoding decoder table linearization johnson (right) playing for { manchester city } in 2010. <@xmath0.c. f.c. w.f.c. city college @xmath1: overall gdmm architecture (left) and constrained decoding illustration (right). dataset modality # sentences / tables train valid test gerbil @xmath2,448 4,791 19,614 wikidiverse @xmath3,312 755 757 melbench @xmath4,216 1,888 3,776 squall @xmath5,028 2,246 -slsql @xmath6,000 1,034 - we assume that the mention span is given across all datasets, thus (1) on gerbil benchmark, we investigate textual entity disambiguation using the same candidate sets as in @xcite @xcite . (2) on wikidiverse, we investigate entity disambiguation performance with retrieved top-10 candidates by @xcite (3) on melbench-wikidata, we follow the original setting in which the given entity mention will be linked to its referent in the knowledge base. (4) on squall, the given entity mention in the natural question will be linked to column names in the target table. (5) on slsql, the entity mention in the natural question will be linked to the column name within multiple tables. statistics for each individual el task are shown in ## gdmm model we build a generative entity linking model that enables diverse-modal vision, language and table understanding and inference. we show how this can be achieved with a generative encoder-decoder structure.