Original: O PEN - DOMAIN V ISUAL E NTITY L INKING
Revised: O PEN - DOMAIN V ISUAL E NTITY L INKING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We introduce the task of Open-domain Visual Entity Linking (O VEN ), targeting a wide range of entities including animals, plants, buildings, locations and much more.
Revised: We introduce the task of Open-domain Visual Entity Linking (O VEN ), targeting a wide range of entities including animals, plants, buildings, locations and much more.

Original: Given an image ( e.g ., an image of an aircraft), a text query (‘ What is the model? ’
Revised: Given an image ( e.g ., an image of an aircraft), a text query (‘ What is the model? ’

Original: or ‘ What is the airline? ’), and a multi-modal knowledge base ( e.g ., Wikipedia), the goal is to link to an entity ( B OEING -777 or EVA A IR ) out of all entities in the knowledge base.
Revised: or ‘ What is the airline? ’), and a multi-modal knowledge base ( e.g ., Wikipedia), the goal is to link to an entity ( B OEING -777 or EVA A IR ) out of all entities in the knowledge base.

Original: We build a benchmark dataset (O VEN -Wiki), by repurposing 14 existing image classiﬁcation, image retrieval, and visual QA datasets.
Revised: We build a benchmark dataset (O VEN -Wiki), by repurposing 14 existing image classiﬁcation, image retrieval, and visual QA datasets.

Original: We link all existing labels to Wikipedia entities when possible, using a state-of-the-art entity linking system and human annotators, creating a diverse and uniﬁed label space.
Revised: We link all existing labels to Wikipedia entities when possible, using a state-of-the-art entity linking system and human annotators, creating a diverse and uniﬁed label space.

Original: O VEN is a rich and challenging task, which requires models to recognize and link visual content to both a small set of seen entities as well as a much larger set of unseen entities ( e.g ., unseen aircraft models).
Revised: O VEN is a rich and challenging task, which requires models to recognize and link visual content to both a small set of seen entities as well as a much larger set of unseen entities ( e.g ., unseen aircraft models).

Original: O VEN also requires models to generalize to previously unseen intents that may require more ﬁne-grained reasoning (‘ What is the model of the aircraft in the back? ’).
Revised: O VEN also requires models to generalize to previously unseen intents that may require more ﬁne-grained reasoning (‘ What is the model of the aircraft in the back? ’).

Original: We build strong baselines based on state-of-the-art pre-trained models and ﬁnd that current pre-trained models struggle to address the challenges posed by O VEN .
Revised: We build strong baselines based on state-of-the-art pre-trained models and ﬁnd that current pre-trained models struggle to address the challenges posed by O VEN .

Original: We hope O VEN will inspire next-generation pre-training techniques and pave the way to future knowledge-intensive vision tasks.
Revised: We hope O VEN will inspire next-generation pre-training techniques and pave the way to future knowledge-intensive vision tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent interest in knowledge-intensive visual applications such as KVQA (Shah et al., 2019) and
Revised: Recent interest in knowledge-intensive visual applications such as KVQA (Shah et al., 2019) and

Original: OK-VQA (Marino et al., 2019) has demonstrated the value of grounding images to knowledge bases such as Wikipedia.
Revised: OK-VQA (Marino et al., 2019) has demonstrated the value of grounding images to knowledge bases such as Wikipedia.

Original: However, while models for these applications focus on integrating information from the knowledge base, there has been little focus on systematic, broad-coverage approaches to the grounding problem itself.
Revised: However, while models for these applications focus on integrating information from the knowledge base, there has been little focus on systematic, broad-coverage approaches to the grounding problem itself.

Original: Existing work typically combines various closed-set classiﬁers ( e.g .,
Revised: Existing work typically combines various closed-set classiﬁers ( e.g .,

Original: ImageNet (Russakovsky et al., 2015), COCO (Lin et al., 2014), etc .) in an ad-hoc manner, without a clear formulation of image grounding as a general task.
Revised: ImageNet (Russakovsky et al., 2015), COCO (Lin et al., 2014), etc .) in an ad-hoc manner, without a clear formulation of image grounding as a general task.

Original: In this paper, we propose and formally deﬁne the task of O pen-domain V isual En tity
Revised: In this paper, we propose and formally deﬁne the task of O pen-domain V isual En tity

Original: Linking (O VEN ), with the goal of building vision systems that ground visual content to entities in large-scale multi-modal knowledge bases (such as Wikipedia).
Revised: Linking (O VEN ), with the goal of building vision systems that ground visual content to entities in large-scale multi-modal knowledge bases (such as Wikipedia).

Original: In contrast to existing informal setups that rely on closed-set classiﬁers in an adhoc way, O VEN is open-domain where predictions cover a large space of entities governed by a knowledge base.
Revised: In contrast to existing informal setups that rely on closed-set classiﬁers in an adhoc way, O VEN is open-domain where predictions cover a large space of entities governed by a knowledge base.

Original: The O VEN task takes as input with an image, a text query 1 that expresses intent with respect to the image, and a knowledge base which contains the entire set of entities, along with supporting text descriptions and a relevant set of images for each.
Revised: The O VEN task takes as input with an image, a text query 1 that expresses intent with respect to the image, and a knowledge base which contains the entire set of entities, along with supporting text descriptions and a relevant set of images for each.

Original: Given these inputs the goal is to predict an entity that is both physically present in the input image as well as satisﬁes the unambiguous intent expressed in the query.
Revised: Given these inputs the goal is to predict an entity that is both physically present in the input image as well as satisﬁes the unambiguous intent expressed in the query.

Original: For instance, given the same image of an aircraft, different text queries such as ‘ Which model is this? ’
Revised: For instance, given the same image of an aircraft, different text queries such as ‘ Which model is this? ’

Original: or ‘ Which airline is this? ’ can lead to different answers, i.e ., B OEING -777 or EVA A IR .
Revised: or ‘ Which airline is this? ’ can lead to different answers, i.e ., B OEING -777 or EVA A IR .

Original: A strong O VEN model should learn to use all three inputs, i.e . the query, the image and the multi-modal knowledge base when making predictions.
Revised: A strong O VEN model should learn to use all three inputs, i.e . the query, the image and the multi-modal knowledge base when making predictions.

Original: Figure 1 illustrates the input-output mapping of O VEN .
Revised: Figure 1 illustrates the input-output mapping of O VEN .

Original: By connecting images to entities in the knowledge base, O VEN creates a universal entity space that organizes existing image labels.
Revised: By connecting images to entities in the knowledge base, O VEN creates a universal entity space that organizes existing image labels.

Original: This setup poses challenges in the form of (1) generalization to
Revised: This setup poses challenges in the form of (1) generalization to

Original: UNSEEN entities and (2) generalization to UNSEEN queries .
Revised: UNSEEN entities and (2) generalization to UNSEEN queries .

Original: For example, classic object recognition systems trained on FGVCAircraft (Maji et al., 2013) can recognize the ninety-six aircraft categories
Revised: For example, classic object recognition systems trained on FGVCAircraft (Maji et al., 2013) can recognize the ninety-six aircraft categories

Original: [Figure]
Revised: [Figure]

Original: deﬁned in FGVCAircraft, but would necessarily fail to recognize B OEING -787 , as it does not belong to the pre-deﬁned categories.
Revised: deﬁned in FGVCAircraft, but would necessarily fail to recognize B OEING -787 , as it does not belong to the pre-deﬁned categories.

Original: In contrast, models for O VEN are required to recognize entities that were UNSEEN in the training data.
Revised: In contrast, models for O VEN are required to recognize entities that were UNSEEN in the training data.

Original: This requirement reﬂects a more realistic scenario, since there will never be enough training data to cover all knowledge base entities, especially when the number of knowledge base entities is constantly growing.
Revised: This requirement reﬂects a more realistic scenario, since there will never be enough training data to cover all knowledge base entities, especially when the number of knowledge base entities is constantly growing.

Original: 2 O VEN also evaluates a model’s ability to understand
Revised: 2 O VEN also evaluates a model’s ability to understand

Original: UNSEEN queries, since it is impossible to observe all text queries to a real-world KB during training.
Revised: UNSEEN queries, since it is impossible to observe all text queries to a real-world KB during training.

Original: To benchmark O VEN , we construct O VEN -Wiki by repurposing 14 existing image classiﬁcation, image retrieval, and visual QA datasets, and grounding all labels to the most prominent knowledge base, Wikipedia.
Revised: To benchmark O VEN , we construct O VEN -Wiki by repurposing 14 existing image classiﬁcation, image retrieval, and visual QA datasets, and grounding all labels to the most prominent knowledge base, Wikipedia.

Original: The training, validation, and testing splits are formed in a way such that models need to generalize to entities that are unseen during training.
Revised: The training, validation, and testing splits are formed in a way such that models need to generalize to entities that are unseen during training.

Original: Grounding labels from different datasets into Wikipedia and combining datasets is challenging because of the ambiguity of language.
Revised: Grounding labels from different datasets into Wikipedia and combining datasets is challenging because of the ambiguity of language.

Original: For example, due to language polysemy, ‘ Yoke ’ can represent a wooden beam or part of the construction of a garment.
Revised: For example, due to language polysemy, ‘ Yoke ’ can represent a wooden beam or part of the construction of a garment.

Original: ‘ Tornado ’ can be a weather phenomena or a type of airplane ( P ANAVIA T ORNADO ).
Revised: ‘ Tornado ’ can be a weather phenomena or a type of airplane ( P ANAVIA T ORNADO ).

Original: To reduce such ambiguity in the grounding, we take multiple steps to reﬁne the labels, including the use of a state-of-the-art textual entity linking system (De Cao et al., 2020).
Revised: To reduce such ambiguity in the grounding, we take multiple steps to reﬁne the labels, including the use of a state-of-the-art textual entity linking system (De Cao et al., 2020).

Original: For more accurate evaluation, a subset of examples is thoroughly annotated by human annotators: entity linking errors are corrected and the ambiguous queries are rewritten so that no other objects can be the answer.
Revised: For more accurate evaluation, a subset of examples is thoroughly annotated by human annotators: entity linking errors are corrected and the ambiguous queries are rewritten so that no other objects can be the answer.

Original: To ensure that the dataset is computationally manageable for the community, we use a 100k subset of
Revised: To ensure that the dataset is computationally manageable for the community, we use a 100k subset of

Original: Wikipedia entities.
Revised: Wikipedia entities.

Original: Using multiple types of information (image, input query and multi-modal knowledge) is important for succeeding on O VEN , but existing pre-trained models like CLIP (Radford et al., 2021) or
Revised: Using multiple types of information (image, input query and multi-modal knowledge) is important for succeeding on O VEN , but existing pre-trained models like CLIP (Radford et al., 2021) or

Original: SimVLM (Wang et al., 2021) cannot use all of the available information natively.
Revised: SimVLM (Wang et al., 2021) cannot use all of the available information natively.

Original: We experiment with baselines applying different combinations of strong pre-trained models and conduct error analysis to show possible headroom.
Revised: We experiment with baselines applying different combinations of strong pre-trained models and conduct error analysis to show possible headroom.

Original: The contributions of the paper are as follows:
Revised: The contributions of the paper are as follows:

Original: • Towards the broader goal of linking visual entities to large open-domain KBs, we formalize the task of Open-domain Visual Entity Linking (O VEN ) •
Revised: • Towards the broader goal of linking visual entities to large open-domain KBs, we formalize the task of Open-domain Visual Entity Linking (O VEN ) •

Original: We construct the ﬁrst large-scale visual entity linking benchmark, O VEN -Wiki, by re-purposingexisting datasets with all of their labels grounded to Wikipedia.
Revised: We construct the ﬁrst large-scale visual entity linking benchmark, O VEN -Wiki, by re-purposingexisting datasets with all of their labels grounded to Wikipedia.

Original: A subset of the dataset is annotated by human annotators for high quality evaluation.
Revised: A subset of the dataset is annotated by human annotators for high quality evaluation.

Original: • We build strong baselines based on state-of-the-art pre-trained models and ﬁnd that the best performing systems are the ones that better leverage multi-modal knowledge.
Revised: • We build strong baselines based on state-of-the-art pre-trained models and ﬁnd that the best performing systems are the ones that better leverage multi-modal knowledge.

Original:  T ASK F ORMULATION FOR O VEN
Revised:  T ASK F ORMULATION FOR O VEN

Original: The proposed task of open-domain visual entity linking relies on inputs that consist of text queries, images and a multimodal knowledge base (KB), all corresponding to a uniﬁed label space.
Revised: The proposed task of open-domain visual entity linking relies on inputs that consist of text queries, images and a multimodal knowledge base (KB), all corresponding to a uniﬁed label space.

Original: The input image-text pairs x = ( x p , x t ) include a text query x t expressing intent with respect to the corresponding image x p .
Revised: The input image-text pairs x = ( x p , x t ) include a text query x t expressing intent with respect to the corresponding image x p .

Original: Given a uniﬁed label space E which deﬁnes the set of all possible entities, the input knowledge base K = { ( e, p ( e ) , t ( e ))
Revised: Given a uniﬁed label space E which deﬁnes the set of all possible entities, the input knowledge base K = { ( e, p ( e ) , t ( e ))

Original: | e ∈ E} is a set of triples, each containing an entity e , its corresponding text description t ( e ) and a (possibly empty) set of relevant images p ( e ) .
Revised: | e ∈ E} is a set of triples, each containing an entity e , its corresponding text description t ( e ) and a (possibly empty) set of relevant images p ( e ) .

Original: For instance, an entity e = S ABATIA CAMPESTRIS would have a corresponding textual description t ( e ) =
Revised: For instance, an entity e = S ABATIA CAMPESTRIS would have a corresponding textual description t ( e ) =

Original: ‘ Sabatia campestris is a species of Sabatia . . . ’
Revised: ‘ Sabatia campestris is a species of Sabatia . . . ’

Original: and a set p ( e ) containing one or more images from the corresponding Wikipedia page 3 of S ABATIA CAMPESTRIS .
Revised: and a set p ( e ) containing one or more images from the corresponding Wikipedia page 3 of S ABATIA CAMPESTRIS .

Original: We consider the combination of t ( e ) and p ( e ) the multimodal knowledge for the entity e .
Revised: We consider the combination of t ( e ) and p ( e ) the multimodal knowledge for the entity e .

Original: In the current framework, we focus on linking entities that are physically present in the image.
Revised: In the current framework, we focus on linking entities that are physically present in the image.

Original: This means that given an image, the target label is an entity that is present in the image and answers the query.
Revised: This means that given an image, the target label is an entity that is present in the image and answers the query.

Original: There could be many different modeling choices forthe O VEN .
Revised: There could be many different modeling choices for O VEN .

Original: As oneof many modeling choices, we can learn a function f Θ to maximize the score of this target entity for the given imagequery pair, using multimodal knowledge from the knowledge base.
Revised: As one modeling choice, we can learn a function f Θ to maximize the score of the target entity for the given image-query pair, using multimodal knowledge from the knowledge base.

Original: Then, given a test example x = ( x p , x t ) and the knowledge base of triples K , the function is used to make a prediction,
Revised: Then, given a test example x = ( x p , x t ) and the knowledge base of triples K , the function is used to make a prediction,

Original: [Equation]
Revised: [Equation]

Original: where a large set of entities in E are UNSEEN  during training.
Revised: where a large set of entities in E = E SEEN (cid:83) E UNSEEN is a combination of SEEN and UNSEEN entities. Note that all examples available during the training stage have entities in E SEEN .

Original: 2. C OMPARING O VEN TO O THER T ASKS
Revised: 2. C OMPARING O VEN TO O THER T ASKS

Original: O VEN is a challenging recognition task By setting the text query to ‘ What is the main object? ’
Revised: O VEN is a challenging recognition task By setting the text query to ‘ What is the main object? ’

Original: , we can recast O VEN into a general object recognition task.
Revised: , we can recast O VEN into a general object recognition task.

Original: However, O VEN is far more challenging compared to existing benchmarks because knowledge bases such as Wikipedia can provide a uniﬁed label space.
Revised: However, O VEN is far more challenging compared to existing benchmarks because knowledge bases such as Wikipedia can provide a uniﬁed label space.

Original: The number of entities in Wikipedia is orders of magnitude larger than conventional benchmarks.
Revised: The number of entities in Wikipedia is orders of magnitude larger than conventional benchmarks.

Original: We demonstrate the challenge of using a large label space as follows—we start by selecting an image from a standard benchmark and using CLIP (without ﬁne-tuning) to encode it, then we scan 100K Wikipedia titles and make a prediction.
Revised: We demonstrate the challenge of using a large label space as follows—we start by selecting an image from a standard benchmark and using CLIP (without ﬁne-tuning) to encode it, then we scan 100K Wikipedia titles and make a prediction.

Original: Figure 2a  shows that extending the original label space to 100K Wikipedia titles has introduced more closely related hard negative examples that turn many existing features into weaker signals.
Revised: Figure 2a 5 shows that extending the original label space to 100K Wikipedia titles has introduced more closely related hard negative examples that turn many existing features into weaker signals.

Original: For example, models can no longer rely on just the color or the car body conﬁguration to identify the ground-truth category D AEWOO
Revised: For example, models can no longer rely on just the color or the car body conﬁguration to identify the ground-truth category D AEWOO

Original: N UBIRA , confusing it with other cars with similar conﬁgurations.
Revised: N UBIRA , confusing it with other cars with similar conﬁgurations.

Original: Instead, ﬁne-grained reasoning over visual attributes is required to make a correct classiﬁcation.
Revised: Instead, ﬁne-grained reasoning over visual attributes is required to make a correct classiﬁcation.

Original: In addition, the introduction of a large label space also makes generalization to previously UNSEEN entities critical, which is not emphasized in most image recognition benchmarks.
Revised: In addition, the introduction of a large label space also makes generalization to previously UNSEEN entities critical, which is not emphasized in most image recognition benchmarks.

Original: This is important because there would not be enough training data to learn classiﬁers for every possible entity in a
Revised: This is important because there would not be enough training data to learn classiﬁers for every possible entity in a

Original: Web-scale KB such as Wikipedia, especially given that it grows over time.
Revised: Web-scale KB such as Wikipedia, especially given that it grows over time.

Original: Finally, knowledge bases such as Wikipedia cover entities of different granularities ( e.g ., the airplane manufacturer or the aircraft model), such that different entities would be preferred according to different text queries.
Revised: Finally, knowledge bases such as Wikipedia cover entities of different granularities ( e.g ., the airplane manufacturer or the aircraft model), such that different entities would be preferred according to different text queries.

Original: Given the scenario in Figure 2b, O VEN models should predict different entities given the same input image but different text queries, whereas recognition models can not handle this.
Revised: Given the scenario in Figure 2b, O VEN models should predict different entities given the same input image but different text queries, whereas recognition models can not handle this.

Original: [Figure]
Revised: [Figure]

Original: For example D AEWOO L ACETTI does not belong ambiguous— LUGZEUGWERKE is manufacturer. O VEN is a specialized Visual QA task with KB grounding Due to the input text query, we can consider O VEN to be a specialized VQA task with KB grounding.
Revised: O VEN is a specialized Visual QA task with KB grounding Due to the input text query, we can consider O VEN to be a specialized VQA task with KB grounding.

Original: A key difference is  that  O VEN only focuses on the entities that are present in  the image, while VQA tasks questions, or questions referring to entities not in the image.KB grounding also distinguishes O VEN from existing VQA tasks since it opens up possibilities to answer questions on entities that have not been learned before, and are thus UNSEEN . In contrast, existing VQA tasks often presume answers follow the same distribution as those in training.
Revised: However, OVEN is not a VQA task as OVEN models are required to output the entity name inside a KB (same as text entity linking tasks), while VQA models output free-form answers that can contain yes-no/counting questions. Moreover, O VEN only focuses on the entities that are present in or can be directly inferred from the image while VQA tasks can ask questions referring to entities not in the image.

Original: As a consequence, O VEN  modelsare also required to handle UNSEEN text queries associated with UNSEEN entities.
Revised: Another unique feature of O VEN is that it covers over 100K ﬁne-grained entities and this encourages the models to make use of the multimodal knowledge in general KB for unseen entities.

Original: Knowledge Base as Input It is important to note that the knowledge base is an input to the model.
Revised: Knowledge Base as Input It is important to note that the knowledge base is an input to the model.

Original: Therefore, the model should still work if the knowledge base gets updated, as the KB is a constantly evolving knowledge source.
Revised: Therefore, the model should still work if the knowledge base gets updated, as the KB is a constantly evolving knowledge source.

Original: The two possible updates include: (1) editing an existing entity’s text description or image set, and (2) growing the label space by adding new entities.
Revised: The two possible updates include: (1) editing an existing entity’s text description or image set, and (2) growing the label space by adding new entities.

Original: 2. E VALUATION FOR O VEN
Revised: 2. E VALUATION FOR O VEN

Original: We evaluate O VEN with the goal of balancing performance between SEEN and UNSEEN entities using a harmonic mean, as shown below:
Revised: We evaluate O VEN with the goal of balancing performance between SEEN and UNSEEN entities using a harmonic mean, as shown below:

Original: [Equation]
Revised: [Equation]

Original: Here, Acc SEEN and Acc UNSEEN are the Macro F1 score over examples from all SEEN and UNSEEN classes, respectively.
Revised: Here, Acc SEEN and Acc UNSEEN are the Macro accuracy over examples from all SEEN and UNSEEN classes, respectively.

Original: Harmonic mean equally weighs the importance of both SEEN and UNSEEN subsets, and penalizes models with a short barrel.
Revised: Harmonic mean equally weighs the importance of both SEEN and UNSEEN subsets, and penalizes models with a short barrel.

Original: More concrete details are provided in §3.
Revised: More concrete details are provided in §3.

Original:  T HE O VEN -W IKI DATASET
Revised:  T HE O VEN -W IKI DATASET

Original: Based on our task formulation, we create a benchmark dataset using a comibation of automatic entity linking and human annotation.
Revised: Based on our task formulation, we create a benchmark dataset using a comibation of automatic entity linking and human annotation.

Original: More speciﬁcally, we create O VEN -Wiki by combining 14 existing datasets originally created for image recognition, image retrieval, and visual question answering, and creating a uniﬁed label space grounded in Wikipedia entities.
Revised: More speciﬁcally, we create O VEN -Wiki by combining 14 existing datasets originally created for image recognition, image retrieval, and visual question answering, and creating a uniﬁed label space grounded in Wikipedia entities.

Original: This dataset provides training and evaluation examples, and a Wikipedia-based KB for the proposed open-domain visual entity linking (O VEN ) task.
Revised: This dataset provides training and evaluation examples, and a Wikipedia-based KB for the proposed open-domain visual entity linking (O VEN ) task.

Original: The complete list of datasets includes:
Revised: The complete list of datasets includes:

Original: • Image Recognition (or Retrieval) Datasets : ImageNet21k-P (Russakovsky et al., 2015; Ridnik et al., 2021), iNaturalist2017 (Van Horn et al., 2018), Cars196 (Krause et al., 2013), SUN397 (Xiao et al., 2010), Food101 (Bossard et al., 2014), Sports100 (Gerry, 2021), Aircraft (Maji et al., 2013), Oxford Flower (Nilsback & Zisserman, 2008), Google Landmarks v2 (Weyand et al., 2020) • Visual QA Datasets : VQA v2 (Goyal et al., 2017), Visual7W (Zhu et al., 2016), Visual Genome (Krishna et al., 2017), OK-VQA (Marino et al., 2019), Text-VQA (Singh et al., 2019).
Revised: • Image Recognition (or Retrieval) Datasets : ImageNet21k-P (Russakovsky et al., 2015; Ridnik et al., 2021), iNaturalist2017 (Van Horn et al., 2018), Cars196 (Krause et al., 2013), SUN397 (Xiao et al., 2010), Food101 (Bossard et al., 2014), Sports100 (Gerry, 2021), Aircraft (Maji et al., 2013), Oxford Flower (Nilsback & Zisserman, 2008), Google Landmarks v2 (Weyand et al., 2020) • Visual QA Datasets : VQA v2 (Goyal et al., 2017), Visual7W (Zhu et al., 2016), Visual Genome (Krishna et al., 2017), OK-VQA (Marino et al., 2019), Text-VQA (Singh et al., 2019).

Original: These datasets generally belong to two groups: image recognition (or retrieval) which provides diverse visual entities , deﬁned as the Entity Split ; and visualQA which provides visually-situated natural language queries , deﬁned as the Query split .
Revised: These datasets generally belong to two groups: image recognition (or retrieval) which provides diverse visual entities , deﬁned as the Entity Split ; and visual

Revised: QA which provides visually-situated natural language

Original: We describe data collection, ﬁltering and curation in Appendix B.1.
Revised: queries deﬁned as the

Revised: Query split .

Revised: We describe data collection, ﬁltering and curation in Appendix A.1.

Original: In the following, we present the dataset statistics, and introduce evaluation protocols.
Revised: In the following, we present the dataset statistics, and introduce evaluation protocols.

Original: [Figure]
Revised: [Figure]

Original: Dataset Statistics Since our goal is to evaluate generalization to both existing and novel entities, we hold out half of the entities as UNSEEN and only include them for evaluation.
Revised: Dataset Statistics Since our goal is to evaluate generalization to both existing and novel entities, we hold out half of the entities as UNSEEN and only include them for evaluation.

Original: We also hold out examples from SEEN entities to measure SEEN performances in the evaluation.
Revised: We also hold out examples from SEEN entities to measure SEEN performance in the evaluation.

Original: Figure 3 (left) presents the general distribution of the super-categories for our ﬁnal collection of positive Wikipedia entities.
Revised: Figure 3 (left) presents the general distribution of the super-categories for our ﬁnal collection of positive Wikipedia entities.

Original: Figure 3 (right) shows detailed statistics for queries and entities for each of the training (or ﬁne-tuning), validation, test and gold splits.
Revised: Figure 3 (right) shows detailed statistics for queries and entities for each of the training (or ﬁne-tuning), validation, test and gold splits.

Original: 5 The number of SEEN / UNSEEN examples indicates the number of examples of which the positive entity labels are in the SEEN / UNSEEN split.
Revised: 6 The number of SEEN / UNSEEN examples indicates the number of examples of which the positive entity labels are in the SEEN / UNSEEN split.

Original: Human Annotated Gold Evaluation Besides evaluating on the automatically constructed dataset, we further perform a human annotation to create the gold evaluation data.
Revised: Human Annotated Gold Evaluation Besides evaluating on the automatically constructed dataset, we further perform a human annotation to create the gold evaluation data.

Original: Speciﬁcally, we hired over 30 dedicated annotators to validate the entity links in < image, query, answer > triplets sampled from the test split.
Revised: Speciﬁcally, we hired over 30 dedicated annotators to validate the entity links in < image, query, answer > triplets sampled from the test split.

Original: They were asked to re-annotate the triplets, with access to the visual context, ensuring that the query leads to the correct Wikipedia entity answer.
Revised: They were asked to re-annotate the triplets, with access to the visual context, ensuring that the query leads to the correct Wikipedia entity answer.

Original: This process identiﬁed an approximate 8.4% of entity linking errors, and obtained 4,794 natural language queries, equally distributed over triplets originally sampled from the Entity and Query splits ( i.e ., test splits).
Revised: This process identiﬁed an approximate 8.04% of entity linking errors, and obtained 4,794 natural language queries, equally distributed over triplets originally sampled from the Entity and Query splits ( i.e ., test splits).

Original: We asked the annotators to rewrite the queries so that no other object in the image can be the answer.
Revised: We asked the annotators to rewrite the queries so that no other object in the image can be the answer.

Original: As a result, the percentage of unique queries in the total examples (3,775 out of 4,794) as shown in Table 3 (right) is signiﬁcantly higher in the gold set than the other sets, which brings more challenges to query generalization for the gold eval set.
Revised: As a result, the percentage of unique queries in the total examples (3,775 out of 4,794) as shown in Table 3 (right) is signiﬁcantly higher in the gold set than the other sets, which brings more challenges to query generalization for the gold eval set.

Original: We report results using the same evaluation metrics on the gold data, with respect to SEEN and UNSEEN entities.
Revised: We report results using the same evaluation metrics on the gold data, with respect to SEEN and UNSEEN entities.

Original: Figure 1 provides a glance at the human annotated gold data.
Revised: Figure 1 provides a glance at the human annotated gold data.

Original: More details about human annotation is in Appendix B.2.
Revised: More details about human annotation is in Appendix A.2.

Original: Evaluation Protocol To approach our mission of linking images to Wikipedia entities without imposing overwhelming computational costs, we create an approximated Wikipedia knowledge base.
Revised: Evaluation Protocol To approach our mission of linking images to Wikipedia entities without imposing overwhelming computational costs, we create an approximated Wikipedia knowledge base.

Original: Speciﬁcally, we include a total of 85k top frequent and non-overlapping Wikipedia entities into our dataset, which sums to a total of 105k entities.
Revised: Speciﬁcally, we include a total of 85k top frequent and non-overlapping Wikipedia entities into our dataset, which sums to a total of 105k entities.

Original: This step simulates practical entity linking scenarios, where a model is required to contrast against a massive number of Wikipedia entities and infer the correct one for a query on the visual content.
Revised: This step simulates practical entity linking scenarios, where a model is required to contrast against a massive number of Wikipedia entities and infer the correct one for a query on the visual content.

Original: In this paper, all models predict among ∼ 105k candidate entities.
Revised: In this paper, all models predict among ∼ 105k candidate entities.

Original: As stronger models emerge, we can consider a larger knowledge base in the future work.
Revised: As stronger models emerge, we can consider a larger knowledge base in the future work.

Original: We measure the model’s performance using both the Entity Split (ES) and Query Split (QS).
Revised: We measure the model’s performance using both the Entity Split (ES) and Query Split (QS).

Original: Specifically, we ﬁrst compute the harmonic mean of Macro F1 over examples from SEEN and UNSEEN classes, as Acc ES and Acc QS , according to Equation 2.
Revised: Specifically, we ﬁrst compute the harmonic mean of Macro F1 over examples from SEEN and UNSEEN classes, as Acc ES and Acc QS , according to Equation 2.

Original: Then we further calculate the harmonic mean between splits HM ( Acc ES , Acc QS ) to reward models that do well on both splits.
Revised: Then we further calculate the harmonic mean between splits HM ( Acc ES , Acc QS ) to reward models that do well on both splits.

Original: We use the validation data, which contains examples from subsets of both SEEN and UNSEEN entities, for model selection, and report the overall evaluation.
Revised: We use the validation data, which contains examples from subsets of both SEEN and UNSEEN entities, for model selection, and report the overall evaluation.

Original: For ﬁnal performance, we measure performance on the test split, which covers ∼ 98% positive entities (about half of them are unseen).
Revised: For ﬁnal performance, we measure performance on the test split, which covers ∼ 98% positive entities (about half of them are unseen).

Original: We also report results on the Gold set, to measure model’s generalization to a more diverse set of human queries.
Revised: We also report results on the Gold set, to measure model’s generalization to a more diverse set of human queries.

Original:  B ASELINE M ODELS FOR O VEN
Revised:  B ASELINE M ODELS FOR O VEN

Original: As performance on UNSEEN entities and text queries is important for O VEN , we focus on applying pretrained multimodal models which have previously reported promising zero-shot generalization.
Revised: As performance on UNSEEN entities and text queries is important for O VEN , we focus on applying pretrained multimodal models which have previously reported promising zero-shot generalization.

Original: 4. E XISTING P RE - TRAINED M ULTI -M ODAL M ODELS
Revised: 4. E XISTING P RE - TRAINED M ULTI -M ODAL M ODELS

Original: We consider the following two representative pre-trained models.
Revised: We consider the following two representative pre-trained models.

Original: CLIP: Image-to-Text Dual Encoder CLIP (Radford et al., 2021) is the most popular dual encoder model for images and text, where one encoder encodes images and the other encodes text.
Revised: CLIP: Image-to-Text Dual Encoder CLIP (Radford et al., 2021) is the most popular dual encoder model for images and text, where one encoder encodes images and the other encodes text.

Original: In O VEN , we use its image encoder to encode the input image x p , and the text encoder encodes the Wikipedia text t ( e ) .

Original: The biggest advantage of a dual encoder model is that the embedding of all candidates in the knowledge base can be pre-computed, so that when a new image arrives,  a dot product between the image embedding and all the text embeddings can be computed efﬁciently.
Revised: The biggest advantage of a dual encoder model is that the embedding of all candidates in the knowledge base can be pre-computed, so that when a new example arrives, only a dot product is required to compute the similarity between query input and entity representations.

Revised: Based on the different combination of input information and entity information employed, there are fours variants of CLIP models, which make predictions based on cosine similarities between < x t , t ( e ) >

Original: Note that CLIP models ignore the input text queries x t as well as the entity images stored in the knowledge base p ( e ) .
Revised: (CLIP T2T), < x t , p ( e ) > (CLIP T2I), < x p , p ( e ) > (CLIP I2I), and < x p , t ( e ) > (CLIP I2T). Without explicit notation, we refer CLIP I2T model as the default CLIP models. Note that all CLIP variants are depending on partial information to pair queries with entities.

Original: However, CLIP’s large scale contrastive pre-training—with 400M image-caption pairs—makes it a strong baseline in practice.
Revised: However, CLIP’s large-scale contrastive pre-training—with 400M image-caption pairs—makes it a strong baseline in practice.

Original: SimperVLM: Image+Text Encoder with Text Decoder  SimVLM (Wang et al., 2021) is a pretrained model with an encoder that can jointly encode text and images, and outputs text with a decoder.
Revised: SimperVLM: Image+Text Encoder with Text Decoder (SimpVLM) SimVLM (Wang et al., 2021) is a pretrained model with an encoder that can jointly encode text and images, and outputs text with a decoder.

Original: SimVLM has demonstrated strong performance on vision and language tasks, including Visual QA.
Revised: SimVLM has demonstrated strong performance on vision and language tasks, including Visual QA.

Original: Using this model for O VEN is straight-forward, as we use the encoder to encode inputs ( x p , x t ) , and use the decoder to generate the target entity e .
Revised: Using this model for O VEN is straight-forward, as we use the encoder to encode inputs ( x p , x t ) , and use the decoder to generate the target entity e .

Original: However, the generated text is unconstrained and might not belong to the set of possible entities E .
Revised: However, the generated text is unconstrained and might not belong to the set of possible entities E .

Original: In addition, this model also is not capable of using the entity images p ( e ) .
Revised: In addition, this model also is not capable of using the entity images p ( e ) .

Original: For this paper, we implement our version of SimVLM (Wang et al., 2021)
Revised: For this paper, we implement our version of SimVLM (Wang et al., 2021)

Original: (dubbed SimplerVLM) where we remove the Convolutional Neural Networks ( i.e .,
Revised: (dubbed SimplerVLM) where we remove the Convolutional Neural Networks ( i.e .,

Original: Transformer only) and train the simpliﬁed model on the combined datasets of Fit400M (Yu et al., 2022) and C4 (Raffel et al., 2020).
Revised: Transformer only) and train the simpliﬁed model on the combined datasets of Fit400M (Yu et al., 2022) and C4 (Raffel et al., 2020).

Original: The total # of image-text pairs used for training is similar to the
Revised: The total # of image-text pairs used for training is similar to the

Original: CLIP model’s pre-training dataset, and the # of text-only examples is the same as Wang et al.
Revised: CLIP model’s pre-training dataset, and the # of text-only examples is the same as Wang et al.

Original: Note that these two models do not make use of all information provided by O VEN .
Revised: Note that these two models do not make use of all information provided by O VEN .

Original: Next we extend these two pre-trained models to construct our baseline models that cover more types of information.
Revised: Next we extend these two pre-trained models to construct our baseline models that cover more types of information.

Original: 4. O UR B ASELINE M ODELS : M IXED -M ODALITY D UAL E NCODERS
Revised: 4. O UR B ASELINE M ODELS : M IXED -M ODALITY D UAL E NCODERS

Original: In the following, we propose baseline models that can use all the information provided for a given ( x, e ) pair in O VEN : x p , x t , p ( e ) , t ( e ) .
Revised: In the following, we propose baseline models that can use all the information provided for a given ( x, e ) pair in O VEN : x p , x t , p ( e ) , t ( e ) .

Original: Note that it is computationally infeasible to score all possible entities in K through a joint encoder of all components of the input and all components of each candidate entity, due to the large size of the candidate set.
Revised: Note that it is computationally infeasible to score all possible entities in K through a joint encoder of all components of the input and all components of each candidate entity, due to the large size of the candidate set.

Original: We therefore limit attention to scoring functions that decompose as dual encoders, i.e ., f Θ ( x p , x t , p ( e ) , t ( e ))
Revised: We therefore limit attention to scoring functions that decompose as dual encoders, i.e ., f Θ ( x p , x t , p ( e ) , t ( e ))

Original: =
Revised: =

Original: f θ ( x p , x t ) · f φ ( p ( e ) , t ( e )) .
Revised: f θ ( x p , x t ) · f φ ( p ( e ) , t ( e )) .

Original: Given that both the input pair ( x p , x t ) and the knowledge base information ( p ( e ) , t ( e )) are mixed modality, we call these models Mixed-Modality Dual Encoders.
Revised: Given that both the input pair ( x p , x t ) and the knowledge base information ( p ( e ) , t ( e )) are mixed modality, we call these models Mixed-Modality Dual Encoders.

Original: Note that the parameters of Dual
Revised: Note that the parameters of Dual

Original: Encoder Θ has two components, i.e ., left encoder parameters θ and right encoder parameters φ , which are not shared for all the Mixed-Modality Dual Encoders in this paper.
Revised: Encoder Θ have two components, i.e ., left encoder parameters θ and right encoder parameters φ , which are not shared for all the Mixed-Modality Dual Encoders in this paper.

Original: We study three baseline mixed-modality dual-encoders:
Revised: We study three baseline mixed-modality dual-encoders:

Original: • Dual SimplerVLM  converts SimplerVLM into a mixed-modality dual encoder by using its encoder to embed the paired image and text information into a joint space.
Revised: • Dual SimplerVLM (D-SimpVLM) converts SimplerVLM into a mixed-modality dual encoder by using its encoder to embed the paired image and text information into a joint space.

Original: • CLIP Fusion adopts the pre-trained CLIP model as the featurizer to develop this system, via adding a 2-layer Multi-Modal Transformer on top of the CLIP image and text features.
Revised: • CLIP Fusion adopts the pre-trained CLIP model as the featurizer to develop this system, via adding a 2-layer Multi-Modal Transformer on top of the CLIP image and text features.

Original: • CLIP2CLIP adopts the pre-trained CLIP model via introducing a minimum set of non pre-trained parameters, only at for re-weighting the similarity scores.
Revised: • CLIP2CLIP adopts the pre-trained CLIP model via introducing a minimum set of non pre-trained parameters, only at for re-weighting the similarity scores.

Original: Particularly, it computes the cosine similarity between < x p , t ( e ) > , < x t , p ( e ) > , < x p , p ( e ) > , and < x t , t ( e ) > , using the image and text encoders of CLIP, respectively.
Revised: Particularly, it computes the cosine similarity between < x p , t ( e ) > , < x t , p ( e ) > , < x p , p ( e ) > , and < x t , t ( e ) > , using the image and text encoders of CLIP, respectively.

Original: Then it aggregates these similarities by multiplying them with a learnable vector that reﬂects importance weights.
Revised: Then it aggregates these similarities by multiplying them with a learnable vector that reﬂects importance weights.

Original: For all three models, their left encoders and right encoders are using the same architecture, but differently parameterized.
Revised: For all three models, their left encoders and right encoders are using the same architecture, but differently parameterized.

Original: We ﬁne-tune all their parameters (pre-trained or not) on the O VEN -Wiki.
Revised: We ﬁne-tune all their parameters (pre-trained or not) on the O VEN -Wiki.

Original: Implementation Details We process all the images on our dataset by resizing them to 224 × 224, and use standardization to normalize them.
Revised: Implementation Details We process all the images in our dataset by resizing them to 224 × 224, and use standardization to normalize them.

Original: For natural language text, we perform tokenization based on the adopted pre-trained models’ original vocabulary ( e.g ., CLIP).
Revised: For natural language text, we perform tokenization based on the adopted pre-trained model’s original vocabulary ( e.g ., CLIP).

Original: For SimplerVLM, we employ the vocabulary from T5 (Raffel et al., 2020).
Revised: For SimplerVLM, we employ the vocabulary from T5 (Raffel et al., 2020).

Original: Additional details on input processing, model conﬁgurations, and ﬁne-tuning hyper-parameters are provided in the Appendix C.
Revised: Additional details on input processing, model conﬁgurations, and ﬁne-tuning hyper-parameters are provided in the Appendix B.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: We ﬁrst establish the main benchmark results in §5.1.
Revised: We ﬁrst establish the main benchmark results in §5.1.

Original: We then conduct ablation studies to analyze models, and illustrate the key challenges of O VEN -Wiki in §5.2.
Revised: We then conduct ablation studies to analyze models, and illustrate the key challenges of O VEN -Wiki in §5.2.

Original: 5. B ENCHMARK R ESULTS
Revised: 5. B ENCHMARK R ESULTS

Original: Main Results Results on the validation set are presented in Table 1, including evaluation scores on both the Entitysplit and Query split, as well as the overall combined score.
Revised: Main Results Results on the validation set are presented in Table 1, including evaluation scores on both the Entity and Query split, as well as the overall combined score.

Original: We divide the methods into two categories, (1) zero-shot transfer models (using pretrained models only), and (2) ﬁne-tuned models.
Revised: We divide the methods into two categories, (1) zero-shot transfer models (w/ pretrained models only), and (2) ﬁne-tuned models.

Original: There are several interesting observations in in Table 1.
Revised: [Table] [Table] CLIP I2T [Table]

Original: First, not all ﬁne-tuned models are better than their pre-trained variants , particularly on UNSEEN entities. This showcases the unique generalization
Revised: CLIP2CLIP

Original: [Table]
Revised: [Table]

Revised: CLIP I2T

Original: [Table]
Revised: [Table]

Revised: There are several interesting observations in Table 1.

Revised: First, not all ﬁne-tuned models are better than their pre-trained variants , particularly on UNSEEN entities.

Original:  challenge in O VEN -Wiki, where models cannot be over-optimized for a speciﬁc set of entities that have more training data.
Revised: This showcases the unique generalization challenge in O VEN -Wiki, where models cannot be over-optimized for a speciﬁc set of entities that have more training data.

Original: Next, comparing encoder-decoder models such as SimplerVLM to other methods, we observe that the former struggles to link entities accurately.
Revised: Next, comparing encoder-decoder models such as SimplerVLM to other methods, we observe that the former struggles to link entities accurately.

Original: We ﬁnd that the zero-shot
Revised: We ﬁnd that the zero-shot

Original: SimplerVLM model often does not produce entity names, as the pretrained model never observed
Revised: SimplerVLM model often does not produce entity names, as the pretrained model never observed

Original: Wikipeida titles.
Revised: Wikipeida titles.

Original: The ﬁne-tuned SimplerVLM models do not handle unseen entities well, since the models never see the UNSEEN Wikipedia titles from the KB while training on O VEN -Wiki data.
Revised: The ﬁne-tuned SimplerVLM models do not handle unseen entities well, since the models never see the UNSEEN Wikipedia titles from the KB while training on O VEN -Wiki data.

Original: Comparing all CLIP-based models, we observe that CLIP Fusion, where two new layers have been added on top of pretrained CLIP, shows very strong results on SEEN entities, for both the Entity and the Query splits but weak results on UNSEEN entities, thus leading to lower overall performance.
Revised: Comparing all CLIP-based models, we observe that CLIP Fusion, where two new layers have been added on top of pretrained CLIP, shows very strong results on SEEN entities, for both the Entity and the Query splits but weak results on UNSEEN entities, thus leading to lower overall performance.

Original: The CLIP2CLIP model, on the other hand, is capable of retaining the cross-entity generalization performance while improving its prediction accuracies on SEEN entities.
Revised: The CLIP2CLIP model, on the other hand, is capable of retaining the cross-entity generalization performance while improving its prediction accuracies on SEEN entities.

Original: This suggests that adding more new (but non-pre-trained) parameters can harm the pre-trained models.
Revised: This suggests that adding more new (but non-pre-trained) parameters can harm the pre-trained models.

Original: Unsurprisingly, methods with large pre-trained models outperforms their smaller counterparts.
Revised: Unsurprisingly, methods with large pre-trained models outperforms their smaller counterparts.

Original: We select the best performing methods and evaluate them on the test data, as well as the human annotated gold evaluation set.
Revised: We select the best performing methods and evaluate them on the test data, as well as the human annotated gold evaluation set.

Original: Table 2 shows that the results on the Test set are generally aligned with observations on the validation set.
Revised: Table 2 shows that the results on the Test set are generally aligned with observations on the validation set.

Original: In the goldevaluation set, annotators provide more varieties of the input queries, which can challenge the model more.
Revised: In the gold set, annotators provide more varieties of the input queries, which can challenge the model more.

Original: Interestingly, results on the goldevaluation set suggest that methodswhich perform better on the SEEN entities of the test split do not necessarily transfer to gold evaluation ( e.g ., CLIP Fusion vs. CLIP2CLIP), perhaps due to the aforementioned variation in queries.
Revised: Interestingly, results on the gold set suggest that methods perform better on the SEEN entities of the test split do not necessarily transfer to gold data split ( e.g ., CLIP Fusion vs. CLIP2CLIP), perhaps due to the aforementioned variation in queries.

Revised: Human Performance We conduct a study to estimate the human performance on O VEN -Wiki, via requesting 3 dedicated human annotators to answer 100 examples (sampled from gold evaluation set, answers are non-overlapping).

Revised: Because humans are ineffective in searching 7 , we allow the annotators to use search engines ( e.g ., Google Image Search, Wikipedia Search, etc.), as long as the annotators can provide a valid Wikipedia entity name as answer (among 105K candidates).

Revised: As a result, human achieves 76% average accuracy, which is signiﬁcantly higher than the best comparing systems 8 .

Revised: [Table]

Original: 5. A BLATION S TUDIES AND A NALYSIS
Revised: 5. A BLATION S TUDIES AND A NALYSIS

Original: We perform empirical studies to analyze O VEN -Wiki using the best performing model, CLIP2CLIP.
Revised: We perform empirical studies to analyze O VEN -Wiki using the best performing model, CLIP2CLIP.

Original: Particularly, these studies are related to (1) the importance of input representations for O VEN , (2) the effects of different knowledge information for entities, (3) how ﬁne-tuning inﬂuences generalization, (4) how the size of the Wikipedia knowledge base inﬂuences model performance, and (5) error analysis on the best performing model.
Revised: Particularly, these studies are related to (1) the importance of input representations for O VEN , (2) the effects of different knowledge information for entities, (3) how ﬁne-tuning inﬂuences generalization, (4) how the size of the Wikipedia knowledge base inﬂuences model performance, and (5) error analysis on the best performing model.

Original: Which input information is important for O VEN ?
Revised: Which input information is important for O VEN ?

Original: Table 3 (left) shows scores for CLIP2CLIP variants using different combinations of input query representations.We observe that the input Query Repr. Entity Split
Revised: Table 3 (left) shows scores for CLIP2CLIP variants using different combinations of input query representations.

Original: Query Split Overall x p x t

Original: SEEN UNSEEN SEEN UNSEEN

Original: HM18.

Original: Entity Repr.

Original: Entity Split

Original: Query Split Overall p ( e ) t ( e )

Original: SEEN UNSEEN SEEN UNSEEN HM4. image is most essential for the model when making predictions.
Revised: We observe that the input image is most essential for the model when making predictions.

Original: Meanwhile, we see that query in combination with the image, can provide improvements to results on both the Entity split and the
Revised: Meanwhile, we see that query in combination with the image, can provide improvements to results on both the Entity split and the

Original: Query split, with a larger margin on the latter.
Revised: Query split, with a larger margin on the latter.

Original: Additionally, models that only look at the question can also make a reasonable guess on the Query split, but not the Entity split.
Revised: Additionally, models that only look at the question can also make a reasonable guess on the Query split, but not the Entity split.

Original: Which entity knowledge is important for O VEN ?
Revised: Which entity knowledge is important for O VEN ?

Original: Table 3 (right) provides a study on different combinations of knowledge from Wikipedia, used as the representation for an entity (for the CLIP2CLIP model).
Revised: Table 3 (right) provides a study on different combinations of knowledge from Wikipedia, used as the representation for an entity (for the CLIP2CLIP model).

Original: Particularly, we observe that the Wikipedia entity image itself does not provide sufﬁcient information to support strong O VEN performance, but it contains complementary information that helps the CLIP2CLIP model to better understand the entity text.
Revised: Particularly, we observe that the Wikipedia entity image itself does not provide sufﬁcient information to support strong O VEN performance, but it contains complementary information that helps the CLIP2CLIP model to better understand the entity text.

Original: As a result the best performing model is achieved via combining Wikipedia entity image and entity title, which is also serves as the de facto setting for all Mixed-Modality Dual Encoders in this paper.
Revised: As a result the best performing model is achieved via combining Wikipedia entity image and entity title, which is also serves as the de facto setting for all Mixed-Modality Dual Encoders in this paper.

Original: We note that there is other useful content on Wikipedia pages, such as the summary text.
Revised: We note that there is other useful content on Wikipedia pages, such as the summary text.

Original: However, current experiments show that including summary text does not further improve Mixed-Modality Dual Encoders ( i.e ., overall performances drops ∼ 0.5%), potentially because they are often longer pieces of text with complex structure, which is out-of-domain for the pre-trained models.
Revised: However, current experiments show that including summary text does not further improve Mixed-Modality Dual Encoders ( i.e ., overall results drops ∼ 0.5%), potentially because they are often longer pieces of text with complex structure, which is out-of-domain for the pre-trained models.

Original: The complete study is included in Appendix D.
Revised: Appendix C included the complete study.

Original: [Figure]
Revised: [Figure]

Original: Does ﬁne-tuning always help generalization?
Revised: Does ﬁne-tuning always help generalization?

Original: Figure 4 (a) shows the validation scores of the
Revised: Figure 4 (a) shows the validation scores of the

Original: CLIP2CLIP model, during ﬁne-tuning on O VEN -Wiki’s training split.
Revised: CLIP2CLIP model, during ﬁne-tuning on O VEN -Wiki’s training split.

Original: Note that we only plot the results on Entity split as the representative, because the trends are similar on the Query split.
Revised: Note that we only plot the results on Entity split as the representative, because the trends are similar on the Query split.

Original: It shows that longer training schedule does not converge to better generalization performance, particularly when evaluated on the UNSEEN entities.
Revised: It shows that longer training schedule does not converge to better generalization performance, particularly when evaluated on the UNSEEN entities.

Original: Because of this, we employ the early stopping strategy for the CLIP2CLIP model for model selection, and pick the model that reaches the best UNSEEN performance on validation data.
Revised: Because of this, we employ the early stopping strategy for the CLIP2CLIP model for model selection, and pick the model that reaches the best UNSEEN performance on validation data.

Original: How would the number of entities in KB inﬂuence the model’s prediction?
Revised: How would the number of entities in KB inﬂuence the model’s prediction?

Original: Figure 4 (b) presents the accuracy of CLIP and CLIP2CLIP, as a function of the # of  candidates to contrast against.
Revised: Figure 4 (b) presents the accuracy of CLIP and CLIP2CLIP, as a function of the # of total candidates to retrieve from.

Original: Here, we compute the accuracy byrepeatedly sub-sampling the  KB to different sizes.
Revised: Here, we compute the accuracy by sub-sampling the negative candidates from KB to different sizes.

Original: We observe that when the retrieval candidate entities are only the positive entities (with 0 additional negatives), the performance of both CLIP and CLIP2CLIP is signiﬁcantly higher than the 100K setting.
Revised: We observe that when the retrieval candidate entities are only the positive entities (with 0 additional negatives), the performance of both CLIP and CLIP2CLIP is signiﬁcantly higher than the 100K setting.

Original: Beyond this, as the KB size increases, model accuracy decreases.
Revised: Beyond this, as the KB size increases, model accuracy decreases.

Original: Error analysis We show top 5 predictions of CLIP2CLIP on O VEN -Wiki in Table 4, with the left three examples being the erroneous predictions and the right example the correct prediction.

Original: From left two examples, CLIP2CLIP made mistakes among hard negative entities similar to the ground truth.

Original: On the third example, CLIP2CLIP seems to fail to reason about the image and the query jointly.

Original: [Table]

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Learning to Recognize UNSEEN Categories There has been a signiﬁcant amount of prior work (Lampert et al., 2014;
Revised: Learning to Recognize UNSEEN Categories There has been a signiﬁcant amount of prior work (Lampert et al., 2014;

Original: Vinyals et al., 2016; Liu et al., 2019) focusing on the generalization situation where information of novel categories are presented at test time.
Revised: Vinyals et al., 2016; Liu et al., 2019) focusing on the generalization situation where information of novel categories are presented at test time.

Original: Zero-shot learning (ZSL) is one of such attempts that tackles learning new categories with zero images for training.
Revised: Zero-shot learning (ZSL) is one of such attempts that tackles learning new categories with zero images for training.

Original: To achieve such transfer, ZSL methods typically relies generating UNSEEN image classiﬁers based on corresponding semantic representations, in the format of manually labeled attributes (Lampert et al., 2014), unsupervised learned word vectors (Changpinyo et al., 2016), or pre-trained sentence embeddings (Kil & Chao, 2021;
Revised: To achieve such transfer, ZSL methods typically relies generating UNSEEN image classiﬁers based on corresponding semantic representations, in the format of manually labeled attributes (Lampert et al., 2014), unsupervised learned word vectors (Changpinyo et al., 2016), or pre-trained sentence embeddings (Kil & Chao, 2021;

Original: Radford et al., 2021).
Revised: Radford et al., 2021).

Original: Few-shot learning (FSL) (Vinyals et al., 2016) proposes a more realistic setup, where learners have access to a limited number of visual exemplars during the model deployment.
Revised: Few-shot learning (FSL) (Vinyals et al., 2016) proposes a more realistic setup, where learners have access to a limited number of visual exemplars during the model deployment.

Original: With this goal, FSL methods aim to extract the inductive bias of learning from the SEEN classes, such that the model can leverage it in learning the UNSEEN classes, to avoid severe over-ﬁtting.
Revised: With this goal, FSL methods aim to extract the inductive bias of learning from the SEEN classes, such that the model can leverage it in learning the UNSEEN classes, to avoid severe over-ﬁtting.

Original: Particularly, prior works either use adapted non-parametric classiﬁers (Snell et al., 2017; Rusu et al., 2019;
Revised: Particularly, prior works either use adapted non-parametric classiﬁers (Snell et al., 2017; Rusu et al., 2019;

Original: Ye et al., 2020), or meta-optimized linear classiﬁers (Finn et al., 2017;
Revised: Ye et al., 2020), or meta-optimized linear classiﬁers (Finn et al., 2017;

Original: Raghu et al., 2020) to incorporate the few-shot UNSEEN support examples.
Revised: Raghu et al., 2020) to incorporate the few-shot UNSEEN support examples.

Original: Comparing to them, our proposed task exposes different challenges as we ask the model to make the best use of open-world
Revised: Comparing to them, our proposed task exposes different challenges as we ask the model to make the best use of open-world

Original: Web knowledge ( i.e ., Wikipedia pages with images & ﬁgures), which contains textual semantic information and visual appearance of the entities in the open world.
Revised: Web knowledge ( i.e ., Wikipedia pages with images & ﬁgures), which contains textual semantic information and visual appearance of the entities in the open world.

Original: Vision and Language + Knowledge There has been efforts in combining knowledge into vision and language tasks, such as Visual QA (Shah et al., 2019; Marino et al., 2019) and entity-focused image captioning (Liu et al., 2020; Biten et al., 2019).
Revised: Vision and Language + Knowledge There has been efforts in combining knowledge into vision and language tasks, such as Visual QA (Shah et al., 2019; Marino et al., 2019; Chang et al., 2022; Chen et al., 2021) and entity-focused image captioning (Liu et al., 2020; Biten et al., 2019).

Original: Among them, knowledge-based VQA is most related to O VEN , but also differ in many aspects.
Revised: Among them, knowledge-based VQA is most related to O VEN , but also differ in many aspects.

Original: Speciﬁcally, Shah et al.
Revised: Speciﬁcally, Chang et al.

Revised: (2022) presents a text QA dataset that requires understanding multi-modal knowledge in a KB.

Revised: Shah et al.

Original: (2019) propose to perform knowledge-based question answer tasks, centered around questions that resolves relational query over public Figures.
Revised: (2019) propose to perform knowledge-based question answer tasks, centered around questions that resolves relational query over public Figures.

Original: Meanwhile, Marino et al.
Revised: Meanwhile, Marino et al.

Original: (2019) propose to answer questions where the answer is outside of the image context, to assess model’s capability in understanding real-world knowledge.
Revised: (2019) propose to answer questions where the answer is outside of the image context, to assess model’s capability in understanding real-world knowledge More recently, Chen et al.

Revised: (2021) studies the zero-shot visual

Revised: QA setting where some answers (out of a total of 500 frequent answers of general concepts) are unseen during the training, where a KB is supplied to assist the model in answering unseen answers.

Original: Comparing to them, O VEN steps back to the more fundamental problem of establishing the link between visual content and entity in the KB, but at a larger scale and broader coverage.
Revised: Comparing to them, O VEN steps back to the more fundamental problem of establishing the link between visual content and entity in the KB, but at a larger scale and broader coverage.

Original: We believe that stronger models developed on O VEN would beneﬁts such knowledge intensive visual QA tasks.
Revised: We believe that stronger models developed on O VEN would beneﬁts such knowledge intensive visual QA tasks.

Original: D ISCUSSION
Revised: Entity Linking Entity linking (EL) is the task of grounding entity mentions in text by linking them to entries in a given knowledge base. Supervised EL (Milne & Witten, 2008) has demonstrated its strong performance when all entities are in-distribution during the evaluation. Because KB is updating all the time, recent works (Logeswaran et al., 2019; Botha et al., 2020; Zhang et al., 2021; De Cao et al., 2020; 2022) focus on a more realistic setting where entity linking needs to be achieved in the zero-shot, with a large portion of entities (to be evaluated) completely unseen during the training.

Revised: O VEN is a visual analog of zero-shot EL, and targets at developing generalizable models that recognize entities unseen in the training.

Revised: Among all EL literature, visually assisted EL Zheng et al.

Revised: (2022) is most relevant to this work, whose goal is to use associated image of text to improve the precision of text EL.

Revised: O VEN is different as its text queries do not mention the name of the entities, which put visual understanding and reasoning into the central position.

Revised:  D ISCUSSION

Original: We introduced O VEN , a task that aims to link visual content to the corresponding entities in a knowledge base.
Revised: We introduced O VEN , a task that aims to link visual content to the corresponding entities in a knowledge base.

Original: While the focus of this paper is mostly on the task and the benchmark, our results point to several interesting future avenues for representation learning research.
Revised: While the focus of this paper is mostly on the task and the benchmark, our results point to several interesting future avenues for representation learning research.

Original: For example, the fact that Mixed-Modality Dual Encoders outperform CLIP models suggests that future work on large scale contrastive pretraining could beneﬁt from jointly encoding or mixing modalities.
Revised: For example, the fact that Mixed-Modality Dual Encoders outperform CLIP models suggests that future work on large scale contrastive pretraining could beneﬁt from jointly encoding or mixing modalities.

Original: This is especially critical for unseen entities whose representations are typically learned only during pretraining.
Revised: This is especially critical for unseen entities whose representations are typically learned only during pretraining.

Original: In textual entity linking, De Cao et al.
Revised: In textual entity linking, De Cao et al.

Original: (2021) has shown that it is possible to obtain a high-performing seq2seq model for entity linking.
Revised: (2021) has shown that it is possible to obtain a high-performing seq2seq model for entity linking.

Original: O VEN now provides a benchmark for evaluating such approaches for visual entity linking.
Revised: O VEN now provides a benchmark for evaluating such approaches for visual entity linking.

Original: In summary, we hope O VEN will drive future research on knowledge-infused multimodal representation learning via visual entity linking.
Revised: In summary, we hope O VEN will drive future research on knowledge-infused multimodal representation learning via visual entity linking

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: As our dataset, i e ., O VEN -Wiki, is composed of existing image recognition, image retrieval, and visual question answering datasets, we have introduced minimum risk of exposing additional social bias in our data.
Revised: As our dataset, i e ., O VEN -Wiki, is composed of existing image recognition, image retrieval and visual question answering datasets, we have introduced minimum risk of exposing additional social bias in our data.

Original: However, O VEN -Wiki is still at the risk of inheriting existing dataset biases.
Revised: However, O VEN -Wiki is still at the risk of inheriting existing dataset biases.

Original: As a result, we employed existing data curation strategies (Yang et al., 2020) to reduce such potential risks.
Revised: As a result, we employed existing data curation strategies (Yang et al., 2020) to reduce such potential risks.

Original: Besides such risk O VEN -Wiki also opens up new possibilities that can alleviate ethical concerns in
Revised: Besides such risk, O VEN -Wiki also opens up new possibilities that can alleviate ethical concerns in

Original: AI systems.
Revised: AI systems.

Original: Speciﬁcally, O VEN -Wiki is a dataset that targets advancing research for establishing stronger grounding between the visual content and knowledge base, which can potentially contribute to building more attributed visual systems, such as a visual question answering model that produces answers based on the linked Wikipedia page, with improved interpretability and controllability.
Revised: Speciﬁcally, O VEN -Wiki is a dataset that targets advancing research for establishing stronger grounding between the visual content and knowledge base, which can potentially contribute to building more attributed visual systems, such as a visual question answering model that produces answers based on the linked Wikipedia page, with improved interpretability and controllability.

Original: A E XTENDED L ITERATURE R EVIEW

Original: In this section, we include an additional review to discuss the Textual Entity Linking.

Original: Textual Entity Linking Entity linking (EL) is the task of grounding entity mentions by linking them to entries in a given knowledge base or dictionary of entities, and has been well studied in the NLP communities.

Original: Current supervised EL models can achieve strong performance (Milne & Witten, 2008) on most cases that entities are seen during training.

Original: However, as the KB is updated frequently and demand of linking to specialized domain KB increases, zero-shot EL has gain more attention in the community.

Original: Recent work (Logeswaran et al., 2019) formulates the zero-shot EL task as entity retrieval where each entity is featurized with a text description.

Original: At the same time, many works worked on unifying the zero-shot and supervised EL setup and achieved remarkable performance gain using large models to both seen and unseen entities from the text (Botha et al., 2020; Zhang et al., 2021;

Original: De Cao et al., 2020; 2022).

Original: As we propose O VEN as a visual analog of the EL, we believe that similar trends could happen to this line of research in the future as well.

Original: B D ATASET C ONSTRUCTION , A NNOTATION , AND A DDITIONAL S TATISTICS
Revised: A D ATASET C ONSTRUCTION , A NNOTATION , AND A DDITIONAL S TATISTICS

Original: In this section, we describes the complete details on data collection, curation, entity linking, and show additional statistics of the processed dataset (§B.1).
Revised: In this section, we describes the complete details on data collection, curation, entity linking, and show additional statistics of the processed dataset (§A.1).

Original: Then we also discuss how we train annotators to annotate our task, and provide the concrete annotation interface(§B.2).
Revised: Then we also discuss how we train annotators to annotate our task, and provide the concrete annotation interface(§A.2).

Original: B. D ATA C OLLECTION & P RE - PROCESSING
Revised: A. D ATA C OLLECTION & P RE - PROCESSING

Original: Data Filtering Some of our member datasets have been reported to include non-imageable classes, classes with undesired social bias (Yang et al., 2020), or non-entity classes ( e.g ., numbers).
Revised: Data Filtering Some of our member datasets have been reported to include non-imageable classes, classes with undesired social bias (Yang et al., 2020), or non-entity classes ( e.g ., numbers).

Original: Therefore, we apply a ﬁltering process to compose our dataset, based on the individual condition of each source dataset.
Revised: Therefore, we apply a ﬁltering process to compose our dataset, based on the individual condition of each source dataset.

Original: Overall, to create the Entity split, we ﬁrst apply a general safety ﬁlter (Yang et al., 2020) to remove non-imageable labels, non-entity labels, and labels with social bias.
Revised: Overall, to create the Entity split, we ﬁrst apply a general safety ﬁlter (Yang et al., 2020) to remove non-imageable labels, non-entity labels, and labels with social bias.

Original: To create the Query split, we employed three expert annotators to write heuristic policies to ﬁlter each VQA dataset, and ensure our task is focusing on entity related questions.
Revised: To create the Query split, we employed three expert annotators to write heuristic policies to ﬁlter each VQA dataset, and ensure our task is focusing on entity related questions.

Original: Concretely, questions related to counting, veriﬁcation, or querying non-entity attributes ( e.g ., dates), are removed.
Revised: Concretely, questions related to counting, veriﬁcation, or querying non-entity attributes ( e.g ., dates), are removed.

Original: Then we apply the same safety ﬁlter.
Revised: Then we apply the same safety ﬁlter.

Original: We provide more details in Appendix.
Revised: We provide more details in Appendix.

Original: Entity Linking Based on the ﬁltered data, we developed a two-staged entity linking strategy to connect the label text to Wikipedia entities, on both Entity and Query splits.
Revised: Entity Linking Based on the ﬁltered data, we developed a two-staged entity linking strategy to connect the label text to Wikipedia entities, on both Entity and Query splits.

Original: First, we obtain exact match based entity candidates by querying the Wikipedia search API (with the auto-suggestion disabled) with the raw label text.
Revised: First, we obtain exact match based entity candidates by querying the Wikipedia search API (with the auto-suggestion disabled) with the raw label text.

Original: We reject candidates whose landing pages are identiﬁed as disambiguation pages.
Revised: We reject candidates whose landing pages are identiﬁed as disambiguation pages.

Original: The Wikipedia API 6 automatically redirects queries (in our case, labels) matching entity aliases to their canonical form.
Revised: The Wikipedia API 9 automatically redirects queries (in our case, labels) matching entity aliases to their canonical form.

Original: For the labels which do not have an exact match in
Revised: For the labels which do not have an exact match in

Original: Wikipedia, we use a state-of-the-art text-based entity linker ( i.e ., GENRE (De Cao et al., 2021)) to obtain top candidate Wikipedia entity names.
Revised: Wikipedia, we use a state-of-the-art text-based entity linker ( i.e ., GENRE (De Cao et al., 2021)) to obtain top candidate Wikipedia entity names.

Original: Finally, we link the label to the top ranked entity whose landing page is not a disambiguation page.
Revised: Finally, we link the label to the top ranked entity whose landing page is not a disambiguation page.

Original: Preparing Multi-Modal Knowledge Using the entity linking process described earlier, we successfully connect a total of 24,895 class labels in O VEN -Wiki to corresponding Wikipedia entities.
Revised: Preparing Multi-Modal Knowledge Using the entity linking process described earlier, we successfully connect a total of 24,895 class labels in O VEN -Wiki to corresponding Wikipedia entities.

Original: Overall, our dataset contains 20,801 unique entities.
Revised: Overall, our dataset contains 20,801 unique entities.

Original: For the Entity split data, we generate a synthetic text query based on the super-category information of the label (either provided by source dataset or mined from Wikidata 7 ), using templated language.
Revised: For the Entity split data, we generate a synthetic text query based on the super-category information of the label (either provided by source dataset or mined from Wikidata 10 ), using templated language.

Original: For example, iNaturalist has provided detailed supercategory annotation on each class, such as Plantae , Reptilia , etc .
Revised: For example, iNaturalist has provided detailed supercategory annotation on each class, such as Plantae , Reptilia , etc .

Original: For dataset that do not provide this information, we use the super-category mined from Wikidata, which is publiclly crowd sourced and maintained.
Revised: For dataset that do not provide this information, we use the super-category mined from Wikidata, which is publiclly crowd sourced and maintained.

Original: As a result, our templated query generator produces the query “what is the species of the plant in this image?”
Revised: As a result, our templated query generator produces the query “what is the species of the plant in this image?”

Original: for the entity “Eryngium alpinum” , whose super-category is Plantae .
Revised: for the entity “Eryngium alpinum” , whose super-category is Plantae .

Original: Due to space limit, we provide more explanation in Appendix.
Revised: Due to space limit, we provide more explanation in Appendix.

Original: For all Wikipedia entities, we use the corresponding Wikipedia page and its associating multi-media content ( e.g ., information box images, etc .)
Revised: For all Wikipedia entities, we use the corresponding Wikipedia page and its associating multi-media content ( e.g ., information box images, etc .)

Original: as the source of multi-modal knowledge about entities.
Revised: as the source of multi-modal knowledge about entities.

Original: [Figure]
Revised: [Figure]

Original: Statistics on Entities Speciﬁcally, Figure 5 shows the number of unique entities in both the Entity and Query splits, where we compare the total number of entities in each source dataset against its original population (after applied safety ﬁlter).
Revised: Statistics on Entities Speciﬁcally, Figure 5 shows the number of unique entities in both the Entity and Query splits, where we compare the total number of entities in each source dataset against its original population (after applied safety ﬁlter).

Original: Note that for the Google Landmarks v2 (Gldv2) dataset, we employed the cleaned data split from Yokoo et al.
Revised: Note that for the Google Landmarks v2 (Gldv2) dataset, we employed the cleaned data split from Yokoo et al.

Original: (2020), where the total number of unique entities is signiﬁcantly reduced.
Revised: (2020), where the total number of unique entities is signiﬁcantly reduced.

Original: Because Gldv2 is automatically generated and has reported to contain noises particularly with tail entities (Yokoo et al., 2020), we removed entities with less than 50 instances for a improved precision (further reduces the # of entities in Gldv2 to ∼ 6k).
Revised: Because Gldv2 is automatically generated and has reported to contain noises particularly with tail entities (Yokoo et al., 2020), we removed entities with less than 50 instances for a improved precision (further reduces the # of entities in Gldv2 to ∼ 6k).

Original: Entity Super-Categories To give more details for the Figure 3 in the main text, we further present full super-category grouping information in Figure 6.
Revised: Entity Super-Categories To give more details for the Figure 3 in the main text, we further present full super-category grouping information in Figure 6.

Original: As aforementioned, we have combined entities that belongs to general groups ( e.g ., “object”, “item” groups) or unpopular groups ( e.g ., groups with less than 5 entities) into the “others” group.
Revised: As aforementioned, we have combined entities that belongs to general groups ( e.g ., “object”, “item” groups) or unpopular groups ( e.g ., groups with less than 5 entities) into the “others” group.

Original: We also merged some sub-categories into super-categories, e.g ., “location”+“park”+“lake”+“river”+“mountain” → “location”, “building”+“bridge” → “building”.[Figure]
Revised: We also merged some sub-categories into super-categories, e.g ., “location”+“park”+“lake”+“river”+“mountain” → “location”, “building”+“bridge” → “building”.

Original: [Figure]
Revised: [Figure]

Original: B. H UMAN A NNOTATION P ROCEDURE & I NTERFACE
Revised: A. H UMAN A NNOTATION P ROCEDURE & I NTERFACE

Original: In order to verify the quality of O VEN -Wiki and to provide a human veriﬁed test set to evaluate on, we conduct human annotation on a subset of test set.
Revised: In order to verify the quality of O VEN -Wiki and to provide a human veriﬁed test set to evaluate on, we conduct human annotation on a subset of test set.

Original: The annotators are asked to correct the errors in the <image, query, answer> triplets.
Revised: The annotators are asked to correct the errors in the <image, query, answer> triplets.

Original: The details are as follows.
Revised: The details are as follows.

Original: Annotation interface Figure 7 illustrates the annotation interface.
Revised: Annotation interface Figure 7 illustrates the annotation interface.

Original: The left side of Figure 7 are the input to the annotators which includes the original question, image and the answer (together with the wikipedia hyperlink).
Revised: The left side of Figure 7 are the input to the annotators which includes the original question, image and the answer (together with the wikipedia hyperlink).

Original: The annotators are asked to complete the following questions:
Revised: The annotators are asked to complete the following questions:

Original: Does the Wikipedia represent the correct meaning of the answer?
Revised: Does the Wikipedia represent the correct meaning of the answer?

Original: Provide the Wikipedia link if not.
Revised: Provide the Wikipedia link if not.

Original: This question requires the annotators to correct the entity linking errors.
Revised: This question requires the annotators to correct the entity linking errors.

Original: The annotators use
Revised: The annotators use

Original: Google search to ﬁnd the most suitable Wikipedia link if the provided one is not adequate.
Revised: Google search to ﬁnd the most suitable Wikipedia link if the provided one is not adequate.

Original: In our dataset, 8.4% of the entity links are reported wrong by more than 2 annotators.
Revised: In our dataset, 8.4% of the entity links are reported wrong by more than 2 annotators.

Original: Is the Wikipedia answer physically present in the image.
Revised: Is the Wikipedia answer physically present in the image

Original: This question is mainly aimed at ﬁltering out the OCR examples which are out of our scope.
Revised: This question is mainly aimed at ﬁltering out the OCR examples which are out of our scope.

Original: One example is that the image about a wall painted with the word “love" and the linked entity is the “love" Wikipedia.
Revised: One example is that the image about a wall painted with the word “love" and the linked entity is the “love" Wikipedia.

Original: In our dataset, 10.3% of the answers are reported not physically present in the image by more than 2 annotators.
Revised: In our dataset, 10.3% of the answers are reported not physically present in the image by more than 2 annotators.

Original: Rewrite the question so that no other object can be the answer.
Revised: Rewrite the question so that no other object can be the answer.

Original: The annotators will rewrite the question is the answer is wrong or ambiguous.
Revised: The annotators will rewrite the question is the answer is wrong or ambiguous.

Original: Annotators will make sure that the question can not be answered without the image and that the answers can not be included in the rewritten questions.
Revised: Annotators will make sure that the question can not be answered without the image and that the answers can not be included in the rewritten questions.

Original: In our annotation, 99.9% of the questions are being rewritten.
Revised: In our annotation, 99.9% of the questions are being rewritten.

Original: Instruction and Training We carefully design the training procedure to improve the annotation quality.
Revised: Instruction and Training We carefully design the training procedure to improve the annotation quality.

Original: We ﬁrst conduct a “Self-study session” where the annotators will read the instructions and annotate a few toy examples.
Revised: We ﬁrst conduct a “Self-study session” where the annotators will read the instructions and annotate a few toy examples.

Original: Then we conduct a “In-person tutorial” where we have an online video session in which we walk annotators through the full version of the instructions and discuss mistakes made in the self-study annotations.
Revised: Then we conduct a “In-person tutorial” where we have an online video session in which we walk annotators through the full version of the instructions and discuss mistakes made in the self-study annotations.

Original: Finally we conduct a “Test exam” and the qualiﬁed annotators are accepted.
Revised: Finally we conduct a “Test exam” and the qualiﬁed annotators are accepted.

Original: In total, 30 annotators went through our training procedure and all of them were eventually accepted to work full-time on the main task.
Revised: In total, 30 annotators went through our training procedure and all of them were eventually accepted to work full-time on the main task.

Original: Quality control We have a three way annotations where each examples are annotated by three annotators.
Revised: Quality control We have a three way annotations where each examples are annotated by three annotators.

Original: We were giving regular feedback on the questions the annotators may have during the annotation and pointed out mistakes identiﬁed in annotators’ past answers.
Revised: We were giving regular feedback on the questions the annotators may have during the annotation and pointed out mistakes identiﬁed in annotators’ past answers.

Original: On average, it took annotators 4.6 minutes to answer each question with the time consumption slightly decreasing as annotators get familiar with the task.
Revised: On average, it took annotators 4.6 minutes to answer each question with the time consumption slightly decreasing as annotators get familiar with the task.

Original: The compensation rate for the task was set to be $17.8/hour which is higher than the minimum hourly wage in the US.
Revised: The compensation rate for the task was set to be $17.8/hour which is higher than the minimum hourly wage in the US.

Original: We ﬁltered out all the examples where the wikipedia links are marked as wrong or the Wikipedia answers are marked as “Not physically present in the image".
Revised: We ﬁltered out all the examples where the wikipedia links are marked as wrong or the Wikipedia answers are marked as “Not physically present in the image".

Original: C I MPLEMENTATION D ETAILS OF THE BASELINE SYSTEMS
Revised: B I MPLEMENTATION D ETAILS OF THE BASELINE SYSTEMS

Original: In this section, we provide complete impelmentation details on the baselines for the O VEN task.
Revised: In this section, we provide complete impelmentation details on the baselines for the O VEN task.

Original: C. S IMPLER VLM & D UAL S IMPLER VLM
Revised: B. S IMPLER VLM & D UAL S IMPLER VLM

Original: As aforementioned, we have simpliﬁed the architecture of original SimVLM (Wang et al., 2021) via using a Multi-Modal Transformer model (the same architecture as T5- Base by Raffel et al.
Revised: As aforementioned, we have simpliﬁed the architecture of original SimVLM (Wang et al., 2021) via using a Multi-Modal Transformer model (the same architecture as T5- Base by Raffel et al.

Original: Speciﬁcally, we ﬁrst extract non-overlapping 16 × 16 patches from the input image, and perform a linear transformation on them to get a sequendce of embeded patch embeddings, which is then combined with the word embeddings of a tokenized text sequence to serve as the input.
Revised: Speciﬁcally, we ﬁrst extract non-overlapping 16 × 16 patches from the input image, and perform a linear transformation on them to get a sequendce of embeded patch embeddings, which is then combined with the word embeddings of a tokenized text sequence to serve as the input.

Original: The Multi-Modal Transformer then take the combined sequence and contextualize the embeddings through self-attention (Vaswani et al., 2017).
Revised: The Multi-Modal Transformer then take the combined sequence and contextualize the embeddings through self-attention (Vaswani et al., 2017).

Original: In the re-implemented SimplerVLM, we then feed the output contextualized embeddings into a
Revised: In the re-implemented SimplerVLM, we then feed the output contextualized embeddings into a

Original: Text decoder (of same architecture), which can then produce text prediction.
Revised: Text decoder (of same architecture), which can then produce text prediction.

Original: We ﬁrst pre-train this model on the combined Fit400M (Yu et al., 2022) and C4 (Raffel et al., 2020) dataset, under the
Revised: We ﬁrst pre-train this model on the combined Fit400M (Yu et al., 2022) and C4 (Raffel et al., 2020) dataset, under the

Original: PreﬁxLM objective deﬁned in Wang et al.
Revised: PreﬁxLM objective deﬁned in Wang et al.

Original: We optimize the model for 500K steps (with batch size = 4,096), using the Adafactor (Shazeer & Stern, 2018) optimizer, a initial learning rate of 0.01 and square root learning rate decay policy (with the ﬁnal learning rate goes to 1e-4).
Revised: We optimize the model for 500K steps (with batch size = 4,096), using the Adafactor (Shazeer & Stern, 2018) optimizer, a initial learning rate of 0.01 and square root learning rate decay policy (with the ﬁnal learning rate goes to 1e-4).

Original: Then we ﬁne-tune the pre-trained SimplerVLM on the proposed O VEN -Wiki dataset, for 20k steps in total.
Revised: Then we ﬁne-tune the pre-trained SimplerVLM on the proposed O VEN -Wiki dataset, for 20k steps in total.

Original: The optimization is also done with the Adafactor optimizer, with the initial learning rate to be 0.001, the learning rate scheduler to be square root LR decay, and the ﬁnal learning rate to be 1e-6.
Revised: The optimization is also done with the Adafactor optimizer, with the initial learning rate to be 0.001, the learning rate scheduler to be square root LR decay, and the ﬁnal learning rate to be 1e-6.

Original: During the inference, the model is then directly predicting the name of the entity, given the input text query and the context image.
Revised: During the inference, the model is then directly predicting the name of the entity, given the input text query and the context image.

Original: W count the exact matched entity name to compute the Macro F1 accuracy.
Revised: W count the exact matched entity name to compute the Macro F1 accuracy.

Original: In the Dual SimplerVLM, we take the pre-trained SimplerVLM (same as above), remove its Text decoder, and use it for both the left and right encoders of the dual encoder model.
Revised: In the Dual SimplerVLM, we take the pre-trained SimplerVLM (same as above), remove its Text decoder, and use it for both the left and right encoders of the dual encoder model.

Original: Therefore, the left query encoder has the same initialization as the right entity encoder.
Revised: Therefore, the left query encoder has the same initialization as the right entity encoder.

Original: We then ﬁne-tune the result dual encoders model on the training set of O VEN -Wiki, under a in-batch contrastive learning objective (Radford et al., 2021), with a batch size of 4,096.
Revised: We then ﬁne-tune the result dual encoders model on the training set of O VEN -Wiki, under a in-batch contrastive learning objective (Radford et al., 2021), with a batch size of 4,096.

Original: Speciﬁcally, we optimize the model for 20K steps during the ﬁne-tuning, again with Adafactor optimizer and a initial learning rate of 0.001.
Revised: Speciﬁcally, we optimize the model for 20K steps during the ﬁne-tuning, again with Adafactor optimizer and a initial learning rate of 0.001.

Original: C. CLIP F USION M ODEL
Revised: B. CLIP F USION M ODEL

Original: As aforementioned, we implemented this Mixed-Modal Dual Encoder via taking pre-trained CLIP image and text encoders as featurizers.
Revised: As aforementioned, we implemented this Mixed-Modal Dual Encoder via taking pre-trained CLIP image and text encoders as featurizers.

Original: Particularly, we build two 2-layer Transformer models, on top of two CLIP models as the left and right encoder, for encoding the query representation and the entity representation, respectively.
Revised: Particularly, we build two 2-layer Transformer models, on top of two CLIP models as the left and right encoder, for encoding the query representation and the entity representation, respectively.

Original: The 2-layer Transformers follows the same architecture as T
Revised: The 2-layer Transformers follows the same architecture as T

Original: Transformer Radford et al.
Revised: Transformer Radford et al.

Original: , but with 2 layers, 12 attention heads, with each attention head of 64 dimensions, and the embedding size of 768.
Revised: , but with 2 layers, 12 attention heads, with each attention head of 64 dimensions, and the embedding size of 768.

Original: We then ﬁne-tune this composed model on the
Revised: We then ﬁne-tune this composed model on the

Original: O VEN -Wiki’s training data, using the same contrastive objective as Dual SimplerVLM.
Revised: O VEN -Wiki’s training data, using the same contrastive objective as Dual SimplerVLM.

Original: We optimize the model for 20K steps in the ﬁne-tuning stage, following the same hyper-parameter conﬁgurations as Dual SimplerVLM.
Revised: We optimize the model for 20K steps in the ﬁne-tuning stage, following the same hyper-parameter conﬁgurations as Dual SimplerVLM.

Original: C. CLIP2CLIP M ODEL
Revised: B. CLIP2CLIP M ODEL

Original: Different from CLIP Fusion, CLIP2CLIP is a model that adds minimum new parameters to the pre-trained CLIP encoders.
Revised: Different from CLIP Fusion, CLIP2CLIP is a model that adds minimum new parameters to the pre-trained CLIP encoders.

Original: Same as other models, we initialize both the query encoder and the target encoder separately with the pre-trained CLIP model.
Revised: Same as other models, we initialize both the query encoder and the target encoder separately with the pre-trained CLIP model.

Original: Speciﬁcally, we use the pre-trained CLIP encoders for both left and right encoders, to encode the image and text modality for both the query representation and the entity representation.
Revised: Speciﬁcally, we use the pre-trained CLIP encoders for both left and right encoders, to encode the image and text modality for both the query representation and the entity representation.

Original: We then compute the four dot product similarity scores on the < input image, target text > , < input text, target image > , < input image, target image > , and < input text, target text > pairs, which is then combined via a learnable similarity weights into one logit score.
Revised: We then compute the four dot product similarity scores on the < input image, target text > , < input text, target image > , < input image, target image > , and < input text, target text > pairs, which is then combined via a learnable similarity weights into one logit score.

Original: The make sure that the learnable similarity weights is initialized properly, we perform a grid search to ﬁnd a roughly good similarity weights for the CLIP2CLIP model (using O VEN -Wiki’s training data).
Revised: The make sure that the learnable similarity weights is initialized properly, we perform a grid search to ﬁnd a roughly good similarity weights for the CLIP2CLIP model (using O VEN -Wiki’s training data).

Original: Then we took this similarity weights to initialize the CLIP2CLIP model and ﬁne-tune all parameters on O VEN -Wiki’s training set, under the same contrastive learning objective.
Revised: Then we took this similarity weights to initialize the CLIP2CLIP model and ﬁne-tune all parameters on O VEN -Wiki’s training set, under the same contrastive learning objective.

Original: Different from other models, given that this model has most of its parameters pre-trained, we realized that it works the best to early stop the model.
Revised: Different from other models, given that this model has most of its parameters pre-trained, we realized that it works the best to early stop the model.

Original: As a result, we only ﬁne-tune this model for 2k steps, with an initial learning rate of 1e-4, and a square root LR decay schedule with ﬁnal learning rate of 1e-6.
Revised: As a result, we only ﬁne-tune this model for 2k steps, with an initial learning rate of 1e-4, and a square root LR decay schedule with ﬁnal learning rate of 1e-6.

Original: We select the model along the way with the best UNSEEN entity performances.
Revised: We select the model along the way with the best UNSEEN entity performances.

Original: D A DDDITIONAL E XPERIMENTS
Revised: C A DDDITIONAL E XPERIMENTS

Original: In this section [Table] we provide experiments that is omitted in the main text due to the space limit.
Revised: In this section, we provide experiments that is omitted in the main text due to the space limit.

Original: [Table]
Revised: Alternative Evaluation Metrics on O VEN -Wiki In addition to the primary evaluation metric, we also report the performance of dual encoder models using ranking based metrics, such as Recall@ (R10) and Mean reciprocal rank@10 (MRR10). The results are shown in Table 4 and Table 5. Overall speaking, the results are showing the same trend as our primary metric ( i.e ., Macro accuracy).

Original: Complete Studies for Entity Representation Table 5 (right) provides a study on different combination of knowledge from Wikipedia to use as the representation for an entity (for CLIP2CLIP model).
Revised: Complete Studies for Entity Representation Table 6 (right) provides a study on different combination of knowledge from Wikipedia to use as the representation for an entity (for CLIP2CLIP model).

Original: Particularly, we observe that image is always beneﬁcial to include as it improve the results when combined with text description.
Revised: Particularly, we observe that image is always beneﬁcial to include as it improve the results when combined with text description.

Original: Comparing to summary, the title information can be better used by the CLIP2CLIP model.
Revised: Comparing to summary, the title information can be better used by the CLIP2CLIP model.

Original: The best combination is to have both Wikipedia image and Wikipedia title as the entity representation, which is the default conﬁguration in this paper.
Revised: The best combination is to have both Wikipedia image and Wikipedia title as the entity representation, which is the default conﬁguration in this paper.

Original: We note that there are many other useful content on Wikipedia pages that we have not yet exploited ( e
Revised: We note that there are many other useful content on Wikipedia pages that we have not yet exploited ( e.g ., Section text, tables, other ﬂoating images, etc .), which can potentially bring poential oportunity for further improvement. [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: g ., Section text, tables, other ﬂoating images, etc .), which can potentially bring poential oportunity for further improvement.
Revised: Error analysis We show top 5 predictions of CLIP2CLIP on O VEN -Wiki in Table 7, with the left three examples being the erroneous predictions and the right example the correct prediction. From left two examples, CLIP2CLIP made mistakes among hard negative entities similar to the ground truth. On the third example, CLIP2CLIP seems to fail to reason about the image and the query jointly. [Table]