Original: E VALUATING LANGUAGE - BIASED IMAGE CLASSIFICATION BASED ON SEMANTIC REPRESENTATIONS
Revised: E VALUATING LANGUAGE - BIASED IMAGE CLASSIFICATION BASED ON SEMANTIC REPRESENTATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Humans show language-biased image recognition for a word-embedded image, known as picture-word interference.
Revised: Humans show language-biased image recognition for a word-embedded image, known as picture-word interference.

Original: Such interference depends on hierarchical semantic categories and reﬂects that human language processing highly interacts with visual processing.
Revised: Such interference depends on hierarchical semantic categories and reﬂects that human language processing highly interacts with visual processing.

Original: Similar to humans, recent artiﬁcial models jointly trained on texts and images, e.g., OpenAI CLIP, show language-biased image classiﬁcation.
Revised: Similar to humans, recent artiﬁcial models jointly trained on texts and images, e.g., OpenAI CLIP, show language-biased image classiﬁcation.

Original: Exploring whether the bias leads to interferences similar to those observed in humans can contribute to understanding how much the model acquires hierarchical semantic representations from joint learning of language and vision.
Revised: Exploring whether the bias leads to interferences similar to those observed in humans can contribute to understanding how much the model acquires hierarchical semantic representations from joint learning of language and vision.

Original: The present study introduces methodological tools from the cognitive science literature to assess the biases of artiﬁcial models.
Revised: The present study introduces methodological tools from the cognitive science literature to assess the biases of artiﬁcial models.

Original: Speciﬁcally, we introduce a benchmark task to test whether words superimposed on images can distort the image classiﬁcation across different category levels and, if it can, whether the perturbation is due to the shared semantic representation between language and vision.
Revised: Speciﬁcally, we introduce a benchmark task to test whether words superimposed on images can distort the image classiﬁcation across different category levels and, if it can, whether the perturbation is due to the shared semantic representation between language and vision.

Original: Our dataset is a set of word-embedded images and consists of a mixture of natural image datasets and hierarchical word labels with superordinate/basic category levels.
Revised: Our dataset is a set of word-embedded images and consists of a mixture of natural image datasets and hierarchical word labels with superordinate/basic category levels.

Original: Using this benchmark test, we evaluate the CLIP model.
Revised: Using this benchmark test, we evaluate the CLIP model.

Original: We show that presenting words distorts the image classiﬁcation by the model across different category levels, but the effect does not depend on the semantic relationship between images and embedded words.
Revised: We show that presenting words distorts the image classiﬁcation by the model across different category levels, but the effect does not depend on the semantic relationship between images and embedded words.

Original: This suggests that the semantic word representation in the
Revised: This suggests that the semantic word representation in the

Original: CLIP visual processing is not shared with the image representation, although the word representation strongly dominates for word-embedded images.
Revised: CLIP visual processing is not shared with the image representation, although the word representation strongly dominates for word-embedded images.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Language is not only a fundamental tool enabling humans to communicate with each other: it also shapes other aspects of perception and cognition, ranging from inﬂuencing visual processing to enabling abstract thinking (Vygotsky, 1934; Miller, 1951; Waxman & Markow, 1995; Colas et al., 2021).
Revised: Language is not only a fundamental tool enabling humans to communicate with each other: it also shapes other aspects of perception and cognition, ranging from inﬂuencing visual processing to enabling abstract thinking (Vygotsky, 1934; Miller, 1951; Waxman & Markow, 1995; Colas et al., 2021).

Original: Analogous to this human ability, recent artiﬁcial intelligence technology has also utilized language to drive the training of visual skills in machines.
Revised: Analogous to this human ability, recent artiﬁcial intelligence technology has also utilized language to drive the training of visual skills in machines.

Original: For instance, the OpenAI team introduced CLIP (Contrastive Language–Image Pre-training), consisting of joint learning of language and vision (Radford et al., 2021).
Revised: For instance, the OpenAI team introduced CLIP (Contrastive Language–Image Pre-training), consisting of joint learning of language and vision (Radford et al., 2021).

Original: The CLIP model efﬁciently learns visual concepts from natural language supervision and can be applied to various visual tasks in a zero-shot manner.
Revised: The CLIP model efﬁciently learns visual concepts from natural language supervision and can be applied to various visual tasks in a zero-shot manner.

Original: Joint learning of language and vision has also been utilized in the Visual Question Answering (VQA) literature, where artiﬁcial models are asked to answer questions about visual content (Antol et al., 2015;
Revised: Joint learning of language and vision has also been utilized in the Visual Question Answering (VQA) literature, where artiﬁcial models are asked to answer questions about visual content (Antol et al., 2015;

Original: Lu et al., 2016; Fukui et al., 2016).
Revised: Lu et al., 2016; Fukui et al., 2016).

Original: While joint training with language effectively fosters the acquisition of general abilities for visual recognition/understanding, its abstraction can produce biased recognition for either humans or machines.
Revised: While joint training with language effectively fosters the acquisition of general abilities for visual recognition/understanding, its abstraction can produce biased recognition for either humans or machines.

Original: The Stroop effect known in human cognitive science literature is one of such biases (Stroop, 1935).
Revised: The Stroop effect known in human cognitive science literature is one of such biases (Stroop, 1935).

Original: When human participants observe a mismatched word, where the word “red” is printed with the “blue” color, and are asked to answer the “color” of the word, their response to the word can be slower than when they see the word “red” with the “red” color.
Revised: When human participants observe a mismatched word, where the word “red” is printed with the “blue” color, and are asked to answer the “color” of the word, their response to the word can be slower than when they see the word “red” with the “red” color.

Original: In addition to the color-word interference, language also interferes with human image category recognition, known as picture-word interference (Rosinski, 1977; Lupker, 1979).
Revised: In addition to the color-word interference, language also interferes with human image category recognition, known as picture-word interference (Rosinski, 1977; Lupker, 1979).

Original: When a participant observes an image coupled with an incongruent category word (e.g., Figure 1, right), the image categorization speed can be delayed.
Revised: When a participant observes an image coupled with an incongruent category word (e.g., Figure 1, right), the image categorization speed can be delayed.

Original: Similar to the effect in humans, recent deep learning systems jointly trained on language and vision, e.g., CLIP, also show object recognition interference when words are embedded in images (Goh et al., 2021).
Revised: Similar to the effect in humans, recent deep learning systems jointly trained on language and vision, e.g., CLIP, also show object recognition interference when words are embedded in images (Goh et al., 2021).

Original: Speciﬁcally, by simply adding a handwritten word to an image, the object recognition can be biased to the superimposed word.
Revised: Speciﬁcally, by simply adding a handwritten word to an image, the object recognition can be biased to the superimposed word.

Original: A critical aspect of picture-word interference in humans is that it depends on the semantic relationship between images and superimposed words.
Revised: A critical aspect of picture-word interference in humans is that it depends on the semantic relationship between images and superimposed words.

Original: For instance, the reaction time on the image classiﬁcation for word-superimposed images is slower when the image category is semantically similar to the word one (Rosinski, 1977).
Revised: For instance, the reaction time on the image classiﬁcation for word-superimposed images is slower when the image category is semantically similar to the word one (Rosinski, 1977).

Original: Based on the accumulated ﬁndings, previous works have commonly suggested that at least two different processes mediate the effect in humans, as introduced in detail in Section 2.2.
Revised: Based on the accumulated ﬁndings, previous works have commonly suggested that at least two different processes mediate the effect in humans, as introduced in detail in Section 2.2.

Original: Speciﬁcally, when a participant observes a word-superimposed image, an activation process synthesizes semantic representations corresponding to the superimposed word and image (Figure 2, activation process).
Revised: Speciﬁcally, when a participant observes a word-superimposed image, an activation process synthesizes semantic representations corresponding to the superimposed word and image (Figure 2, activation process).

Original: Based on the representations, it has been assumed that a selection process decides which possible activation is the answer for the current task.(Figure 2, selection process).
Revised: Based on the representations, it has been assumed that a selection process decides which possible activation is the answer for the current task.(Figure 2, selection process).

Original: The activated representation is shared across visual word forms and images, and the dual activation by words and images confuses the decision in the selection process.
Revised: The activated representation is shared across visual word forms and images, and the dual activation by words and images confuses the decision in the selection process.

Original: These human cognitive ﬁndings raise the question of whether interferences in artiﬁcial models have some similarities with those observed in humans.
Revised: These human cognitive ﬁndings raise the question of whether interferences in artiﬁcial models have some similarities with those observed in humans.

Original: Indeed, some artiﬁcial models show languagebiased image classiﬁcation, but one possibility is that the bias might emerge irrespective of the semantic interaction between words and images.
Revised: Indeed, some artiﬁcial models show languagebiased image classiﬁcation, but one possibility is that the bias might emerge irrespective of the semantic interaction between words and images.

Original: When we consider a model with text and image encoders for picture-word interference (Figure 2), we can regard the functional role of the image encoder as synthesizing semantic representations for word-superimposed images.
Revised: When we consider a model with text and image encoders for picture-word interference (Figure 2), we can regard the functional role of the image encoder as synthesizing semantic representations for word-superimposed images.

Original: Although pictureword interference in humans depends on shared semantic representations of written words and images, this is not the only mechanism that can produce interference.
Revised: Although pictureword interference in humans depends on shared semantic representations of written words and images, this is not the only mechanism that can produce interference.

Original: For instance, the image encoder of the model may assign the written word ”dog” a different semantic representation than the visual image ”dog” and show strong preferential activation to written words while ignoring image contents.
Revised: For instance, the image encoder of the model may assign the written word ”dog” a different semantic representation than the visual image ”dog” and show strong preferential activation to written words while ignoring image contents.

Original: Since joint learning of language and vision is expected to enable machines to acquire highly interacted representations of texts and images to solve various general tasks, observing such independent representations and a bias for one of them can be problematic and show a limit of the joint training process.
Revised: Since joint learning of language and vision is expected to enable machines to acquire highly interacted representations of texts and images to solve various general tasks, observing such independent representations and a bias for one of them can be problematic and show a limit of the joint training process.

Original: In this study, we import methodological tools from the cognitive science literature to assess biases of artiﬁcial models in image classiﬁcation for word-superimposed images.
Revised: In this study, we import methodological tools from the cognitive science literature to assess biases of artiﬁcial models in image classiﬁcation for word-superimposed images.

Original: Speciﬁcally, we introduce a paradigm to evaluate picture-word interference, in which an agent has to predict the image superordinate/basic category for images with superordinate/basic words (Figure 1).
Revised: Speciﬁcally, we introduce a paradigm to evaluate picture-word interference, in which an agent has to predict the image superordinate/basic category for images with superordinate/basic words (Figure 1).

Original: Our benchmark dataset is a set of images with superimposed words and consists of a mixture of natural image datasets (Cichy et al., 2016; Mohsenzadeh et al., 2019) and hierarchical word labels (Lin et al., 2014; Krizhevsky et al., 2009).
Revised: Our benchmark dataset is a set of images with superimposed words and consists of a mixture of natural image datasets (Cichy et al., 2016; Mohsenzadeh et al., 2019) and hierarchical word labels (Lin et al., 2014; Krizhevsky et al., 2009).

Original: Our benchmark task aims to test artiﬁcial models with a text encoder and an image encoder (Figure 2).
Revised: Our benchmark task aims to test artiﬁcial models with a text encoder and an image encoder (Figure 2).

Original: The task enables us to test 1) whether language-biased decisions happen across different category levels and 2) the extent to which picture-word interference in artiﬁcial models depends on the semantic similarity between superimposed words and images.
Revised: The task enables us to test 1) whether language-biased decisions happen across different category levels and 2) the extent to which picture-word interference in artiﬁcial models depends on the semantic similarity between superimposed words and images.

Original: Furthermore, by importing a methodological tool from the neuroscience literature, called representational similarity analysis (RSA) (Kriegeskorte et al., 2008), we can also test 3) whether the image encoder has a common semantic representation for superimposed words and images.
Revised: Furthermore, by importing a methodological tool from the neuroscience literature, called representational similarity analysis (RSA) (Kriegeskorte et al., 2008), we can also test 3) whether the image encoder has a common semantic representation for superimposed words and images.

Original: Out of various joint learning models of language and vision, the present study evaluates the CLIP model using the dataset because the model consists of dual processing of visual and text encoders and is developed to be applied to various general visual tasks without ﬁne-tuning the dataset.
Revised: Out of various joint learning models of language and vision, the present study evaluates the CLIP model using the dataset because the model consists of dual processing of visual and text encoders and is developed to be applied to various general visual tasks without ﬁne-tuning the dataset.

Original: To anticipate the result, we show that 1) presenting words disturbs the CLIP image classiﬁcation even across different category levels, 2) the effect does not depend on the semantic relationship between images and words, and 3) the superimposed word representation in the CLIP image encoder is not shared with the image representation.
Revised: To anticipate the result, we show that 1) presenting words disturbs the CLIP image classiﬁcation even across different category levels, 2) the effect does not depend on the semantic relationship between images and words, and 3) the superimposed word representation in the CLIP image encoder is not shared with the image representation.

Original: The present study provides an open-source code to reproduce the results and the full dataset.
Revised: The present study provides an open-source code to reproduce the results and the full dataset.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Joint learning of language and vision in artiﬁcial models: Some recent works in machine learning have reported language-biased image recognition for artiﬁcial models trained with joint learning of language and vision.
Revised: Joint learning of language and vision in artiﬁcial models: Some recent works in machine learning have reported language-biased image recognition for artiﬁcial models trained with joint learning of language and vision.

Original: One example is the CLIP model (Radford et al., 2021).
Revised: One example is the CLIP model (Radford et al., 2021).

Original: The CLIP architecture consists of an image encoder and a text encoder (Figure 2).
Revised: The CLIP architecture consists of an image encoder and a text encoder (Figure 2).

Original: The image encoder is based on a ResNet (He et al., 2016) or Vision Transformer
Revised: The image encoder is based on a ResNet (He et al., 2016) or Vision Transformer

Original: (Dosovitskiy et al., 2020), and the text encoder is a Language
Revised: (Dosovitskiy et al., 2020), and the text encoder is a Language

Original: Transformer model (Vaswani et al., 2017).
Revised: Transformer model (Vaswani et al., 2017).

Original: The model is trained on a large dataset of image-text pairs with contrastive objectives, where it learns to align text and image representations for each pair.
Revised: The model is trained on a large dataset of image-text pairs with contrastive objectives, where it learns to align text and image representations for each pair.

Original: The pre-trained model can be used for various visual tasks in a zero-shot manner.
Revised: The pre-trained model can be used for various visual tasks in a zero-shot manner.

Original: For instance, when the model is applied for the image classiﬁcation task, the learned text encoder synthesizes a zero-shot linear classiﬁer for the descriptions, including classiﬁcation labels, e.g., ”a photograph of [label]” (Figure 2).
Revised: For instance, when the model is applied for the image classiﬁcation task, the learned text encoder synthesizes a zero-shot linear classiﬁer for the descriptions, including classiﬁcation labels, e.g., ”a photograph of [label]” (Figure 2).

Original: The learned image encoder outputs the representation for an input image, and the text classiﬁer decides which label representation is the closest to the image representation.
Revised: The learned image encoder outputs the representation for an input image, and the text classiﬁer decides which label representation is the closest to the image representation.

Original: Languagebiased image classiﬁcation is reported in the CLIP image classiﬁcation task as a typographic attack (Goh et al., 2021).
Revised: Languagebiased image classiﬁcation is reported in the CLIP image classiﬁcation task as a typographic attack (Goh et al., 2021).

Original: As shown in Figure 1, when a written word is superimposed on an image, the
Revised: As shown in Figure 1, when a written word is superimposed on an image, the

Original: CLIP misclassiﬁes the image category as the superimposed word.
Revised: CLIP misclassiﬁes the image category as the superimposed word.

Original: Another example of language-biased image classiﬁcation is reported in the context of the Visual
Revised: Another example of language-biased image classiﬁcation is reported in the context of the Visual

Original: Question Answering (VQA) task (Antol et al., 2015; Lu et al., 2016; Fukui et al., 2016).
Revised: Question Answering (VQA) task (Antol et al., 2015; Lu et al., 2016; Fukui et al., 2016).

Original: This task’s goal is to answer questions about visual content, e.g., a text question ”What color is the dog?”
Revised: This task’s goal is to answer questions about visual content, e.g., a text question ”What color is the dog?”

Original: for a dog image.
Revised: for a dog image.

Original: The task also needs joint learning of language and vision, i.e., an image encoder to recognize what the visual content is and a text encoder to understand the question.
Revised: The task also needs joint learning of language and vision, i.e., an image encoder to recognize what the visual content is and a text encoder to understand the question.

Original: However, joint learning tends to be biased by superﬁcial correlations between texts and images in the training data and ignore visual input to solve a task (Agrawal et al., 2016).
Revised: However, joint learning tends to be biased by superﬁcial correlations between texts and images in the training data and ignore visual input to solve a task (Agrawal et al., 2016).

Original: For instance, even when a model sees a green (untypical) banana and is asked about its color, the model tends to answer ”the color is yellow” while ignoring the actual image content.
Revised: For instance, even when a model sees a green (untypical) banana and is asked about its color, the model tends to answer ”the color is yellow” while ignoring the actual image content.

Original: Based on the observations, recent VQA models have attempted to remove the biases (Agrawal et al., 2018; Ramakrishnan et al., 2018).
Revised: Based on the observations, recent VQA models have attempted to remove the biases (Agrawal et al., 2018; Ramakrishnan et al., 2018).

Original: Picture-word interference in humans: It is essential to understand how the language-biased processing in artiﬁcial models is related to the human mechanisms discussed in the cognitive science literature.
Revised: Picture-word interference in humans: It is essential to understand how the language-biased processing in artiﬁcial models is related to the human mechanisms discussed in the cognitive science literature.

Original: This subsection reviews the functional connection between text/image encoders in artiﬁcial models and picture-word interference in humans (Figure 2).
Revised: This subsection reviews the functional connection between text/image encoders in artiﬁcial models and picture-word interference in humans (Figure 2).

Original: Picture-word interference refers to language-biased recognition for word-superimposed images, and previous works in cognitive science have investigated this effect using various experimental paradigms (reviewed by B ¨ urki et al.
Revised: Picture-word interference refers to language-biased recognition for word-superimposed images, and previous works in cognitive science have investigated this effect using various experimental paradigms (reviewed by B ¨ urki et al.

Original: Here, we focus on interference when a participant observes a word-superimposed image and answers the image category for the image to compare the underlying cognitive mechanisms with artiﬁcial models (Rosinski, 1977).
Revised: Here, we focus on interference when a participant observes a word-superimposed image and answers the image category for the image to compare the underlying cognitive mechanisms with artiﬁcial models (Rosinski, 1977).

Original: The discussion about what type of mechanisms mediate picture-word interference is still controversial in the cognitive science literature, especially the stage at which the interference emerges.
Revised: The discussion about what type of mechanisms mediate picture-word interference is still controversial in the cognitive science literature, especially the stage at which the interference emerges.

Original: However, we can summarize common functional mechanisms shared across different theoretical views and ask whether artiﬁcial models acquire these fundamental components in the current study.
Revised: However, we can summarize common functional mechanisms shared across different theoretical views and ask whether artiﬁcial models acquire these fundamental components in the current study.

Original: First, most views assume that visual words and images activate common semantic representations, i.e., a written word ”dog” activates similar semantic representations to an image ”dog.”
Revised: First, most views assume that visual words and images activate common semantic representations, i.e., a written word ”dog” activates similar semantic representations to an image ”dog.”

Original: Second, they commonly assume that interference emerges due to some types of selection process.
Revised: Second, they commonly assume that interference emerges due to some types of selection process.

Original: The type of selection process is extensively debated and differs depending on the views.
Revised: The type of selection process is extensively debated and differs depending on the views.

Original: One is the lexical selection hypothesis, and the selection works to compare the semantic representation competitively (Levelt et al., 1999; Roelofs, 1992).
Revised: One is the lexical selection hypothesis, and the selection works to compare the semantic representation competitively (Levelt et al., 1999; Roelofs, 1992).

Original: This selection relies on relative activation in semantic representations, and picture-word interference emerges because the activation by words induces misselection when it is semantically close to the activation by images.
Revised: This selection relies on relative activation in semantic representations, and picture-word interference emerges because the activation by words induces misselection when it is semantically close to the activation by images.

Original: However, the lexical selection is inconsistent with some behavioral ﬁndings, e.g., the activation amplitude does not always correspond to the effect size of interference.
Revised: However, the lexical selection is inconsistent with some behavioral ﬁndings, e.g., the activation amplitude does not always correspond to the effect size of interference.

Original: Another theoretical view assumes the response selection, which is closer to the decision stage than the lexical selection (Finkbeiner & Caramazza, 2006).
Revised: Another theoretical view assumes the response selection, which is closer to the decision stage than the lexical selection (Finkbeiner & Caramazza, 2006).

Original: Since the response selection assumes additional processing for relative activated representations, it can explain the inconsistency of the phenomenon with the lexical selection assumption.
Revised: Since the response selection assumes additional processing for relative activated representations, it can explain the inconsistency of the phenomenon with the lexical selection assumption.

Original: We ﬁrst ask whether one observes patterns of interference at the behavioral level that are similar to humans and then study more precisely the learned representations to see if the mechanisms proposed to explain human interference could also be at play in the artiﬁcial model.
Revised: We ﬁrst ask whether one observes patterns of interference at the behavioral level that are similar to humans and then study more precisely the learned representations to see if the mechanisms proposed to explain human interference could also be at play in the artiﬁcial model.

Original: To investigate machine mechanisms in terms of the cognitive science ﬁndings, we regard an image encoder in the artiﬁcial model as the processing to activate semantic representations from images (Figure 2).
Revised: To investigate machine mechanisms in terms of the cognitive science ﬁndings, we regard an image encoder in the artiﬁcial model as the processing to activate semantic representations from images (Figure 2).

Original: We also consider a text encoder as the selection process, which tries to ﬁnd the answers according to the task objectives (superordinate/basic selection).
Revised: We also consider a text encoder as the selection process, which tries to ﬁnd the answers according to the task objectives (superordinate/basic selection).

Original:  L ANGUAGE - BIASED IMAGE CLASSIFICATION TEST
Revised:  L ANGUAGE - BIASED IMAGE CLASSIFICATION TEST

Original: Our dataset is a set of word-superimposed images.
Revised: Our dataset is a set of word-superimposed images.

Original: We use the two image datasets from the cognitive neuroscience literature on object recognition (Cichy et al., 2016; Mohsenzadeh et al., 2019), which include 118 and 156 images with natural object categories.
Revised: We use the two image datasets from the cognitive neuroscience literature on object recognition (Cichy et al., 2016; Mohsenzadeh et al., 2019), which include 118 and 156 images with natural object categories.

Original: Previous works recorded the spatiotemporal neural activity using fMRI and MEG measurements for the images.
Revised: Previous works recorded the spatiotemporal neural activity using fMRI and MEG measurements for the images.

Original: The ﬁrst dataset was annotated with 118 basic categories (Cichy et al., 2016) and the second one with ﬁve superordinate categories (faces, bodies, animals, objects, and scenes) (Mohsenzadeh et al., 2019).
Revised: The ﬁrst dataset was annotated with 118 basic categories (Cichy et al., 2016) and the second one with ﬁve superordinate categories (faces, bodies, animals, objects, and scenes) (Mohsenzadeh et al., 2019).

Original: For the word labels, we extracted the superordinate category words from the MS-COCO dataset and the basic category words from the MS-COCO and CIFAR-100 datasets (Lin et al., 2014; Krizhevsky et al., 2009).
Revised: For the word labels, we extracted the superordinate category words from the MS-COCO dataset and the basic category words from the MS-COCO and CIFAR-100 datasets (Lin et al., 2014; Krizhevsky et al., 2009).

Original: The MS-COCO dataset had the ”person” label for superordinate and basic categories.
Revised: The MS-COCO dataset had the ”person” label for superordinate and basic categories.

Original: We used this label only for the superordinate category because it was a higher category than other person-related basic labels in CIFAR-100.
Revised: We used this label only for the superordinate category because it was a higher category than other person-related basic labels in CIFAR-100.

Original: We removed multi-word labels and duplicates (including the plural of an already existing label) when compounding the MS-COCO and CIFAR-100 basic word label sets.
Revised: We removed multi-word labels and duplicates (including the plural of an already existing label) when compounding the MS-COCO and CIFAR-100 basic word label sets.

Original: The numbers of superordinate and basic word labels were 12 and 138, respectively.
Revised: The numbers of superordinate and basic word labels were 12 and 138, respectively.

Original: We superimposed all labels on each image, and thus, our word-superimposed image dataset included 41,100 images in total.
Revised: We superimposed all labels on each image, and thus, our word-superimposed image dataset included 41,100 images in total.

Original: The color of superimposed words was red and bordered by white lines to enhance the word visibility considering future usage in human experiments.
Revised: The color of superimposed words was red and bordered by white lines to enhance the word visibility considering future usage in human experiments.

Original: Since the image dataset is supposed to observe objects in the foveal vision, we put the words in the center of the image with a size that largely does not hide the original object.
Revised: Since the image dataset is supposed to observe objects in the foveal vision, we put the words in the center of the image with a size that largely does not hide the original object.

Original: The classiﬁcation task was to answer the image label for word-superimposed images.
Revised: The classiﬁcation task was to answer the image label for word-superimposed images.

Original: The classiﬁcation label belonged either to the superordinate or the basic category.
Revised: The classiﬁcation label belonged either to the superordinate or the basic category.

Original: The set of labels in the superordinate and basic categories were the same as the words used to make the word-superimposed images.
Revised: The set of labels in the superordinate and basic categories were the same as the words used to make the word-superimposed images.

Original: Therefore, the stimulus conditions were divided into four, corresponding to two classiﬁcation types and two superimposed word categories (Figure 1).
Revised: Therefore, the stimulus conditions were divided into four, corresponding to two classiﬁcation types and two superimposed word categories (Figure 1).

Original: The former index S of ”S/S” in
Revised: The former index S of ”S/S” in

Original: Figure 1 indicates the classiﬁcation task level (superordinate or basic), and the latter shows the category level of the word superimposed on the image (superordinate or basic).
Revised: Figure 1 indicates the classiﬁcation task level (superordinate or basic), and the latter shows the category level of the word superimposed on the image (superordinate or basic).

Original: In addition to these conditions, we used no-word images to evaluate the original prediction by the model.
Revised: In addition to these conditions, we used no-word images to evaluate the original prediction by the model.

Original:  E XPERIMENTS AND RESULTS 4. E XPERIMENTAL SETUP
Revised:  E XPERIMENTS AND RESULTS 4. E XPERIMENTAL SETUP

Original: Performing image classiﬁcation with the pre-trained CLIP model needs the input of a list of candidate classiﬁcation labels (or a textual representation of them) for the text encoder and an image for the image encoder (Figure 2).
Revised: Performing image classiﬁcation with the pre-trained CLIP model needs the input of a list of candidate classiﬁcation labels (or a textual representation of them) for the text encoder and an image for the image encoder (Figure 2).

Original: We used the sentence of “a photo of a [label]” for the text encoder.
Revised: We used the sentence of “a photo of a [label]” for the text encoder.

Original: The two encoders output the representations of texts and the image respectively.
Revised: The two encoders output the representations of texts and the image respectively.

Original: We then computed the similarities of text and image representations for a set of labels, passed them through the softmax function to obtain the probability of each label for a given input image, and extracted the label with the highest value for the ﬁnal classiﬁcation (Radford et al., 2021).
Revised: We then computed the similarities of text and image representations for a set of labels, passed them through the softmax function to obtain the probability of each label for a given input image, and extracted the label with the highest value for the ﬁnal classiﬁcation (Radford et al., 2021).

Original: We present the results of the pre-trained CLIP image encoder using a Vision Transformer (ViT-B/32) in the main text and those of the other image encoders in Appendix C.
Revised: We present the results of the pre-trained CLIP image encoder using a Vision Transformer (ViT-B/32) in the main text and those of the other image encoders in Appendix C.

Original: 4. L ABEL SWITCHING RATE
Revised: 4. L ABEL SWITCHING RATE

Original: To test whether superimposing the word affects the classiﬁcation, we ﬁrst evaluated the label switching rate.
Revised: To test whether superimposing the word affects the classiﬁcation, we ﬁrst evaluated the label switching rate.

Original: We deﬁned a label-switched image as a word-superimposed image with a different image classiﬁcation than the corresponding the no-word image.
Revised: We deﬁned a label-switched image as a word-superimposed image with a different image classiﬁcation than the corresponding the no-word image.

Original: The label switching rate refers to the ratio of the label-switched images and all images under each classiﬁcation/superimposed-word condition.
Revised: The label switching rate refers to the ratio of the label-switched images and all images under each classiﬁcation/superimposed-word condition.

Original: Table 1 shows the results of the label switching rates for each condition.
Revised: Table 1 shows the results of the label switching rates for each condition.

Original: When the classiﬁcation task type is consistent with the superimposed word category (i.e., the S/S and B/B conditions), the choices of the classiﬁcation labels include the same word as that superimposed on the image.
Revised: When the classiﬁcation task type is consistent with the superimposed word category (i.e., the S/S and B/B conditions), the choices of the classiﬁcation labels include the same word as that superimposed on the image.

Original: For these conditions, the CLIP model switched the original classiﬁcation with high rates.
Revised: For these conditions, the CLIP model switched the original classiﬁcation with high rates.

Original: These ﬁndings suggest that the model can detect the visual word form on the image and match it with the classiﬁcation label.
Revised: These ﬁndings suggest that the model can detect the visual word form on the image and match it with the classiﬁcation label.

Original: These are consistent with the previous work reporting the language-biased image classiﬁcation of the CLIP model (Goh et al., 2021).
Revised: These are consistent with the previous work reporting the language-biased image classiﬁcation of the CLIP model (Goh et al., 2021).

Original: In contrast, when the classiﬁcation type is different from the superimposed word category (i.e., the S/B and B/S conditions), the classiﬁcation labels include semantically related words with the superimposed one but not the same word.
Revised: In contrast, when the classiﬁcation type is different from the superimposed word category (i.e., the S/B and B/S conditions), the classiﬁcation labels include semantically related words with the superimposed one but not the same word.

Original: Even under these conditions, the CLIP model showed label-switching for some of the word-superimposed images.
Revised: Even under these conditions, the CLIP model showed label-switching for some of the word-superimposed images.

Original: Furthermore,  the switched new labels were related to the superimposed words (Appendix E).
Revised: Furthermore, some of the switched new labels were related to the superimposed words (Appendix G).

Original: The ﬁndings suggest that the model can match the meaning of the visual word form extracted from the imageencoder with the label categoryfrom the text encoder across different levels.
Revised: The ﬁndings suggest that the model can match the meaning of the written word in the image with the label category across different levels.

Original: However, it remains unclear whether the image encoder 
Revised: However, it remains unclear whether the image encoder acquires a common semantic

Original: [Table]
Revised: [Table]

Original: in the CLIP model acquires a common semantic representation for the images and  visual word forms.
Revised: representation for the images and the visual word forms.

Original: There is also a possibility that the semantic word representation of superimposed words is separated from the image representation in theCLIP image encoder (e.g., a superimposed word ”dog” is differently represented from an image of a dog).
Revised: There is also a possibility that the semantic word representation of superimposed words is separated from the image representation in the image encoder (e.g., a superimposed word ”dog” is differently represented from an image of a dog).

Revised: Even so, the label switching across different categories can emerge when the label activation of the text encoder can match with the representation of the superimposed word.

Original: To understand the semantic relationship, we evaluated the following semantic similarity between images and superimposed words for the label-switched and unswitched images.
Revised: To understand the semantic relationship, we evaluated the following semantic similarity between images and superimposed words for the label-switched and unswitched images.

Original: 4. S EMANTIC SIMILARITY
Revised: 4. S EMANTIC SIMILARITY

Original: We evaluated the semantic similarity between the original prediction labels for no-word images and the superimposed words to test the semantic dependency in picture-word interference.
Revised: We evaluated the semantic similarity between the original prediction labels for no-word images and the superimposed words to test the semantic dependency in picture-word interference.

Original: Speciﬁcally, we calculated the vector representations of these labels using a pre-trained Word2Vec model (Mikolov et al., 2013b;a), and then measured the cosine similarity between the two vector representations.
Revised: Speciﬁcally, we calculated the vector representations of these labels using a pre-trained Word2Vec model (Mikolov et al., 2013b;a), and then measured the cosine similarity between the two vector representations.

Original: If picture-word interference in CLIP depends on the semantic similarity between superimposed words and images, we would expect the similarity in the label-switched condition would be higher than that in the unswitched condition.
Revised: If picture-word interference in CLIP depends on the semantic similarity between superimposed words and images, we would expect the similarity in the label-switched condition would be higher than that in the unswitched condition.

Original: Figure 3a shows the semantic similarity distribution for the label-switched and unswitched images in the CLIP model.
Revised: Figure 3a shows the semantic similarity distribution for the label-switched and unswitched images in the CLIP model.

Original: The results show that for all task type and word category conditions, the semantic similarity distributions on the label-switched condition were similar to those on the unswitched condition.
Revised: The results show that for all task type and word category conditions, the semantic similarity distributions on the label-switched condition were similar to those on the unswitched condition.

Original: This ﬁnding suggests that picture-word interference in the CLIP model does not depend on the semantic relationship between images and superimposed words, suggesting that the CLIP image encoder does not acquire a common semantic representation across superimposed words and images.
Revised: This ﬁnding suggests that picture-word interference in the CLIP model does not depend on the semantic relationship between images and superimposed words, suggesting that the CLIP image encoder does not acquire a common semantic representation across superimposed words and images.

Original: 4. S PELLING SIMILARITY
Revised: 4. S PELLING SIMILARITY

Original: In addition to the semantic similarity, we evaluated the spelling similarity using the Jaro-Winkler metric (Winkler, 1990) to test how the superimposed word is related to the image category.
Revised: In addition to the semantic similarity, we evaluated the spelling similarity using the Jaro-Winkler metric (Winkler, 1990) to test how the superimposed word is related to the image category.

Original: If the
Revised: If the

Original: CLIP image encoder acquires a common representation of superimposed words and images based on the edit distance, this metric should change with the label-switched/unswitched condition.
Revised: CLIP image encoder acquires a common representation of superimposed words and images based on the edit distance, this metric should change with the label-switched/unswitched condition.

Original: Figure 3b shows the result of the CLIP model on the spelling similarity for the label-switched and unswitched images.
Revised: Figure 3b shows the result of the CLIP model on the spelling similarity for the label-switched and unswitched images.

Original: As in the semantic similarity, the distribution median for the task conditions was always similar between the label-switched and unswitched conditions.
Revised: As in the semantic similarity, the distribution median for the task conditions was always similar between the label-switched and unswitched conditions.

Original: 4. R EPRESENTATIONAL SIMILARITY ANALYSIS (RSA)
Revised: 4. R EPRESENTATIONAL SIMILARITY ANALYSIS (RSA)

Original: We also used a methodological tool from the cognitive neuroscience literature, called representational similarity analysis (RSA) (Kriegeskorte et al., 2008), to understand what is represented in the image encoder for word-superimposed images.
Revised: We also used a methodological tool from the cognitive neuroscience literature, called representational similarity analysis (RSA) (Kriegeskorte et al., 2008), to understand what is represented in the image encoder for word-superimposed images.

Original: RSA refers to an image-by-image similarity assessment of intermediate representations in a brain or model.
Revised: RSA refers to an image-by-image similarity assessment of intermediate representations in a brain or model.

Original: By computing representational dissimilarity matrices (RDMs) for no-word images and word-superimposed images, we can evaluate how the representation for the no-word images in the CLIP image encoder is affected by adding words on the image.
Revised: By computing representational dissimilarity matrices (RDMs) for no-word images and word-superimposed images, we can evaluate how the representation for the no-word images in the CLIP image encoder is affected by adding words on the image.

Original: We input one of the two datasets (156 images) (Mohsenzadeh et al., 2019) for this analysis because it consisted of the previously deﬁned superordinate categories.
Revised: We input one of the two datasets (156 images) (Mohsenzadeh et al., 2019) for this analysis because it consisted of the previously deﬁned superordinate categories.

Original: In addition to the CLIP model, we evaluated the two feed-forward convolutional networks (i.e., VGG-19 and
Revised: In addition to the CLIP model, we evaluated the two feed-forward convolutional networks (i.e., VGG-19 and

Original: ResNet-152) trained for object category classiﬁcation using ImageNet without joint learning of language and vision to conﬁrm whether presenting words as image manipulation can affect the image encoder without joint learning.
Revised: ResNet-152) trained for object category classiﬁcation using ImageNet without joint learning of language and vision to conﬁrm whether presenting words as image manipulation can affect the image encoder without joint learning.

Original: Figure 4 shows the results of the RSA.
Revised: Figure 4 shows the results of the RSA.

Original: Each matrix indicates the image-by-image cosine dissimilarity between intermediate features for the paired images.
Revised: Each matrix indicates the image-by-image cosine dissimilarity between intermediate features for the paired images.

Original: For the features, we used the last fully connected layer for VGG-19/ResNet-152 and the visual encoder output for the CLIP model.
Revised: For the features, we used the last fully connected layer for VGG-19/ResNet-152 and the visual encoder output for the CLIP model.

Original: The top, middle, and bottom rows show the results for the original images, the word-superimposed images with the ”animal” label, and the word-superimposed images with the ”person” label, respectively.
Revised: The top, middle, and bottom rows show the results for the original images, the word-superimposed images with the ”animal” label, and the word-superimposed images with the ”person” label, respectively.

Original: Each left image is one example in the 156 images.
Revised: Each left image is one example in the 156 images.

Original: The results of other word-embeddings are shown in Appendix B1.
Revised: The results of other word-embeddings are shown in Appendices B1 and H1.

Original: For VGG-19 and ResNet-152, we observed some representational clusters according to the superordinate categories (i.e., the squared clusters around the diagonal line in each panel with lower
Revised: For VGG-19 and ResNet-152, we observed some representational clusters according to the superordinate categories (i.e., the squared clusters around the diagonal line in each panel with lower

Original: [Table]
Revised: [Table]

Original: dissimilarity values) (Figure 4, top, VGG-19 and ResNet-152).
Revised: dissimilarity values) (Figure 4, top, VGG-19 and ResNet-152).

Original: The result is consistent with the previous ﬁnding that the higher-order processing of feed-forward convolutional networks is correlated with human and monkey brain activities on the cortical area related to object recognition (Yamins &
Revised: The result is consistent with the previous ﬁnding that the higher-order processing of feed-forward convolutional networks is correlated with human and monkey brain activities on the cortical area related to object recognition (Yamins &

Original: DiCarlo, 2016).
Revised: DiCarlo, 2016).

Original: Similar to these feed-forward models, the CLIP image encoder also showed representational clusters depending on the superordinate categories (Figure 4, top, CLIP), suggesting that the encoder has a hierarchical structure for visual images.
Revised: Similar to these feed-forward models, the CLIP image encoder also showed representational clusters depending on the superordinate categories (Figure 4, top, CLIP), suggesting that the encoder has a hierarchical structure for visual images.

Original: In contrast, the results of the CLIP model for word-superimposed images showed different representations from that of original images (Figure 4, middle and bottom, CLIP).
Revised: In contrast, the results of the CLIP model for word-superimposed images showed different representations from that of original images (Figure 4, middle and bottom, CLIP).

Original: First, the dissimilarity values for word-superimposed images were generally lower than those for original images (i.e., the representations of all images are similar to each other).
Revised: First, the dissimilarity values for word-superimposed images were generally lower than those for original images (i.e., the representations of all images are similar to each other).

Original: The results are consistent with the strong language bias of the CLIP model.
Revised: The results are consistent with the strong language bias of the CLIP model.

Original: As in the other analyses, presenting words on the image shift the
Revised: As in the other analyses, presenting words on the image shift the

Original: CLIP image classiﬁcation to the direction of the superimposed word.
Revised: CLIP image classiﬁcation to the direction of the superimposed word.

Original: This means that when we put a single word on the different images, their representations can be similar.
Revised: This means that when we put a single word on the different images, their representations can be similar.

Original: Most importantly, our analysis showed that we can observe the original representational clusters even when the superimposed words changed the representation for original images.
Revised: Most importantly, our analysis showed that we can observe the original representational clusters even when the superimposed words changed the representation for original images.

Original: Speciﬁcally, we observed category-related squares similar to the original one around the diagonal line of each representational matrix for word-superimposed images (Figure 4, middle and bottom, CLIP).
Revised: Speciﬁcally, we observed category-related squares similar to the original one around the diagonal line of each representational matrix for word-superimposed images (Figure 4, middle and bottom, CLIP).

Original: In addition, even when we put different word labels, ”person” and ”animal,” these representational matrices remained similar.
Revised: In addition, even when we put different word labels, ”person” and ”animal,” these representational matrices remained similar.

Original: If the superimposed words were similarly represented in the image encoder to the image category, presenting words on the image should selectively affect the original image category representation and depend on what words were superimposed.
Revised: If the superimposed words were similarly represented in the image encoder to the image category, presenting words on the image should selectively affect the original image category representation and depend on what words were superimposed.

Original: The current results are inconsistent with the hypothesis and suggest that the CLIP image encoder has different representations of word forms (i.e., superimposed words) from visual image representations.
Revised: The current results are inconsistent with the hypothesis and suggest that the CLIP image encoder has different representations of word forms (i.e., superimposed words) from visual image representations.

Original: 4. P ROMPT DEPENDENCY
Revised: 4. P ROMPT DEPENDENCY

Original: The CLIP model is not trained for image classiﬁcation but just for matching between images and texts.
Revised: The CLIP model is not trained for image classiﬁcation but just for matching between images and texts.

Original: One may consider that the language-biased classiﬁcation emerges because the task objective for the model, which is deﬁned by the prompt ”a photo of X,” is ambiguous.
Revised: One may consider that the language-biased classiﬁcation emerges because the task objective for the model, which is deﬁned by the prompt ”a photo of X,” is ambiguous.

Original: The prompt can be both ”a photo of the written word” and ”a photo of the image content.”
Revised: The prompt can be both ”a photo of the written word” and ”a photo of the image content.”

Original: To investigate such prompt ambiguity, we tested various prompts.
Revised: To investigate such prompt ambiguity, we tested various prompts.

Original: First, we created three prompts that specify that the image content is what should be predicted by the CLIP model (red in Table 2) and three prompts that specify that the superimposed word is what should be predicted by the CLIP model (blue in Table 2).
Revised: First, we created three prompts that specify that the image content is what should be predicted by the CLIP model (red in Table G1) and three prompts that specify that the superimposed word is what should be predicted by the CLIP model (blue in Table G1).

Original: We used each prompt for all word-superimposed images.
Revised: We used each prompt for all word-superimposed images.

Original: These prompts did not change according to what superimposed word was presented.
Revised: These prompts did not change according to what superimposed word was presented.

Original: For instance, we used the same prompt, ”a red word label over a picture of a X” for all images.
Revised: For instance, we used the same prompt, ”a red word label over a picture of a X” for all images.

Original: In addition to such ﬁxed prompts, we created a prompt that varies according to the superimposed word on each image.
Revised: In addition to such ﬁxed prompts, we created a prompt that varies according to the superimposed word on each image.

Original: We used a prompt ”a photo of the word Y written in a red font over a picture of a X,” where X is the label to be answered by the model and Y is the same variable word as superimposed on each image (orange in Table 2).
Revised: We used a prompt ”a photo of the word Y written in a red font over a picture of a X,” where X is the label to be answered by the model and Y is the same variable word as superimposed on each image (orange in Table G1).

Original: For instance, we used a prompt ”a photo of the word electric in a red font over a picture of a X” when the superimposed word is ”electric,” and a prompt ”a photo of the word dog in a red font over a picture of a X” when the superimposed word is ”dog.”
Revised: For instance, we used a prompt ”a photo of the word electric in a red font over a picture of a X” when the superimposed word is ”electric,” and a prompt ”a photo of the word dog in a red font over a picture of a X” when the superimposed word is ”dog.”

Original: Table 2 shows the results of the label-switching rate for different prompts.
Revised: Table G1 shows the results of the label-switching rate for different prompts.

Original: We used ”a photo of X” to predict no-word (original) image labels on the label-switching rate.
Revised: We used ”a photo of X” to predict no-word (original) image labels on the label-switching rate.

Original: The results showed that the rate changed with the different prompts, i.e., the focus of the task (image content or superimposed word).
Revised: The results showed that the rate changed with the different prompts, i.e., the focus of the task (image content or superimposed word).

Original: When the prompt was focused on the image content (red in Table 2) the switching rates were lower than when it was focused on the superimposed word (blue in Table 2).
Revised: When the prompt was focused on the image content (red in Table G1) the switching rates were lower than when it was focused on the superimposed word (blue in Table G1).

Original: However, the language-biased classiﬁcation was high for all conditions.
Revised: However, the language-biased classiﬁcation was high for all conditions.

Original: The results on the variable prompt (orange in Table 2) showed that the label switching rate for the S/S and B/B conditions decreased more than the prompt conditions that specify the image content (red in Table 2).
Revised: The results on the variable prompt (orange in Table G1) showed that the label switching rate for the S/S and B/B conditions decreased more than the prompt conditions that specify the image content (red in Table G1).

Original: The ﬁnding suggests that describing the superimposed word in the prompt is effective in suppressing the language-biased classiﬁcation.
Revised: The ﬁnding suggests that describing the superimposed word in the prompt is effective in suppressing the language-biased classiﬁcation.

Original: Even so, the label-switching rates were still high, more than 30% for all conditions.
Revised: Even so, the label-switching rates were still high, more than 30% for all conditions.

Original: In addition, we analyzed the semantic similarity between the image and superimposed words for label-switched and unswitched images.
Revised: In addition, we analyzed the semantic similarity between the image and superimposed words for labelswitched and unswitched images.

Original: The results showed that, in any prompts, the semantic relationship did not explain the label-switching effect (Appendix Figure D1).
Revised: The results showed that, in any prompts, the semantic relationship did not explain the label-switching effect (Appendix Figure D1).

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: The present study developed a benchmark test for language-supervised visual models to evaluate the language biases in image classiﬁcation.
Revised: The present study developed a benchmark test for language-supervised visual models to evaluate the language biases in image classiﬁcation.

Original: The CLIP evaluation using the test showed that the label switching rate increased even across different categories.
Revised: The CLIP evaluation using the test showed that the label switching rate increased even across different categories.

Original: The ﬁnding suggests that the CLIP image encoder can recognize the visual word form as a word, and the model can match the visual word representation with the classiﬁcation label representations across different category levels.
Revised: The ﬁnding suggests that the CLIP image encoder can recognize the visual word form as a word, and the model can match the visual word representation with the classiﬁcation label representations across different category levels.

Original: However, the semantic similarity analysis showed that the language-biased image classiﬁcation in the CLIP model was irrespective of the semantic/spelling relationship between image and superimposed word categories.
Revised: However, the semantic similarity analysis showed that the language-biased image classiﬁcation in the CLIP model was irrespective of the semantic/spelling relationship between image and superimposed word categories.

Original: This ﬁnding is different from human ﬁndings and suggests that the CLIP image encoder does not acquire the semantic word representation shared with the image representation.
Revised: This ﬁnding is different from human ﬁndings and suggests that the CLIP image encoder does not acquire the semantic word representation shared with the image representation.

Original: The RSA results supported the hypothesis by showing that presenting words on images did not affect the no-word image representation selectively to speciﬁc categories.
Revised: The RSA results supported the hypothesis by showing that presenting words on images did not affect the no-word image representation selectively to speciﬁc categories.

Original: We consider multiple steps for an artiﬁcial model with joint learning of language and vision to acquire semantic compositionality for visual word and image categories.
Revised: We consider multiple steps for an artiﬁcial model with joint learning of language and vision to acquire semantic compositionality for visual word and image categories.

Original: First, an image encoder needs to recognize the visual word form as a word.
Revised: First, an image encoder needs to recognize the visual word form as a word.

Original: Next, the semantic representation for words and images needs to be shared in the image encoder.
Revised: Next, the semantic representation for words and images needs to be shared in the image encoder.

Original: That is, the written word ”dog” needs to have the same semantic representation as the image ”dog.”
Revised: That is, the written word ”dog” needs to have the same semantic representation as the image ”dog.”

Original: The present study showed that the CLIP model acquires the ﬁrst step, but the second step remains challenging.
Revised: The present study showed that the CLIP model acquires the ﬁrst step, but the second step remains challenging.

Original: Our language-biased image classiﬁcation test enables researchers to evaluate step-by-step semantic interactions in joint learning of language and vision.
Revised: Our language-biased image classiﬁcation test enables researchers to evaluate step-by-step semantic interactions in joint learning of language and vision.

Original: A remaining question is how the CLIP model acquires the biased representations in the image encoder.
Revised: A remaining question is how the CLIP model acquires the biased representations in the image encoder.

Original: The presented results may be improved by further studying them as a function of potential biases in the training data of CLIP itself: for example, it would be useful to understand in the ﬁrst place how CLIP acquired the ability to read superimposed words and why we observed a bias favoring the superimposed text over the whole image.
Revised: The presented results may be improved by further studying them as a function of potential biases in the training data of CLIP itself: for example, it would be useful to understand in the ﬁrst place how CLIP acquired the ability to visually recognize superimposed words and why we observed a bias favoring the superimposed words over the whole image.

Original: In particular, we can ask whether the written words are in the training data and if they are, whether there are adversarial examples in the training data like our word-superimposed images.
Revised: In particular, we can ask whether the written words are in the training data and if they are, whether there are adversarial examples in the training data like our word-superimposed images.

Original: Picture-word interference in humans is a phenomenon showing a language-biased effect in terms of image classiﬁcation.
Revised: Picture-word interference in humans is a phenomenon showing a language-biased effect in terms of image classiﬁcation.

Original: However, this can also be considered as a developmental product humans acquire, where such bias could have functionalities.
Revised: However, this can also be considered as a developmental product humans acquire, where such bias could have functionalities.

Original: Some human studies have shown that pictureword interference is inextricably linked to facilitation effects of language priming for image category recognition (Finkbeiner & Caramazza, 2006).
Revised: Some human studies have shown that pictureword interference is inextricably linked to facilitation effects of language priming for image category recognition (Finkbeiner & Caramazza, 2006).

Original: The ﬁnding reminds us that acquiring human-like biased representations for artiﬁcial models can have a functional meaning to process world information efﬁciently in natural environments.
Revised: The ﬁnding reminds us that acquiring human-like biased representations for artiﬁcial models can have a functional meaning to process world information efﬁciently in natural environments.

Original: The current efforts to manage language-biased image recognition in artiﬁcial models are mainly aimed to remove the biases (Agrawal et al., 2018; Ramakrishnan et al., 2018).
Revised: The current efforts to manage language-biased image recognition in artiﬁcial models are mainly aimed to remove the biases (Agrawal et al., 2018; Ramakrishnan et al., 2018).

Original: However, one direction might be to make the biases more human-like by controlling joint learning curricula of language and vision (c.f., Sigaud et al.
Revised: However, one direction might be to make the biases more human-like by controlling joint learning curricula of language and vision (c.f., Sigaud et al.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: All the source codes are available to create the word-superimposed images from the image datasets (Cichy et al., 2016; Mohsenzadeh et al., 2019) and to reproduce the current analysis results, which are shared in the supplementary materials of the ICLR submission.
Revised: All the source codes are available to create the word-superimposed images from the image datasets (Cichy et al., 2016; Mohsenzadeh et al., 2019) and to reproduce the current analysis results, which are shared in the supplementary materials of the ICLR submission.

Original: The analysis relies on the pretrained models shared in the previous studies (e g., CLIP).
Revised: The analysis relies on the pretrained models shared in the previous studies (e g., CLIP).

Original: We describe how to import them for our analysis using a python notebook and include the description in the supplementary materials
Revised: We describe how to import them for our analysis using a python notebook and include the description in the supplementary materials

Original: A P REDICTION EXAMPLE FOR WORD - SUPERIMPOSED IMAGES .
Revised: A P REDICTION EXAMPLE FOR WORD - SUPERIMPOSED IMAGES .

Original: B R EPRESENTATIONAL SIMILARITY ANALYSIS USING WORD - SUPERIMPOSED IMAGES .
Revised: B R EPRESENTATIONAL SIMILARITY ANALYSIS USING WORD - SUPERIMPOSED IMAGES .

Original: C E VALUATION OF DIFFERENT IMAGE ENCODERS OF THE CLIP MODEL
Revised: C E VALUATION OF DIFFERENT IMAGE ENCODERS OF THE CLIP MODEL

Original: The image encoder of the CLIP pre-trained model is based on ResNets (He et al., 2016) or Vision
Revised: The image encoder of the CLIP pre-trained model is based on ResNets (He et al., 2016) or Vision

Original: Transformers (Vaswani et al., 2017).
Revised: Transformers (Vaswani et al., 2017).

Original: To explore the extent to which the language-biased image classiﬁcation depends on the network architecture, we investigated the misclassiﬁcation rate and the semantic relationship using various encoders in addition to a Vision Transformer model (ViT-B/32) used in the main text.
Revised: To explore the extent to which the language-biased image classiﬁcation depends on the network architecture, we investigated the misclassiﬁcation rate and the semantic relationship using various encoders in addition to a Vision Transformer model (ViT-B/32) used in the main text.

Original: Speciﬁcally, for the ResNets, we used a ResNet-50, a ResNet-101, and two models which follow EfﬁcientNet-style model scaling (a ResNet 50x4 and a ResNet 50x16).
Revised: Speciﬁcally, for the ResNets, we used a ResNet-50, a ResNet-101, and two models which follow EfﬁcientNet-style model scaling (a ResNet 50x4 and a ResNet 50x16).

Original: For the
Revised: For the

Original: Vision Transformer, we used a ViT-B/16 in addition to a ViT-B/32.
Revised: Vision Transformer, we used a ViT-B/16 in addition to a ViT-B/32.

Original: We used the pre-trained CLIP networks using these architectures, which are shared by the previous work (Radford et al., 2021).
Revised: We used the pre-trained CLIP networks using these architectures, which are shared by the previous work (Radford et al., 2021).

Original: The experimental procedure is the same as that of the main text.
Revised: The experimental procedure is the same as that of the main text.

Original: Figure C1 shows the results of misclassiﬁcation rates for the CLIP models using various image encoders.
Revised: Figure C1 shows the results of misclassiﬁcation rates for the CLIP models using various image encoders.

Original: Even when the other architectures were used for the image encoder training, the results were consistent with the performance of the ViT-B/32 version.
Revised: Even when the other architectures were used for the image encoder training, the results were consistent with the performance of the ViT-B/32 version.

Original: The misclassiﬁcation rate was high when the classiﬁcation type was consistent with the superimposed word level (S/S and B/B).
Revised: The misclassiﬁcation rate was high when the classiﬁcation type was consistent with the superimposed word level (S/S and B/B).

Original: In addition, the CLIP models showed signiﬁcant misclassiﬁcation across different category conditions under the other conditions (S/B and B/S).
Revised: In addition, the CLIP models showed signiﬁcant misclassiﬁcation across different category conditions under the other conditions (S/B and B/S).

Original: The results of semantic similarity analysis and the RDA also did not depend on the architecture types (Figures C2 and C3, respectively).
Revised: The results of semantic similarity analysis and the RDA also did not depend on the architecture types (Figures C2 and C3, respectively).

Original: The semantic similarity analysis showed the median similarity of the label-switched images was not different from that of the unswitched images Figures C2 for all architectures.
Revised: The semantic similarity analysis showed the median similarity of the label-switched images was not different from that of the unswitched images Figures C2 for all architectures.

Original: The RDA showed that the image encoder representation for no-word images changed with the architectures (Figures C3, ﬁrst and fourth rows).
Revised: The RDA showed that the image encoder representation for no-word images changed with the architectures (Figures C3, ﬁrst and fourth rows).

Original: However, the effect of attaching words did not change with the architecture.
Revised: However, the effect of attaching words did not change with the architecture.

Original: For all architectures, the overall dissimilarity across imageby-image pairs decreased for word-superimposed images, but the representation displacements were not selective to the original image representation.
Revised: For all architectures, the overall dissimilarity across imageby-image pairs decreased for word-superimposed images, but the representation displacements were not selective to the original image representation.

Original: E P REDICTION PROBABILITY
Revised: E P REDICTION PROBABILITY

Original: The prediction probability analysis indicates how label switching relies on the prediction conﬁdence (i.e., the probability of the classiﬁed label in the label list) for no-word images and how the original prediction conﬁdence decreases by presenting words.
Revised: The prediction probability analysis indicates how label switching relies on the prediction conﬁdence (i.e., the probability of the classiﬁed label in the label list) for no-word images and how the original prediction conﬁdence decreases by presenting words.

Original: Figure E1 shows the results of the prediction probability analysis.
Revised: Figure E1 shows the results of the prediction probability analysis.

Original: Different panels show different stimulus conditions, as indicated in Figure 1.
Revised: Different panels show different stimulus conditions, as indicated in Figure 1.

Original: The original prediction means the probabilityof the classiﬁcation label for no-word images, categorized into label-switched or unswitched  conditions depending on if the label is switched in the new prediction.
Revised: The original prediction means the probability for no-word images. In the ﬁgure, the original prediction is categorized into label-switched or unswitched conditions. The label-switched and unswitched conditions refer to if the original prediction is switched in the new prediction for word-superimposed images.

Original: The new prediction shows the probability of the label before and after adding the superimposed word, for the label-switched word-superimposed images.
Revised: The new prediction shows the probability of the label before and after adding the superimposed word, for the label-switched word-superimposed images.

Original: In the original prediction, the probabilities for the label-switched images were lower than those of the unswitched images.
Revised: In the original prediction, the probabilities for the label-switched images were lower than those of the unswitched images.

Original: The trend was notable when the task type condition was inconsistent with the superimposed word category (the B/S and S/B conditions).
Revised: The trend was notable when the task type condition was inconsistent with the superimposed word category (the B/S and S/B conditions).

Original: These results suggest that picture-word interference is more effective when the original prediction is not conﬁdent.
Revised: These results suggest that picture-word interference is more effective when the original prediction is not conﬁdent.

Original: In addition, the original prediction probability drastically decreased in the new prediction for the word-superimposed image (two red plots in each panel of Figure E1).
Revised: In addition, the original prediction probability drastically decreased in the new prediction for the word-superimposed image (two red plots in each panel of Figure E1).

Original: However, we must be aware that a label prediction probability depends on other label predictions and that it can be lower if the other label prediction is high.
Revised: However, we must be aware that a label prediction probability depends on other label predictions and that it can be lower if the other label prediction is high.

Original: We show how the original image representation is changed/unchanged by presenting words in the next section.
Revised: We show how the original image representation is changed/unchanged by presenting words in the next section.

Original: Furthermore, the prediction probability analysis showed that the new label conﬁdence for the label-switched images (the blue plot in Figure E1) was high when the classiﬁcation task type was consistent with the superimposed word category (the S/S and B/B conditions).
Revised: Furthermore, the prediction probability analysis showed that the new label conﬁdence for the label-switched images (the blue plot in Figure E1) was high when the classiﬁcation task type was consistent with the superimposed word category (the S/S and B/B conditions).

Original: This trend is consistent with the results of the label switching rate analysis.
Revised: This trend is consistent with the results of the label switching rate analysis.

Revised: F E FFECT OF PSEUDOWORD

Revised: [Table]

Revised: We conducted an additional experiment using nonsense pseudowords and random words.

Revised: We created two pseudowords from the word ”animal” by a multilingual pseudoword generator, Wuggy (Keuleers & Brysbaert, 2010).

Revised: We also used one random word used in a previous human study (Finkbeiner & Caramazza, 2006).

Revised: Results showed that the label switching rate decreased for these words (Table G1).

Revised: We also analyzed the RDM for the images on which these words were superimposed.

Revised: Our results showed that when nonsense pseudo and random words were superimposed on the images, the effect of words on the representation was weaker than meaningful words.

Revised: The ﬁnding suggests that presenting meaningful words on the images can be a factor affecting the representation of the image encoder.

Revised: G R ELATIONSHIP BETWEEN SWITCHED LABEL PREDICTION AND SUPERIMPOSED WORDS

Revised: [Table]

Revised: H R ELATIONSHIP BETWEEN NO - WORD IMAGES AND WORD - SUPERIMPOSED IMAGES

Revised: similarity, and the RDMs were similar even different words were superimposed.

Revised: The results are consistent with that the CLIP image encoder has different representations of word forms from visual image representations.