Original:  I NFERRING O FFENSIVENESS I N I MAGES F ROM N ATURAL L ANGUAGE S UPERVISION
Revised: A NTICIPATING E THICAL I SSUES IN I MAGE T RAININGD ATA USING L ARGE - SCALE T RANSFORMERS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT (cid:34)
Revised: A BSTRACT (cid:34)

Original: This paper contains images and descriptions that are offensive in nature.
Revised: This paper contains images and descriptions that are offensive in nature

Original: Probing or fine-tuning (large-scale) pre- trained models results in state -of- the -art performance for many NLP tasks and, more recently, even for computer vision tasks when combined with image data.
Revised: Probing or fine-tuning (large-scale) pre-trained models results in state-of-the-art performance for many NLP tasks and, more recently, even for computer vision tasks when combined with image data.

Original: Unfortunately, these approaches also entail severe risks.
Revised: Unfortunately, these approaches also entail severe risks.

Original: In particular, large image datasets automatically scraped from the web may contain derogatory terms as categories and offensive images, and may also underrepresent specific classes.
Revised: In particular, large image datasets automatically scraped from the web may contain derogatory terms as categories and offensive images and may also underrepresent specific classes.

Original: Consequently, there is an urgent need to carefully document datasets and curate their content.
Revised: Consequently, there is an urgent need to carefully document datasets and curate their content.

Original: Unfortunately, this process is tedious and error-prone.
Revised: Unfortunately, this process is tedious and error-prone.

Original: We show that pre-trained transformers themselves provide a methodology for the automated curation of large-scale vision datasets.
Revised: We show that pre-trained transformers themselves provide a methodology for the automated curation of large-scale vision datasets.

Original: Based on human-annotated examples and the implicit knowledge of a CLIP based model, we demonstrate that one can select relevant prompts for rating the offensiveness of an image.
Revised: Based on human-annotated examples and the implicit knowledge of a CLIP based model, we demonstrate that one can select relevant prompts for rating the offensiveness of an image.

Original: In addition to e.g. privacy violation and pornographic content previously identified in ImageNet, we demonstrate that our approach identifies further inappropriate and potentially offensive content.
Revised: In addition to e.g. privacy violation and pornographic content previously identified in ImageNet, we demonstrate that our approach identifies further inappropriate and potentially offensive content

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep learning models yielded many improvements in several fields.
Revised: Deep learning models yielded many improvements in several fields.

Original: Particularly, transfer learning from models pre-trained on large-scale supervised data has become common practice in many tasks both with and without sufficient data to train deep learning models.
Revised: Particularly, transfer learning from models pre-trained on large-scale supervised data has become common practice in many tasks both with and without sufficient data to train deep learning models.

Original: While approaches like semisupervised sequence learning (Dai & Le, 2015) and datasets such as ImageNet (Deng et al., 2009), especially the ImageNet-ILSVRC-2012 dataset with 1.2 million images, established pre-training approaches, in the following years, the training data size increased rapidly to billions of training examples (Brown et al., 2020; Jia et al., 2021), steadily improving the capabilities of deep models.
Revised: Whereas approaches like semisupervised sequence learning (Dai & Le, 2015) and datasets such as ImageNet (Deng et al., 2009), especially the ImageNet-ILSVRC-2012 dataset with 1.2 million images, established pre-training approaches, in the following years, the training data size increased rapidly to billions of training examples (Brown et al., 2020; Jia et al., 2021), steadily improving the capabilities of deep models.

Original: Recently, autoregressive (Radford et al., 2019), masked language modeling (Devlin et al., 2019) as well as natural language guided vision models (Radford et al., 2021) have enabled zero-shot transfer to downstream datasets removing the need for dataset-specific customization.
Revised: Recently, autoregressive (Radford et al., 2019), masked language modeling (Devlin et al., 2019) as well as natural language guided vision models (Radford et al., 2021) have enabled zero-shot transfer to downstream datasets removing the need for dataset-specific customization.

Original: Besides the parameter size of these models, the immense size of training data has enabled deep learning models to achieve high accuracy on specific benchmarks in natural language processing (NLP) and computer vision (CV) applications.
Revised: Besides the parameter size of these models, the immense size of training data has enabled deep learning models to achieve high accuracy on specific benchmarks in natural language processing (NLP) and computer vision (CV) applications.

Original: However, in both application areas, the training data has been shown to have problematic characteristics resulting in models that encode e.g. stereotypical and derogatory associations (Gebru et al., 2018; Bender et al., 2021).
Revised: However, in both application areas, the training data has been shown to have problematic characteristics resulting in models that encode e.g. stereotypical and derogatory associations (Gebru et al., 2018; Bender et al., 2021).

Original: Unfortunately, the curation of these large datasets is tedious and error-prone.
Revised: Unfortunately, the curation of these large datasets is tedious and error-prone.

Original: Pre-trained models (PM) used for downstream tasks such as face detection propagate retained knowledge to the downstream module e.g. the classifier.
Revised: Pre-trained models (PM) used for downstream tasks such as face detection propagate retained knowledge to the downstream module e.g. the classifier.

Original: To raise the awareness of such issues, Gebru et al.
Revised: To raise the awareness of such issues, Gebru et al.

Original: (2018) describe how large, uncurated, Internetbased datasets encode e.g. dominant and hegemonic views, which further harms people at the margins.
Revised: (2018) described how large, uncurated, Internetbased datasets encode e.g. dominant and hegemonic views, which further harms people at the margins.

Original: The authors urge researchers and dataset creators to invest significant resource allocation towards dataset curation and documentation practices.
Revised: The authors urge researchers and dataset creators to invest significant resource allocation towards dataset curation and documentation practices.

Original: As a result, Birhane & Prabhu (2021) provided modules to detect faces and post-process them to provide privacy, as well as a pornographic content classifier to remove inappropriate images.
Revised: As a result, Birhane & Prabhu (2021) provided modules to detect faces and post-process them to provide privacy, as well as a pornographic content classifier to remove inappropriate images.

Original: Furthermore, Birhane & Prabhu (2021) conduct a hand surveyed image selection to identify misogynistic images in the ImageNet-ILSVRC-2012 dataset.
Revised: Furthermore, Birhane & Prabhu (2021) conducted a hand surveyed image selection to identify misogynistic images in the ImageNet-ILSVRC-2012 dataset.

Original: Unfortunately, such a curation process is tedious and does not scale to current dataset sizes.
Revised: Unfortunately, such a curation process is tedious and does not scale to current dataset sizes.

Original: Moreover, misogynistic images, as well as pornographic content, are only two subsets of offensive images.
Revised: Moreover, misogynistic images, as well as pornographic content, are only two subsets of offensive images.

Original: It remains an open question how to infer general offensiveness in images, including abusive, indecent, obscene, or menacing content, and how to identify them in an automated dataset curation process.
Revised: It remains an open question how to infer general offensiveness in images, including abusive, indecent, obscene, or menacing content, and how to identify them in an automated dataset curation process.

Original: While large image datasets automatically scraped from the web may contain derogatory terms as categories and offensive images, which results in models with undesirable behavior, pre-trained models may also reflect desirable implicit knowledge and biases such as our social, ethical, and moral choices (Jentzsch et al., 2019; Schramowski et al., 2020) reflected within the training data.
Revised: Whereas large image datasets automatically scraped from the web may contain derogatory terms as categories and offensive images, which results in models with undesirable behavior, pre-trained models may also reflect desirable implicit knowledge and biases such as our social, ethical, and moral choices (Jentzsch et al., 2019; Schramowski et al., 2020) reflected within the training data.

Original: In our study, we investigate modern vision PMs trained on large-scale datasets, in particular, the
Revised: In our study, we investigate modern vision PMs trained on large-scale datasets, in particular, the

Original: Contrastive Language-Image Pre-trained model (CLIP) (Radford et al., 2021) and argue that they themselves pave a way to mitigate the associated risks.
Revised: Contrastive Language-Image Pre-trained model (CLIP) (Radford et al., 2021) and argue that they themselves pave a way to mitigate the associated risks.

Original: Specifically, we show that they encode implicit knowledge to infer offensiveness in images overcoming previous issues, namely the lack of adequate and sufficient training data.
Revised: Specifically, we show that they encode implicit knowledge to infer offensiveness in images overcoming previous issues, namely the lack of adequate and sufficient training data.

Original: Furthermore, we demonstrate that our approach can be utilized to annotate offensive images in vision datasets and, therefore, reliably assist the curation process of such datasets.
Revised: Furthermore, we demonstrate that our approach can be utilized to annotate offensive images in vision datasets and, therefore, reliably assist the curation process of such datasets.

Original: We illustrate our approach on the popular ImageNet-ILSVRC-2012 dataset and show that large computer vision datasets contain additional inappropriate content, which previous documentations had not detected.
Revised: We illustrate our approach on the popular ImageNet-ILSVRC-2012 dataset and show that large computer vision datasets contain additional inappropriate content, which previous documentations had not detected.

Original: With our proposed method this content can be automatically and reliably pre-selected.
Revised: With our proposed method this content can be automatically and reliably pre-selected.

Original: As an example, Fig. 1(left) shows an exemplary image from the ImageNet-ILSVRC-2012 validation set identified as misogynistic content  by a hand-surveyed image selection in (Birhane & Prabhu, 2021).
Revised: As an example, Fig. 1(left) shows an exemplary image from the ImageNet-ILSVRC-2012 validation set identified as misogynistic content —categorized as “beach voyeur”— by a hand-surveyed image selection in (Birhane & Prabhu, 2021).

Original: Next to this human-selected image, Birhane & Prabhu (2021) applied different models to detect visible faces (thus violating privacy rights) and pornographic content.
Revised: Next to this human-selected image, Birhane & Prabhu (2021) applied different models to detect visible faces (thus violating privacy rights) and pornographic content.

Original: However, as we will show with our study, further inappropriate images, which we refer to as offensive, can be identified within the dataset.
Revised: However, as we will show with our study, further inappropriate images, which we refer to as offensive, can be identified within the dataset.

Original: For instance, Fig. 1(right) shows sixteen hand-picked images from a set of automatically detected possibly offensive images, utilizing our proposed approach.
Revised: For instance, Fig. 1(right) shows sixteen handpicked images from a set of automatically detected possibly offensive images, utilizing our proposed approach.

Original: Depending on the task and stakeholders, this ranges from offensive objects such as weapons (first row, first and fifth image) and dead animals (first row, sixth image) to immoral actions such as harming or even killing animals (second row, second image) and humans (second row, seventh image), as well as offensive text and symbols (first row, third image).
Revised: Depending on the task and stakeholders, this ranges from offensive objects such as weapons (second row, first and fifth image) and dead animals (first row, sixth image) to immoral actions such as harming or even killing animals (second row, second image) and humans (second row, seventh image), as well as offensive text and symbols (first row, third image).

Original: With our study we therefore strongly advocate for curating and documenting a dataset by the categories and models provided by Birhane & Prabhu (2021) but also by taking the possible general offensiveness in images into account.
Revised: With our study we therefore strongly advocate for curating and documenting a dataset by the categories and models provided by Birhane & Prabhu (2021) but also by taking the possible general offensiveness in images into account.

Original: To this end, we provide our models and the necessary data to reproduce our experiments and utilize our proposed method 1 .
Revised: To this end, we provide our models and the necessary data to reproduce our experiments and utilize our proposed method 1 .

Original: We proceed as follows.
Revised: We proceed as follows.

Original: We start with a brief overview of related work and required background introducing pre-trained models and their successes as well as concerns raised.
Revised: We start with a brief overview of related work and required background introducing pre-trained models and their successes as well as concerns raised.

Original: Next, we describe the term offensiveness and show that common deep models can not reliably detect offensive image content due to the lack of sufficient data.
Revised: Next, we describe the term offensiveness and show that common deep models can not reliably detect offensive image content due to the lack of sufficient data.

Original: We then continue by demonstrating that recent models, guided by natural language during the pre-training phase, can infer offensiveness in images based on their implicit knowledge.
Revised: We then continue by demonstrating that recent models, guided by natural language during the pre-training phase, can infer offensiveness in images based on their implicit knowledge.

Original: Before concluding, we present our automated dataset curation exemplary on the ImageNet-ILSVRC-2012 dataset.
Revised: Before concluding, we present our automated dataset curation exemplary on the ImageNet-ILSVRC-2012 dataset.

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: Concerns about large-scale data sets.
Revised: Concerns about large-scale data sets.

Original: Pre-training has become an essential approach in many vision and language tasks.
Revised: Pre-training has become an essential approach in many vision and language tasks.

Original: In the vision domain, pre-training on large-scale supervised data such as
Revised: In the vision domain, pre-training on large-scale supervised data such as

Original: ImageNet (Deng et al., 2009) has shown to be crucial for enhancing performance on downstream tasks via transfer learning.
Revised: ImageNet (Deng et al., 2009) has shown to be crucial for enhancing performance on downstream tasks via transfer learning.

Original: Since these datasets contain millions of data samples, curating such pre-training datasets requires heavy work on data gathering, sampling, and human annotation, making it error-prone and difficult to scale.
Revised: Since these datasets contain millions of data samples, curating such pre-training datasets requires heavy work on data gathering, sampling, and human annotation, making it error-prone and difficult to scale.

Original: Moreover, in the language domain, task-agnostic objectives such as autoregressive (Radford et al., 2019) and masked language modeling (Devlin et al., 2019) have scaled across many orders of magnitude, especially in model capacity and data, steadily increasing performance but also the capabilities of deep models.
Revised: Moreover, in the language domain, task-agnostic objectives such as autoregressive (Radford et al., 2019) and masked language modeling (Devlin et al., 2019) have scaled across many orders of magnitude, especially in model capacity and data, steadily increasing performance but also the capabilities of deep models.

Original: With their standardized input-output (text-totext) interface Radford et al.
Revised: With their standardized input-output (text-totext) interface Radford et al.

Original: (2019) have enabled zero-shot transfer to downstream datasets.
Revised: (2019) have enabled zero-shot transfer to downstream datasets.

Original: Recent systems like GPT-3 (Brown et al., 2020) are now competitive across many tasks with specialized models while requiring only a small amount to no task-specific training data.
Revised: Recent systems like GPT-3 (Brown et al., 2020) are now competitive across many tasks with specialized models whereas requiring only a small amount to no task-specific training data.

Original: Based on these advances, more recently, Radford et al.
Revised: Based on these advances, more recently, Radford et al.

Original: and Jia et al.
Revised: and Jia et al.

Original: (2021) introduce models with similar capabilities in the vision domain.
Revised: (2021) introduced models with similar capabilities in the vision domain.

Original: However, pre-training such models requires particularly large-scale training data, and the datasets’ curation process is tedious and error-prone.
Revised: However, pre-training such models requires particularly large-scale training data, and the datasets’ curation process is tedious and error-prone.

Original: To tackle this issue Gebru et al.
Revised: To tackle this issue Gebru et al.

Original: (2018) suggest to provide dataset audit cards to document datasets.
Revised: (2018) suggest to provide dataset audit cards to document datasets.

Original: This provides stakeholders the ability to understand training data characteristics in order to alleviate known as well as unknown issues.
Revised: This provides stakeholders the ability to understand training data characteristics in order to alleviate known as well as unknown issues.

Original: The authors argue that while documentation allows for potential accountability, undocumented training data perpetuates harm without recourse.
Revised: The authors argue that whereas documentation allows for potential accountability, undocumented training data perpetuates harm without recourse.

Original: Birhane & Prabhu (2021) provide such a dataset card for the popular computer-vision ImageNetILSVRC-2012 dataset, including several metrics and the hand surveyed identification of images with misogynistic content.
Revised: Birhane & Prabhu (2021) provided such a dataset card for the popular computer-vision ImageNetILSVRC-2012 dataset, including several metrics and the hand surveyed identification of images with misogynistic content.

Original: More importantly, the authors raised the awareness of polluted image datasets by the example of pornographic content inside several popular computer vision benchmark datasets.
Revised: More importantly, the authors raised the awareness of polluted image datasets by the example of pornographic content inside several popular computer vision benchmark datasets.

Original: Although the authors raised criticism against ImageNet and identified several inappropriate images, the ImageNet-ILSVRC-2012 dataset —and the pre-trained models— are still under the most popular datasets in the ML community.
Revised: Although the authors raised criticism against ImageNet and identified several inappropriate images, the ImageNet-ILSVRC-2012 dataset —and the pre-trained models— are still under the most popular datasets in the ML community.

Original: In line with Gebru et al.
Revised: In line with Gebru et al.

Original: (2018), Birhane & Prabhu (2021) urge that ethics checks for future dataset curation endeavors become an integral part of the human-in-the-loop validation phase.
Revised: (2018), Birhane & Prabhu (2021) urge that ethics checks for future dataset curation endeavors become an integral part of the human-in-the-loop validation phase.

Original: ImageNet.
Revised: ImageNet.

Original: The ImageNet (Deng et al., 2009) data collection is one of the most popular datasets in the computer vision domain and mostly refers to the subset ImageNet1k dataset with 1.2 million images across 1000 classes.
Revised: The ImageNet (Deng et al., 2009) data collection is one of the most popular datasets in the computer vision domain and mostly refers to the subset ImageNet1k dataset with 1.2 million images across 1000 classes.

Original: This was introduced in 2012 for the classification challenge in the
Revised: This was introduced in 2012 for the classification challenge in the

Original: ImageNet Large Scale Visual Recognition Challenge (ILSVRC).
Revised: ImageNet Large Scale Visual Recognition Challenge (ILSVRC).

Original: However, in total the collection (ImageNet21k) covers over 14 million images spread across 21,841 classes.
Revised: However, in total the collection (ImageNet21k) covers over 14 million images spread across 21,841 classes.

Original: As Birhane & Prabhu (2021) state, the ImageNet dataset remains one of the most influential and powerful image databases available today, although it was created over a decade ago.
Revised: As Birhane & Prabhu (2021) state, the ImageNet dataset remains one of the most influential and powerful image databases available today, although it was created over a decade ago.

Original: To apply transfer learning, the most popular deep learning frameworks provide downloadable pre-trained models for ImageNet1k.
Revised: To apply transfer learning, the most popular deep learning frameworks provide downloadable pre-trained models for ImageNet1k.

Original: Recently, Ridnik et al.
Revised: Recently, Ridnik et al.

Original: (2021) provided a novel scheme for high-quality, efficient pre-training on ImageNet21k and, along with it, the resulting pre-trained models.
Revised: (2021) provided a novel scheme for high-quality, efficient pre-training on ImageNet21k and, along with it, the resulting pre-trained models.

Original: Pre-training vision models with natural language supervision.
Revised: Pre-training vision models with natural language supervision.

Original: Pre-training methods that learn directly from raw data have revolutionized many tasks in natural language processing and computer vision over the last few years.
Revised: Pre-training methods that learn directly from raw data have revolutionized many tasks in natural language processing and computer vision over the last few years.

Original: Radford et al.
Revised: Radford et al.

Original: (2021) propose visual representation learning via natural language supervision in a contrastive learning setting.
Revised: (2021) propose visual representation learning via natural language supervision in a contrastive learning setting.

Original: The authors collected over 400M image-text pairs (WebImageText dataset) to show that the improvement with large-scale transformer models in NLP can be transferred to vision.
Revised: The authors collected over 400M image-text pairs (WebImageText dataset) to show that the improvement with large-scale transformer models in NLP can be transferred to vision.

Original: More precisely, while typical vision models jointly train an image feature extractor and a linear classifier, CLIP jointly trains an image encoder and a text encoder to predict the correct pairings of a batch of (image, text) training examples.
Revised: More precisely, whereas typical vision models jointly train an image feature extractor and a linear classifier, CLIP jointly trains an image encoder and a text encoder to predict the correct pairings of a batch of (image, text) training examples.

Original: At test time the authors propose to synthesize the learned text encoder with a (zero
Revised: At test time the authors propose to synthesize the learned text encoder with a (zero

Original: For simplicity, we refer to a model trained in a contrastive language-image pre-training setting and fine-tuned or probed for a downstream task as CLIP model.
Revised: For simplicity, we refer to a model trained in a contrastive language-image pre-training setting and fine-tuned or probed for a downstream task as CLIP model.

Original: Closely related to CLIP, the ALIGN (Jia et al., 2021) model is a family of multimodal dual encoders that learn to represent images and text in a shared embedding space.
Revised: Closely related to CLIP, the ALIGN (Jia et al., 2021) model is a family of multimodal dual encoders that learn to represent images and text in a shared embedding space.

Original: Instead of Vision-Transformers (ViT) or ResNet models ALIGN uses the
Revised: Instead of Vision-Transformers (ViT) or ResNet models ALIGN uses the

Original: EfficientNet (Tan & Le, 2019) and BERT (Devlin et al., 2019) models as vision and text encoders.
Revised: EfficientNet (Tan & Le, 2019) and BERT (Devlin et al., 2019) models as vision and text encoders.

Original: These encoders are trained from scratch on image-text pairs (1.8B pairs) via contrastive learning.
Revised: These encoders are trained from scratch on image-text pairs (1.8B pairs) via contrastive learning.

Original: These models and their zero-shot capabilities display significant promise for widely-applicable tasks like image retrieval or search
Revised: These models and their zero-shot capabilities display significant promise for widely-applicable tasks like image retrieval or search

Original: (Radford et al., 2021).
Revised: (Radford et al., 2021).

Original: For instance, since image and text are encoded in the same representational space, these models can find relevant images in a database given text or relevant text given an image.
Revised: For instance, since image and text are encoded in the same representational space, these models can find relevant images in a database given text or relevant text given an image.

Original: More importantly, the relative ease of steering CLIP toward various applications with little or no additional data or training unlocks novel applications that were difficult to solve with previous methods, e.g., as we show, inferring the offensiveness in images.
Revised: More importantly, the relative ease of steering CLIP toward various applications with little or no additional data or training unlocks novel applications that were difficult to solve with previous methods, e.g., as we show, inferring the offensiveness in images.

Original: Carried Knowledge of Pre-trained Large-Scale Models.
Revised: Carried Knowledge of Pre-trained Large-Scale Models.

Original: As already described, with training on raw text, large-scale transformer-based language models revolutionized many NLP tasks.
Revised: As already described, with training on raw text, large-scale transformer-based language models revolutionized many NLP tasks.

Original: Recently,
Revised: Recently,

Original: Radford et al.
Revised: Radford et al.

Original: (2021), Ramesh et al.
Revised: (2021), Ramesh et al.

Original: and Jia et al.
Revised: and Jia et al.

Original: (2021) showed encouraging results that a similar breakthrough in computer vision will be possible.
Revised: (2021) presented encouraging results that a similar breakthrough in computer vision will be possible.

Original: Besides the performance improvements in generation, regression, and classification tasks, these large-scale language models show surprisingly strong abilities to recall factual knowledge present in the training data (Petroni et al., 2019).
Revised: Besides the performance improvements in generation, regression, and classification tasks, these large-scale language models show surprisingly strong abilities to recall factual knowledge present in the training data (Petroni et al., 2019).

Original: Further,
Revised: Further,

Original: Roberts et al.
Revised: Roberts et al.

Original: (2020) showed that large-scale pre-trained language models’ capability to store and retrieve knowledge scales with model size.
Revised: (2020) showed that large-scale pre-trained language models’ capability to store and retrieve knowledge scales with model size.

Original: Since such models are often trained on unfiltered data, the kind of knowledge acquired is not controlled, leading to possibly undesirable behavior such as stereotypical and derogatory associations.
Revised: Since such models are often trained on unfiltered data, the kind of knowledge acquired is not controlled, leading to possibly undesirable behavior such as stereotypical and derogatory associations.

Original: However, Schick et al.
Revised: However, Schick et al.

Original: (2021) demonstrated that these models can recognize, to a considerable degree, their undesirable retained knowledge and the toxicity of the content they produce.
Revised: (2021) demonstrated that these models can recognize, to a considerable degree, their undesirable retained knowledge and the toxicity of the content they produce.

Original: The authors further showed that a language model with this ability can perform self-debiasing to reduce its probability of generating offensive text.
Revised: The authors further showed that a language model with this ability can perform self-debiasing to reduce its probability of generating offensive text.

Original: Furthermore, Jentzsch et al.
Revised: Furthermore, Jentzsch et al.

Original: and Schramowski et al.
Revised: and Schramowski et al.

Original: (2020) even show that the retained knowledge of such models carries information about moral norms aligning with the human sense of “right” and “wrong” expressed in language.
Revised: (2020) even showed that the retained knowledge of such models carries information about moral norms aligning with the human sense of “right” and “wrong” expressed in language.

Original: Similar to (Schick et al., 2021), Schramowski et al. (2021) demonstrate how to utilize this knowledge to guide autoregressive language models’ text generation to prevent their toxic degeneration.
Revised: Similar to (Schick et al., 2021), Schramowski et al. (2021) demonstrated how to utilize this knowledge to guide autoregressive language models’ text generation to prevent their toxic degeneration.

Original: In this work, we investigate if we are able to utilize the carried knowledge of large-scale vision models in a similar way, i.e. detecting possible offensive images in large-scale vision datasets.
Revised: In this work, we investigate if we are able to utilize the carried knowledge of large-scale vision models in a similar way, i.e. detecting possible offensive images in large-scale vision datasets.

Original:  P RE - TRAINED MODELS ARE ABLE TO INFER OFFENSIVENESS IN IMAGES .
Revised:  P RE - TRAINED MODELS ARE ABLE TO INFER OFFENSIVENESS IN IMAGES .

Original: Inspired by these previous results, we utilize a pre-trained multi-modal (language-vision) model to investigate its carried visual knowledge.
Revised: Inspired by these previous results, we utilize a pre-trained multi-modal (language-vision) model to investigate its carried visual knowledge.

Original: We make use of the multimodality by prompting the model with natural language to analyze if and to which extent it is able to infer the offensiveness in images.
Revised: We make use of the multimodality by prompting the model with natural language to analyze if and to which extent it is able to infer the offensiveness in images.

Original: Offensive images.
Revised: Offensive images.

Original: Let us start by defining the term “offending” and describing it in the context of images.
Revised: Let us start by defining the term “offending” and describing it in the context of images.

Original: According to the Cambridge dictionary 2 , “offending” can be phrased as “unwanted, often because unpleasant and causing problems”.
Revised: According to the Cambridge dictionary 2 , “offending” can be phrased as “unwanted, often because unpleasant and causing problems”.

Original: Additionally, in the context of images and text, according to Law Insider 3 : Offending Materials means any material, data, images, or information which is (a) in breach of any law, regulation, code of practice or acceptable use policy; or (b) defamatory, false, inaccurate, abusive, indecent, obscene or menacing or otherwise offensive; or (c) in breach of confidence, copyright or other intellectual property rights, privacy or any other right of any third party.
Revised: Additionally, in the context of images and text, according to Law Insider 3 : Offending Materials means any material, data, images, or information which is (a) in breach of any law, regulation, code of practice or acceptable use policy; or (b) defamatory, false, inaccurate, abusive, indecent, obscene or menacing or otherwise offensive; or (c) in breach of confidence, copyright or other intellectual property rights, privacy or any other right of any third party.

Original: In this work, we focus on images following the definition (b).
Revised: In this work, we focus on images following the definition (b).

Original: This definition aligns with definitions of previous work detecting hate speech (Gomez et al., 2020) and offensive product images (Gandhi et al., 2020).
Revised: This definition aligns with definitions of previous work detecting hate speech (Gomez et al., 2020) and offensive product images (Gandhi et al., 2020).

Original: As Gandhi et al.
Revised: As Gandhi et al.

Original: (2020) describe, technical challenges of building such a system are, among others, the lack of adequate training data, an extreme class imbalance, and changing test distribution.
Revised: (2020) have described, technical challenges of building such a system are, among others, the lack of adequate training data, an extreme class imbalance, and changing test distribution.

Original: However, we will showcase that recent pre-trained models trained on large-scale data guided by natural language supervision are able to distinguish between inappropriate, possible offending image content and other images based on their carried knowledge acquired during the pre-training phase.
Revised: However, we will showcase that recent pre-trained models trained on large-scale data guided by natural language supervision are able to distinguish between inappropriate, possible offending image content and other images based on their carried knowledge acquired during the pre-training phase.

Original: This is due to the self-supervised learning and the implied task-independence of the involved transformer
Revised: This is due to the self-supervised learning and the implied task-independence of the involved transformer.

Original: [Figure]
Revised: [Figure]

Original: Socio-Moral Image Database.
Revised: Note that offensiveness is a concept that is a function of social norms and people have diverse sentiments. In the present study, we define offensive content based on a data-centric approach. Therefore, the investigated “offensiveness” may primarily surfaces from the group of people that have generated the selected data and the annotators but also the pre-trained model’s implicit knowledge. Both, the model and the data could contain biases which can result in ethical issues.

Original:  To steer the model towards detecting  offensive objects, symbols as well as actions portrayed in images, we use the images contained in the Socio-Moral Image Database (SMID) (Crone et al., 2018) as samples for offensive image content.
Revised: We refer to the ethics statement (Section 6) for details. To steer the model towards detecting (morally) offensive image content, we used the images contained in the Socio-Moral Image Database (SMID) (Crone et al., 2018).

Original: The dataset contains 2 ,images annotated by attributes such as care, valence, arousal, and moral. According to the creators, the SMID dataset is the largest freely available moral stimulus database.
Revised: The dataset contains 2 , 941 images.

Original: It covers both the morally good and bad poles of a range of content dimensions, including the portrayal of moral actions and images of objects and symbols related to morality.
Revised: It covers both the morally good and bad poles of a range of content dimensions including objects, symbols as well as actions portrayed.

Original: In a large-scale survey, these images were annotated by participants.
Revised: In a large-scale survey, these images were annotated by 2 ,participants.

Original: In the case of rating the morality, the participants decided between the following statements: “This image portrays something < immoral/blameworthy > and < moral/praiseworthy > ”.

Original: Fig.
Revised: Fig.

Original: 2 shows the density distribution of the annotated labels.
Revised: 2 shows the density distribution of the annotated labels.

Original: Based on their findings, Crone et al.
Revised: Based on their findings, Crone et al.

Original: (2018) divided the data into good (green; meanmoral rating > 3 . 5 ), bad (red; meanmoral rating < 2 . 5 ), and neutral (grey; rest) images.
Revised: (2018) divided the data into good (green; mean rating > 3 . 5 ), bad (red; mean rating < 2 . 5 ), and neutral (grey; rest) images.

Original: The annotations describe what a human could —depending on the context— perceive as offending.

Original: Therefore, we train models to distinguish between moral and immoral content to infer the offensiveness in images.

Original: As the creators suggest, we discretised a rating < 2 . 5 as immoral, in our case offending, and rating > 3 . [-
-] as moral content (not offending), cf.
Revised: As the creators suggest, we discretised a rating < 2 . 5 as (morally) offending, and rating > 3 . [-
-]5 as not offending, cf.

Original: Fig.2.
Revised: Fig.

Original: We split the dataset (962 negative images and 712 positive images) into two random splits for our experiments.
Revised: We split the dataset (962 negative images and 712 positive images) into two random splits for our experiments.

Original: The training set contains 90%, and the test set the remaining 10% images.
Revised: The training set contains 90%, and the test set the remaining 10% images.

Original: In the following experiments, 10 -fold cross-validated results are reported.
Revised: In the following experiments, 10 -fold cross-validated results are reported.

Original: Deep Learning to classify offensive image content.
Revised: Deep Learning to classify offensive image content.

Original: In the context of the Christchurch mosque shooting video streamed on Facebook, officials at Facebook replied that one reason the video was not detected and removed automatically is that artificial intelligence systems are trained with large volumes of similar content.
Revised: In the context of the Christchurch mosque shooting video streamed on Facebook, officials at Facebook replied that one reason the video was not detected and removed automatically is that artificial intelligence systems are trained with large volumes of similar content.

Original: However, in this case, there was not enough comparable content because such attacks are rare 4 .
Revised: However, in this case, there was not enough comparable content because such attacks are rare 4 .

Original: Also Gandhi et al.
Revised: Also Gandhi et al.

Original: (2020) describe the lack of adequate training data to train a classifier to, in their case, detect offensive product content.
Revised: (2020) have described the lack of adequate training data to train a classifier to, in their case, detect offensive product content.

Original: We further investigate this issue with our next experiment by fine-tuning a common pre-trained model on few offensive labeled images.
Revised: We further investigate this issue with our next experiment by fine-tuning a common pre-trained model on few offensive labeled images.

Original: To measure how well a model can encode what a human could consider to be offending, we consider the above-mentionedSocio-Moral Image Database (in total 1 , 674 images).
Revised: To measure how well a model can encode what a human could consider to be offending, we considered the above-mentioned Database (in total 1 , 674 images).

Original: We start by training a deep vision model on this dataset.
Revised: We start by training a deep vision model on this dataset.

Original: Similar to Gandhi et al.
Revised: Similar to Gandhi et al.

Original: we chose the ResNet50 architecture (He et al., 2016) pre-trained on ImageNet datasets (Deng et al., 2009).
Revised: we chose the ResNet50 architecture (He et al., 2016) pre-trained on ImageNet datasets (Deng et al., 2009).

Original: Fig. 2(b) shows a dimension reduction via PCA of the embedded representations of the pre-trained model, i.e. before trained on the SMID dataset.
Revised: Fig. 2(b) shows a dimension reduction via PCA of the embedded representations of the pre-trained model, i.e. before trained on the SMID dataset.

Original: Based on this dimension reduction, it is unclear if the ImageNet1k pre-trained ResNet50 variant is able to infer offensiveness in images reliably.
Revised: Based on this dimension reduction, it is unclear if the ImageNet1k pre-trained ResNet50 variant is able to infer offensiveness in images reliably.

Original: Also, after training the network, the performance of the fine-tuned (training all model parameters), as well as linear probed model ( cf.
Revised: Also, after training the network, the performance of the fine-tuned (training all model parameters), as well as linear probed model ( cf.

Original: Tab. 1), shows inconclusive results; even if the performance increases when a larger dataset (ImageNet21k) is used.
Revised: Tab. 1), shows inconclusive results; even if the performance increases when a larger dataset (ImageNet21k) is used.

Revised: Specifically, the resulting weak precision and weak recall of probing the ImagNet1k based models shows us that it has issues classifying truly offensive images as well as distinguishing between truly

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  WebTextImage dataset (Radford et al., 2021).
Revised: the WebTextImage dataset (Radford et al., 2021).

Original: The ViT is pre-trained on the WebTextImage dataset.
Revised: The ViT is pre-trained on the WebTextImage dataset.

Original: On the ImageNet datasets, we applied linear probing (top), and on the WebImageText-based models soft-prompt tuning.
Revised: On the ImageNet datasets, we applied linear probing (top), and on the WebImageText-based models soft-prompt tuning.

Original: Tuning was performed on different sizes of the SMID training set where 100% corresponds to 1506 images.
Revised: Tuning was performed on different sizes of the SMID training set where 100% corresponds to 1506 images.

Original: One can see that steering CLIP towards inferring the offensiveness in images requires only little additional data.
Revised: One can see that steering CLIP towards inferring the offensiveness in images requires only little additional data.

Original: In contrast to other pre-trained models, it therefore provides a reliable method to detect offending images.
Revised: In contrast to other pre-trained models, it therefore provides a reliable method to detect offending images.

Revised: non-offensive and offensive images.

Revised: After fine-tuning the whole model, it can increase its recall; however, the precision stays comparable low.

Original: This supports the previous findings mentioned above.
Revised: This supports the previous findings mentioned above.

Original: Next, we will consider these models as baselines to investigate if more advanced PMs trained on larger unfiltered datasets carry knowledge about offensiveness.
Revised: Next, we will consider these models as baselines to investigate if more advanced PMs trained on larger unfiltered datasets carry knowledge about offensiveness.

Original: Pre-trained, natural language guided models carry knowledge about offensiveness.
Revised: Pre-trained, natural language guided models carry knowledge about offensiveness.

Original: By training on over 400M data samples and with natural language supervision, CLIP Radford et al. (2021), and other similar models Jia et al. (2021), acquire (zero-)few-shot capabilities displaying a significant promise for applications with little or no additional data.
Revised: By training on over 400M data samples and with natural language supervision, CLIP (Radford et al., 2021), and other similar models (Jia et al., 2021), acquire (zero-)few-shot capabilities displaying a significant promise for applications with little or no additional data.

Original: Next, we  investigate if this includes the detection of offensive image content.
Revised: Next, we will investigate if this includes the detection of offensive image content.

Original: Fig. 2(c) shows the embedded representations of the ViT-B/16 model pre-trained on WebImageText via Contrastive Language-Image Pre-training (Radford et al., 2021).
Revised: Fig. 2(c) shows the embedded representations of the ViT-B/16 model pre-trained on WebImageText via Contrastive Language-Image Pre-training (Radford et al., 2021).

Original: One can observe that the ViT’s learned representation encodes knowledge of the underlying task, i.e. distinguish offensive and not offensive images without being explicitly trained to do so.
Revised: One can observe that the ViT’s learned representation encodes knowledge of the underlying task, i.e. distinguish offensive and not offensive images without being explicitly trained to do so.

Original: These results confirm our assumption that due to the natural language supervision, CLIP implicitly acquired knowledge about what a human could —depending on the context— perceive as offending.
Revised: These results confirm our assumption that due to the natural language supervision, CLIP implicitly acquired knowledge about what a human could —depending on the context— perceive as offending.

Original: Furthermore, the natural language supervision of CLIP allows us to probe the model without training it (zero-shot).
Revised: Furthermore, the natural language supervision of CLIP allows us to probe the model without training it (zero-shot).

Original: More precisely, as with the previous model (ResNet50 pre-trained on ImageNet), the images are encoded via the pre-trained visual encoder.
Revised: More precisely, as with the previous model (ResNet50 pre-trained on ImageNet), the images are encoded via the pre-trained visual encoder.

Original: Instead of training a linear classifier, we operate on the similarity of samples, in this case, the cosine similarity, in the representational space:
Revised: Instead of training a linear classifier, we operate on the similarity of samples, in this case, the cosine similarity, in the representational space:

Original: [Equation]
Revised: [Equation]

Original: where E visual and E text are the visual and text encoders, and x an image sample and z a prompt.
Revised: where E visual and E text are the visual and text encoders, and x an image sample and z a prompt.

Original: We embedded the classes, as suggested by Radford et al.
Revised: We embedded the classes, as suggested by Radford et al.

Original: (2021), into natural language prompts such as “This image is about something < label > .”, which has shown to be a good default helping to specify that the text is about the content of the image.
Revised: (2021), into natural language prompts such as “This image is about something < label > .”, which has shown to be a good default helping to specify that the text is about the content of the image.

Original: Following the collection of human annotations contained in the SMID dataset Crone et al. (2018), we applied various prompt classes: bad/good behavior , blameworthy/praiseworthy , positive/negative and moral/immoral .
Revised: Following the collection of human annotations contained in the SMID dataset (Crone et al., 2018), we applied various prompt classes: bad/good behavior , blameworthy/praiseworthy , positive/negative and moral/immoral .

Original: Whereby, the labels positive and negative resulted in the best zero-shot performance.
Revised: Whereby, the labels positive and negative resulted in the best zero-shot performance.

Original: Fig.
Revised: Fig.

Original: 3 ( 0% ) shows that this zero-shot approach utilizing the implicit knowledge of the CLIP models is already performing on par with the ImageNet-based PMs which were fine-tuned on SMID.
Revised: 3 ( 0% ) shows that this zero-shot approach utilizing the implicit knowledge of the CLIP models is already performing on par with the ImageNet-based PMs which were fine-tuned on SMID.

Original: However, we noticed that the zero-shot approach is able to classify true-negative samples well but performs less well on classifying positives.
Revised: However, we noticed that the zero-shot approach is able to classify true-negative samples well but performs less well on classifying positives.

Original: This suggests that both, or at least the prompt corresponding to the positive class label, are not chosen optimally.
Revised: This suggests that both, or at least the prompt corresponding to the positive class label, are not chosen optimally.

Original: The nearest image neighbors extracted from the SMID dataset ( cf.
Revised: The nearest image neighbors extracted from the SMID dataset ( cf.

Original: Fig.
Revised: Fig.

Original: 4 top-right) confirm this observation.
Revised: 4 top-right) confirm this observation.

Original: No need to learn new features: Learning how to ask the model.
Revised: No need to learn new features: Learning how to ask the model.

Original: The previously defined prompts may not be the optimal way to query the model’s implicit knowledge to infer the offensiveness in images.
Revised: The previously defined prompts may not be the optimal way to query the model’s implicit knowledge to infer the offensiveness in images.

Original: To further steer the model, we, therefore, searched for optimal text embeddings, i.e. optimize the prompts, which is also called (soft-) prompt-tuning (Zhong et al., 2021; Qin & Eisner, 2021).
Revised: To further steer the model, we, therefore, searched for optimal text embeddings, i.e. optimize the prompts, which is also called (soft-) prompt-tuning (Zhong et al., 2021; Qin & Eisner, 2021).

Original: As an optimization task we define the distinction of offensive and not offensive images and optimize the prompts by gradient descent as follows:
Revised: As an optimization task we defined the distinction of offensive and not offensive images and optimized the prompts by gradient descent as follows:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]

Original: ˆy

Original: [Equation]
Revised: [Equation]

Original: Note, that we do not update the parameters, θ , of E visual and E text .
Revised: Note, that we do not updated the parameters, θ , of E visual and E text .

Original: The term y is the ground truth class label and X a batch during the stochastic gradient descent optimization.
Revised: The term y is the ground truth class label and X a batch during the stochastic gradient descent optimization.

Original: The resulting prompts are shown in Fig.
Revised: The resulting prompts are shown in Fig.

Original: 4 and clearly portray on the one side possible offending image content and the other side positive content.
Revised: 4 and clearly portray on the one side possible offending image content and the other side positive content.

Original: Next, we  evaluate the resulting CLIP model equipped with the newly designed prompts.
Revised: Next, we will evaluate the resulting CLIP model equipped with the newly designed prompts.

Original: Fig. 3 shows that even a small portion of the training data ( e.g. 4%, 60 images) increases the vision transformer’s (ViT-B) performance to over 90%.
Revised: Fig.shows that even a small portion of the training data ( e.g. 4%, 60 images) increases the vision transformer’s (ViT-B) performance to over 90%.

Original: In general, the vision transformer outperforms the pre-trained ResNet50 models.
Revised: In general, the vision transformer outperforms the pre-trained ResNet50 models.

Original: Furthermore, the vision transformer with higher model capacity outperforms the smaller variant, indicating that not only the dataset’s size is important, but also the size of the model.
Revised: Furthermore, the vision transformer with higher model capacity outperforms the smaller variant, indicating that not only the dataset’s size is important, but also the capacity of the model.

Original: Training with the full training set reaches a final test accuracy of 96 . 30% ± 1 .[-
-]
Revised: Training with the full training set reaches a final test accuracy of 96 . 30% ± 1 .[-
-]

Original: ( cf.
Revised: ( cf.

Original: Tab. 1).
Revised: Tab. 1).

Original: These results clearly show that large-scale pre-trained transformer models are able to infer the offensiveness in images and that they already acquire this required knowledge during their pre-training phase guided by natural language supervision.
Revised: These results clearly show that large-scale pre-trained transformer models are able to infer the offensiveness in images and that they already acquire this required knowledge during their pre-training phase guided by natural language supervision.

Original: Summarized, the results clearly show that our approach provides a reliable method to identify offensive image content.

Original:  M ACHINES A SSIST TO D ETECT O FFENDING I MAGES IN CV B ENCHMARKS
Revised:  M ACHINES A SSIST TO D ETECT O FFENDING I MAGES IN CV B ENCHMARKS

Original: Next, we utilized the pre-trained CLIP model, and the SMID-based selected prompts to identify possible offending images from popular computer vision benchmark datasets.
Revised: Next, we utilize the pre-trained CLIP model, and the SMID-based selected prompts to identify possible offending images from popular computer vision benchmark datasets.

Original: As Birhane & Prabhu (2021) we focus on ImageNet and use its most-popular subset the ImageNet-ILSVRC-2012 dataset as an example.
Revised: As Birhane & Prabhu (2021) we focus on ImageNet and use its most-popular subset the ImageNet-ILSVRC-2012 dataset as an example.

Original: Using our previously described approach the pre-selection by CLIP extracts possible offensive images.
Revised: Using our previously described approach the pre-selection by CLIP extracts possible offensive images.

Original: However, offensiveness is subjective to the user and, importantly, the task at hand.
Revised: However, offensiveness is subjective to the user and, importantly, the task at hand.

Revised: In our case, the steered model may primarily mirror the moral compass and social expectations of the 2 ,annotators.

Original: Therefore, it is required that humans and machines interact with each other, and the human user can select the images based on a given setting and requirements.
Revised: Therefore, it is required that humans and machines interact with each other, and the human user can select the images based on a given setting and requirements.

Original: Hence, we do not advise removing specific images but investigate the range of examples and offensiveness selected by the system and thereby document the dataset.
Revised: Hence, we do not advise removing specific images but investigate the range of examples and offensiveness selected by the system and thereby document the dataset.

Original: We here provide an exemplary list of contents and disguised images (Fig. 5).
Revised: We here provide an exemplary list of contents and disguised images (Fig. 5).

Original: Additionally, we provide Python notebooks with the corresponding images along with the classifier in the supplemental material.
Revised: Additionally, we provide Python notebooks with the corresponding images along with the classifier in the supplemental material.

Original: Moreover, to enforce targeting possible strongly offensive images, we determined the prompts by shifting the negative threshold to a rating of 1 . 5 instead of 2 . 5 .
Revised: Moreover, to enforce targeting possible strongly offensive images, we determined the prompts by shifting the negative threshold to a rating of 1 . 5 instead of 2 . 5 .

Original: Due to the complexity of offensive context, we separate the identified offensiveness into offending objects, symbols, and actions in images.
Revised: Due to the complexity of offensive context, we separate the identified offensiveness into offending objects, symbols, and actions in images.

Original: Objects.
Revised: Objects.

Original: The ImageNet1k dataset, also known as ImageNet-ILSVRC-2012, formed the basis of task-1 of the ImageNet Large Scale Visual Recognition Challenge.
Revised: The ImageNet1k dataset, also known as ImageNet-ILSVRC-2012, formed the basis of task-1 of the ImageNet Large Scale Visual Recognition Challenge.

Original: Hence, all images display animals or objects.
Revised: Hence, all images display animals or objects.

Original: Therefore it is not surprising that the largest portion of offensive content concerns negative associated objects and animals.
Revised: Therefore it is not surprising that the largest portion of offensive content concerns negative associated objects and animals.

Original: In total, 40 , 501 images were identified by the offensiveness classifier, where the objects “gasmask” (797 images), “guillotine” (783), and “revolver” (725) are the top-3 classes.
Revised: In total, 40 , 501 images were identified by the offensiveness classifier, where the objects “gasmask” (797 images), “guillotine” (783), and “revolver” (725) are the topclasses.

Original: However, while most people would assign these objects as morally questionable and offensive, they can not be treated as offensive when training a general object classifier.
Revised: However, whereas most people would assign these objects as morally questionable and offensive, they can not be treated as offensive when training a general object classifier.

Original: The same applies to the animal-classes tick (554) and spider (397).
Revised: The same applies to the animal-classes tick (554) and spider (397).

Original: To infer the offensiveness of images contained in ImageNet, it may be more applicable to investigate classes with only a small portion of possible offensive images.
Revised: To infer the offensiveness of images contained in ImageNet, it may be more applicable to investigate classes with only a small portion of possible offensive images.

Original: Next to injured ( e.g. “koala”, “king penguin”) and aggressive animals ( e.g. “pembroke”, “redbone”), our proposed classifier detects caged ( e.g. “great pyrenees”, “cock”) and dead animals ( e.g. “squirrel monkey”, “african elephant”).
Revised: Next to injured ( e.g. “koala”, “king penguin”) and aggressive animals ( e.g. “pembroke”, “redbone”), our proposed classifier detects caged ( e.g. “great pyrenees”, “cock”) and dead animals ( e.g. “squirrel monkey”, “african elephant”).

Original: Additionally, objects in inappropriate, possible offensive scenes, like a bathtub tainted with blood (“tub”) are extracted.
Revised: Additionally, objects in inappropriate, possible offensive scenes, like a bathtub tainted with blood (“tub”) are extracted.

Original: Symbols.
Revised: Symbols.

Original: Furthermore, one is able to identify offensive symbols and text on objects: several 
Revised: Furthermore, one is able to identify offensive symbols and text on objects: several National

Original: National Socialist symbols especially swastika ( e.g. “mailbag”, “military uniform”), persons in Ku-Klux-Klan uniform ( e.g. “drum”), insults by e.g. showing the middle finger ( e.g. “miniature pinscher”, “gorilla”, “lotion”), and inappropriate text such as “child porn” (“file”) and “bush=
Revised: Socialist symbols especially swastika ( e.g. “mailbag”, “military uniform”), persons in Ku-KluxKlan uniform ( e.g. “drum”), insults by e.g. showing the middle finger ( e.g. “miniature pinscher”, “gorilla”, “lotion”), and inappropriate text such as “child porn” (“file”) and “bush=

Original: i***t f*** off USA” (“pay-phone”).
Revised: i***t f*** off USA” (“pay-phone”).

Original: Actions.
Revised: Actions.

Original: In addition to objects and symbols, our proposed classifier is able to interpret scenes in images and hence identify offensive actions shown in images.
Revised: In addition to objects and symbols, our proposed classifier is able to interpret scenes in images and hence identify offensive actions shown in images.

Original: Scenes such as burning buildings ( e.g. “church”) and catastrophic events ( e.g. “airliner”, “trailer truck”) are identified.
Revised: Scenes such as burning buildings ( e.g. “church”) and catastrophic events ( e.g. “airliner”, “trailer truck”) are identified.

Original: More importantly, offensive actions with humans involved are extracted such as comatose persons ( e.g. “apple”, “brassiere”, “tub”), persons involved in an accident ( e.g. “mountain bike”), the act of hunting animals ( e.g. “African elephant”, “impala”), a terrifying person hiding under a children’s crib (“crib”), scenes showing weapons or tools used to harm, torture and kill animals ( e.g. “hamster”) and people ( e.g. “hatchet”, “screwdriver”, “ballpoint”, “tub”).
Revised: More importantly, offensive actions with humans involved are extracted such as comatose persons ( e.g. “apple”, “brassiere”, “tub”), persons involved in an accident ( e.g. “mountain bike”), the act of hunting animals ( e.g. “African elephant”, “impala”), a terrifying person hiding under a children’s crib (“crib”), scenes showing weapons or tools used to harm, torture and kill animals ( e.g. “hamster”) and people ( e.g. “hatchet”, “screwdriver”, “ballpoint”, “tub”).

Original: Furthermore, derogative scenes portraying men and women wearing muzzles (“muzzle”), clearly misogynistic images e.g. harmed women wearing an abaya, but also general nudity with exposed genitals ( e.g. “bookshop”, “bikini”, “swimming trunks”) and clearly derogative nudity ( e.g. “plastic bag”) are automatically selected by our proposed method.
Revised: Furthermore, derogative scenes portraying men and women wearing muzzles (“muzzle”), clearly misogynistic images e.g. harmed women wearing an abaya, but also general nudity with exposed genitals ( e.g. “bookshop”, “bikini”, “swimming trunks”) and clearly derogative nudity ( e.g. “plastic bag”) are automatically selected by our proposed method.

Original: Note that e.g. the misogynistic image showing a harmed woman wearing an abaya was not identified by the human hand surveyed image selection of Birhane & Prabhu (2021).
Revised: Note that e.g. the misogynistic image showing a harmed woman wearing an abaya was not identified by the human hand surveyed image selection of Birhane & Prabhu (2021).

Original: Therefore, we strongly advocate utilizing the implicit knowledge of large-scale state-of-the-art models in a human-in-the-loop curation process to not only partly automatize the process but also to reduce the susceptibility to errors.
Revised: Therefore, we strongly advocate utilizing the implicit knowledge of large-scale state-of-the-art models in a human-in-the-loop curation process to not only partly automatize the process but also to reduce the susceptibility to errors.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In recent years, deep learning approaches, especially transfer learning from models pre-trained on large-scale supervised data, have become standard practice for many applications.
Revised: In recent years, deep learning approaches, especially transfer learning from models pre-trained on large-scale supervised data, have become standard practice for many applications.

Original: To train such models, a tremendous amount of data is required.
Revised: To train such models, a tremendous amount of data is required.

Original: As a result, these datasets are insufficiently filtered collections crawled from the web.
Revised: As a result, these datasets are insufficiently filtered collections crawled from the web.

Original: Recent studies (Gebru et al., 2018; Birhane & Prabhu, 2021;
Revised: Recent studies (Gebru et al., 2018; Birhane & Prabhu, 2021;

Original: Bender et al., 2021) have revealed that models trained on such datasets, and the resulting models for downstream tasks benefiting from these pre-trained models, implicitly learn undesirable behavior, e.g., stereotypical associations or negative sentiment towards certain groups.
Revised: Bender et al., 2021) have revealed that models trained on such datasets, and the resulting models for downstream tasks benefiting from these pre-trained models, implicitly learn undesirable behavior, e.g., stereotypical associations or negative sentiment towards certain groups.

Original: Consequently, there is an urgent need to document datasets and curate their content carefully.
Revised: Consequently, there is an urgent need to document datasets and curate their content carefully.

Original: Unfortunately, current processes are tedious, error-prone, and do not scale well to large datasets.
Revised: Unfortunately, current processes are tedious, error-prone, and do not scale well to large datasets.

Original: To assist humans in the dataset curation process, we, therefore, introduced a novel approach utilizing the implicit knowledge of large-scale pre-trained models and illustrated its benefits.
Revised: To assist humans in the dataset curation process, we, therefore, introduced a novel approach utilizing the implicit knowledge of large-scale pre-trained models and illustrated its benefits.

Original: We showed that
Revised: We showed that

Original: CLIP (Radford et al., 2021) retains the required knowledge about what a human would consider to be offending during its pre-training phase.
Revised: CLIP (Radford et al., 2021) retains the required knowledge about what a human would consider to be offending during its pre-training phase.

Original: As a result, it offers a solution to overcome previous issues, namely the lack of sufficient training data to identify offensive material automatically.
Revised: As a result, it offers a solution to overcome previous issues, namely the lack of sufficient training data to identify offensive material automatically.

Original: In this regard, we have outlined a new solution to assist the curation process on large-scale datasets.
Revised: In this regard, we have outlined a new solution to assist the curation process on large-scale datasets.

Original: On the example of the ImageNet-ILSVRC2012 dataset, we showcased that our proposed approach can identify additional inappropriate content compared to previous studies.
Revised: On the example of the ImageNet-ILSVRC2012 dataset, we showcased that our proposed approach can identify additional inappropriate content compared to previous studies.

Original: Our approach can be transferred to any other vision dataset.
Revised: Our approach can be transferred to any other vision dataset.

Original: In future work, we thus plan to extend our analysis to other datasets such as the OpenImage dataset (Kuznetsova et al., 2020) and multi-modal datasets (Jia et al., 2021).
Revised: In future work, we thus plan to extend our analysis to other datasets such as the OpenImage dataset (Kuznetsova et al., 2020) and multi-modal datasets (Jia et al., 2021).

Original: Further possible avenues for future work are the extensions of the proposed method to multi-label classification to directly separate offensive objects, symbols, and actions or derive other categories of offensive content.
Revised: Further possible avenues for future work are the extensions of the proposed method to multi-label classification to directly separate offensive objects, symbols, and actions or derive other categories of offensive content.

Original: Moreover, classifying different levels of offensiveness could further provide details to document datasets; however, this could require additional data.
Revised: Moreover, classifying different levels of offensiveness could further provide details to document datasets; however, this could require additional data.

Original: Since the underlying model of our proposed classifier is a deep learning model, it inherits its black-box properties.
Revised: Since the underlying model of our proposed classifier is a deep learning model, it inherits its black-box properties.

Original: This makes it hard to understand why the model is identifying specific images.
Revised: This makes it hard to understand why the model is identifying specific images.

Original: Applying explainable AI methods such as (Chefer et al., 2021) to explain the reasoning process could lead to further improvement of the curation process.
Revised: Applying explainable AI methods such as (Chefer et al., 2021) to explain the reasoning process could lead to further improvement of the curation process.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: Our proposed method provides a solution to automatically infer offensiveness in images with the intention to assist the curation process and documentation of datasets.
Revised: Our proposed method provides a solution to automatically infer offensiveness in images with the intention to assist the curation process and documentation of datasets.

Original: However, we strongly advise applying such methods in a human-in-the-loop setting.
Revised: However, we strongly advise applying such methods in a human-in-the-loop setting.

Original: Since CLIP models themselves are trained with weak supervision on not freely accessible data sources and only the pre-trained models are provided by its’ creators, it is unclear if e.g. social biases are inherent to the model.
Revised: Since CLIP models themselves are trained with weak supervision on not freely accessible data sources and only the pre-trained models are provided by its’ creators, it is unclear if e.g. social biases are inherent to the model.

Original: Details about possible biases and other potential misuses ( e.g. surveillance) of the CLIP models can be found in the original work of Radford et al.
Revised: Details about possible biases and other potential misuses ( e.g. surveillance) of the CLIP models can be found in the original work of Radford et al.

Revised: Furthermore, in the present study, offensive content is detected by a data-centric AI system, where the model is steered towards the sentiment reflected in the data.

Revised: Therefore, next to the model’s contained biases, the investigated “offensiveness” may primarily surfaces from the group of people that have generated the selected data and the annotators but also the pre-trained model’s implicit knowledge.

Revised: An interesting and important avenue for future work is addressing what different groups of society, e.g. different cultures, would consider as offensive by the creation of datasets representing their corresponding sentiments.

Revised: As other social norms, offensiveness is constantly evolving.

Revised: This makes it necessary to update the data, system and documentations.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: The code to reproduce the figures and results of this article, including pre-trained models, can be found in the publicly available repository.
Revised: The code to reproduce the figures and results of this article, including pre-trained models, can be found in the publicly available repository.

Original: Furthermore, we provide the source code needed in order to determine prompts to steer the CLIP model towards inferring offensiveness in images and apply it to detect possible offending images in vision datasets.
Revised: Furthermore, we provide the source code needed in order to determine prompts to steer the CLIP model towards inferring offensiveness in images and apply it to detect possible offending images in vision datasets.

Original: The figures with disguised images are provided in original form in the supplement material.
Revised: The figures with disguised images are provided in original form in the supplement material.