Knowledge Distillation with BERT for Image Tag-Based Privacy Prediction (2021)

Abstract: Text in the form of tags associated with online images is often informative for predicting private or sensitive content from images. When using privacy prediction systems running on social networking sites that decide whether each uploaded image should get posted or be protected, users may be reluctant to share real images that may reveal their identity but may share image tags. In such cases, privacy-aware tags become good indicators of image privacy and can be utilized to generate privacy decisions. In this paper, our aim is to learn tag representations for images to improve tag-based image privacy prediction. To achieve this, we explore self-distillation with BERT, in which we utilize knowledge in the form of soft probability distributions (soft labels) from the teacher model to help with the training of the student model. Our approach effectively learns better tag representations with improved performance on private image identification and outperforms state-of-the-art models for this task. Moreover, we utilize the idea of knowledge distillation to improve tag representations in a semi-supervised learning task. Our semi-supervised approach with only 20% of annotated data achieves similar performance compared with its supervised learning counterpart. Last, we provide a comprehensive analysis to get a better understanding of our approach.

## introduction with the rapid growth of the number of users on online social networking sites, image privacy has become a major concern @xcite . users may accidentally disclose their sensitive information such as locations, habits or personal relationships from images that they post to their social networking sites @xcite , which could be used in the detriment of the users @xcite . @xcite defines private images as ones that belong to the private sphere (e.g., portraits, family, home) or capture sensitive contents that can not be shared with everyone on the internet. the remaining images are considered to be public. binary image privacy classifiers are developed @xcite @xcite aiming to identify whether images belong to the public class or the private class. however, the access to the image content is not always allowed since users may be reluctant to share the real images (revealing user's identity through the face, and friends, etc.) for visual content analysis. in such cases, tags attached by users to describe their images are found to be informative about the image contents and are good indicators of the privacy settings and improve the privacy prediction methods @xcite . privacy prediction models trained with image tags achieve competitive results compared with vision-based privacy prediction models @xcite . therefore, our goal is to learn good tag representations for images to further improve the performance of tag-based privacy prediction. pre-trained language models have been extensively studied in nlp communities @xcite @xcite . bert @xcite is a pre-trained language model based on a multi-layer bidirectional transformer, and has shown to be effective for generating universal language representations and attains state-of-the-art performance on many natural language processing tasks @xcite . in our work, we fine-tune bert for the task of tag-based image privacy prediction to generate better tag representations. in addition, we propose to use self knowledge distillation with bert @xcite @xcite to further improve the performance of tag-based privacy prediction. specifically, we first train a bert @xcite teacher model for privacy prediction, and then train a bert student model using both class labels and the pre-trained teacher model's output probability distributions. the student model can thus learn from not only the ground truth class information, but also how to assign compatible probabilities according to various input examples. experimental results show that knowledge distillation effectively improves tag representations and achieves boosted prediction performance. moreover, training a classifier often requires a large amount of annotated data. however, the annotation process is very time consuming and requires a significant human effort in many cases @xcite . thus, we investigate knowledge distillation in a semi-supervised learning approach @xcite . to do this, we first train a bert teacher model using limited amount of labeled image tags, and use it to annotate a large amount of unlabeled data, which is further used to train another bert student model for privacy prediction. experimental results show that our semi-supervised approach with bert learns good tag representations and achieves comparable performance with its supervised counterpart with only 20% annotated image tags. last, we provide a comprehensive analysis for our tag-based privacy prediction. first, we perform a calibration analysis to show that models trained by improved tag representations with knowledge distillation yield better calibration (the alignment between prediction confidence and correctness likelihood @xcite ). second, neural models are sensitive to small perturbation in the input and a small perturbation on the input may fool a well-trained neural network @xcite @xcite . we analyze the robustness of our privacy classification models trained by tag representations learned with knowledge distillation against adversarial attacks. the results show that our approach shows the most robustness against adversarial attacks over compared baselines. third, we perform a statistical analysis on the correlation between the privacy and sentiment and emotion of image tags. ## methods in this work, we adopt knowledge distillation with bert to learn better tag representations for tagbased image privacy prediction. the idea behind knowledge distillation is that: soft probability distributions generated by a pre-trained image privacy prediction model carries additional privacy information compared with hard class labels. specifically, hard labels can only reflect the class information (either private or public) of input image tags, while soft probability distributions can further reveal the confidence of the privacy classification model toward each prediction. a proper usage of such additional information, in combination with hard labels, can help learn better tag representations to boost the performance of tag-based image privacy prediction models. the goal of this work is to distill knowledge using bert to transfer knowledge (in the form of soft probability distributions) from a strong, pre-trained bert teacher model to a bert student model to boost the performance of the latter for tag-based privacy prediction. ## conclusion in this work, we explore learning tag representations with knowledge distillation approach based on bert for tag-based image privacy prediction. our approach significantly outperforms the state-ofart models for tag-based image privacy prediction. we also perform a bert-based semi-supervised learning approach using only a small amount of annotated data, where bert achieves comparable performance with its supervise counterpart with only 20% of labeled data provided. moreover, we also perform calibration analysis and show that tag representations learned with knowledge distillation yield better calibration. we also study the robustness of our learned tag representations against some adversarial attacks for image tags. our results show that our approach show stronger robustness over compared baselines against random and synonymbased attacks. last, we analyze the correlation between the privacy and the emotion of image tags and use some examples in the picalert dataset @xcite to help us understand the underlying reasons. our future direction is to integrate deep cnn models for image processing with bert to develop a multi-modal image privacy prediction model with both images and tags as inputs.