firmed cases worldwide 1 [10] . Following WHO health advisories, most countries have declared national emergencies, closed borders, and restricted public movement [13] . Many studies have recognized the importance of community-wide wearing masks for controlling the pandemic [3] [19]. Masks have been found to reduce potential exposure risk from an infected person, proving to be a successful measure to suppress transmission and save lives [8] [4] . With its ease of access and global outreach, Social Media has a disproportionate influence on disseminating information during a pandemic. In recent times of the pandemic, social media has become a popular platform for people to express their thoughts, opinions and broadcast activities. The general public and authorities have been using hashtags like #CoronaOutbreak, #COVID19, and #mask to disseminate important information and health advisories. This provides us with an opportunity to analyze behaviors and the impact of such advisories worldwide. This study analyzes social media images from Instagram for (not) using masks and evaluation of mask-fit using deep neural networks. The major hurdle in using deep neural networks for face mask detection is the limited availability of annotated datasets with masked and unmasked faces. We curated a dataset of 2.04 million location-tagged images scraped from the internet. We focused our study on six US cities: New York City, Seattle, Boston, Dallas, Minneapolis, New Orleans. Then, we did a comparative analysis from images on each of these cities in terms of the number of mask wearers, non-mask wearers. Then, we also studied the effect of the Government's mask-wearing guidelines on Instagram images from these cities during the pandemic's early months. We also studied the mask-wearing behaviors, including mask-fit, and derived a fit score for the same. Following data collection from Instagram, we used face detec-tion, facial landmarking, and semantic segmentation algorithms to quantify these indices in the six cities. The facial landmarking algorithm detected the jaw region (mask location), which was then fed as input to the deep learning models for classifying each face as "masked" or "unmasked" and semantic segmentation of mask v/s rest. Since protests involve large physical grouping of people, we calculated our novel indices upon images from these events, thus providing a significant opportunity to study compliance and analyze social media trends. The rest of this paper is organized in the following manner. Section 2 talks about related works in the field of mask classification and face detection. Section 3 presents datasets, models, and methods that were used for the study. Section 4 talks about our experiments, results, and the derived analysis.
Social media has been extensively explored and analyzed for patterns that emerged during the COVID-19 pandemic. Ritterman et al. [14] showed how prediction market models and social media analysis can be used to model public sentiment on the spread of a pandemic. Signorini et al. [16] examined Twitter based-information to track the swiftly-evolving public sentiment regarding Swine Flu in 2011 as well as correlate the H1N1 related activity to accurately track reported disease levels in the US. Realizing the potential of social media in understanding such events, in this study, we utilize the social media images from Instagram, a popular imagesharing social media platform. We found two datasets previously published for mask classification tasks, MAFA[7] and RMFD [9] . MAFA dataset contains 35,805 masked images, with many wrong annotations. Since, the MAFA Dataset was curated and released in 2017, it couldn't capture different varieties and types of masks that have been in use during the pandemic period. Also MAFA dataset is highly biased towards one kind of mask, it majorly consists of medical staff wearing disposable medical grade masks. RMFD contains 7,959 masked images containing a variety of masks used during the Covid period. However, the images were of low quality and dimensions rendering them useless for analysing high quality social media images. After training our model on our dataset we were able to easily beat RMFD model's results on their test dataset. To our best knowledge, no work exists which is trying to analyse mask fit using semantic segmentation and a corresponding annotated dataset.
For masked face images, around 30,000 images were collated from Google Search Images and Instagram explore tag pages 2 . Tags like "people wearing mask", "children wearing masks", etc. [full list in Table 6 ] were used to scrape images from google. Images from Instagram were also collected with the Tag explore feature, using these three tags: "mask", "masked", "covidmask". Post data collection, images with width and height of at least 50 pixels were kept. Then Face Detector from section 3.2.1 was used on these images to extract faces. Then the images of faces were then distributed between 5 annotators, and annotators were asked to classify the face as either "masked" or "unmasked". Images annotated as "masked" were cross-validated by another annotator. After the annotations, images of 9055 masked faces were obtained, which were used in section 3.2.2 to train the classifier. This data-set is publicly available for use in use and further research at precog.iiitd.edu.in/resources.html.
For un-masked face images we made a random sample(without replacement) of size 9055 from VGGFace2 [2] Dataset . This, along with the dataset from section 3.1.1 was used to train the Unmask-Mask Classifier in section 4.1.1.
The goal was to evaluate whether the masks are being worn effectively or not using the images from social media. We employ 'semantic segmentation' to solve this problem (Details about how we use semantic segmentation are in 3.2.3). Semantic segmentation requires pixel-level annotations for the entire image. Out of 9055 masked faces that we obtained from section 3.1.1, we selected 504 images with different poses and a wide variety of mask designs and then annotated these images for semantic segmentation using Label Studio 3 . This dataset was then used to train a semanticsegmentation model in section 4.1.2. This data-set is publicly available for use in use and further research at precog.iiitd.edu.in/resources.html.
For the analysis phase, we collected the location tagged public posts from Instagram between 1 st Feb 2020 and 31 st May fig. 1 ). This collection was done for six major US cities. These 6 cities were selected to represent different geographical sections of the country. We collected a total of 2 million public posts from these 6 cities. These posts were collected via Instagram's explore location. Instagram's GraphQL API was employed for this purpose. The tools used have been published as a python PyPI package 4 . The states and the summary statistics can be found in the Table  1 .
The proposed framework includes face detection followed by landmark detection followed by classification and/or segmentation, as shown in 
Images, as obtained from sources mentioned in section 3.1.4 consisted of various individuals. Thereby, to detect face masks in these images, the pipeline's first task involved face detection. CNN models have performed significantly well in face detection tasks. The first step in our approach was detecting faces, which was 4 https://pypi.org/project/instagram-scraper/ An example of output of the facial landmarks on a face is shown in fig. 4 .
Our goal with this network was to classify whether a given image of the jaw from section 3.2.1 contained a mask over it or not. The following architectures were used for classification -MobileNet, NasNetMobile, EfficientNet, and DenseNet since these have significantly fewer parameters than most other architectures 2.
The input image size for all the models was 224 x 224. Weight initialization of all models was done using ImageNet. All the models were truncated at the last fully connected layer. The following layers were added: (i) Average Pooling with 5 x 5 Pool size, (ii) Flatten Layer, (iii) Dense Layer with 128 hidden units and reLU activation (iv) Dropout layer of 0.5, and (v) Dense Layer of 2 hidden units and Softmax Activation. Adam optimizer with an initial learning rate of 1e-4 and Binary Cross-Entropy loss function was used (formula for the same is given in Equation 1 ) and each model was trained for 30 epochs with a batch size of 64.
Our goal with this network was to determine the fit of the mask on the appropriate region covering the nose and mouth 5 https://github.com/ternaus/retinaface 6 https://paperswithcode.com/paper/190500641 7 http://dlib.net/ 
= ∑ N i=1 −(y i log(y i ) + (1 − y i )log(1 − y i )) N (1) Precision = |T P| |T P ∪ FP| (2) Recall = |T P| |T P ∪ FN| (3) Accuracy = |T P ∪ T N| |T P ∪ T N ∪ FP ∪ FN| (4) IntersectionOverUnion(IOU) = |T P| |T P ∪ FP ∪ FN| (5) FitScore = |T P ∪ FP ∩ ROI| |ROI|(6)
The model deployed was a U-net [15] . with ResNet 32 and 50 encoder, pre-trained on ImageNet data [5] . The Layers 
This section presents the experimental results to evaluate patterns of people wearing masks in the six cities across the selected time frame, 1 st Feb 2020 and 31 st May 2020. 1.64 million faces and 0.23 million masks across the six cities were detected. Table 5 shows the City-wise distribution of the detected faces and masks. 57 %(out of 2 million collected posts) of the total posts had no faces while 93%(out of 2 million collected posts) of the total posts had no masks. One or more faces were detected in 43%(out of 2 million collected posts) of the posts, of which 30%(out of 2 million collected posts) had a single face detected, and 13%(out of 2 million collected posts) had multiple faces. While in 7%(out of 2 million collected posts), one or more masks were detected, out of which 6%(out of 2 million collected posts) of which had a single mask. It is evident from Fig. 5 that the number of posts drastically decreases as the number of faces and masks increases. To study how social distancing measures and lockdown affected online posting in the 6 cities, we plotted the percentage of group pictures against time, across all the states. The time period was divided into weeks and the group pictures detected over all the cities were added for a given week, and the percentage was calculated out of the total posts for that week. A group photo is described as a picture with at least two faces detected. As expected, Fig. 6 , indicates that as the pandemic grew and lockdown came into effect, group posting decreased significantly. We studied and compared the number of people wearing masks across the six cities using monthly percentages of masked faces. For each city, the data-set was divided into months, and the percentages of people wearing masks were computed. Fig. 7 shows the monthly percentages of maskwearers for each of the six cities. A general increasing trend is seen across all the cities.
This subsection analyzes how social media behavior changed when wearing masks guidelines were put in place by respective state Governments. For Boston Department of Public Health issued a two-week stay-at-home advisory on March 23 [1] following which a drop of 2% was seen in the percentage of group pictures. Residents were asked to wear a mask in public places starting May 6, in Boston. An increase of 3% was seen in the percentage of people wearing masks, in the collected posts for Boston. For Minneapolis, stay-at-home advisory came into effect on March 27 following which a drop of 1.6% was observed in percentage of group pictures. Mask wearing guidelines were put in place on April 30 for Minneapolis, encouraging people to wear face coverings. An increase of 7.4% was seen in the percentage of people wearing masks,on Instagram for Minneapolis. New Orleans observed a drop of 0.58% in number of group pics, post stay-at-home orders starting March 20, while Dallas saw a drop of 2.8 % in the number of group pics after March 23, after application of stay-at-home orders.Seattle observed a drop of 1.3 % after stay-at-home order on May 31.The dates when mask guidelines were put in place for New Orleans, Seattle and Dallas, are beyond our chosen time frame, hence were not considered for masked face percentage comparison. In New York State state-wide stay-athome order was declared on March 20, following which a drop of 1% was seen in percentage group pics. State residents were asked to wear masks in public starting April 15. Seattle observed a drop of 1.3 % after stay-at-home order on May 31. Fig. 9 shows the percentage of people wearing masks, before and after the date of mask-wearing guidelines for the respective states. We observe that there was a significant jump in the percentage in most of the states, showing adherence to the state policies. Fig. 8 shows the percentage of group pictures before and after stay-at-home policies were put in place for the respective states.
The killing of George Floyd on 25 th May sparked a series of protests [18] and agitation across the country. Protests are designed to stimulate public action for social justice. Such protests involve the physical gathering of people, making it difficult to adhere to social distancing and to wear masks. These protests provide an opportunity to observe how people reacted in terms of wearing masks and social distancing during a pandemic. To study these patterns on Instagram. We collected and used data for New York City and Minneapolis for the time period May 25 th -July 15 th . These two cities observed major protests [12] .
We curate a list of trending tags and keywords during this period, refer Table7 that related to the pandemic. We refer to the posts whose captions included these tags as BLM posts and the rest as Non-BLM posts. The classification and segmentation pipelines were applied to the selected posts for the two cities. Fig. 12 shows the difference in the percentage of group pics between BLM and non-BLM posts, which implies that a significant number of posts collected under the BLM tags had groups of people present. To get insight into how many people were wearing masks during the protests, we compare non-BLM group pics and BLM-group pics. Fig. 11 shows percentages of faces covered with masks in group pictures for Non-BLM, and BLM posts. There was a significant increase in the percentage of people detected with masks, showing that a high percentage of people were abiding to mask-wearing laws during the protests.
We employed the mask-fit frame-work from section 3 on the BLM posts to get an insight into how well people wore masks in groups during the protests. Fig. 10 shows the distribution of the fit scores, for people wearing masks in BLM-posts. Around only 45% of the detected faces had a fit score of ≥ 80%. This means that the remaining 55% had some major part of their nose-mouth region not covered.
This article presents two datasets, VAMA-C and VAMA-S, for classification and segmentation tasks respectively. We develop a novel face mask classifier to classify masked and unmasked faces with an accuracy of 98%. We further propose a mask-fit metric and a semantic segmentation based model with an accuracy of 98%. Using these models, we analyzed how people wore masks in images of six cities in the United States using Instagram posts. A general increasing trend in the percentage of people wearing masks is found as the pandemic grows. We also find that state policies proved useful in imposing mask-wearing and stay-at-home recommendations. A significant decrease in the posting of group pictures was found for every city, post stay-at-home guidelines were put in place. A significant increase in the percentage of mask-wearers is observed in the states, where state wise mask-wearing orders were applied. We further analyzed the George Floyd protests in New York City and Minneapolis through Instagram and found that there were significantly more group pictures for the protest-related posts than otherwise. 40% of people in protest-related posts were found to be wearing masks, which is much greater than what was observed on average for the unrelated posts for both the cities. On analyzing the mask-fit of people in protest-related posts, only 45% of the people wore masks with a fit score of >80% 
The authors did not receive support from any organization for the submitted work.
The authors declare that they have no conflict of interest.
The datasets, VAMA-C and VAMA-S, are available at http: //precog.iiitd.edu.in/resources.html. Instagram Images Analysis dataset (sec. 3.1.4) and Un-Masked Face dataset (sec. 3.1.2) are available from the corresponding author on reasonable request.
The code is available from the corresponding author on reasonable request.