The COVID-19 outbreak has redirected the efforts of the scientific community as a whole to help relieve the pressures on medical staff, create new effective treatments, limit the spread of the disease, and, most importantly, create a vaccine to end the pandemic. Our group has focused on the first issue, in trying to aide in the identification of COVID-19 afflicted individual from non-COVID-19 viral pneumonia or healthy individuals. Using unsupervised machine learning, or clustering techniques, we aim to create a method that can diagnose patients given a CT lung scan. This is an important, novel, and difficult issue because of the nature of clustering algorithms; these algorithms are extremely sensitive and require a great deal of fine-tuning in order to work properly. Clustering is an essential data analysis task where objects are placed into groups based on similarity of features. Popular supervised classification models would likely yield more accurate results, but implementing supervised learning requires labeled data in every application of a given method-a potential limiting factor for novel data [12, 20] . Creating an unsupervised model with high accuracy would be a significant accomplishment because of the greater range of real-world applicability the model would have. This is namely due to the fact that it can process unlabeled data and because of the objective nature of its solution, which simply deciphers data for what it is and not by reference to any labeling [4] . The work we have done focuses on using clustering as a means to identify patterns of pulmonary tissue sequelae in X-ray image visualizations via a 2-D representation of the data. This is an important stipulation as it provides the potential for direct application of our experiment to real world scenarios. Lastly, the proposed method can be used as an important first step in creating labeled data for supervised classification.
In this work, we utilize the new COVID-19 Radiography Database by Chowdhury et al. [11, 3] . The dataset is composed of X-ray and CT scan images separated into three classes: COVID-19 cases, Viral Pneumonia cases, and Normal lungs. We applied the K-Means++ (KM++) [1] and Robust Continuous Clustering (RCC) [13] algorithms to this database of X-ray images of patients labeled by cause of illness with the goal of exploring the clustering properties latent in the images. We conducted two tests using this data, the first was processing the dataset as it stands; while the second was creating a binary class dataset by combining both the Viral Pneumonia and Healthy lung classes into one non-COVID-19 class. The images were then isolated from their class labelling in processing and the classes were only used as a means to label the visualization/graphical representation. As KM++ was found to be one of the top 10 algorithms in unsupervised machine learning [8] , we wanted to test whether the claims of A. Shah and V. Koltun were true [13] and seen if RCC could compete with KM++ in terms of performance and clustering capabilities. Through our work, we have found that RCC has dramatically increased clustering performance in the COVID-19 database. We attribute this performance gain to the connectivity structure that the algorithm relies on [13] . It is able to identify the group of COVID-19 pneumonia cases with high accuracy, suggesting that there is information encoded in the image data set that differentiates cases of COVID-19 from other types of viral pneumonia. Figure 1 showcases lung scans of patients where the top row contains scans of those with COVID-19. The bottom row consists of non-COVID-19 cases, meaning they are patients with either normal lungs or viral pneumonia.
The database used is composed of many different image databases of recently developed COVID-19 and lung/CT scan databases of varying quality, resolution, and focuses [11] . The authors of the database complied the images to test supervised learning techniques and published their database on Kaggle to aid other researchers. Despite the varied origins and quality of the images, our findings are nonetheless interesting from an exploratory perspective as they illuminate that there are inherent differences between COVID-19 and non-COVID-19 lung/CT scans. This has no impact on the optimization of the RCC algorithm as it is not parameterized by the number of expected clusters. It does have an impact on our performance metrics.
Visualizing high dimensional data is a difficult challenge in many data science problems. Techniques like Principal Component Analysis (PCA) [10, 6] have been used for dimension reduction for quite some time. Recently, the method of mapping higher dimensional data to a lower dimensional representation that preserves neighbor structure in t-Distributed Stochastic Neighbor Embedding (t-SNE) was introduced by van der Maaten and Hinton [16] . The t-SNE method has the ability to exhibit global arrangement of data in the sense of clusters at a varying range of scales. This is extremely useful since high dimensional data, like image data, can be viewed in as few as 2 or 3 dimensions with similar images clustering in this lower dimensional space. More details and related work to t-SNE can be found in [5, 14, 17, 15] .
To evaluate clustering performance of our experiment, we utilize the Adjusted Mutual Information (AMI) [9, 18, 19] criterion. AMI is useful in this setting as we have access to the ground truth categorization of the data set. This criterion gives us an insight into the efficacy of our clustering routines while accounting for random chance. We will go into more details later.
The images were PCA reduced to 80 components and then normalized before being fed into the RCC routine. The RCC routine has a connectivity structure built with the Mutual k-Nearest Neighbors algorithm [2] , which we set to k=30.
We noticed that image normalization is very important to obtain convergence of the RCC algorithm and is done so in the empirical results of the original paper, although not explicitly mentioned.
We make use of four main data processing techniques in our experiment. These are Principal Component Analysis (PCA) for dimensionality reduction, t-Distributed Stochastic Neighbor Embedding (t-SNE) [16] for two-dimensional visualization, and K-Means++ (KM++) along with Robust Continuous Clustering (RCC) as clustering methods. Using these techniques, we are able to process the vector representations of our images in a meaningful manner.
t-SNE is a dimension reduction technique that creates a low dimension representation of high dimensional data points. This technique is primarily used for visualization in two or three dimensions as it preserves pairwise distances well while reducing dimension [16] . This is accomplished by computing the Euclidean distances between data points and constructing probability distributions in both high and low dimensions such that if two high dimensional data points are close together it is likely that their low dimensional representations are as well. We start by defining the probabilities as follows:
., x N } are the initial data points and Y = {y 1 , y 2 , ..., y N } are their low dimensional counterparts, paired by index, σ i which is computed using bisection search to best represent the distances of the data points. These conditional probabilities are then symmetrized for each pair of data points in order to obtain the pairwise similarity:
in the high dimension, and the same for the low dimensional conditional probability q. The algorithm then calls for minimization of the Kullback-Leibler divergence between the two pairwise probabilities:
using gradient descent. The gradient is as follows:
. This technique allows for visualization of high dimensional data that enables a visual and spacial intuition into the relationship of the data points and performance of clustering and classification algorithms.
RCC is a recently developed clustering technique that evolves a continuous representation of the input data set such that similar data points form tight clusters [13] . The objective function follows as:
..], x ∈ R D , and U is the set of representative points, which are initially set as X. Most importantly, ξ is the set of edges connecting data points to one another-this is constructed by mutual-K Nearest Neighbors (m-KNN). By m-KNN, in order for two or more points to be grouped together, both points must consider each other as the nearest point, thus the points have a "mutual neighbor." This allows RCC to have a parameter that defines the maximum number of neighbors to look for, highlighting another feature of RCC: the number of clusters does not need to be known ahead of time, setting it apart from KM++ greatly. The ρ term is used to represent estimator functions, in our case the Geman-McClure estimator is used with scale parameter µ. The term λ balances the contributions of terms to the objective. Below is another form of the RCC objective function, which takes into account the connections formed by m-KNN:
RCC can be optimized via alternating minimization, which sees the formula become:
while the optimal value of l p,q becomes:
We calculated the Adjusted Mutual Information Score (AMI) between the dataset's ground truth labeling and the labeling produced by our algorithm to evaluate clustering performance. Mutual Information (MI) is an entropy based measure which quantifies the amount of information given by a random variable in a particular clustering based on the probability of a particular point lying in any given cluster. The formulation of MI is given by:
where U i and V j are clusters in separate partitions of the same set of data ranging from {U 1 , ..., U R } and {V 1 , ..., V C } respectively. The probability of a data point lying in a given cluster is denoted by P U and P V , and the joint probability between partitions is labeled P U V . To obtain the AMI, the adjustment of an index for chance proposed by Hubert and Arabie [7] is then applied to the formulation of the MI:
Using this formula, a number between 0 and 1 is obtained, which determines the algorithm's effectiveness is displayed. The closer the number is to 1, the more effective that algorithm is at clustering the given dataset.
It is clear that the COVID-19 data points naturally cluster with our experiment. This was initially observed in the t-SNE reduction of the true labeling of the dataset in Figure 2 (left) where COVID-19 points are colored red. The set of COVID-19 data points are similar enough to each other to have been grouped together, and different enough from the other cases to have been remapped to the exterior of the set of all considered data points. This suggests that there is enough easily accessible information to cluster our data without complex feature engineering. So the first cluster analysis technique to try is KM++, a robust technique that affords an established baseline. The results from KM++, which are discussed in detail below, produced a rather unsuccessful clustering. Next up is RCC, which produced very intriguing results. It was able to successfully identify the COVID-19 cases as separate from the other cases, yet was unable to separate the viral pneumonia from the healthy cases in any significant way. We suspect that this behavior is due to the similarity of COVID-19 images and the connectivity structure that RCC uses. Nonetheless, the result is notable and quite useful in its ability to discern between COVID-19 and non-COVID-19. This is significant as KM++ was unable to discriminate between any of the cases. Now the AMI score for each of the clustering algorithms shows that RCC produces a significantly better clustering in this case. In order to interpret the data, we define a mapping from cluster labels to true labels. This map is defined in a manner similar to cluster purity, where labels are assigned to clusters by majority rule. Using such a mapping enables us to see if like elements cluster; while also not penalizing for increased cluster count. This mapping allows us to consider the confusion matrix of the clustering results. In the tables above, the confusion matrices for both Figure 2 and 3 are shown. For Table 1 , you can see that the sensitivity is 92.8%, while the specificity is 98.1%; as for T able 2, here are the sensitivity and specificity for each category (COVID-19, Viral Pneumonia, Healthy) are 92.8% and 51%, 68.75% and 96.8%, and 49.75% and 93.5%, respectively. This displays that RCC is more successful in two class situations, rather than three class situation. This fact can be seen in the three class case's sensitivity and specificity being 70.43% and 80.43%, a bit less favorable performance than the two class case. Figures 2 and 3 show the clustering of the COVID-19 dataset by KM++ (center) and RCC (right). Figure 2 shows the dataset and KM++ prediction in 3 clusters (Viral Pneumonia, Normal, or COVID-19), while Figure 3 shows the true dataset and KM++ prediction using 2 clusters (non-COVID19 or COVID-19); both figures show the same clustering prediction by RCC. In both cases, the performance of KM++ is left wanting in this dataset as the AMI for each figure falls below 0.1. On the other hand, RCC performs remarkably well in both cases, where it scores 0.5044 and 0.267 respectively. It can be seen that RCC does not separate data into three or even two distinct classes, but it is able to distinguish COVID-19 cases from other cases quite well. As seen in both figures, it is apparent that the "red" cluster captures most of the true COVID-19, while the "purple" cluster captures much of the non-COVID-19 cases, whether or not they were grouped as such initially. Another thing of note is that, other clusters predicted by RCC are able to pick out some COVID-19 cases within the larger purple cluster as well, making note of these outliers in a way KM++ does not.
The results from our experiment showed that the COVID-19 data points in the data set are clearly different from the non-COVID-19 cases. Even though RCC does not separate the COVID-19 cases and non-COVID-19 cases into two distinct classes, it is still able to create clusters that distinguish both cases. Thus, revealing that there is latent information within COVID-19 images and there is an underlying similarity between many COVID-19 cases.
RCC likely performs well because it uses a more refined measure of pairwise similarity than simply 2 distance. The connectivity structure constructed by m-KNN places a stricter requirement on determining similarities between data points, enabling the algorithm to perform more informed data clustering.
Potential limitations of our work come from the relative ambiguity as to the reasoning for classification and the, overall, small sample of COVID-19 cases in the data which may not be representative of all real COVID-19 cases. It can be inferred that the COVID-19 cases present in the dataset are likely severe as they warranted a professional lung scan, meaning that our findings may be limited to only the more severe COVID-19 cases. Considering this possibility, our findings are nonetheless relevant as they still display an inherent similarity between the COVID-19 cases in the data and distinguishes them from other viral pneumonia. In order to combat this potential limitation, future implementations of these techniques can be performed on cleaner, more rigorously constructed datasets.
One direction for future work is to explore the effectiveness of different feature engineering techniques on this data set. Consideration of other dimensionality reduction techniques, other than PCA, that may be more finely tuned to the COVID-19 dataset is a worthwhile future route. Moreover, as new data emerges, we can further refine our experimental methodology.