An AI-Assisted Labeling Tool for Cataloging High-Resolution Images of Galaxies
Abstract
The Hubble Space Telescope (HST), the recently launched James Web Space
Telescope (JWST), and many earth-based observatories collect data allowing astronomers to answer fundamental questions about the Universe.
In this work we focus on an ecosystem of AI tools for cataloging bright sources within galaxies, and use them to analyze young star clusters – groups of stars held together by their gravitational fields.
Their age, mass among other properties provide insights into the process of star formation and the birth and evolution of galaxies.
Significant domain expertise and resources are required to discriminate star clusters among tens of thousands of sources that may be extracted for each galaxy.
To accelerate this step we propose: 1) a web-based annotation tool to label and visualize highresolution astronomy data, encouraging efficient labeling and consensus building; and 2) techniques to reduce the annotation cost by leveraging recent advances in unsupervised representation learning on images.
We present case studies where we work with astronomy researchers to validate the annotation tool and find that the proposed tools can reduce the annotation effort by 3 × on existing HST catalogs, while facilitating accelerated analysis of new data.
Introduction
The origin of the universe, the existence of extraterrestrial life, and numerous other questions motivate the study of the cosmos.
More than three decades of data from the Hubble Space Telescope (HST), and the newly launched James Webb Space Telescope (JWST), are expected to provide fundamental insights into many of these questions.
One such question is the process of birth, spatial and temporal evolution, and chemical properties of star clusters within galaxies.
Star clusters are groups of stars, typically consisting of thousands to millions of stars, held together by their mutual gravitational attraction.
Since the majority of star formation in a galaxy occurs in star clusters [30], the distribution of their ages, masses, and other properties provide insights into the formation, evolution, and appearance of galaxies [30, 7, 37, 3].
The HST has produced high-resolution images of many nearby galaxies, and the data is of sufficient quality to resolve these galaxies into their individual sources.
Astronomers often collaborate in large teams to produce a survey or a catalog documenting individual sources within these images.
For instance, The HST Legacy ExtraGalactic UV Survey (LEGUS) [11] consists of more thanhuman-labeled stars and star clusters, the Galaxy Zoo [36] contains around 1.5 million galaxies,
[Figure]
while the USNO-B Catalog [39] comprises one billion cataloged objects. Yet, producing catalogs can be a challenging and time-consuming task – a nearby galaxy might contain ten of thousands of bright sources and classifying these requires considerable domain expertise. LEGUS represents about 9000 person-hours of estimated manual annotation effort, which is hard to replicate as finer-grained categorization will be enabled as new measurements become available, e.g., from
Our work focuses on constructing star-cluster catalogs, though the tools are general and can be applied to construct other surveys.
The process typically involves the identification of sufficiently bright sources within the galaxy using a tool such as the Source Extractor [8], followed by heuristics to remove sources that are unlikely to be clusters based on their brightness profile (to a rough approximation clusters are more spatially extended than individual stars).
Subsequently, these are manually inspected to identify the star clusters.
The process requires careful inspection of the spectral and spatial properties of their brightness distributions of several tens of thousands of sources extracted in the previous step.
The task is so challenging and cumbersome that the existing LEGUS star-cluster catalogs have about a ≈ 87% consensus among humans for the binary (cluster/non-cluster) classification [45, 35].
To address these challenges we propose an ecosystem of AI tools along two major thrusts (§ 3).
First is a web-based annotation tool allowing visualization of high-resolution images and supporting various forms of annotations (See Figure 1–bottom).This serves as a platform for collaborative labeling, visualization of existing labels, and building consensus for labels themselves.
Currently, no tools exist to support this due to the challenges for data visualization and labeling needs unique to the community (e.g., current systems are desktop-based and are not amenable to collaborative labeling efforts).
Second, are techniques to produce catalogs with less labeling effort by utilizing techniques for unsupervised representation learning in visual data (See Figure 1–top).
A unique aspect of this problem is that astronomy data is heterogeneous, where the spatial resolution, wavelength coverage and number of filters vary from task to task.
Frequently, there are no existing networks that transfer well to these domains, making learn from little data challenging.
We find that domain-specific representations trained using contrastive learning are effective, which also allows us to integrate new sensory data as they arrive without labeling effort.
We pilot the proposed tool in two case studies.
First, we show that existing catalogs in LEGUS can be constructed with a near-human agreement with only 32% of the labeling effort (§ 4).
Second, we collaborate with astronomy researchers to construct new catalogs involving new modalities to understand and improve the entire workflow from data acquisition, to initial labeling, AI-assisted catalog construction, to enabling basic science (§ 5).
Related work
Astronomy image analysis Computer vision tools have been applied to numerous astronomy applications, including galaxy classification [18, 28, 6], galaxy morphology [16, 53], and galaxy mergers [1, 65].
More recently, machine learning has been applied to the classification of young star clusters [21, 56, 9, 10, 45].
In [9, 10], the authors select a subset of channels based on domain knowledge to classify star cluster images using transfer learning [59] from pretrained networks on color images.
Later, [56] incorporates all the channels from the multi-spectral data to classify star clusters.
The authors use an ensemble [32] of pretrained networks in parallel and split the image into subsets of three channels.
More recently, [45] trains a multi-scale CNN from scratch using all channels in the star cluster images.
The success of these methods is underscored by the availability of significant training data, while our work focuses on learning with limited data.
Labeling and visualization tools Tools for labeling and visualizing training data are indispensable for training and validating modern AI systems.
While a large number of tools existing for labeling images or video (e.g.,[51, 50, 19, 49]), similar tools are lacking for Astronomy imagery.
They are similar to satellite imagery data (e.g., they are multi-spectral and of high resolution) or medical images and tools in these domains can be repurposed for our goals.
For instance, in [27] and [47], the authors propose web-based labeling tools with dynamic tiling to handle high-resolution remote sensing data.
In medicine, multiple labeling interfaces have been proposed [60, 44] for 3D medical data.
Our web-based labeling tool for high-resolution galaxy images is based on [47], allowing dynamic tiling for high-resolution imagery such as HST and JWST images, and supports tools for point, bounding box, and segmentation annotations.
Unsupervised representation learning The labeled data bottleneck can also be approached from the perspective of learning useful representations from few labeled samples using semi-supervised learning [33, 31, 62, 52, 46] methods, or with only unlabeled images using self-supervised learning [17, 55, 40, 61, 20, 42, 26, 63, 5].
In particular, contrastive learning [14, 25, 13, 22, 64] have performed particularly well in learning features from unlabeled data that are useful for downstream tasks and have been applied to other domains such as medical imaging [12, 58], remote sensing [34, 4, 57], and astronomy [23, 38].
An AI-assisted Labeling Tool for Building Catalogues of Galaxies
Thrust 1: Labeling interface Our labeling interface is based on on Microsoft’s AI4Earth satellite imagery labeling tool [48], an open-source web-based tool that supports dynamic tiling for highresolution remote sensing imagery (see Figure 1).
We modified and added new features based on the goals of astronomy researchers.
The tool supports different renditions which are helpful for annotating objects (e.g., ultra-violet, optical, and infrared bands).
These renditions are saved as a cloud-optimized GeoTIFF (COG) and hosted on a web server.
The tool supports dynamic tiling, allowing handling of high-resolution images.
Labels are managed using GeoJSON files, so previously created catalogs (e.g., uncurated catalogs generated with SExtractor) can be imported and modified to later export them as a new GeoJSON file.
The web based interface allows distributed labeling and visualization on browsers without the need to install custom software packages.
The labeling interface can be customized to work with any high-resolution image, and to annotate different types of astronomical sources, as it allows the user to quickly mark objects by their center coordinates, bounding box, or select irregular regions.
Furthermore, the implementation only requires a standard web server to host the data renditions and a tiling application for dynamically rendering the high-resolution images.
Thrust 2: Learning with limited data Our framework supports different aspects of learning with astronomy data:
• Deep networks, pre-training and transfer Our framework supports various forms of pre-trained models (e
g., ImageNet [15] pretrained) and self-supervised contrastive learning customized for astronomy data.
One challenge is that astronomy data is of a different shape than color images (e
g., there are more channels) and some modification of the network is necessary (e
g., splitting the channels into groups of three or an adaptor [43]).
Pre-training might improve generalization when a few labels are available for the initial training.
See Appendix C for how contrastive learning from natural images can be adapted to astronomy data.
• Active and semi-supervised learning Active and semi-supervised learning can be used to efficiently label the unlabeled samples.
For active learning we explore the effectiveness of entropy sampling (i
e., sampling points that maximize the entropy of the predicted probabilities) over random sampling, while for semi-supervised learning we explore a label propagation method based on pseudo-labels [33].
Case Study 1: Building Star Cluster Catalogs with LEGUS
The HST Treasury Program Legacy ExtraGalactic UV Survey (LEGUS)
[11] consist of 50 galaxies spanning distances from 3.5 Mpc and ≈ 16 Mpc using HST imagery in five bands:
NUV (F275W filter), U (F336W filter), B (either F438W or F435W filter), V (either F555W or F606W filter) and I (F814W filter).
When building the star cluster catalogs for each galaxy, the raw images are initially preprocessed and aligned.
Then, a preliminary set of detections is generated with SExtractor [8] using a set of basic selection functions to remove the majority of stars and artifacts.
A set of distribution fits are applied to rule-out additional non-cluster objects before a final visual inspection on objects brighter than V= − 6 mag 1 by at least three human annotators.
The total number of objects in LEGUS catalog is 15471, and each one is classified into four classes based on morphology [2] (See Figure 2):
• Class 1: compact objects with symmetric density profile.
• Class 2: compact objects with asymmetric or elongated density profile.
• Class 3: multi-peak associations with a shared diffuse underlying emission.
• Class 4: foreground stars, background galaxies, artifacts, and any other spurious detection.
Cluster/non-cluster variant For most applications, a binary classification of the sources to a cluster/non-cluster is sufficient.
From LEGUS, classes 1 and 2 are combined into the cluster class, and classes 3 and 4 are combined into the non-cluster class.
LEGUS contains a total of 5573 (36%) cluster sources and 9898 (64%) non-cluster sources.
We use the binary classification scheme in our case studies (§ 4–5).
[Figure]
Human agreement Each source was labeled with a number of annotators.
We calculate human agreement as the fraction of annotations that agree with the final classification (mode) from all the annotators.
The agreement is 75 .
0% for four-class task and 87 .
2% for binary task.
4.1 Problem formulation
Initially a small labeled dataset D L =
( x i , y i ) N L
i =1 and a unlabeled dataset D U
=
( x i ) N U i =1 where
N U ≫ N L are available, which can inform choices for pre-training, active learning, and selfsupervised learning.
We also assume a separate validation set D V =
( x i , y i ) N V
i
=1 , such that D V ∩
D U ∩ D L = ∅ .
We use the training-validation split proposed in [45], which consists of ≈ 11000 and ≈ 1200 training and validation objects respectively, and report the mean result for each experiment after four runs with four different seeds.
See training hyper-parameters in Appendix B.
4.2 Results
Role of representation learning.
Training a deep network for star cluster classification that matches human agreement ( ≈ 85%) requires 5000 labeled samples, amounting to 32% of the entire LEGUS catalog.
Thus indicating that the labeling effort can be reduced by 3 × on the LEGUS dataset using our proposed approach.
All baselines perform similarly down to 500 labeled images, as shown in
Figure 3.
On the other hand, for very low data regimes (less than 100 labeled images) unsupervised learning with contrastive pre-training (using BYOL [22]) leads to significantly higher performance initially ( ≈ 75% with 20 labeled samples) compared to training a network from scratch or transfer learning from an ImageNet pretrained ResNet50 [24] (62-66% which is close to chance performance).
On this dataset the performance of the linear classifier is comparable to fine-tuning the entire network (light green).
Training the linear classifier is significantly faster and can be done interactively with little computational resources within the labeling tool.
Applying semi-supervised learning on top of self-supervision increases performance slightly, as shown in Figure 3(black).
On the active learning side, sampling based on entropy of the predictions did not improve accuracy compared to random sampling for any number of labeled images.
Case Study 2: Building Catalogs of Young Star Clusters from IR data
Stars are born inside dense dust and gas concentrations known as molecular clouds.
When these young and massive stars ionize the surrounding gas clouds with high-energy UV radiation, ionized hydrogen (HII) regions are formed.
These very young regions produce a sufficient amount of ionizing radiation that their high-redshift counterparts could be responsible for the reionization at redshifts 6- (age of the Universe: 0.5-1 Gyr), the last phase transition our Universe underwent.
For this to be the case, these regions need to emerge quickly from their dense and opaque natal clouds, so that enough
[Figure] [Figure]
V a li d a t i o n a cc u r a c y
[Figure]
massive stars are still alive to contribute to the reionization.
The timescale for this ‘emergence’ remain highly debated, and is one of the questions the JWST is posed to address.
Because these very young star clusters reside in very dense regions, optical light cannot emerge and they need to be investigated using infrared (IR) light.
Figure 4(a) provides and example, from HST imaging, of a young star cluster surrounded by its HII region.
In preparation for the upcoming JWST images, we design a pilot study to classify young star clusters in HII regions using the near-IR filters from the HST 2 .
Our pilot consists of five rounds of annotations with 3-4 human annotators on each round.
We perform the study on galaxy NGC4449 due to its high rate of star formation and the presence of a population of massive and young star clusters.
For this case study, we use a preliminary catalog of 1119 sources generated with SExtractor on galaxy
NGC4449, similar to the procedure in Section 4.
These initial sources generally focus on the peaks in the emission line image 3 and are marked with yellow rings in the labeling interface, as shown in Figure 4(c).
In addition to the near-UV and optical bands used in LEGUS (i.e., F275W, F336W,
F435W, F555W, F657N, and F814W), we include near-infrared channels to our image data: F1100W,
F1280W, and F1600W.
5.1 Labeling interface for young star clusters in near-IR images
We customize the tool to support emission lines and near IR renditions to the visualization (See
Figure 4(a)).
Also, we added a marker to sources that are also found in LEGUS catalogs; given that
N ea r-
i n f r a r ed
E m i ss i on li ne s
U V + op t i c a l
[Figure]
(c)
N ea r-
i n f r a r ed
E m i ss i on li ne s
LEGUS clusters are already visually inspected (sources with blue circle markers in Figure 4(c)).
Finally, we display specific information about each source, including the prediction score and the class the human annotators gave in the previous round with a majority voting consensus (See Figure 1).
We sample and mark the sources to be annotated in the following rounds to measure the agreement between human annotators (sources marked with a large white circle marker in Figure 4(c)).
These sources are sampled randomly from high-informative areas selected manually by a user (i.e., from high HII concentration regions as shown in Figure 4(b)).
5.2 Results
Contrastive learning for near-IR images We use the entire set of 1119 sources obtained with
SExtractor to pre-train the network using contrastive learning.
On the other hand, we annotatefrom the 1119 sources where 398 (60.5%) sources were labeled as clusters and 260 (39.5%) as non-clusters and split the labeled samples into 90% for training and 10% for validation.
Finally, using the same problem formulation as in Section 4.1, the same hyperparameters from Appendix B, and following the evaluation procedure described in Appendix C, we get a classification accuracy of 82.7%, demonstrating the generalizability of our approach to additional modalities and different image resolutions.
Human agreement Following the procedure in Section 4, we calculate a human-to-mode agreement of 93.3% averaged over the five rounds.
Conclusion
The analysis of star clusters has proven critical in answering scientific questions regarding stellar evolution and how galaxies are born.
In this work we collaborate with astronomy researchers to propose an ecosystem of AI tools for cataloging bright sources in high-resolution images of nearby galaxies, reducing to 32% the labeling effort on the star cluster classification task with the LEGUS dataset.
In addition, we demonstrate the robustness of our approach to new modalities and different image resolutions anticipating the upcoming images from the JWST.
A Color renditions for astronomical data
Astronomical multi-spectral images are often combined to form color renditions.
For visualization purposes, in this work we merge the two reddest channels into a single red channel and the two bluest channels as a single blue channel as described below:
• R =
( γ v V + γ i I ) /•
G = ( γ b B ) • B
= ( γ n NUV + γ u U ) /
where γ is the inverse gain of the filters used by the HST.
B Supervised and sel-supervised training hyper-parameters
Contrastive learning pretraining We use BYOL PyTorch [41] implementation by [54].
We pretrain our network with the adapted version of BYOL for 170,000 iterations ( ≈ 1000 epochs) with a batch size of 128 samples and a learning rate of 3E-4.
We train for approximately 7 hours using a single GPU 1080ti .
Supervised training We use the CNN optimized for star cluster classification proposed by [45].
We train for ten epochs and a batch size of 64 images using Adam optimizer [29].
We use a learning rate of η =1E-04 with a learning schedule η ′ =
η ∗ 0 .
1 in the 6 th epoch.
C BYOL adaptation for multi-spectral images and ablation
For our self-supervised learning experiments, we use an adapted version of BYOL [22] for multi-spectral star cluster data, which achieves good performance without needing negative pairs.
In addition, BYOL is more robust to lower batch sizes, which allows training models with good performance using a single GPU.
Off-the-shelf BYOL (with the backbone input adapted to work with multi-spectral star cluster images) does not converge during training.
This is not surprising since BYOL was designed with ResNet encoders for color images.
Our approach uses an encoder that works on multi-spectral images.
Empirically, we find that the projector’s size affects the network’s training considerably in our approach (See Table C.2a).
This is expected since the size of our CNN’s output features f θ ( v ) is 256 (compared to 2048 of
ResNet).
Using the default projector size of BYOL with our encoder and input data yields poor performance.
Therefore, we reduce the projector hidden size while still emulating the dimensional increase from f θ ( v ) to the projector hidden size (See second row in Table C.2a).
This configuration improves over the default settings, but the performance is still low.
We get the best results using a projector size of (256-256).
See Table C.2a for more configurations.
Contrastive learning methods are sensitive to the choice of image augmentations [13, 22], and star cluster data is very different from color images like ImageNet.
We apply modifications to the data augmentations that are also key for the model to converge during training.
First, we limit the lower-bound scale of the random resize crop augmentation (See Table C.2c).
The scale specifies the lower and upper bounds for the random area of the crop before resizing.
Astronomical object images contain the object of interest centered in the image with multiple other objects in the close surroundings.
For that reason, we hypothesize that small-scale values might result in two different centered objects signaled as a positive pair.
Thus, confusing the model.
For the BYOL ablation experiments, we divide the LEGUS dataset into a 90-10% training-validation split.
From the training set, we select 100 samples as our ‘labeled dataset’, the remaining samples as the ‘unlabeled dataset’, and evaluate over the complete validation set to compare performances between models.
We follow the experimental setup as the main experiments in Figure 3.
First, we pretrain a network from scratch with the ‘unlabeled dataset’ using BYOL.
Then, we finetune the model with the ‘labeled dataset’.
After the training is complete, we evaluate the validation set.
[Table]