An AI-Assisted Labeling Tool for Cataloging High-Resolution Images of Galaxies
Abstract
The Hubble Space Telescope (HST), the recently launched James Web Space
Telescope (JWST), and many earth-based observatories collect data allowing astronomers to answer fundamental questions about the Universe.
In this work wefocus on an ecosystem of AI tools for cataloging bright sources within galaxies,and use them to analyze young star clusters – groups of stars held together by theirgravitational fields.
Their age, mass among other properties provide insights intothe process of star formation and the birth and evolution of galaxies.
Significantdomain expertise and resources are required to discriminate star clusters amongtens of thousands of sources that may be extracted for each galaxy.
To acceleratethis step we propose: 1) a web-based annotation tool to label and visualize highresolution astronomy data, encouraging efficient labeling and consensus building;and 2) techniques to reduce the annotation cost by leveraging recent advances inunsupervised representation learning on images.
We present case studies where wework with astronomy researchers to validate the annotation tool and find that theproposed tools can reduce the annotation effort by 3 × on existing HST catalogs,while facilitating accelerated analysis of new data.
Introduction
The origin of the universe, the existence of extraterrestrial life, and numerous other questions motivatethe study of the cosmos.
More than three decades of data from the Hubble Space Telescope (HST),and the newly launched James Webb Space Telescope (JWST) are expected to provide fundamentalinsights into many of these questions.
One such question is the process of birth, spatial and temporalevolution, and chemical properties of star clusters within galaxies.
Star clusters are groups of stars,typically consisting of thousands to millions of stars, held together by their mutual gravitationalattraction.
Since the majority of star formation in a galaxy occurs in star clusters [30], the distributionof their ages, masses and other properties provide insights into the formation, evolution, andappearance of galaxies [30, 7, 36, 3].
The HST has produced high-resolution images of many nearby galaxies and the data is of sufficientquality to resolve these galaxies into their individual sources.
Astronomers often collaborate in largeteams to produce a survey or a catalog documenting individual sources within these images.
Forinstance, The HST Legacy ExtraGalactic UV Survey (LEGUS) [11] consists of more than 15000human-labeled stars and star clusters while the USNO-B Catalog [38] comprises one billion catalogedobjects.
Yet, producing catalogs can be a challenging and time-consuming task – a nearby galaxy might contain ten of thousands of bright sources and classifying these requires considerable domain
[Figure]
effort enabled new become
Our work focuses on constructing star-cluster catalogs, though the tools are general and can beapplied to construct other surveys.
The process typically involves the identification of sufficientlybright sources within the galaxy using a tool such as the Source Extractor [8], followed by heuristicsto remove sources that are unlikely to be clusters based on their brightness profile (to a roughapproximation clusters are more spatially extended than individual stars).
Subsequently, these aremanually inspected to identify the star clusters.
The process requires careful inspection of thespectral and spatial properties of their brightness distributions of several tens of thousands of sourcesextracted in the previous step.
The task is so challenging and cumbersome that the existing LEGUSstar-cluster catalogs have about a ≈ 87% consensus among humans for the binary (cluster/non-cluster)classification [44, 35].
To address these challenges we propose an ecosystem of AI tools along two major thrusts (§ 3).
First is a web-based annotation tool allowing visualization of high-resolution images and supportingvarious forms of annotations (See Figure 1–bottom).This serves as a platform for collaborativelabeling, visualization of existing labels, and building consensus for labels themselves.
Currently, notools exist to support this due to the challenges for data visualization and labeling needs unique to thecommunity (e.g., current systems are desktop-based and are not amenable to collaborative labeling efforts).
Second, are techniques to produce catalogs with less labeling effort by utilizing techniquesfor unsupervised representation learning in visual data (See Figure 1–top).
A unique aspect of thisproblem is that astronomy data is heterogeneous, where the spatial resolution, wavelength coverageand number of filters vary from task to task.
Frequently, there are no existing networks that transferwell to these domains, making learn from little data challenging.
We find that domain-specificrepresentations trained using contrastive learning are effective, which also allows us to integrate newsensory data as they arrive without labeling effort.
We pilot the proposed tool in two case studies.
First, we show that existing catalogs in LEGUS canbe constructed with a near-human agreement with only 32% of the labeling effort (§ 4).
Second,we collaborate with astronomy researchers to construct new catalogs involving new modalities tounderstand and improve the entire workflow from data acquisition, to initial labeling, AI-assistedcatalog construction, to enabling basic science (§ 5).
Related work
Astronomy image analysis Computer vision tools have been applied to numerous astronomyapplications, including galaxy classification [18, 28, 6], galaxy morphology [16, 52], and galaxymergers [1, 64].
More recently, machine learning has been applied to the classification of youngstar clusters [21, 55, 9, 10, 44].
In [9, 10], the authors select a subset of channels based on domainknowledge to classify star cluster images using transfer learning [58] from pretrained networks oncolor images.
Later, [55] incorporates all the channels from the multi-spectral data to classify starclusters.
The authors use an ensemble [32] of pretrained networks in parallel and split the imageinto subsets of three channels.
More recently, [44] trains a multi-scale CNN from scratch using allchannels in the star cluster images.
The success of these methods is underscored by the availabilityof significant training data, while our work focuses on learning with limited data.
Labeling and visualization tools Tools for labeling and visualizing training data are indispensablefor training and validating modern AI systems.
While a large number of tools existing for labelingimages or video (e.g.,[50, 49, 19, 48]), similar tools are lacking for Astronomy imagery.
They aresimilar to satellite imagery data (e.g., they are multi-spectral and of high resolution) or medicalimages and tools in these domains can be repurposed for our goals.
For instance, in [27] and [46],the authors propose web-based labeling tools with dynamic tiling to handle high-resolution remotesensing data.
In medicine, multiple labeling interfaces have been proposed [59, 43] for 3D medicaldata.
Our web-based labeling tool for high-resolution galaxy images is based on [46], allowingdynamic tiling for high-resolution imagery such as HST and JWST images, and supports tools forpoint, bounding box, and segmentation annotations.
Unsupervised representation learning The labeled data bottleneck can also be approached fromthe perspective of learning useful representations from few labeled samples using semi-supervisedlearning [33, 31, 61, 51, 45] methods, or with only unlabeled images using self-supervised learning [17, 54, 39, 60, 20, 41, 26, 62, 5].
In particular, contrastive learning [14, 25, 13, 22, 63] have performed particularly well in learning features from unlabeled data that are useful for downstream tasksand have been applied to other domains such as medical imaging [12, 57], remote sensing [34, 4, 56],and astronomy [23, 37].
An AI-assisted Labeling Tool for Building Catalogues of Galaxies
Thrust 1: Labeling interface Our labeling interface is based on on Microsoft’s AI4Earth satelliteimagery labeling tool [47], an open-source web-based tool that supports dynamic tiling for highresolution remote sensing imagery (see Figure 1).
We modified and added new features based onthe goals of astronomy researchers.
The tool supports different renditions which are helpful forannotating objects (e.g., ultra-violet, optical, and infrared bands).
These renditions are saved as acloud-optimized GeoTIFF (COG) and hosted on a web server.
The tool supports dynamic tiling, allowing handling of high-resolution images.
Labels are managed using GeoJSON files, so previouslycreated catalogs (e.g., uncurated catalogs generated with SExtractor) can be imported and modifiedto later export them as a new GeoJSON file.
The web based interface allows distributed labeling andvisualization on browsers without the need to install custom software packages.
The labeling interface can be customized to work with any high-resolution image, and to annotatedifferent types of astronomical sources, as it allows the user to quickly mark objects by their centercoordinates, bounding box, or select irregular regions.
Furthermore, the implementation only requiresa standard web server to host the data renditions and a tiling application for dynamically renderingthe high-resolution images.
Thrust 2: Learning with limited data Our framework supports different aspects of learning withastronomy data:
• Deep networks, pre-training and transfer Our framework supports various forms of pre-trainedmodels (e
g., ImageNet [15] pretrained) and self-supervised contrastive learning customized for
astronomy data.
One challenge is that astronomy data is of a different shape than color images(e.g.
, there are more channels) and some modification of the network is necessary (e
g., splitting the
channels into groups of three or an adaptor [42]).
Pre-training might improve generalization when
a few labels are available for the initial training.
See Appendix C for how contrastive learning from
natural images can be adapted to astronomy data.
• Active and semi-supervised learning Active and semi-supervised learning can be used to effi-
ciently label the unlabeled samples.
For active learning we explore the effectiveness of entropy
sampling (i
e., sampling points that maximize the entropy of the predicted probabilities) overrandom sampling, while for semi-supervised learning we explore a label propagation method based
on pseudo-labels [33].
Case Study 1: Building Star Cluster Catalogs with LEGUS
The HST Treasury Program Legacy ExtraGalactic UV Survey (LEGUS)
[11] consist of 50 galaxiesspanning distances from 3.5 Mpc and ≈ 16 Mpc using HST imagery in five bands:
NUV
(F275Wfilter), U (F336W filter), B (either F438W or F435W filter), V (either F555W or F606W filter) and I(F814W filter).
When building the star cluster catalogs for each galaxy, the raw images are initiallypreprocessed and aligned.
Then, a preliminary set of detections is generated with SExtractor [8] usinga set of basic selection functions to remove the majority of stars and artifacts.
A set of distributionfits are applied to rule-out additional non-cluster objects before a final visual inspection on objectsbrighter than V= − 6 mag 1 by at least three human annotators.
The total number of objects in LEGUScatalog is 15471, and each one is classified into four classes based on morphology [2] (See Figure 2):
• Class 1: compact objects with symmetric density profile.
• Class 2: compact objects with asymmetric or elongated density profile.
• Class 3: multi-peak associations with a shared diffuse underlying emission.
• Class 4: foreground stars, background galaxies, artifacts, and any other spurious detection.
Cluster/non-cluster variant For most applications, a binary classification of the sources to acluster/non-cluster is sufficient.
From LEGUS, classes 1 and 2 are combined into the cluster class,and classes 3 and 4 are combined into the non-cluster class.
LEGUS contains a total of 5573 (36%)cluster sources and 9898 (64%) non-cluster sources.
We use the binary classification scheme in ourcase studies (§ 4–5).
[Figure]
Human agreement Each source was labeled with a number of annotators.
We calculate humanagreement as the fraction of annotations that agree with the final classification (mode) from all theannotators.
The agreement is 75 .
0% for four-class task and 87 .
2% for binary task.
4.1 Problem formulation
Initially a small labeled dataset D L =
( x i , y i ) N L
i =1 and a unlabeled dataset D U
=
( x i ) N U i =1 where
N U ≫ N L are available, which can inform choices for pre-training, active learning, and selfsupervised learning.
We also assume a separate validation set D V =
( x i , y i ) N V
i
=1 , such that D V ∩
D U ∩ D L = ∅ .
We use the training-validation split proposed in [44], which consists of ≈ 11000 and≈ 1200 training and validation objects respectively, and report the mean result for each experimentafter four runs with four different seeds.
See training hyper-parameters in Appendix B.
4.2 Results
Role of representation learning.
Training a deep network for star cluster classification that matcheshuman agreement ( ≈ 85%) requires 5000 labeled samples, amounting to 32% of the entire LEGUScatalog.
Thus indicating that the labeling effort can be reduced by 3 × on the LEGUS dataset usingour proposed approach.
All baselines perform similarly down to 500 labeled images, as shown in
Figure 3.
On the other hand, for very low data regimes (less than 100 labeled images) unsupervisedlearning with contrastive pre-training (using BYOL [22]) leads to significantly higher performanceinitially ( ≈ 75% with 20 labeled samples) compared to training a network from scratch or transferlearning from an ImageNet pretrained ResNet50 [24] (62-66% which is close to chance performance).
On this dataset the performance of the linear classifier is comparable to fine-tuning the entire network(light green).
Training the linear classifier is significantly faster and can be done interactively withlittle computational resources within the labeling tool.
Applying semi-supervised learning on top of self-supervision increases performance slightly, asshown in Figure 3(black).
On the active learning side, sampling based on entropy of the predictionsdid not improve accuracy compared to random sampling for any number of labeled images.
Case Study 2: Building Catalogs of Young Star Clusters from IR data
Stars are born inside dense dust and gas concentrations known as molecular clouds.
When theseyoung and massive stars ionize the surrounding gas clouds with high-energy UV radiation, ionizedhydrogen (HII) regions are formed.
These very young regions produce a sufficient amount of ionizingradiation that their high-redshift counterparts could be responsible for the reionization at redshifts 6-9(age of the Universe: 0.5-1 Gyr), the last phase transition our Universe underwent.
For this to be thecase, these regions need to emerge quickly from their dense and opaque natal clouds, so that enough
[Figure] [Figure]
V a li d a t i o n a cc u r a c y
[Figure]
massive stars are still alive to contribute to the reionization.
The timescale for this ‘emergence’remain highly debated, and is one of the questions the JWST is posed to address.
Because thesevery young star clusters reside in very dense regions, optical light cannot emerge and they need to beinvestigated using infrared (IR) light.
Figure 4(a) provides and example, from HST imaging, of ayoung star cluster surrounded by its HII region.
In preparation for the upcoming JWST images, we design a pilot study to classify young star clustersin HII regions using the near-IR filters from the HST 2 .
Our pilot consists of five rounds of annotationswith 3-4 human annotators on each round.
We perform the study on galaxy NGC4449 due to its highrate of star formation and the presence of a population of massive and young star clusters.
For this case study, we use a preliminary catalog of 1119 sources generated with SExtractor on galaxy
NGC4449, similar to the procedure in Section 4.
These initial sources generally focus on the peaksin the emission line image 3 and are marked with yellow rings in the labeling interface, as shownin Figure 4(c).
In addition to the near-UV and optical bands used in LEGUS (i.e., F275W, F336W,
F435W, F555W, F657N, and F814W), we include near-infrared channels to our image data: F1100W,
F1280W, and F1600W.
5.1 Labeling interface for young star clusters in near-IR images
We customize the tool to support emission lines and near IR renditions to the visualization (See
Figure 4(a)).
Also, we added a marker to sources that are also found in LEGUS catalogs; given that
N ea r-
i n f r a r ed
E m i ss i on li ne s
U V + op t i c a l
[Figure]
(b) (c)
N ea r-
i n f r a r ed
E m i ss i on li ne s
LEGUS clusters are already visually inspected (sources with blue circle markers in Figure 4(c)).
Finally, we display specific information about each source, including the prediction score and the classthe human annotators gave in the previous round with a majority voting consensus (See Figure 1).
We sample and mark the sources to be annotated in the following rounds to measure the agreementbetween human annotators (sources marked with a large white circle marker in Figure 4(c)).
Thesesources are sampled randomly from high-informative areas selected manually by a user (i.e., fromhigh HII concentration regions as shown in Figure 4(b)).
5.2 Results
Contrastive learning for near-IR images We use the entire set of 1119 sources obtained with
SExtractor to pre-train the network using contrastive learning.
On the other hand, we annotate 658from the 1119 sources where 398 (60.5%) sources were labeled as clusters and 260 (39.5%) asnon-clusters and split the labeled samples into 90% for training and 10% for validation.
Finally,using the same problem formulation as in Section 4.1, the same hyperparameters from Appendix B,and following the evaluation procedure described in Appendix C, we get a classification accuracyof 82.7%, demonstrating the generalizability of our approach to additional modalities and differentimage resolutions.
Human agreement Following the procedure in Section 4, we calculate a human-to-mode agreementof 93.3% averaged over the five rounds.
Conclusion
The analysis of star clusters has proven critical in answering scientific questions regarding stellarevolution and how galaxies are born.
In this work we collaborate with astronomy researchers topropose an ecosystem of AI tools for cataloging bright sources in high-resolution images of nearbygalaxies, reducing to 32% the labeling effort on the star cluster classification task with the LEGUSdataset.
In addition, we demonstrate the robustness of our approach to new modalities and differentimage resolutions anticipating the upcoming images from the JWST.
A Color renditions for astronomical data
Astronomical multi-spectral images are often combined to form color renditions.
For visualization purposes, in 341 this work we merge the two reddest channels into a single red channel and the two bluest channels as a singleblue channel as described below:
[Equation]
=
[Equation]
γ v
[Equation]
+
[Equation]
i
[Equation]
• G =
( γ b B )
• B = ( γ n NUV + γ u U ) /
where γ is the inverse gain of the filters used by the HST.
B Supervised and sel-supervised training hyper-parameters
Contrastive learning pretraining We use BYOL PyTorch [40] implementation by [53].
We pretrain our 349 network with the adapted version of BYOL for 170,000 iterations ( ≈ 1000 epochs) with a batch size of 128samples and a learning rate of 3E-4.
We train for approximately 7 hours using a single GPU 1080ti .
Supervised training We use the CNN optimized for star cluster classification proposed by [44].
We train 352 for ten epochs and a batch size of 64 images using Adam optimizer [29].
We use a learning rate of η =1E-04 witha learning schedule η ′ =
η ∗ 0 .
1 in the 6 th epoch.
C BYOL adaptation for multi-spectral images and ablation
For our self-supervised learning experiments, we use an adapted version of BYOL [22] for multi-spectral star 356 cluster data, which achieves good performance without needing negative pairs.
In addition, BYOL is more robustto lower batch sizes, which allows training models with good performance using a single GPU.
Off-the-shelf BYOL (with the backbone input adapted to work with multi-spectral star cluster images) does 359 not converge during training.
This is not surprising since BYOL was designed with ResNet encoders for colorimages.
Our approach uses an encoder that works on multi-spectral images.
Empirically, we find that the projector’s size affects the network’s training considerably in our approach (See 362 Table C.2a).
This is expected since the size of our CNN’s output features f θ ( v ) is 256 (compared to 2048 of
ResNet).
Using the default projector size of BYOL with our encoder and input data yields poor performance.
Therefore, we reduce the projector hidden size while still emulating the dimensional increase from f θ ( v ) to theprojector hidden size (See second row in Table C.2a).
This configuration improves over the default settings, but 366 the performance is still low.
We get the best results using a projector size of (256-256).
See Table C.2a for moreconfigurations.
Contrastive learning methods are sensitive to the choice of image augmentations [13, 22], and star cluster data is 369 very different from color images like ImageNet.
We apply modifications to the data augmentations that are alsokey for the model to converge during training.
First, we limit the lower-bound scale of the random resize crop 371 augmentation (See Table C.2c).
The scale specifies the lower and upper bounds for the random area of the cropbefore resizing.
Astronomical object images contain the object of interest centered in the image with multiple 373 other objects in the close surroundings.
For that reason, we hypothesize that small-scale values might result intwo different centered objects signaled as a positive pair.
Thus, confusing the model.
For the BYOL ablation experiments, we divide the LEGUS dataset into a 90-10% training-validation split.
376 From the training set, we select 100 samples as our ‘labeled dataset’, the remaining samples as the ‘unlabeleddataset’, and evaluate over the complete validation set to compare performances between models.
We follow 378 the experimental setup as the main experiments in Figure 3.
First, we pretrain a network from scratch with the‘unlabeled dataset’ using BYOL.
Then, we finetune the model with the ‘labeled dataset’.
After the training is 380 complete, we evaluate the validation set.
[Table]