Unsupervised Object Representation Learning using Translation and Rotation Group Equivariant VAE
Abstract
In many imaging modalities, objects of interest can occur in a variety of locationsand poses (i.e. are subject to translations and rotations in 2d or 3d), but the locationand pose of an object does not change its semantics (i.e. the object’s essence).
Thatis, the specific location and rotation of an airplane in satellite imagery, or the 3drotation of a chair in a natural image, or the rotation of a particle in a cryo-electronmicrograph, do not change the intrinsic nature of those objects.
Here, we considerthe problem of learning semantic representations of objects that are invariant topose and location in a fully unsupervised manner.
We address shortcomings inprevious approaches to this problem by introducing TARGET-VAE, a translationand rotation group-equivariant variational autoencoder framework.
TARGET-VAEcombines three core innovations: 1) a rotation and translation group-equivariantencoder architecture, 2) a structurally disentangled distribution over latent rotation,translation, and a rotation-translation-invariant semantic object vector, which arejointly inferred by the approximate inference network, and 3) a spatially equivariantgenerator network.
In comprehensive experiments, we show that TARGET-VAElearns disentangled representations without supervision that significantly improveupon and avoid the pathologies of previous methods.
Semantic representationslearned by TARGET-VAE on images highly corrupted by rotation and translationapproach those learned on consistently posed objects, dramatically improvingclustering and pose inference on multiple datasets.
Introduction
In many imaging modalities, objects of interest are arbitrarily located and oriented within the imageframe.
Examples include airplanes in satellite images, galaxies in astronomy images [1] and evenparticles in single-particle cryo-electron microscopy (cryo-EM) micrographs [2].
However, neitherthe location nor the rotation of an object within an image frame changes the nature (i e. the semantics)of
the object itself.
An airplane is an airplane regardless of where it is in the image, and differentrotations of a particle in a cryo-EM micrograph are still projections of the same protein.
Hence, thereis great interest in learning semantic representations of objects that are invariant to their location andpose.
In general, unsupervised representation learning methods do not recover representations that disentangle the semantics of an object from its location or its pose.
Popular unsupervised deep learningmethods for images, such as variational autoencoders [3], usually use encoder-decoder frameworksin which an encoder network is learned to map an input image to an unstructured latent variableor distribution over latent variables which is then decoded back to the input image by the decodernetwork.
However, the unstructured nature of the latent variables means that they do not separateinto any specific and interpretable sources of variation.
To achieve disentanglement, unstructured methods have been proposed that encourage independence between latent variables [4, 5].
However,these methods make no prior or structural assumptions about what these latent variables shouldencode, even when some sources of variation e.g. an object’s location or pose are prevalent andwell-known.
Recently, several methods have proposed structured models that explicitly model rotation or translation within their generative networks [6, 7, 8] by formulating the image generativemodel as a function mapping coordinates in space to pixel values.
Although promising, only thegenerative portion of these methods is equivariant to rotation and translation and inference networkshave lackluster performance due to poor inductive bias for these structured latents.
To address this, we propose TARGET-VAE, a Translation and Rotation Group Equivariant Variational
Auto-Encoder.
TARGET-VAE is able to learn semantic object representations that are invariantto pose and location from images corrupted by these transformations by structurally decomposingthe image generative factors into semantic, rotation, and translation components.
We performapproximate inference with a group equivariant convolutional neural network [9] and speciallyformulated approximate posterior distribution that allows us to disentangle the latent variables into arotationally equivariant latent rotation, translationally equivariant latent translation, and rotation andtranslation invariant semantic latent variables.
By combining this with a spatial generator network[6, 8, 7], our framework is completely invariant to rotation and translation, unsupervised, and fullydifferentiable; the model is trained end-to-end using only observed images.
In comprehensiveexperiments, we show that TARGET-VAE accurately infers the rotation and translation of objectswithout supervision and learns high quality object representations even when training images areheavily confounded by rotation and translation.
We then show that this framework can be used tomap continuous variation in 2d images of proteins collected with cryo-EM.
Related Work
In the recent years, there has been a great progress in developing models for unsupervised semanticanalysis of images using VAEs [10, 11, 6, 12], flow-based methods [13], and Generative Adversarial
Networks (GAN) [14, 15, 16].
In these methods, usually the goal is to build a low-dimensionalrepresentation of the images, which can be sampled to reconstruct similar images with some variations,or to use the representations to identify clusters in the data.
The unsupervised learning methodsused for clustering objects, need to address the variations caused by the common transformationsin the objects.
One simple approach to building invariant representations is data augmentation.
Inthis approach, training dataset plus samples generated by the applying multiple transformations onthem, are used to learn the transformation-invariant representations in a lower-dimensional latentspace [17, 18].
This training approach is not efficient and when it comes to severe transformationssuch as rotations with high angles, the model might not be able to map the original data samplesand the rotated ones to the same cluster in the latent space.
Another group of methods that usedata augmentation for learning transformation-invariant features, are using contrastive learning[19, 20, 21].
These models learn to output similar encoding for the different transformed versions ofthe same data sample.
Even though this training scheme has made great progress in the recent years,it is considered to be pre-training method which needs to be combined with some supervised trainingand is not completely unsupervised.
In some methods, the model is trained to learn an invariant representation by constraining the latentspace in the different variations of beta-VAE [15, 4, 5].
In these methods, the latent space is notexplicitly structured to separate the representation variable from the transformation ones.
A moreexplicit constraint is imposed on the latent space in Spatial-VAE [6], where the latent space is dividedto variables encoding the representation and transformations. The latent space that can capture eachtargeted transformation in a structured space, can theoretically disentangle them from the content ofthe image.
To be able to disentangle the representation, from the transformations, the inference model shouldbe equivariant to the transformations.
Convolutional layers are one of the most popular featureextraction methods in image processing, due their translation-equivariant features.
But they arenot rotation-equivariant.
There has been many studies in designing transformation-equivariant andspecially rotation-equivariant neural networks [9, 22, 23]. In Group equivariant Convolutional Neural
Networks (G-CNNs) [9], the 2D rotation space is discretized and filters are also applied on thisrotation dimension. In this method, without adding anymore parameters and by small computationaloverhead, the regular convolutional layers can reflect the rotation of the input images. None of the
Group-Convolution’s Feature Maps
Encoder
Decoder
Attention on x and r (attn x,r )
Inference module for attention
Rotation distribution (theta) content distribution(z) 𝒒 𝜽 𝒙,𝒓,𝒚)~(𝓝 𝝁 𝒊𝜽 ,𝝈 𝒊𝜽 +𝜽 𝒐𝒇𝒇𝒔𝒆𝒕 ) 𝒒 𝒛 𝒅𝒙,𝒓,𝒚)~𝓝 𝝁 𝒛𝒊 ,𝝈 𝒛𝒊 Sampled z
Fully connected layer on spatial coordinates concatenated f(x 1 , x 2 ) pixel-value for (x 1 , x 2 ) x 1 , x
Fully connected layers
Inference module for z
Inference module for 𝜃
Fully connected layer on z 𝒒(𝒅𝒙,𝒓|𝒚)~𝒑 𝒅𝒙 𝒚 𝒑(𝒓|𝒚)
Rotated and translated previous methods in unsupervised object detection have included equivariance in their inferencemodel.
Some image processing methods have proposed to train their model as a function that maps spatialcoordinates of a pixel to their values [24, 25, 8, 7]. In these models, the generator learns to map the2D or 3D coordinates of the image to its pixel values.
However, in these studies, no direct inferenceon the rotation is performed.
To the best of our knowledge, we are the first method to establishrotation and translation equivariance in both the inference and generator models of our proposed VAEframework, along with disentangling the mentioned transformations as structured variables in thelatent space from the continuous representation variable of the objects.
Method
[Figure] In this section, we present in details our proposed VAE framework that enables an unsuperviseddisentanglement of the rotation and translation of the objects from their representations.
Figure 1displays our proposed VAE-based framework.
3.1 Image generation process
[Figure]
Images can be considered as a combination of values for discreetly identifiable pixels.
Transformationof an image, can also be considered as transforming the spatial coordinates of the pixels in the image.
To have a generation process adherent to the transformations identified in the latent space, we defineour generator as a function, which is mapping the spatial coordinates of the pixels to their values.
The generator outputs a probability distribution as p (ˆ y i | x i , z ) , where z , x i , and ˆ y i are the latentrepresentation variable, the spatial coordinates of a point, and the value for that point, respectively.
Similar to [3], for an image with n pixels, we can define the probability of the image generated inthis manner as sum over probability of its pixel values:
[Equation]
We use a Cartesian coordinates with the origin at the center of the image.
The translation and rotationof the coordinates can be done by shifting the origin, and rotating coordinates around it, respectively.
Depending on the value of the representation variable z , the generator acts as a function over thepixel coordinates to reconstruct the input image.
In this setup the generation process is equivariantwith regard to the transformations on the coordinate system.
Assuming R ( θ ) as the rotation matrixfor angle θ and translation value of t , the probability of the generated image from Formula 1, can berewritten as
[Equation]
In this study, we focus on disentangling the rotation and translation from the representation of data,but the image generation probability from Formula 2, can be extended the other transformations too.
3.2 Approximate inference on latent variables
We implement a VAE framework to perform approximate inference on representation, rotation, andtranslation latent variables.
Since rotation optimization is non-convex, we have discretized 2D rotationspace to r values, each related to a specific k 2 πr angle, where k ∈ { 0 , 1 , ..., r − 1 } .
As a result,our posterior distributions are conditioned on an additional factor r .
We represent the approximateposterior as q ( z, θ, t, r | y ) , where z is the representation value, θ is the rotation angle, and t and r referto the translation and discretized rotation values, and y is the input image.
This posterior distributionis intractable, and making the simplifying assumption that q(z|t,r,y) and q( θ |t,r,y) are independent , wecan factorize the posterior distribution as
[Equation]
, where q ( t, r | y ) identifies a 3D distribution over t and r . The other two posterior distributions,q ( θ | t, r, y ) and q ( z | t, r, y ) , are mixture distributions conditioned on r , t , and the input y .
We use N (0 , I ) for the prior distribution over z .
The prior over r and θ can be uniform or Gaussiandepending on our knowledge about the distribution of rotation values in the dataset.
To calculate theprior over r , we calculate the probability of the discrete rotation angles related to each r dimension,which we call θ offset . For example, when r = 4 , we have rotation values of θ offset = { 0 , π 2 , π, 3 π 2 } .
The prior over translation values x is set to be Gaussian with mean 0, which denotes the center ofthe 2D coordinate system.
The standard deviation of this prior depends on the number of pixels inour fixed-size coordinate system and whether we want the model to be flexible with large translationvalues or not.
Lower standard deviation for the prior over translations, will cause the model topose higher loss towards larger predicted translation values.
Using these priors, we calculate the Kullback-Leibler (KL) divergence for the latent variables:
[Equation]
To simplify this equation, we us the factorization from Equation 3:
[Equation]
, where KL t,r = (cid:80) t,r q ( t, r | y ) log q ( t,r | y ) p ( t,r )
, KL θ = KL ( q ( θ | t, r, y ) || p ( θ | t, r, y )) , and KL z =
KL ( q ( z | t, r, y ) || p ( z | t, r, y )) .
We give a detailed calculation of the KL-divergence in Appendix A1.
Multiplication of Kl θ and KL z with q ( t, r ) weights the KL-divergence on θ and z based on the joint posterior distributionsover the discretized rotation and translation values.
We can calculate p ( t, r | y ) by multiplying p ( t | y )and p ( r | y ) , and p ( z | t, r, y ) is the same as as p ( z | y ) which is N (0 , I ) . The p ( θ | t, r, y ) cannot besimplified to p ( θ | y ) since the value of θ is dependent on the θ offset of r . However, p ( θ | t, r, y )can be written as p ( θ | r, y ) since it is independent of t translation values.
We define this prior as
N ( θ offset , πr ) , which means that the Gaussian prior for each r dimension has a mean equal to therotation of filters in that dimension.
Also, we are tightening the prior distributions by lowering theirstandard deviation as r grows, so that the prior distribution for the r dimension would not overlap.
The full variational lower-bound for our model in this VAE framework is calculated based on thereconstruction loss and the Kl-divergence:
[Equation]
, where we calculate the KL-divergence value in closed form using Equation 5 during the training.
3.3 Neural network architecture
Convolutional layers are translation equivariant meaning that the spatial shift in the image causesthe same shift in their output feature maps.
To incorporate rotation-equivariance in the inferencemodel, we choose to use group convolutional layers from [9].
A set X with an operation . , whichis associative on the set, and has inverse and identity elements, specifies a symmetry group G .
Furthermore, combination of two symmetry transformations on a set, also creates a symmetry group.
We can consider the regular convolutional layers as a function f : Z 2 → R k , that maps every pixel toa vector of k values where k is the number of kernels in the convolutional layer.
The convolutionallayer does this by sliding the kernels on the input image.
Replacing this shifting operation withtransformations from group G , changes the function to a function over group G [26].
We define p r group where r ∈ N , and the operation consists of all translations, and r rotations, aboutany center of rotation in a 2D space.
In the group convolutional layer, each kernel is rotated byk 2 πr angles, where k ∈ { 0 , 1 , ..., r − 1 } and then we calculate its dot product with the input imageor feature map.
Rotating kernels happens in addition to the translation of them across the input.
As a result, the output feature map of the p n group convolutions has one extra dimension whichcorresponds to the dot-product of the kernels with different rotations.
As shown in Figure 1, we define three parallel 1x1 3D convolutional layers on the translation androtation equivariant feature maps of the group convolutional layers.
These layers output three logposterior distributions of log q ( t, r | y ) , log q ( z | t, r, y ) , and log q ( θ | t, r, y ) .
To build the associationbetween the θ values of each r dimension and the rotation of the kernel in that r dimension, we addthe mean values from the q ( θ | t, r, y ) with the respective θ offset values of their r dimension.
During the training, we sample from the joint posterior distribution of log q ( t, r ) using Gumbelsoftmax [27], and use the sample to obtain z and θ values from the other two posterior distributions.
We define a coordinate system where x is in range.
Then we translate and rotate this coordinatesystem using the sampled t and θ values.
The generator receives the sampled z value along withthe coordinates a of point at every pass and outputs the value from that point in the image.
Weuse random Fourier feature expansion [28] on the coordinates of the pixel before feeding it to thefully-connected layers.
As shown in Figure 1, there are two sets of parallel fully-connected layerseach with 512 hidden units, where one receives the coordinates of the pixel (or its Fourier featureexpanded version), and the other one takes the z as input.
The output of these layers are concatenatedbefore being input to a final set of fully-connected layers to output the value for the pixel.
Results
4.1 Experiment setup
We run our experiments with p 4 , p 8 , and p 16 group convolutional layers in the inference model.
Theinput images are normalized to the range of [0,1].
We use 128 rectangular filters where size of thekernels in the first layer needs to be set to a value larger than the size of the targeted objects.
Weuse leaky-ReLU activation functions. The batch size for all the experiments is set to 100.
We use ADAM optimizer [29], with learning rate of 2e-4, and the learning rate is reduced by factor of 0.5after no improvements in the loss for 10 epochs.
We run the training for a maximum of 500 epochswith early-stopping in case of no improvements in the loss for 20 epochs.
Most of our experimentsconverge in less than 200 epochs due to the early-stopping condition.
We run all our experiments in asingle NVIDIA A100 gpu, with 80 GB memory.
The fully-connected layers in the generator, each has 512 hidden units, and the output can be oneprobability value in the case of binary valued pixels, or mean and standard deviation if the pixels arereal values.
For the RGB images, the dimension of output for each pixel can be easily extended tothree.
[Table]
We evaluate how accurately, our proposed model can identify the translation and rotation valuesof the objects in the images.
For this purpose, we create two datasets using hand-written digits in
MNIST.
In the first one, the digits are randomly rotated by sampling from N (0 , π 2 16 ) and randomlytranslated by sampling from N (0 , 5 2 ) pixels.
We call this dataset MNIST(N), because of the normaldistribution over the rotation angles.
In the second dataset, the digits are translated similar to the MNIST(N) , but the rotations are randomly sampled from U (0 , 2 π ) . We call this dataset MNIST(U).
In both datasets, the training and test sets have 60000, and 10000 images where each image is 50x50pixels.
Appendix A2 shows some examples of the data in these datasets.
We train TARGET-VAE with p 4 , p 8 , and p 16 on these generated datasets.
For the prior over θ , weuse N (0 , π 4 ) for the MNIST(N) dataset, and U (0 , 2 π ) for the MNIST(U).
For p ( θ | r ) we calculatep ( θ offset ( r ) ) ∼ p ( θ ) , where θ offset ( r ) is the rotation angle of the kernels in each dimension of r. Inthe training for both of the datasets, we set the prior over t to N (0 , 5 2 ) .
After the training, to evaluate our model’s performance on the prediction of the translation androtation of the objects, we use the samples from the test set as the input to the inference model toobtain q ( t, r | y ) and q ( θ | t, r, y ) .
To identify the most-likely ( t, r ) , we apply the softmax functionon q ( t, r | y ) . By marginalize the q ( t, r ) over r , we identify the translation value t , and by using theresult of this softmax to sample from q ( θ | t, r, y ) , we identify the rotation value θ .
For the translationvalues, we calculate Pearson correlation, but because of the circular nature of the rotations values,we calculate the circular correlation [30] for them.
Table 1, shows the correlation values on the testset for MNIST(N) and MNIST(U), respectively.
The two values in the translation column, showthe translation correlation for x 1 and x 2 coordinates.
The values in these tables are related to themodel trained with dimension of the representation space ( z _ dim ) equal to 2.
Our experimentsshow that changing dimension of the representation space, does not have a significant effect onthe translation and rotation correlation values, but increasing the level of the discretization overthe rotation dimension usually improves the rotation correlation.
To the best of our knowledge,the only other method that performs direct inference on the rotation value to identify the invariantrepresentation for the objects is spatial-VAE [6].
To compare our results, we run spatial-VAE, with thesame number of parameters and z_dim=2.
As shown in Table 1, in both MNIST(N) and MNIST(U)datasets, our proposed method, TARGET-VAE significantly improves the rotation correlation value.
Some of the digits in MNIST datasets, have natural ambiguity, with regard to their rotations.
Investigating the correlation between the predicted rotation values, and the ground-truth ones, shows adifference about π between the some of the predicted angles and their ground-truth ones.
Figure 2shows this difference.
By drawing digitwise rotation correlation figures in appendix A.3, we showthat this π difference mostly belongs to the three digits of 0, 1, and 8.
This shows how our model canacknowledge the rotation symmetry in these digits.
4.3 TARGET-VAE improves clustering accuracy by learning translation and rotationinvariant features
By disentangling rotation and translation from the representation values, TARGET-VAE can identifythe different clusters of the objects.
Similar to 4.2, we use softmax ( q ( t, r )) to identify the mostprobable representation from q ( z | t, r, y ) .
We train TARGET-VAE with p 4 , p 8 , and p 16 and evaluatetheir clustering performance using K-means algorithm [31] on MNIST(N) and MNIST(U). Table [Figure]
[Table] 2 shows the clustering accuracies on these two datasets [Table] where increasing the dimension of therepresentation space, helps the model with the separation of different digits and achieving higherclustering accuracy.
Based on the accuracies in this table and rotation correlations in Table 1, higherdiscretization of the rotation space [Table] allows the TARGET-VAE to have a better estimate of the rotationof the objects [Table]
and this is turn results in more rotation-invariant representation encoding and higherclustering accuracy.
To compare the clustering performance of TARGET-VAE with other similarmethods, we train a standard VAE [3], beta-VAE [4] [Table] and spatial-VAE [6] with about same number ofparameters as TARGET-VAE, on MNIST(N) and MNIST(U) datasets.
We set the z_dim=2 for bothspatial-VAE and TARGET-VAE.
Since the standard VAE and the beta-VAE do not have dedicatedlatent variable for identifying translation and rotation, we set the dimension of latent space to 4, sothat it would have equal dimensions with the total size of the latent spaces in our model.
Also, forthe beta-VAE, we set β value to 4 based on the proposed settings in [4].
As shown in the Table2, all variations of the TARGET-VAE outperform the other methods.
We believe that the directinference on the rotation and dedicating a structured latent variable to encode the rotation of theobjects in TARGET-VAE, plus the transformation-equivariant generator, are the main reason forthe dramatic increase in the clustering accuracy with our proposed method.
In MNIST(N), sincethere are limited range for the rotation values, increasing the size of the group convolution, does notnecessarily increase the rotation correlation or the clustering accuracy, where in MNIST(U) due tothe larger variations in the rotation values, increasing the size of the group convolution leads to abetter estimate of rotations and improved clustering accuracy.
Figure 3 shows the confusion matrix and t-sne plot for TARGET-VAE with p 8 group-convolution,on MNIST(N).
Based on the confusion matrix, most of the digits are clustered correctly, and somemisclassification is related to the confusion between rotated digits 6 and 9.
In appendix A.4, we perform experiments using images with multiple digits to show that TARGETVAE can detect multiple objects in the evaluation stage.
Furthermore, in appendix A.5 and A.6,we train and evaluate TARGET-VAE on the dSprites [32], and galaxy zoo [1].
We also perform anablation study on the important components of our proposed model in Appendix A.7.
4.4 Identifying class-averages and protein conformations in the electron-microscopy imageswith TARGET-VAE
In the single particle electron microscopy images, biological particles can appear with differentrotations and in different locations of the images.
To be able to identify different types of particlesor different conformations of the protein, the representation vector should be invariant to thesetransformations.
Here, we demonstrate the capacity of TARGET-VAE in capturing different classesand variable conformations of particles in the electron microscopy images.
Identifying different classes of particles in EMPIAR-10025.
The dataset EMPIAR-10025 [33]contains T20S Proteasome micrographs with 2.8 Å resolution.
We use Topaz [34] package to create161292 images of 400x400 pixels, where each image contains a particle.
After downsampling byfactor of 4 and normalizing the images, we train TARGET-VAE with p 8 and z_dim=2 on them.
Dueto the high-complexity of the structures of the proteins, we use 6 fully-connected layers with 512units in the generator.
In the rotation inference our prior over r is uniform, and the prior over θ , is
N (0 , π ) .
We randomly select 10% of images as validation set to monitor the training process.
Asshown in Figure 4, our model identifies three clusters in this dataset.
The image on the left side of the Figure 4 contains instances of input images which are extremely noisy.
Identifying different views of particles in EMPIAR-10029.
The EMPIAR-10029 dataset containssimulated EM dataset of GroEL particles.
This dataset has 10000 images of shape 200x200, whichwe downsample to 100x100 and normalize.
Similar to EMPIAR-10025, we train TARGET-VAE withthe same settings as the EMPIAR-10025.
The latent space does not identify separate clusters in thisdataset, but we can perform a random walk in the latent space to identify the different projections of this biological particle.
We have also tested our model in identifying the heterogeneity in some otherdatasets, which we discuss in appendix A.8.
Conclusion
In this paper, we presented TARGET-VAE, a translation and rotation group equivariant variationalautoencoder framework for learning to detect and represent objects in images without supervision.
By structuring the encoder to be translation and rotation equivariant, the model can identify thelatent rotation and translation variables and disentangle them from the latent variables encoding thecontent of the image.
The spatially equivariant generator samples from the posterior distributionto reconstruct the image.
We showed that this approach not only accurately estimates the rotationand translation of objects, but also learns content representations that reflect known semantics (i.e.,clusters match known semantic labels) across multiple datasets.
Although we only consider 2d imagescontaining single objects with rotation and translation, this framework can be extended to other objecttransformations by adopting encoder networks with the proper equivariance and adapting the latentdistribution.
In the future, we expect this framework can also be extended to multi-object detection,object tracking over time, and to 3d environments.
TARGET-VAE lays the groundwork for a newgeneration of fully unsupervised object detection and semantic analysis methods for cryo-EM andother imaging modalities.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes] Our proposed framework can perform inference on the
rotation and translation values of the object and use this inference to learn a rotation
and translation invariant representation value for them.
(b) Did you describe the limitations of your work?
[Yes] We mentioned the limitations in the conclusion in the form the future directions.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
To the
best of our knowledge, our work does not have any negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] We have read the ethics review guidelines and we believe that our paperconforms to that.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] In the parts that we made any assumptions, we clearly explained them.
(b) Did you include complete proofs of all theoretical results?
[Yes] We included proof forone of our equations in section 3.2 which needs proof.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)?
[Yes] The code isavailable at —————————– (b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] All the details are either explained in the paper or can be found inthe code.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] The variation in the results is not large enough to justifyusing error bars.(d)
Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We included all the processinginformation.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A] All the data that we are using is the public domain.
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
The data that we are using does not containany personally identifiable information or offensive content.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
N/A
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]