Paper:
SEMI-SUPERVISED DETECTION OF EXTREME WEATHER EVENTS IN LARGE CLIMATE DATASETS
The detection and identification of extreme weather events in large scale climate simulations is an important problem for risk management, informing governmental policy decisions and advancing our basic understanding of the climate system. Recent work has shown that fully supervised convolutional neural networks (CNNs) can yield acceptable accuracy for classifying well-known types of extreme weather events when large amounts of labeled data are available. However, there are many different types of spatially localized climate patterns of interest (including hurricanes, extra-tropical cyclones, weather fronts, blocking events, etc.) found in simulation data for which labeled data is not available at large scale for all simulations of interest. We present a multichannel spatiotemporal encoder-decoder CNN architecture for semi-supervised bounding box prediction and exploratory data analysis. This architecture is designed to fully model multichannel simulation data, temporal dynamics and unlabelled data within a reconstruction and prediction framework so as to improve the detection of a wide range of extreme weather events. Our architecture can be viewed as a 3D convolutional autoencoder with an additional modified one-pass bounding box regression loss. We demonstrate that our approach is able to leverage temporal information and unlabelled data to improve localization of extreme weather events. Further, we explore the representations learned by our model in order to better understand this important data, and facilitate further work in understanding and mitigating the effects of climate change.
1 INTRODUCTION
Climate change is one of the most important challenges facing humanity in the 21st century and climate simulations are one of the only viable mechanisms for understanding the future impact of various carbon emission scenarios and intervention strategies. Large climate simulations produce massive datasets: a single 30-year run from a 25-km resolution model produces on the order of 10TB of multi-variate data. This scale of data makes post-processing such datasets to make quantitative assessments challenging and as a result, climate analysts and policy makers typically take global and annual averages of temperature or sea-level rise. These quantities are very coarse measurements and while they are perfect for public and media consumption, they ignore spatially (and temporally) resolved extreme weather events such as extra-tropical cyclones and tropical cyclones (hurricanes). Because the general public and policy makers are concerned about the local impacts of climate change, it is critical that we be able to analyze trends in extreme weather events which can have dramatic and tragic impacts on local and national populations and economies.
The task of finding extreme weather events in climate data has some similarities to the task of detecting objects and activities in video streams - a popular application area for deep learning techniques. An important difference is that in the case of climate data the ’video’ has 16 or more ’channels’ of information (such as water vapour, pressure and temperature), while conventional video only has 3 (RGB). In addition, these climate simulations do not share the same statistics as natural images. As a result, we cannot build off of some of the computer vision communities’ efforts, such as pretrained weights from VGG or AlexNet with ImageNet.
Deep neural networks, especially deep convolutional neural networks, have enjoyed breakthrough success in recent recent years, achieving state-of-the-art results on many benchmark datasets (Krizhevsky et al. (2012); He et al. (2015); Szegedy et al. (2015)) and also compelling results on many practical tasks such as disease diagnosis (Hosseini-Asl et al. (2016)), facial recognition (Parkhi et al. (2015)), autonomous driving (Chen et al. (2015)), and many others. Furthermore, deep neural networks have also been very effective in the context of unsupervised and semi-supervised learning; some recent examples include variational autoencoders (Kingma & Welling (2013)), adversarial networks (Goodfellow et al. (2014); Makhzani et al. (2015)), and what-where autoencoders (Zhao et al. (2015)).
In this paper we present a 3D convolutional architecture for the task of semi-supervised climate event detection. Our contribution is the creation of a 3D spatiotemporal autoencoding CNN architecture for bounding box prediction and the exploration of its use for this important problem. Our architecture is capable of semi-supervised event detection, as well as exploratory data analysis through clustering representations learned using this approach. Our framework allows climate researchers to identify important climate events which have not yet been precisely defined, and could provide a basis for mining large climate simulation data archives and potentially near-real-time prediction of extreme weather events.
2 RELATED WORK
2.1 DEEP LEARNING FOR CLIMATE AND WEATHER DATA
The climate science community primarily relies on expert engineered systems and ad-hoc rules for characterizing climate and weather patterns. Prabhat et al. (2012; 2015) have implemented an assortment of heuristics in the TECA MapReduce framework to process large scale climate datasets on high-performance computing (HPC) platforms. Using the output of TECA analysis as ground truth, Liu et al. (2016) demonstrated for the first time that convolutional architectures could be successfully applied to predict the class label for two extreme weather event types. Their work considered the binary classification task on centered, cropped patches from 2D (single-timestep) multi-channel images. Like Liu et al. (2016) we use TECA’s planetary-scale output as ground truth, but we build on this work by: 1) using uncropped images, 2) considering the temporal axis of the data 3) doing multi-class bounding box detection and 4) exploring a semi-supervised approach based on a hybrid predictive and reconstructive model.
Some recent work has applied deep learning methods to weather forecasting. Xingjian et al. (2015) have explored a convolutional LSTM architecture (described in 2.2 for predicting future precipitation on a local scale (i.e. the size of a city) using radar echo data. In contrast, we focus on extreme event detection on planetary-scale data. Our aim is to capture patterns which are very local in time (e.g. a hurricane may be present in half a dozen sequential frames) compared to the scale of our underlying climate data, consisting of global simulations over many years. As such we use full blown 3D CNNs as they make more sense for our detection application compared to LSTMs, whose strength is in capturing long-term dependencies.
2.2 RELATED METHODS AND MODELS
Following the dramatic success of CNNs in static 2D images, a wide variety of CNN architectures have been explored for video, ex. (Karpathy et al., 2014; Yao et al., 2015; Tran et al., 2014). The details of how CNNs are extended to capture the temporal dimension are important and Karpathy et al. (2014) explored different strategies for fusing information from 2D CNN subcomponents. In contrast, Yao et al. (2015) created 3D volumes of statistics from low level image features.
CNNs have been combined with RNNs for modeling video and other sequence data and we briefly review some relevant video models here. The most common approach to modeling sequential images is to feed single-frame representations from a CNN at each timestep to an RNN. This approach has been examined for a number of different types of video (Donahue et al., 2015). Srivastava et al. (2015) have explored an LSTM architecture for the unsupervised learning of video representations using a pretrained CNN representation as input. Another popular model, also used on 1D data, is a convolutional RNN, wherein the hidden-to-hidden transition layer is 1D convolutional (i.e. the state
is convolved over time). Ballas et al. (2016) combine these ideas, applying a convolutional LSTM to frames processed by a (2D) CNN.
The 3D CNNs we use here are based on 3-dimensional convolutional filters, taking the height, width, and time axes into account for each feature map, as opposed to aggregated 2D CNNs. This approach was studied in detail in Tran et al. (2014). 3D convolutional neural networks have been used for various tasks ranging from human activity recognition (Ji et al., 2013), to large-scale YouTube video classification (Karpathy et al., 2014), and video description (Yao et al., 2015). Hosseini-Asl et al. (2016) use a 3D convolutional autoencoder for diagnosing Alzheimer’s disease through MRI - in this case, the 3 dimensions are height, width, and depth. Whitney et al. (2016) use 3D (height, width, depth) filters to predict consecutive frames of a video game for continuation learning. Some recent work has also examined ways to use CNNs to generate animated textures and sounds Xie et al. (2016). This work is similar to our approach in that it uses a 3D convolutional encoder, but they use a stochastic approach whereas we use a deterministic autoencoder with a 3D convolutional decoder for the unsupervised component of our overall objective function.
Stepping back, our approach and model is closely related to Tran et al. (2014) in that we use a 3D deconvolutional CNN approach to construct the autoencoding component of our model. Our approach is related to recent work from Zhang et al. (2016) (and others) in that we also use a hybrid prediction and autoencoding architecture, but our model is for multidimensional 3D data and yields multiple bounding box predictions. Our formulation for the bounding box prediction loss is inspired by the approach in Redmon et al. (2015) and extended in Ren et al. (2015), the single shot multiBox detector formulation used in Liu et al. (2015) and the seminal bounding box work in OverFeat Sermanet et al. (2013).
3 THE MODEL
We use a 3D convolutional encoder-decoder architecture (the code can be found here: https://github.com/eracah/hur-detect/); feature maps are 3D: time, height, and width. The encoder part of this architecture is described in Table 2. The decoder is the equivalent structure in reverse, using tied weights and deconvolutional layers. As we take a semi-supervised approach, the code (bottleneck) layer of the autoencoder is used as the input to the loss layers, which make predictions for bounding box location and size, the class associated with the bounding box, and the confidence (sometimes called ’objectness’) of the bounding box. In training, we input one day’s simulation at a time (8 time steps, 16 variables) and reconstruct all 8 time steps, while only predicting the bounding box for 4 time steps, since only 4 out of 8 time steps are labelled, so a 1:1 ratio of unlabelled data to labelled data. The entire architecture is depicted in Figure 1.
For comparison, to evaluate how useful the time axis is to recognizing extreme weather events, we also run experiments with a 2D (width, height) version of this architecture.
In order to regress bounding boxes, we use some techniques from Redmon et al. (2015), Ren et al. (2015), and Liu et al. (2015). We split the original 768 by 1152 image into a 12x18 grid of 64x64 “anchor” boxes We then make a guess for a box at each grid point by transforming the representation to 216 “scores”. Each score encodes three pieces of information: how much the predicted box differs in size and location from the anchor box, how confident we are that an object of interest is in the predicted box, and the class probability distribution for that object. We compute each component of the score by several 3x3 convolutions applied to the 640 12x18 feature maps from output of the last layer of the encoder (”code layer”). Because each set of pixels in each feature map at a given x, y coordinate can be thought of as a learned representation of the climate data in a 64x64 patch of the input image, we can thinks of the 3x3 convolutions as using the representation of a 192 x 192 neighborhood from the input image as context to determine the box and object centered in the given 64x64 patch. Our approach is similar to Liu et al. (2015) and Sermanet et al. (2013), which use convolutions from small local receptive field filters to regress boxes. This choice is motivated by the fact that extreme weather events occur in relatively small spatiotemporal volumes, with the ’background’ context being highly consistent across event types and between events and non-events. This is in contrast to that of Redmon et al. (2015), which uses a fully connected layer to consider the whole image as context. Their approach is appropriate to the objective of object identification in natural images, where there is often a strong relationship between background and object.
The bounding box regression loss is determined as follows:
Lsup = 1
N (Lbox + Lconf + Lcls), (1)
where N is the number of time steps in the minibatch, and Lbox is defined as: Lbox = α ∑ i 1 obj i R(ui − u ∗ i ) + β ∑ i 1 obj i R(vi − v ∗ i ), (2)
where i ∈ [0, 216) is the index of the anchor box for the ith grid point (12x18 grid, so 216 grid points). where 1obji is 1 if an object is present at the ith grid point and 0 if not, R(z) is the smooth L1 loss (Ren et al. (2015)), ui = (tx, ty)i and u∗i = (t ∗ x, t ∗ y)i, vi = (tw, th)i and v ∗ i = (t ∗ w, t ∗ h)i
where t is the parametrization defined in (Ren et al. (2015)) such that:
tx = (x− xa)/wa, ty = (y − ya)/ha, tw = log(w/wa), th = log(h/ha)
t∗x = (x ∗ − xa)/wa, t∗y = (y∗ − ya)/ha, t∗w = log(w∗/wa), t∗h = log(h∗/ha),
where (xa, ya, wa, ha) is the center coordinates and height and width of the closest anchor box, (x, y, w, h) are the predicted coordinates and (x∗, y∗, w∗, h∗) are the ground truth coordinates.
Lconf is the weighted cross entropy of the object confidence (probability distribution for whether object is present in a grid cell or not):
Lconf = ∑ i 1 obj i [(− log(p(obj)i)] + γ ∗ ∑ i 1 noobj i [− log(p(obj)i] (3)
Lcls is just the cross-entropy between the one-hot encoded class distribution and the softmax predicted class distribution, evaluated only for predicted boxes at the grid points containing a ground truth box:
Lcls = ∑ i 1 obj i ∑ c∈classes −p∗(c) log(p(c)) (4)
This loss is similar in spirit to YOLO (Redmon et al. (2015)), but has a few differences. One, the object confidence and class probability terms in YOLO are squared-differences between ground truth and prediction, so we make the logical change to cross-entropy, which is more appropriate for probabilities and is used in the region proposal network from Faster R-CNN Ren et al. (2015) and the network from Liu et al. (2015) for the object probability term and the class probability term respectively.
In addition, there is a difference in the parametrization for the coordinates and the size of the bounding box.
In YOLO, the parametrizations for x and y are equivalent to Faster R-CNN’s tx and ty for an anchor box the same size as the patch it represents (64x64). For the w and h parametrizations in YOLO, they would only be equivalent to Faster-RCNN’s th and tw for a 64x64 anchor box if the anchor box had a height and width equal to the size of the whole image and if there were no log transform in the faster-RCNN’s parametrization. We found in practice that both these differences in YOLO did not work well in practice. For one, without the log term and using a RELU transform as the nonlinearity with a standard weight initialization scheme sampled from a distribution centered around 0, most outputs (more than half) for the parametrization were zero or close to zero as expected (at least at the beginning of training); when the parametrization was converted to raw width and height, this resulted in 0 height and width boxes, so technically the default boxes were of 0 width and height. In practice, it took many epochs for the network to learn to resize 0 area boxes to the correct size due to all the other terms of the loss that were being optimized. Adding the log term, would in effect make the ”default” box (an output of 0 ) equal to a box of height and width ha and wa, which in YOLO’s case was the height and width of the entire image. This once again proved to be a problem as it took a long time to resize the boxes to the size of the much smaller ground truth boxes. Making ha and wa equal to 64x64 made training a lot more efficient, as the optimization was focused more on picking which box contained an object and not as much on what size the box should be. The last difference was YOLO used squared difference between predicted and ground truth for the coordinate parametrizations, as opposed to smooth L1. We used smooth L1 due its lower sensitivity to outlier predictions Ren et al. (2015).
The loss is a weighted combination of reconstruction error and bounding box regression loss:
L = Lsup + λLrec, (5)
where Lunsup is the classic squared difference between input and reconstruction:
Lrec = ||X −X∗||22, (6)
4 EXPERIMENTS AND DISCUSSION
4.1 CLIMATE DATA
The climate science community utilizes three flavors of global datasets: observational products (satellite, gridded weather station); reanalysis products (obtained by assimilating disparate observational products into a climate model) and simulation products. In this study, we analyze output from the third category: free-running climate simulations with prescribed initial and boundary conditions. In particular, we consider a 27-year run of the CAM5 (Community Atmospheric Model v5); configured at 25-km spatial resolution (Wehner et al. (2015)), with ground-truth labelling for four extreme weather events: Tropical Depressions (TD) Tropical Cyclones (TC), Extra-Tropical Cyclones (ETC) and Atmospheric Rivers (AR).
At 25-km resolution, each snapshot of the global atmospheric state in the CAM5 model output corresponds to a 768x1152 image. Each image is comprised of 16 variables (including surface temperature, surface pressure, precipitation, zonal wind, meridional wind, humidity, cloud fraction, water vapor, etc.) stored in double-precision floating point representation. While climate models are run on a 3D grid, with the vertical dimension corresponding to 30 levels; we only consider surface quantities (i.e. 2D data) in this study. Model output is stored every 3 hours.
The training data (which can be downloaded here: http://portal.nersc.gov/project/dasrepo/climate/) consists of 365 days of simulations from 1979 with 8 frames a day (every three hours), but labels are only every 6 hours, so one day for supervised is 4 frames and 8 for semi-supervised. The test set consisted of 365 days from 1984. Table 3 shows the breakdown of the dataset splits for each class.
TECA, the ground truth labelling framework, tries to implement the state of the art heuristics for determining ’ground truth’ data for the four types extreme weather events. However, it is entirely possible there are errors in the labeling: for instance, there is little agreement in the climate community on a standard heuristic for capturing Extra-Tropical Cyclones (Neu et al. (2013)); Atmospheric Rivers have been extensively studied in the northern hemisphere (Lavers et al. (2012); Dettinger et al. (2011)), but not in the southern hemisphere; and spatial extents of such events not universally agreed upon. As such, there is potential for many false negatives, resulting in partially annotated images. This, in addition to lower representation for classes, like AR’s and Tropical Depressions, is part of our motivation in exploring semi-supervised methods to better understand the features underlying extreme weather events. Moreover, although there is more than enough labelled data for our training set sizes, not all flavors of climate datasets can be labelled by TECA due it’s ”hard-coded” set up for a specific resolution simulation with specific parameters, so demonstrating the success of semi-supervised learning for climate pattern detection is important. Lastly, TECA only labels four types of events, so detection of other climate events would require hand-labelling, which would likely result in only 10’s of labels, necessitating semi-supervised approaches.
4.2 FRAMEWISE RECONSTRUCTION
As a simple experiment, we first train a 2D convolutional autoencoder on the data, treating each timestep as an individual training example (everything else about the model is as described in Section 3), in order to visually assess reconstructions and ensure reasonable accuracy of detection. Figure 2 shows the original and reconstructed feature maps for the 16 climate variables of one image in the training set. Though we are showing reconstructions on the training set, the reconstruction loss on the validation set was also similar.
As the reconstruction visualizations suggest, the 2D convolutional autoencoder architecture does a good job of encoding spatial information from climate images. In the upcoming subsections, we utilize the full 3D architecture for the detection of climate events.
4.3 COMPARING FULLY SUPERVISED AND SEMI-SUPERVISED DETECTION
We run two experiments of 3D supervised and semi-supervised bounding box prediction, where values for α, β, γ were selected with experimentation and some inspiration from (Redmon et al. (2015)) to be 5, 7 and 0.5 respectively.
A lower value for γ also can be interpreted as way to deal with ground truth false negatives, as we want more emphasis on pushing up the confidence of true positive examples in order to learn the features of the four events, as opposed to pushing down the confidence of false negative examples, which may interfere with generalization. We used the Adam optimizer (Kingma & Ba (2014)) with a base learning rate of 0.0001. We used a weight decay coefficient of 0.0005.
In Table 4 and Table 5, we present 2D and 3D supervised and semi-supervised results for various settings of λ. Because the 3D model has inherently higher capacity in terms of number of parameters than the 2D model, we also show some experiments with higher capacity 2D models by doubling the number of filters that are in each layer in Table 2. Average Precision (AP) was calculated in the manner of ImageNet (Russakovsky et al. (2015)), where we integrated the precision-recall curve for each class. Results are shown for two modes of evaluation: denoting a true postive as a bounding box prediction with IOU with the ground truth box of 0.1 and 0.5. Mean average precision was just the average of the AP over the four classes. Furthermore, in Figure 3 we provide bounding box predictions shown on 2 consecutive (6 hours in between) simulation frames comparing the 3D supervised vs 3D semi-supervised model predictions.
As we can see from Table 4, Table 5 and Figure 3, 3D and semi-supervised approaches help to some degree for rough localization of the weather events (IOU=0.1). Namely, semi-supervised approaches help the performance of the 3D model for IOU=0.1. In contrast, semi-supervised approach 2D does not have the same effect for IOU=0.1. It is important to note that a more thorough hyperparameter search for λ and other parameters may yield better results for semi-supervised. Also of note is that the 3D models perform significantly better than their 2D counterparts for ETC and TC (hurricane) classes. This potentially means that the time evolution of these weather events is an important criteria for discriminating them. In addition, the semi-supervised model significantly improves the ETC and TC performance, which suggests unsupervised shaping of the spatio-temporal representation is important for these events. It would be interesting to further explore and confirm the spatio-temporal representation learned for these weather events in the 3D models.
In contrast, Table 4 shows that for a more stringent, but standard detection metric (IOU=0.5), the semi-supervised models do much worse than their semi-supervised counterparts. This could potentially be improved by using a dataset from a different year for obtaining unlabelled data, so as not to
have as much correlation with the labelled data. Moreover, the 3D supervised models perform worse than the 2D supervised models and 3D semi-supervised perform comparably to the corresponding 2D semi-supervised models. The potential reason for this can be explained to a certain degree with Figure 3. As shown in the figure, the 3D models do a good job roughly localizing the various events. However, as mentioned in Section 3, the network has a hard time adjusting the size of the boxes. As such, in this figure we see mostly boxes of size 64x64. For example, for TD’s, which are usually much smaller than 64x64 and for AR’s, which are always much bigger than 64x64, a 64x64 box roughly centered on the event is sufficient to count as a true positive with IOU=0.1, but not for the more stringent IOU=0.5. This lead to a large dropoff in performance for AR’s and TD’s and a sizable dropoff in the variable sized TC’s as well as shown in Table 5. While the model has the capacity to adjust the sizes, it can take a while to converge on more different sizes; with longer training time we expect it (and have seen it for 2D) to be able to detect other size boxes, but due to the size of the model we did not have time for longer training. Hence, we believe the 2D model was able to do more size and location adjustments in its training and as a result, it’s AP dropoff for the 4 classes was not as large. However, the 3D semi-supervised model’s ability to better roughly localize events (IOU=0.1) is still useful for the application and shows promise toward future use in an improved system. Future work would involve harnessing anchor boxes of many shapes and sizes, like (Ren et al. (2015), Liu et al. (2015), which we believe would greatly increase the detection accuracy for IOU=0.5.
In Figure 4 we provide t-SNE visualisation of the first 7 days in the training sets for both 3D supervised (left) and semi-supervised (right) experiments. It is possible to make out some grouping of various spiral patterns in the latent space for the TC’s and ETC’s. Und r review as a confer nce paper at ICLR 2017
8
5 CONCLUSIONS AND FUTURE WORK
We have explored semi-supervised methods for object detection and bounding box prediction using 3D CNNs. These architectures and approaches are motivated by finding extreme weather patterns;
a meaningful and important problem for society. Thus far, the climate science community has used hand-engineered criteria to characterize patterns. Our results indicate that there is much promise in considered deep learning based approaches. In addition, this work represents our first attempt at applying semi-supervised learning and exploratory data analysis (i.e. clustering in a learned representation) to climate data. In the future, we think it would be interesting to delve deeper into interpreting and visualizing what the network has learned. This would not only help open the ”black box” of deep learning for skeptics in the climate community, but also potentially help resolve ambiguities in the definition of extreme weather events in addition to highlighting unknown underlying factors in extreme weather events. On a similar note, another goal would be to be able to segment every pixel/voxel in the multi-variate spatio-temporal volume as belonging to a particular weather pattern. Once we are able to decompose the climate system along various patterns, we will be able to conduct sophisticated conditional analysis: how much of global precipitation can be attributed to various patterns (cyclones vs. atmospheric rivers vs. fronts). Moreover, we think it would be interesting to apply the models learned from this CAM5 model output to satellite, reanalysis and other simulation model output. However, these datasets have different spatial resolution, variables, and missing values (in the case of satellite observations). Lastly, applying this extreme weather detection framework to simulations of years in the future would be a very informative exercise for the climate community and the world.
ACKNOWLEDGMENTS
This research used resources of the National Energy Research Scientific Computing Center (NERSC), a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. Code relies on open-source deep learning frameworks Theano (Bergstra et al.; Team et al., 2016) and Lasagne (Team, 2016), whose developers we gratefully acknowledge. We would also like to thank Yunjie Liu and Michael Wehner for providing access to the climate datasets; Alex Lamb and Thorsten Kurth for helpful discussions; and Samsung for their support of this research.

Review:
[EDIT: The thoughtful author responses addressed my major concerns. The github links for data and code will be really helpful for reproducing results (I haven't looked carefully, but this is great). The revision addressed many issues, including the additional results. As such I am upgrading my rating from a 5 to a 6 and recommend acceptance of the paper.]
The paper proposes to apply deep nets to perform detection and localization of extreme weather events in simulated weather data. The problem is related to object detection in computer vision in that the input is a 2D “image” (multichannel spatial weather data) or 3D “video” (temporal version of the data) and the output is a bounding box (spatial-temporal localization of a weather event) and class label (weather event type). It differs from standard object detection in that the input has multiple heterogenous channels and labeled data is scarce.
A simple but quite reasonable deep net is proposed for the task based on similar approaches in computer vision. While proposal based systems are most popular in vision currently (in particular Faster-RCNN) the proposed approach is simple and a fine starting point. There is little innovation on the part of the detection system, but as noted, it is a valid application of ideas from computer vision to the task at hand. 
The authors propose both a supervised approach (only ground truth bounding box location/label is used) and a semi-supervised approach that additionally incorporates the reconstruction loss as a regularization. In all cases the losses are fairly standard and again, reasonable. The only confusing bit is that the “semi-supervised” loss actually has all the labels used for the “supervised” loss and additionally incorporates the reconstruction loss. Hence, the “semi-supervised” loss is actually stronger, which makes the terminology a bit confusing.
The paper is easy to follow, but notation is sloppy. For example, above equation 5 it states that “the loss is a weighted combination of reconstruction error and bounding box regression loss”; actually it’s a combination of the supervised and unsupervised loss (Lsup and Lunsup), and Lrec is not defined (although I assume Lrec=Lunsup). The paper is fairly non-technical, but nevertheless these minor issues should be fixed. (E.g., see also reference to “figure 4 and 4”). 
The biggest concern w the paper though is experimental results. Only a single figure and table of results are shown (figure 4 and table 4). The metrics are not defined (what is mean average recall?). Only 2D versus 3D version of the model are shown, and supervised and semi-supervised. Moreover, numbers seem a bit all over the place, without consistent patterns (e.g., why is 2D supervised better than the seemingly much strong 3D semi-supervised?). One of the things that is unclear is how many events are actually in the training/testing data, and more importantly, how good are these results in absolute terms? Regardless, the experiments are fairly sparse and ablation studies and more discussion lacking. It is also unclear if future researchers will be able to reproduce the experimental setting (a commitment to open-source the data or a way to reproduce the experiments would be critical for future authors).
Minor nit: the authors use both a classification loss and an “objectness” loss. I’ve never seen both used together like this (normally objectness is used in two-stage object proposal systems where in the first stage class-agnostic proposals are given and in the second stage these are cropped and a class-specific classifier is applied). I strongly suspect removing the objectness loss would not impact results since the classification loss should provide strictly stronger supervisory signal. Regardless, this is a fairly non-standard choice and should be justified (experimentally).
Overall this is a borderline paper. I do believe that it is valuable to apply computer vision techniques to a domain that I’ve see little work on in our community. That being said, I have no expertise on this type of data -- it’s possible this deep learning techniques are now routinely used in the climate science literature (I suspect not, though). Overall, there is little novelty on the algorithmic side in this paper (the equations in section 3 are commonly used in the cv literature). The use of reconstruction loss to improve results in the data-sparse setting is interesting, but the experimental results are inconclusive. The experimental validation is generally insufficient. Reproducibility for future research is difficult unless the data is open-sourced. Overall, I think this paper is a good start, and with improved experiments and more careful writing I think could eventually make for a decent paper.