Paper: Deep probabilistic 3D angular regression for directional dark matter detectors (ICLR-2024)

Reviewer comment (Experiments):
Limited Comparison: While the paper compares the proposed method with a non-machine learning algorithm, it lacks a comprehensive comparison against other deep learning-based methods, especially those for predicting uncertainties on manifolds. See, for example [1] and the references therein.

Author response:
We have clarified the relationship to prior work. Please see “Comparison to related work” in our general response.

Reviewer comment (Experiments):
Real-world Evaluation: The paper primarily relies on a simulated test set. Although simulations are valuable, as this is an application-focussed paper, it would be useful to also evaluate the model on real-world data, possibly obtained from actual detectors, to gauge its performance under realistic conditions.

Author response:
Please see the “Real-World Evaluation and the sim-to-real gap” discussion in our general comment.

Reviewer comment (Writing):
I would like the authors to explain why this method is relevant to the machine learning community and also give a better overview of why the problem is important (so that non-particle physiscists can understand its significance).

Author response:
Please see the “Interest to the community / key Novelties” discussion in our general comment.

Reviewer comment (Experiments):
Further analysis of the model's generalizability could be insightful. This will help readers gauge the model's applicability to other problems within the broader particle detection domain.

Author response:
This is a very general approach. In appendix B (previously Section 5), we demonstrate the model on a toy example with arrows in 3D. For the electron recoils, the shape of the ionization track is unique for each sample and highly variable. The shapes also depend heavily on the energy of the electron recoils and the amount of gaussian smearing applied (a parameter of the detector setup), and in our experiments we have varied both of these to show that our model generalizes well.

Reviewer comment (Presentation):
Section 5 probably should be moved to the supplementary, but in any case, it should not be after Section 4.3. Estimating an arrow might be a sanity check but it is irrelevant once the reader has already seen more complex examples.

Author response:
1. We have moved Section 5 into the appendix.

Reviewer comment (Presentation):
What do the colors in Fig. 1 represent? Ionization? A color bar should be present and this should be mentioned in the caption.

Author response:
2. Yes, the color scale represents ionization. A colorbar has been added and mentioned in the caption.

Reviewer comment (Presentation):
Consider replacing Table 1 with a more useful and digestible visual representation, like a block diagram.

Author response:
3. Table 1 has been replaced with a block diagram, Figure 2.

Reviewer comment (Theory):
While the model includes an uncertainty estimate and is therefore heteroscedastic across different trajectories, it should be mentioned that the uncertainty is assumed to be homogeneous along a single trajectory (i.e., all data points within a trajectory have the same uncertainty).

Author response:
4. To clarify, our model predicts the initial direction for a single event, where an event consists of a particle traveling through the detector and causing ionization at many points in 3D space.  
The reviewer might also be referring to the fact that our model represents uncertainty isotropically. This is captured in the statement just above Equation 3, “We simplify Equation 2 to only model uncertainty isotropically about the mean direction by setting \beta = 0.”

Reviewer comment (Presentation):
The term "efficiency cut" is introduced in 4.3 (first paragraph) but only explained later in Fig. 4.

Author response:
5. Thank you for catching this. We updated the sentence to, “... we plot the cosine distance loss versus the percentage of omitted tracks, referred to as the track efficiency cut”

Reviewer comment (Experiments):
Lack of real-world experiments. The effectiveness of the proposed learning-based method is only verified on data from simulation. It also does not explain why the specific choices of simulation parameters, as described in section 4.1, are determined. For example, would performing Gaussian smearing on the simulation data favor the Gaussian-like von Mises-Fisher model, and it is not clear whether the proposed model is still a good choice for real-world data.

Author response:
# Re: Lack of real-world experiments.

Under “Real-World Evaluation and the sim-to-real gap” in our general response, we discuss the scope of our paper.

Please also see “General comments about sim-to-real gap and systematic errors” where we elaborate on the sim-to-real gap.

The specific choice of our simulation parameters is based on Jaegle et al. (2019). We have slightly modified the text on Section 4.1 to emphasize this further. The choice of binning segmentation was deliberately conservative with respect to what is experimentally achievable.

Gaussian smearing, which models the diffusion, is not the only thing influencing the directionality of the track. The multiple scattering that is simulated by Degrad also influences directionality. Furthermore, diffusion is a Gaussian process, so if this process does favor the von Mises-Fisher distribution, then that motivates using the von Mises-Fisher distribution.

Please also see our response “Re: systematic errors” where we demonstrate our model is robust against small fluctuations in the simulation parameters.

Reviewer comment (Novelty):
Technical contribution to the learning community. The 3D sparse convolution and von Mises-Fisher probabilistic model are well-established techniques and do not seem to provide too much insight for the learning community. This work may be better suited for a conference/journal in physics.

Author response:
# Re: Technical contribution to the learning community.

While the von Mises-Fisher distribution is well-established, no other deep learning approaches utilize it in the way we do. We believe this work is well suited for a machine learning conference because

1. Predicting direction is a general problem. 
2. To our knowledge, this is the first deep learning approach which probabilistically predicts 3D direction distributions

Furthermore, our work includes other contributions such such as the approximations needed for stable training and tests of model calibration. Please see “Interest to the community / key Novelties” in our general response comment, where we elaborate on our contribution.

Reviewer comment (Presentation):
Some detailed information in section 3, such as 3.4, would better fit in the experiment section (e.g., section 4.2).

Author response:
# Re: Some detailed information in section 3, such as 3.4, would better fit in the experiment section(e.g., section 4.2).

Thank you, we agree and have moved the text to Section 4.2

Reviewer comment (Experiments):
What considerations are taken to determine the specific simulation parameters used in section 4.1? Why is the test set different from the training set and how are the specific parameters determined?

Author response:
# Re: What considerations are taken to determine the specific simulation parameters are used in section 4.1? Why is the test set different from the training set and how are the specific parameters determined?

The specific simulation parameters are addressed in our response above. 
The model is trained on electron recoil simulations in the 40-50 keV with Gaussian smearing in the  160 − 466 μm range. For Testing, it makes sense to compare performance in specific cases, i.e. low energy versus high energy and low diffusion versus high diffusion. This way, we test our models in cases where it is harder / easier to determine the direction. Aside from targeting specific test cases, our training and testing data is produced in exactly the same way.