From: NIPS.cc/2018/Workshop/IRASL
title: Acceptance Decision
decision: Accept

From: NIPS.cc/2018/Workshop/IRASL/Paper61/AnonReviewer2
title: a cool idea to solve a difficult problem; not clear if lambda is tuned on the evaluation set, though
review: The hardest problem in acoustic event detection is that the labels are so arbitrary.  Even when labelers are given a closed set of 12 labels to choose from, they have trouble choosing; if they are allowed to label sounds without restriction, the result is that one very rarely finds any two labelers agreeing on the label for any one sound.  This paper proposes to use a hierarchical ontology to solve the problem: each sound is given two labels, a hypernym, y2, and a hyponym, y1.  The authors seem to foolishly waste space showing that p(y2) = sum p(y1), a fact known to every high school student.... and then they brilliantly show that that the space was not wasted, because that equation specifies a simple and elegant multi-task training architecture.

The result is a reduction in error rate from 31.4% to 26%.  That's a huge difference, but it may not be statistically significant, with only 500 test files; can the authors test that?  More troubling, the description MSoS only mentions "development" versus "evaluation" data, so it's not quite clear if lambda was tuned on the development set or the evaluation set.  The description of US8K doesn't mention any train/dev/eval split at all.  One wishes to give the authors the benefit of the doubt, but these things should be specified.

There is a second method proposed: the representations of objects in the (same,same), (same,different) and (different,different) hypernym,hyponym were trained to have dissimilarities of 0, 1, and 10 (a type of contrastive divergence).  This method gave results better than baseline, but worse than just using the feedforward ontology-based training.  In my own experiments, I've discovered that triplet loss works better than contrastive divergence, so I recommend that the authors should try triplet loss in future work.

rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper61/Authors
title: Thank you for your detailed feedback, it has helped us to improve the quality of our work.
comment: We will add your feedback to our work and below we include some comments to hopefully clarify your concerns.

I have the feeling that treating the ontology-based clustering of the classes in a probabilistic framework is a bit exaggerated: the relation between classes and super-classes is deterministic, so the probabilistic derivation is just trivial. Similarly, the ontological layer is, basically, an average pooling. So, it sounds to me like an over-complication of the problem.
A: Our framework aims to justify and explain the ontological-layer and output layers in the architecture. The ontological layer enforces the defined structure of the ontology, instead of trying to learn something that resembles it. Also, the network has two output layers, one per level, which are part of the loss function. 

However, in my opinion, the weak part is the experimental analysis, whose results do not really support the claims of the authors. The results in fig.5a) are rather jumpy and the gain of the proposed method with respect to the baseline (\lambda=1) is quite in the range of this ripples. So, I am not sure if one can actually claim that the proposed method is bringing an advantage. In addition, in fig.5b) the difference is very marginal and the best score for level2 is for lambda=0.1.
A: Our objective in Figure 5 is to show that there is a lambda value different from zero that will improve the classification performance of level 2. That means that considering both levels in the loss function is better than considering only level 2. 

Finally, the part on Ontology-based embeddings seems a bit stitched together with the rest and, again, experimental results are not conclusive.
A: Our conclusion for the ontology-based embeddings was that although they provide comparable performance to the FF+Ontology architecture, they have the added benefit of the hierarchical grouping. This grouping suggested that the embeddings are learning ontological relations.

- state of the art for SEC: there are several recently published papers on sound event detection and classifications that would have fit this list, in place of ArXiv papers. Some of them present very advanced and effective neural architectures.
A: The SEC system was treated as feature extractor for our purpose, but our approach can work with any other system. In particular, the system we used, demonstrated SoA performance for SEC in recent months in AudioSet, the major SEC dataset.

- experiments of the DCASE challenges would have lifted the quality of the paper as it would have been possible to compare against publicly evaluated baselines (rather than an internal one).
A: DCASE SEC datasets have helped to push the SoA in SEC. In fact, the MSoS Challenge dataset we used is part of the 2018 DCASE. Moreover, the Urban Sounds dataset is one of the top 3 most cited SEC datasets. The most important aspect is that both come with a defined ontology, which is not the case for other DCASE SEC datasets.

- Table 1 reports results for ontology embeddings. This part is not completely clear to me. When I read sec 2.3 I thought that the ontology-embeddings are vectors Z. But if so, it is not clear what FF+Ontology is. At the end, what I understood is that: in one case the network is trained end-to-end, in the other case the vector Z is replaced by the ontology-embeddings. Is it correct?
A: In the FF+Ontology case, audio features and class labels are passed through a feed forward network (Base Net), which outputs the vector Z. In the SNN case, a pair of audio features associated to ontology-class labels are passed through a Siamese Neural Network, which outputs two vectors Z (ontology-based embeddings). The SNN does not add more parameters. The Z vectors are used to compute a distance or similarity metric. Note that in this case, the loss function depends on the two output layers, one per level, but also on the distance between the Z vectors.

From: NIPS.cc/2018/Workshop/IRASL/Paper61/AnonReviewer1
title: Well-written paper on ontology-based training with non-conclusive experimental resutls
review: The authors present an "ontology-based" training of neural networks for sound event detection. Basically, the classes are categorized in super-classes using some similarities and the training loss is a linear combination of the class-wise and super-class-wise losses.
The paper is clear and easy to follow (mostly conversational). I have the feeling that treating the ontology-based clustering of the classes in a probabilistic framework is a bit exaggerated: the relation between classes and super-classes is deterministic, so the probabilistic derivation is just trivial. Similarly, the ontological layer is, basically, an average pooling. So, it sounds to me like an over-complication of the problem.
I, personally, would have tried to go the other way around, learning the super-classes first and then refining the classification in sub-classes. So, I would have liked to see a discussion about this particular choice. Overall, the paper is well-written and sufficiently rigorous.

However, in my opinion, the weak part is the experimental analysis, whose results do not really support the claims of the authors. 
The results in fig.5a) are rather jumpy and the gain of the proposed method with respect to the baseline (\lambda=1) is quite in the range of this ripples. So, I am not sure if one can actually claim that the proposed method is bringing an advantage. In addition, in fig.5b) the difference is very marginal and the best score for level2 is for lambda=0.1.

Finally, the part on Ontology-based embeddings seems a bit stitched together with the rest and, again, experimental results are not conclusive.

Other comments/typos:
- state of the art for SEC: there are several recently published papers on sound event detection and classifications that would have fit this list, in place of ArXiv papers. Some of them present very advanced and effective neural architectures.
- experiments of the DCASE challenges would have lifted the quality of the paper as it would have been possible to compare against publicly evaluated baselines (rather than an internal one).
- in sec. 3.6 there are several typos: "features..described in the next section" (next section is conclusions", "we used the considered the values", "Figure6".
- Table 1 reports results for ontology embeddings. This part is not completely clear to me. When I read sec 2.3 I thought that the ontology-embeddings are vectors Z. But if so, it is not clear what FF+Ontology is. At the end, what I understood is that: in one case the network is trained end-to-end, in the other case the vector Z is replaced by the ontology-embeddings. Is it correct?
In any case, please try to clarify this part, possibly using a more "linear" description.

rating: 2: Marginally below acceptance threshold
confidence: 2: The reviewer is fairly confident that the evaluation is correct

From: NIPS.cc/2018/Workshop/IRASL/Paper61/AnonReviewer3
title: The authors propose a new ontology-based neural network, to suitably deal with the ontological relationship among data. The contribution is interesting and faces an important problem in sound event classification. There are some issues regarding the experimental part of the work, for which the authors are requested to provide more details and clarifications. 
review: The authors propose a new type of neural network (NN), namely ontology-based neural network, characterized by a specific neural layer to better deal with the ontological relationship among data, with application to the sound event classification problem. The authors also describe how to obtain ontology-based embeddings by using Siamese NNs and how to use them for sound event classification tasks.

The work is well-written and well-organized, pretty clear in all its parts. The usage of English is surely satisfactory. The contribution is interesting and addresses an important problem in sound event classification, i.e. the existence of ontological relationships among data and the way to deal with them in order to improve the overall classification performance. 

The experimental tests, carried out by employing two well-known sound event classification datasets comparing the proposed algorithm with standard baselines, seem to confirm the effectiveness of the idea. In particular the ontology-based neural network outperforms the ontology-based embeddings
However, the are some issues to face in this part of the work, as detailed in the following:

1- It seems that no hyper-parameter optimization procedure has been performed in computer simulations and, since the base network architecture is used for both the new ontology-based NN and also for the baseline solution, it would be fairer to consider a pool of neural architectures and training parameters for both technical solutions under comparison.
Moreover, there is a strange sentence in Section 3.6: “We also tuned the hyper-parameters…described in the next section”. First of all there is no section in the paper which these details are provided. Second, no information is given about such hypothetical hyper-parametrization optimization. Third, it is not clear to the reviewer why such an optimization procedure has been performed for the SNN embeddings case study and not for the base NET. Some explanations should be provided on purpose.

2- The results given in Table 1 are the ones corresponding to the optimal \lambda value. The accuracy performance for different \lambda values are reported in Figure 5. This is fine and it allows to show that selecting a proper \lambda value we are able to get better results with respect to the standard methods. However, it would be nice and also more interesting from an application point of view to select the \lambda value to be used on testing by employing a suitable validation dataset, extracted from the development dataset used for training.

3- How and to what extent do the obtained results depend on the NET weight initialization? Have you performed different simulation runs in this sense? If so, also the accuracy standard deviation values should be reported.

4- It would be nice to add some remarks on the conclusive statement in Section 3.6. in which the superiority of FF+Ontology with respect to the Ontology-embeddings approach is claimed. How come? Is there any motivation behind that behaviour?

Some minor changes to apply:
- A paper outline description should be added at the end of Section 1
- Page 2, line 67: “straigh forward” -> “straighforward”
- Plots and writings in Figure 6 are not really readable: please, enlarge the plots and increase the fontsize.


rating: 4: Top 50% of accepted papers, clear accept
confidence: 2: The reviewer is fairly confident that the evaluation is correct