Sound event classiﬁcation using ontology-based neural networks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
State of the art sound event classiﬁcation relies in neural networks to learn theassociations between class labels and audio recordings within a dataset.
Thesedatasets typically deﬁne an ontology to create a structure that relates these soundclasses with more abstract super classes.
Hence, the ontology serves a source ofdomain knowledge representation of sounds.
However, the ontology information israrely considered, and specially under explored to model neural network architectures.
We propose ontology-based neural network architectures for sound eventclassiﬁcation.
We deﬁned a framework to design simple network architectures thatpreserve an ontological structure.
The networks are trained and evaluated usingtwo of the most common sound event classiﬁcation datasets.
Results show animprovement in accuracy demonstrating the beneﬁts of the ontology.
Introduction
Humans can identify a large number of sounds in their environments e.g., a baby crying , a wailingambulance siren , microwave bell .
These sounds can be related to more abstract categories thataid interpretation e.g., humans , emergency vehicles , home .
These relations and structures canbe represented by ontologies [1], which are deﬁned for most of the available datasets for soundevent classiﬁcation (SEC).
However, sound event classiﬁcation rarely exploits this already availableinformation.
Moreover, although neural networks are the state of the art for SEC [2–4], they arerarely designed considering such ontologies.
An ontology is a formal representation of domain knowledge through categories and relationshipsthat can provide structure to the training data and the neural network architecture.
The most commontype of ontologies are based on abstraction hierarchies deﬁned by linguistics, where a super categoryrepresents its subcategories.
Generally, the taxonomies are deﬁned by either nouns or verbs e.g.,animal contains dog and cat , dog contains dog barking and dog howling .
Examples of datasets are
ESC-50 [5], UrbanSounds [6], DCASE [7], AudioSet [8].
Another taxonomy can be deﬁned byinteractions between objects and materials, actions and descriptors e.g., contains Scraping , whichcontains Scraping Rapidly and Scraping a Board [9–11].
Another example of this type is givenby physical properties, such as frequency and time patterns [12–14].
There are multiple beneﬁtsof considering hierarchical relations in sound event classiﬁers.
They can allow the classiﬁer toback-off to more general categories when encountering ambiguity among subcategories.
They candisambiguate classes that are acoustically similar, but not semantically.
They can be used as domainknowledge to model neural networks.
In fact, ontological information has been evaluated in computervision [15] and music [16], but has rarely been used for sound event classiﬁcation.
Ontology-based network architectures have showed improvement in performance along with otherbeneﬁts.
Authors in [17] proposed an ontology-based deep restricted Boltzmann machine for textualtopic classiﬁcation.
The architecture replicates the tree-like structure adding intermediate layers to
[Figure]
model the transformation from a superclass to its subclasses.
Authors showed improved performanceand reduced overﬁtting in training data.
Another example used a perceptron for each node of thehierarchy, which classiﬁed whether an image corresponded to such class or not [18].
Authors showedan improvement in performance due to the ability of class disambiguation by comparing predictionsof classes and subclasses.
Motivated by these approaches and by the ﬂexibility to adapt structures ina deep learning model we propose our ontology-based networks detailed in the following section.
Methods
In this section we present a framework to deal with ontological information using deep learningarchitectures.
First, we describe a set of assumptions we consider along this paper.
In particular,we describe the type of ontologies we work and some of their implications.
Later, we present a
Feed-forward model that includes the discussed constraints, deﬁning our proposed ontological layer.
Second, in order to ﬁnd more discriminative features, which are consistent with the ontologicalstructure, we extended the learning model to compute ontology-based embeddings using Siamese
Neural Networks.
2.1 Framework and Assumptions
The framework is deﬁned to make use of the ontology structure and to model the neural networkarchitectures.
It should be noted that we considered ontologies with two levels, which are the mostcommon in sound event datasets.
Nevertheless, the presented framework can be easily generalized tomore levels.
In our framework, we considered the training data { ( x 1 , y 1 ) , ..., ( x n , y n ) } , where x i ∈
X is an audiorepresentation, which is associated to a set of labels given by the ontology y i ∈
C 1 × C 2 × ...
× C k .
In this case, C i is the set of possible classes at i -level.
Assuming a hierarchical relation, we canconsider that each possible class in C i is mapped to one element in C
i +1 .
The higher the value of i ,the higher the level in the ontology.
For an example, consider the illustration of an ontology in Figure 1.
In this case k
= 2 , C 1 = { cat, dog , breathing , eating , sneezing , violin , drums , piano , beep , boing , train , siren } and
C 2 = { nature , human , music , effects , urban } .
As the ﬁgure shows, every element in C 1 is relatedto one element in C 2 ; e.g., cat belongs to nature , or drums belongs to music .
Furthermore, for a given representation x ∈ X , if we know the corresponding label y 1 in C 1 , we caninfer its label in C 2 .
This intuition can be formalized using a probabilistic formulation, where it isstraight forward to see that, assuming p ( y 2 | y 1 , x )
= p ( y 2 | y 1 ) , the following is satisﬁed:
[Equation]
Therefore, if we want to estimate p ( y 2 | x ) using a model, we just need to compute the estimation ofp ( y 1 | x ) and sum the values corresponding to the children of y 2 .
This case is valid for inference time,
[Figure]
however, it is not clear that using the representation and label ( x , y 1 ) should be enough to train themodel.
If at training time we can make use of knowledge to relate the different classes in y 1 , it shouldimprove the performance of the model, specially at making predictions for classes y 2 .
In the following sections we take our proposed framework and use it to design ontology-based neuralnetwork architectures.
2.2 Feed-forward Network with Ontological Layer
In this section, we describe how we use our proposed framework to design the architecture.
Also, weintroduce the ontological layer , which makes use of the ontology structure.
The Feed-forward Network (FFN) with Ontological Layer consists of a base network (Net), anintermediate vector z , and two outputs, one for each ontology level.
The base network weights arelearned at every parameter update.
The base network utilizes an input vector of audio features x andgenerates a vector z .
This vector is used to generate two outputs, p ( y 1 | x ) a probability vector for C 1and p ( y 2 | x ) a probability vector for C 2 .
First, the vector z is passed to a softmax layer of the size of
C 1 .
Second, the same vector z is multiplied by the ontological layer M and generates a layer of sizeof C 2 .
Once the FFN is trained, it can be used to predict any class C 1 and C 2 for any input x .
The ontological layer reﬂects the relations between classes and sub classes given by the ontology.
Todescribe how we used this layer, we refer to Equation 3, where p ( y 2 | x ) is the sum of all the values ofp ( y 1 | x ) corresponding to the children of y 2 .
If we consider this equation as a directed graph where
M is the | C 2 | × | C 1 | incidence matrix, then, it is clear that Equation 3 can we rewritten as,
[Equation]
Note that the ontological layer M deﬁnes the weights of a standard layer connection.
Although wedo not consider that these weights are trainable, they are part of our training data.
In order to train this model, we simply propose to apply gradient-based method to minimize the lossfunction L , which is a convex combination between two categorical cross-entropy functions; L 1 thecategorical cross entropy corresponding to p ( y 1 | x ) and L 2 corresponding to p ( y 2 | x ) .
Formally,
[Equation] [Equation]
Hence, we consider λ ∈ [0 , 1] as a hyper parameter to be tuned.
Note that, when λ = 1 , we arereducing the problem to train a standard classiﬁer just using the information from the ﬁrst level of theontology.
[Figure]
2.3 Ontology-based embeddings
[Figure]
In this section, we describe how we learned the ontology-based embeddings.
The embeddings are computed using a Siamese neural network (SNN), shown in Fig.
3, consistingof twin networks that have the same base architecture (Net) with shared weights.
The weights arelearned simultaneously at every parameter update.
Each base network utilizes an input vector of audiofeatures x .
Then, for the inputs x 1 and x 2 , we can obtain the outpus p ( y 1 | x 1 ) , p ( y 1 | x 2 ) , p ( y 2 | x 1 )and p ( y 2 | x 2 ) .
In addition, we consider as output the similarity metric between the hidden vectors z
1and z 2 as illustrates Fig 3.
For training, ﬁrst, we need to associate a loss function to every output.
The innovation of this modelis the loss for the similarity metric.
Our attempt is that the similarity metric describes somehow theontology; the difference of the embeddings z 1 and z 1 should indicate how different x 1 and x 2 arewith respect to the ontology.
In a 2-level ontology there are 3 possible distances, for this work wechose 0 or 1 or 10 depending on whether the inputs are from the same subclass, or different subclass,but same super class, or different super class.
Hence, the model attempts to approximate the distancewithin the ontology using the distance between embeddings.
In order to train the full model, we need to provide pairs of audio examples and apply a gradientbased method to minimize the loss function L .
We propose to use a linear combination betweenfour categorical cross-entropy functions: L 11 and L 21 the categorical cross-entropy corresponding top ( y 1 | x 1 ) and p ( y 1 | x 2 ) respectively, and L 12 and L 22 corresponding to p ( y 2 | x 1 ) and p ( y 2 | x 2 ) , andﬁnally the similarity metric D w given by Euclidean Distance.
Formally,
[Equation]
Experimental Results
In this section, we evaluate the sound event classiﬁcation performance of the ontological-basedneural network architectures.
We present the datasets and its ontologies, the baseline and proposedarchitectures, and the improvement in classiﬁcation at different levels of the hierarchy.
[Figure]
3.1 Datasets and Ontologies
[Figure]
Making Sense of Sounds Challenge - MSoS:
The dataset is designed for a challenge which objectiveis to classify the most abstract classes or highest level in its taxonomy.
The ontology, illustratedin Fig.
1 has two levels, the lowest level 1, has 97 classes and the highest level 2, has 5 classes.
The audio ﬁles were taken from Freesound data base, the ESC-50 dataset and the Cambridge-MT
Multitrack Download Library.
The development dataset consists of 1500 audio ﬁles divided into theﬁve categories, each containing 300 ﬁles.
The number of different sound types within each categoryis not balanced.
The evaluation dataset consists of 500 audio ﬁles, 100 ﬁles per category.
All ﬁleshave an identical format: single-channel 44.1 kHz, 16-bit .wav ﬁles.
All ﬁles are exactly 5 secondslong, but may feature periods of silence.
Urban Sounds - US8K: The dataset is designed to evaluate classiﬁcation of urban sounds, whichare organized using a taxonomy with more nodes than the annotated number of classes.
Due to thisreason, we adjusted the taxonomy to avoid redundant levels with only one annotated child.
Theresulting ontolgy is illustrated in Fig.
4, with two levels, the lowest level 1, has 10 classes and thehighest level 2, has 4 classes.
The audio ﬁles were taken from Freesound data base and correspondedto real ﬁeld recordings.
All ﬁles have an identical format: single-channel 44.1 kHz, 16-bit .wavﬁles.
The dataset contains 8,732 audio ﬁles divided into 10 stratiﬁed subsets.
All ﬁles are of up to 4seconds in duration.
3.2 Audio Features
We used state-of-the-art Walnet features [2] to represent audio recordings.
For each audio, wecomputed a 128-dimensional logmel-spectrogram vector and transformed it via a convolutionalneural network (CNN) that was trained separately on the balanced set of AudioSet.
The networkcomprised 8 convolutional layers, resulting in an output feature vector of dimensionality 527.
To this,we concatenated intermediate outputs from the 8 th layer of the CNN with dimensionality of 1024.
3.3 Base Network Architecture (Net)
The architecture of the base network (Net) considered in this experiment, shown in Fig.
2, is afeed-forward multi-layer perceptron network.
It consists of 4 layers: the input layer of dimensionality1024, which takes audio feature vectors, 2 dense layers of dimensionality 512 and 256, respectively,and the output layer of dimensionality 128, which is the dimensionality of the vector z .
The denselayers utilize Batch Normalization, a dropout rate of 0 .
5 and the ReLU activation function; max (0 , x ) ,where x is input to the function.
We tuned the parameters in the ‘Net’ box as well as the parametersthat transform z into p ( y 1 | x ) .
3.4 Performance of Baseline Models
We considered baseline models both in level 1 and 2 in different data sets.
In this case, the baselinemodels do not consider any ontological structure, so the models take the Base Network Architectureadding an output layer to either level 1 or 2.
[Figure]
[Table]
Note that for level 1 this is equivalent to train the Feed-forward model with Ontological Layer usingλ = 1 .
Indeed, with λ = 1 the loss function associated to level 2 is not considered.
For level 2, thebaseline model is different from the Feed-forward model with λ = 0 , because in the baseline modelthere is no layer corresponding to the prediction of y 1 .
Table 1 shows the results of baseline modelsfor both MSoS and US8K data set in level 1 and level 2.
The baseline performance in the MSoS challenge was reported to be 0.81 for level 2 and no baselinewas provided for level 1.
3.5 Performance of Feed-forward Model with Ontological layer
To validate the architecture presented in Section 2.2 and analyze the utility of the ontological layer,we trained models taking different values of λ .
Figure 5 shows the effect of λ in both data sets.
Ingeneral, we observe that considering values different from 0 and 1 helps to increase the performance.
Note that the classiﬁcation in both level is affected by the ontological layer.
In the case of MSoS data set, the best performance was obtained using λ = 0 . 8 , getting 74.0% and91.3% of accuracy in level 1 and 2 respectively.
Thus, using the ontological structure we can get anabsolute improvement of 5.4% and 6% respect baseline models.
Running the same experiment on the US8K data set, we observe a smaller improvement.
The bestperformance was obtained using λ = 0 . 7 , being the accuracy of 82.5% and 86.3% for level 1 and 2respectively.
This means an improvement of 2.5% and 0.2% only, respect baseline models.
[Figure]
We tested the architecture described in Section 2.3 to evaluate the performance of the ontology-basedembeddings for sound event classiﬁcation.
Additionally, we include t-SNE plots to illustrate thesuper classes clustered with our embeddings.
We processed the Walnet audio features and chose different super and sub class pairs to train the
Siamese neural network to produce the ontology-based embeddings.
The architecture of the base network (Net) is the same as the one used in the previous section.
We trained the SNN forepochs using the Adam algorithm.
We also tuned the hyper-parameters of the SNN to achieve good performance with the input features that are described in the next section.
We also tried different number of pairs for the input training data, from 100 to 1,000,000 pairs and found that 100, yielded the best performance.
For the loss function we used the considered the values computed in the previous experiment.
We used the value of 0.8 for the λ of the classiﬁers of level 2 and 0.2 for the classiﬁers in level 1, and 0.2 for the similarity metric.
The results in Table 1 show that the accuracy performance of MSoS and US8K were respectively asfollows, in level 1 73.6% and 81.3%, and in level 2 88.6% and 86.1%.
Based on these results wemake the following conclusions.
The performance of this architecture is better than the baseline,but slightly under performs the original method of FF+Ontology.
Nevertheless, the ontology-basedembeddings have the added beneﬁt of better grouping as illustrated in the t-SNE plots of the level 2classes of the MSoS ontology in Figure6.
We also noticed that modifying the lambdas in the lossfunction affected the overall performance.
Conclusions and Future work
In this paper we proposed a framework to learn classiﬁcation task using hierarchical ontologies.
Wehave shown that it is possible to add ontogical information to a deep learning model in a simplemanner, without adding more learnable parameters, but considering a structure between prediction ofdifferent levels in the ontology in the learning phase.
This seems to be fundamental for Sound eventclassiﬁcation, where within a class we can ﬁnd a lot of diverse sounds.
Hence, the proposed modelshelp to deal with the class ambiguity including extra knowledge.
Our experimental results showsthat using the ontological structure it is possible to obtain an improvement of an absolute 5% to 6%percent approximately in different levels of prediction in the MSoS data set.
However, the improvement is smaller in the case of the US8K data set.
We think this is because thenumber of subclasses is comparable with the number of classes; we have 10 subclasses and 4 classes,unlike the case of the MSoS data set, where we have 97 subclasses and 5 classes.
It seems when theratio between the number of subclasses and the number of classes is not large, the contribution of theontology is minor.
Currently, we have many possible directions to continue this work.
One is the study of non-hierarchicalontologies.
To do so, one alternative is to generalize the ontological layers, adding some degreeof freedom to the matrix M but keeping the probabilistic interpretation.
Other alternative is usingdistances on the graph that represents the ontology, similarly to the approach taken in the Ontologybased embedding model, and use those distances as a measure of the relation between classes.
Other questions are about the way we select the parameters λ in these models.
Is there conditiondependening on the number of classes and subclasses that determines the optimal value of λ ?.
Finally, we think necessary to study the extension to multiclass detection, especially for audio, sincemultiple sounds can appear in the same audio recording.
How can we model this type of data into anontological framework?
These and other questions are part of future work