S OFT L OC : R OBUST T EMPORAL L OCALIZATION UNDER L ABEL M ISALIGNMENT
Anonymous authors
Paper under double-blind review
A BSTRACT
This work addresses the long-standing problem of robust event localization in the presence of temporally misaligned labels in the training data.
We propose a novel versatile loss function that generalizes a number of training regimes from standard fully-supervised cross-entropy to count-based weakly-supervised learning.
Unlike classical models which are constrained to strictly ﬁt the annotations during training, our soft localization learning approach relaxes the reliance on the exact position of labels instead.
Training with this new loss function exhibits strong robustness to temporal misalignment of labels, thus alleviating the burden of precise annotation of temporal sequences.
We demonstrate state-of-the-art performance against standard benchmarks in a number of challenging experiments and further show that robustness to label noise is not achieved at the expense of raw performance
I NTRODUCTION
The surge of deep neural networks (LeCun et al., 2015; Schmidhuber, 2015) has accentuated the evergrowing need for large corpora of data (Banko & Brill, 2001; Halevy et al., 2009).
The main bottleneck for the efﬁcient creation of datasets remains the annotation process.
Over the years, while new labeling paradigms have emerged to alleviate this issue (e.g., crowdsourcing (Deng et al., 2009) or external information sources (Abu-El-Haija et al., 2016)), these methods have also highlighted, and emphasized, the prevalence of label noise .
Deep neural networks are unfortunately not immune to these perturbations as their intrinsic ability to memorize and learn label noise (Zhang et al., 2017) can be the cause of training robustness issues and poor generalization performance.
In this context, the development of models robust to label noise is essential.
This work tackles the problem of precise temporal localization of events (i.e., determining when and which events occur) in sequential data (e.g. time series, video or audio sequences) despite only having access to poorly aligned annotations for training (see Figure 1).
This task is characterized by the discrepency between the precision required of the predictions during inference and the noisiness of the training labels. Indeed, while models are trained on inaccurate data, they are evaluated on their ability to predict event occurences as precisely as possible with respect to the ground-truth. In such a setting, effective models have to infer event locations more accurately than the labels they relied on for training. This requirement is particularly challenging for most classical approaches that are designed to learn localization by strictly mimicking the provided annotations.
Indeed, as the training labels themselves do not accurately reﬂect the event location, focusing on replicating these unreliable patterns is incompatible with the overall objective of learning the actual ground-truth. These challenges highlight the need for more relaxed learning approaches that are less dependent on the exact location of labels for training. The presence of temporal noise in localization tasks is ubiquitous given the continuous nature of the perturbation, in contrast to classiﬁcation noise where only a fraction of the samples are misclassiﬁed.
Temporal labeling is further characterized by an inevitable trade-off between annotation precision and time investment.
For instance, while a coarse manual transcription of a minute of complex piano music might be achieved within a moderate time frame, a millisecond precision requirement — a common assumption for deep learning models — signiﬁcantly increases the annotation burden.
In this respect, models alleviating the need for costly annotations are key for a wide and efﬁcient deployment of deep learning models in temporal localization applications.
This work introduces a novel model-agnostic loss function that relaxes the reliance of the learning process on the exact temporal location of the annotations.
This softer learning approach inherently makes the model more robust to temporally misaligned labels.
Contributions
This work: a) proposes a novel loss function for robust temporal localization under label misalignment, b) presents a succinct analysis of the loss’ properties, c) evaluates the robustness of state-of-the-art localization models to label misalignment, and d) demonstrates the effectiveness of the proposed approach in various experiments.
P ROBLEM F ORMULATION
The main assumption of this work is the instantaneous nature (i.e., lasting only one time-step in discrete time settings) of the events of interest.
(Durations can nevertheless be modelled in such a framework by labeling the beginning and end of each event class as two separate channels.)
Thus, for each sample, the ground-truth T G :=
{ ( t m ,c m ) | m ≤ M } consists of M event occurrences each deﬁned by its exact timestamp ( t m ∈ R ≥ 0 ) and its class ( c m ∈ [1 ,...,d ] , with d event classes ) .
In this work, temporal label misalignment is then modelled by adding perturbations to the ground-truth timestamps:
[Equation]
(cid:15) [Equation] Although commonly deﬁned as a normal distribution N (0 , σ 2 i ) (see experiments in Section 5.1 and 5.2), the noise distribution E can also represent a wider range of perturbations (see experiments in
Sections 5.2 and 5.3).
The training dataset D := (cid:8)(cid:0)
X i , T Li (cid:1) :
0 < i ≤ N (cid:9) is comprised of N pairs with model input ( X i ) and misaligned labels ( T Li ) .
The aim of this work is the following:
Objective Estimate the true event occurrence times T G of an unseen input sequence X using only the noisy data D for training.
From a practical standpoint, although not necessary for the use of our loss, time is generally discretized.
In such a discrete setting, each predictor X i of the training data D := { ( X i , Y i ) :
<
i ≤ N } is an observable temporal sequence of length T i (i.e., X i =
( x i ( t ))
T i t =1 ∈ IR T i × λ ) such as a DNNlearned representation, a spectrogram or any other λ -dimensional time-series. The label sequence
Y i =
( y i ( t ))
T
i t =1 ∈ { 0 , 1 } T i × d is then the discrete equivalent of T L .
(Note that this last statement assumes that only one event per class can occur at each time-step; in cases where this assumption is violated, the use of smaller temporal granularity solves this issue.)
R ELATED W ORKS
Temporal Localization Under Label Misalignment
The literature on temporal noise robustness is limited despite the critical relevance of this issue.
First, Yadati et al.
(2018) propose solutions combining noisy and expert labels; however, unlike our approach, these methods require a sizable clean subset of annotations.
Second, while Adams & Marlin (2017) achieve increased robustness by augmenting simple classiﬁers with an explicit probabilistic model of the noise structures, the effectiveness of the approach on more complex temporal models (e.g., LSTM) still needs to be demonstrated.
Finally, Lea et al.
(2017) perform robust temporal action segmentation by introducing an encoder-decoder architecture.
However, the coarse temporal encoding comes at the expense of ﬁner-grained temporal information, which is essential for the precise localization of short events (e.g., drum hits).
In this paper, rather than a new architecture, we propose a novel and ﬂexible loss function — agnostic to the underlying network — which allows the robust training of temporal localization networks even in the presence of extensive label misalignment.
Classical Heuristic Our approach is closely linked to the more classical trick of label smoothing or target smearing (e.g., applying a ˜ σ 2 -Gaussian ﬁlter Φ ˜ σ 2 to the labels) which has been considered to increase robustness to temporal misalignment of annotations (Schlüter & Böck, 2014; Hawthorne et al., 2017).
This slight modiﬁcation of the input data converts the original point prediction problem into a distribution prediction problem.
Indeed, the smoothing of the labels transforms the point labels into distributions.
The algorithm is then trained to predict these distributions, which eventually have to be transformed back to point predictions using hand-crafted peak picking heuristics (see Figure 3 ( left )). This methodology is also very common in 2D image keypoint detection applications which deal with spatial uncertainty, e.g. human pose estimation (Tompson et al., 2014; 2015) or facial landmark detection (Merget et al., 2018). However, despite its intuitive nature, this traditional solution presents several inherent drawbacks (see Figure 2): (Issue 1) Even in a noise-free setting, by transforming the impulse-like target into a distribution, the optimal model predictions (with respect to the training loss) differs from the actual goal of the pipeline (i.e., precise localization indicated by the original event label).
(Issue 2)
As the model learns by mimicking the smoothed target throughout the learning phase, the predictions themselves will be spread out over several time-steps.
Hence, additional tailored heuristics, such as peak picking (Böck et al., 2013) or complex thresholding, are required to achieve precise temporal localization.
(Issue 3)
Even advanced peak picking approaches struggle to disentangle close events.
For instance, a unique maximum might emergence in the middle of two events, thus signiﬁcantly disturbing the timeliness of the ﬁnal predictions.
(Issue 4)
Having the label mass dispersed temporally both before and after the event occurrences is problematic not only for causal models (i.e., models that make predictions at time t only with data up to time t − 1 ) but also for one-sided recurrent networks and fully convolutional architectures with limited receptive ﬁelds.
Indeed, all these models have to estimate the left tail of the label distribution before even seeing the event occur.
This requirement compels the model to ﬁnd some structure before the actual event occurrence, leading to poor generalization performance.
Although bidirectional networks do not suffer from it, this issue limits the range of possible architectures.
The presence of strong label misalignment further worsens these four issues as increased noise commonly warrants increased smoothing, dispersing the label (and consequently prediction) mass even more.
Overall, experimental evidence (e.g. Section 5.1.1) shows that the accumulation of these issues proves to be very detrimental to the noise robustness of these classical approaches.
In contrast, this work presents a novel paradigm for dealing with temporal label uncertainty.
The main idea consists in directly inferring point predictions rather than resorting to distributions or heatmaps by fully integrating the modelling of the noise into the loss function.
Such a direct approach allows for an end-to-end learning of localization without the need for additional hand-crafted components.
In addition, the systematic and standalone loss function proposed in this regard (Section 4.2) not only solves all the above-mentioned issues but also scales well to extensive label misalignment.
Weakly-Supervised Learning Some weakly-supervised models leverage weaker annotations to infer more ﬁne-grained concepts.
In such frameworks, noisy labels are implicitly bypassed by the use of higher-level labels — which are more invariant to perturbations.
For instance, some works achieve object detection (Fergus et al., 2003; Bilen & Vedaldi, 2016) or temporal localization (Kumar & Raj, {+
 ;+} Wang et , 2017) using only
However labels even noisy ones, often contain some additional information that is essential for optimal performance
S OFT L OC M ODEL 4. S OFT L OCALIZATION L EARNING L OSS
The general principle of relaxing the localization learning is intuitive and potentially powerful if carefully implemented. However, by smoothing the label only , classical approaches transform the original point prediction problem into a distribution prediction problem which eventually causes issues (see Section 3).
Many of the drawbacks arising from the asymmetric nature of the one-sided smoothing can however be alleviated by ﬁltering not only the labels (i.e., Φ S i 2 ∗ Y i ( · ) ), but also the predictions (i.e., Φ S i 2 ∗ ˆY i ( · ) ) with a unique softness parameter S i .
The comparison of these two smoothed processes yields a relaxed loss function for the soft learning of the location that deals on its own with the temporal uncertainty of the labels.
Indeed, in such a setting, the model is given input sequences of point-like events and directly infers point predictions without having to resort to distributions or heatmaps (see Figure 3 ( right )); it is only the loss function that views these point labels and predictions as smoothed processes.
In discrete time settings, the loss can be written as
[Equation]
where L ( · , · ) can be any measure of distance.
The learning is characterized as soft since the loss is not strictly constraining in terms of precision or mass concentration.
Indeed, the mass of each event can be both scattered over numerous time-steps and slightly shifted temporally without any abrupt increase in loss.
Thus, the model’s reliance on exact label locations is relaxed.
Measure L In noise-free settings, the average stepwise cross-entropy is a common choice of loss function for state-of-the-art models (Lea et al., 2017; Wu et al., 2018; Hawthorne et al., 2019).
While a potentially unbounded penalization of false predictions might be ideal when training on clean datasets, such behavior can be highly detrimental when labels are subject to temporal misalignment.
Therefore, for all experiments in Section 5, L is set to the (bounded) average local mean-squared error.
Properties Symmetrically smoothing both the labels and predictions solves several of the issues highlighted in the previous section (see Figure 4).
First, in a noise-free setting, the optimal predictions with respect to L SLL are the original annotations themselves. (Solves 1) .
Second, since the predictions are also smoothed over time, each trigger adds detection mass not only after, but also before the prediction time.
Therefore, the model is not required the estimate the left-tail of the label distribution before the actual event occurrence (Solves 4) .
The prediction mass for a particular event is not necessarily dispersed over time anymore.
For instance, in noise-free settings, the point-like targets themselves are the solution to the optimization problem.
However, L SLL does not strictly constrain the mass of each event to be contained in a single time-step (Partially Solves 2 & 3) .
4. S OFT L OC L OSS
The potential dispersion of the prediction mass and its direct consequences on localization performance still need to be addressed.
To that end, we propose to leverage the properties of the weaklysupervised model deﬁned in (Schroeter et al., 2019), which achieves precise temporal localization using only occurrence counts for training.
Aside from exhibiting strong localization performance, the loss introduced in that work possesses an implicit mass convergence property, which concentrates the scattered prediction mass toward well-deﬁned single points in time:
[Equation]
where F is the set of all subsets of { 1 , 2 ,...,T i } of size (cid:80)
k
y i ( k ) .
Full S oftLoc Model
Incorporating this mass convergence loss as a regularizer to our soft localization learning loss L SLL allows the model to directly achieve precise impulse-like localization, without weakening its noise robustness properties. Thus, this eliminates prediction ambiguity, as only a single point prediction is outputted per event occurrence (Solves 2 & 3) .
Overall, when trained with the S oftLoc loss,
[Equation]
the model simultaneously softly learns to mimic the localization annotation, while converging the scatter mass toward impulse-like predictions.
In this equation, α τ regulates the predominance of the mass convergence against the soft learning (for training iteration τ ).
From a practical standpoint, starting with a moderate α τ allows an initial relaxed localization learning, before performing stronger mass convergence (see Section 5 for the speciﬁc settings used in this paper).
End-to-end Learning of Localization One of the key factors of the predominance of the deep learning models over classical ones relies on their ability to solve problems in an end-to-end fashion (Collobert et al., 2011; Krizhevsky et al., 2012), without the need to resort to partial optimization or hand-crafted heuristics.
In contrast to more classical approach (see Section 3), our proposed method is an end-to-end solution to the problem of temporal localization in the presence of misaligned labels (see Issue 2).
This solution eliminates the need for hand-crafted components (e.g. peak picking) and is expected to better serve the task at hand.
Continuous Setting While all experiments in Section 5 and most state-of-the-art temporal localization models perform a discretization of time, the loss deﬁnition can easily be adapted to suit continuous-time frameworks.
4. G ENERALIZATION OF P AST W ORKS
Our versatile S oftLoc model is a generalization of several past works.
Indeed, depending on the softness parameter S M , the model encompasses a wide range of training regimes from classical fully-supervised to count-based weakly-supervised.
Softness → 0 → 0 →
By tending S M toward zero, the model becomes similar to a count-aware localization
RNN with soft localization learning loss.
For instance, setting L ( · ) =
− log(1 − |·| ) yields
[Equation]
which corresponds to the sum of all stepwise cross-entropies.
By further setting α τ = 0 (i.e., discarding any count-awareness), our loss function becomes identical to the ones found in numerous temporal detection works (e.g., drum detection (Wu et al., 2018), piano onset detection (Hawthorne et al., 2017), and video action segmentation (Lea et al., 2017)).
→ ∞ Setting S M → ∞ causes the gradient of L SLL ( θ ) to vanish, discarding any prior information of localization, thus making the training weakly-supervised (Schroeter et al., 2019):
[Equation]
4. D EALING WITH U NCERTAINTIES
The introduced softness parameter can be leveraged to deal with different kinds of uncertainties.
First, in contrast to the traditional approach of aggregating the annotations of multiple individuals (thus trading off dataset richness for noise reduction), our model can be trained on all conﬂicting individual sequences, since it can cope with noisy annotations.
Second, an annotator speciﬁc softness S a 2 can further be implemented to model their respective reliability.
Finally, an extract speciﬁc softness can be incorporated to capture the noise or annotation complexity of certain more challenging sequences.
Experiments conducted in the section below show that the performance is robust to variations in the softness parameter.
Indeed, this hyperparameter only acts as a coarse indicator of temporal uncertainty and thus does not need to strictly match the underlying noise distribution.
E XPERIMENTS
In this section, we demonstrate the effectiveness and ﬂexibility of our approach in a broad range of challenging experiments (music event detection, times series detection, video action segmentation). ,
Experiment and implementation details can be found on the paper’s website 1 .
5. M USIC E XPERIMENTS 5.1. P IANO O NSET E XPERIMENT
Piano transcription and more speciﬁcally piano onset detection is a difﬁcult problem as it requires precise and simultaneous detection of hits from 88 different polyphonic channels.
Dataset This experiment is based on the MAPS database (Emiya et al., 2010).
The dataset creation protocol strictly follows the one from Hawthorne et al.
(Only onsets are considered for the comparison.)
To evaluate the robustness, the training labels are artiﬁcially perturbed according to a normal distribution (cid:15) m ∼ N (0 , σ 2 ) , while the test labels are kept intact for unbiased evaluation.
Benchmarks Three different benchmarks are considered.
First, the state-of-the-art model (on clean data) proposed by Hawthorne et al.
(2017) is highly representative of models aiming for optimal performance with little regard for annotation noise ( Hawthorne ).
Second, a smoothed version of the ﬁrst benchmark with extended onset length (i.e., over 96ms ) illustrates the common practice used to achieve robustness ( Hawthorne (smoothed) ).
Finally, as the ﬁrst benchmark performs local classiﬁcation using standard cross-entropy, the soft bootstrapping loss proposed by Reed et al.
(2014) is leveraged instead for increased robustness ( Bootstrap (soft) ).
Architecture, Training and Evaluation Our network is comprised of six convolutional layers (representation learning) followed by a 128-unit LSTM (temporal dependencies learning) and two fully-connected layers (prediction mapping).
The network is trained using mel-spectrograms (Stevens et al., 1937) and their ﬁrst derivatives stacked together as model input, while data augmentation in the form of sample rate variations is applied for increased robustness and performance.
The loss (Equation 4) with softness S M = 100ms is optimized using the Adam algorithm (Kingma & Ba, 2015).
The models are evaluated on the noise-free test set using F 1 -scores computed with the standard mir_eval library (Raffel et al.) and a 50ms tolerance (Hawthorne et al., 2017).
( α τ = max(min( τ − 10 5 10 5 ,. 9) ,. 2) .)
Results As depicted in Figure 5, our proposed S oftLoc approach displays strong robustness against label misalignment; in contrast to all benchmarks, the performance appears almost invariant to the noise level.
(See Appendix A.1 for discussion on the model’s performance for σ > 200ms .)
At σ = 150ms , only 26% of training labels lie within the 50ms tolerance.
In this context, the score achieved by our S oftLoc model (i.e., ∼ 75% ) is unattainable for classical approaches, which do not take label uncertainty into account and attempt to strictly ﬁt the noisy annotations.
While standard tricks, such as label smoothing, slightly improve noise robustness (e.g., Hawthorne (smoothed) ), their effectiveness is limited in contrast to our proposed approach.
Finally, the parameters used throughout this experiment are ﬁxed.
However, as our loss is a strict generalization of the standard cross-entropy loss used by Hawthorne et al.
(2017), the small performance gap for small noise levels can be reduced by setting α τ = 1 , S M 2 → 0ms and L ( · )
= − log(1 − |·| ) .
Ablation Study To assess the usefulness of the different components of L S oftLoc , we repeat the above experiments keeping only individual parts of the loss function.
Table 1 reveals that L SLL is the main driver of performance in noise-free settings, while L MC ensures stability under increased label misalignment.
(A simple threshold-based peak-picking algorithm was implemented to infer localization from the dispersed mass produced by L SLL .)
Overall, while each loss individually produces reasonable predictions, only the combined L S oftLoc yields both competitive scores in noise-free settings and strong robustness to temporal misalignment.
5.1. D RUM D ETECTION E XPERIMENT
The softness S M is a deﬁning model hyperparameter.
In this section, 210 independent runs for the same drum detection experiment are conducted with varying noise and softness levels in order to highlight the correlation between this key parameter, label noise and the ﬁnal localization performance.
Dataset The experiment is based on the D-DTD Eval Random drum detection task (IDMT-SMTDrums dataset (Dittmar & Gärtner, 2014)) performed by Wu et al.
The goal is the correct [Table] temporal localization of three different classes of drum hits — hi-hats (HH), kick drums (KD), and snare drums (SD) — within a 50ms tolerance window.
Normally distributed errors (cid:15) m ∼ N (0 , σ 2 ) are artiﬁcially introduced on all training and validation labels, while the test labels are kept intact for unbiased inference.
The noise level σ for each run is uniformly sampled from the range [0ms , 100ms] .
Architecture, Training and Evaluation The network is similar to the one in Section 5.1.1, except for the number of ﬁlters and nodes.
The model softness S M for each run is uniformly sampled from [0ms , 150ms] .
Training and evaluation are carried out in the same way as in the piano experiment in
Section 5.1.1.
(Learning rate: 10 − 4 , batch size: 32, iterations: 1 . 5 × 10 5 , sample length: 1 . 5s ) .
Results
The results of the 210 runs are displayed in Figure 6.
A Gaussian Nadaraya–Watson kernel regression (Nadaraya, 1964; Watson, 1964) is used to interpolate the F 1 -score, offering a detailed view of the model’s response to varying label noise levels.
This ﬁgure not only conﬁrms the model’s high robustness to label misalignments, but also reveals that these results are very robust to changes in the softness level.
Indeed, a wide range of softnesses yield optimal performance, as long as S M ≥ σ .
Obviously, extreme softness levels (e.g. S M 2 → ∞ ) would however induce a partial or even total loss of the information conveyed by the localization prior, resulting in a decrease in performance (see Table 2).
Robustness considerations aside, our S oftLoc model displays an outstanding overall performance with F 1 -scores over 95% across all noise levels; the model — even when trained on extremely noisy labels (e.g., σ = 100ms ) — outperforms several standard benchmarks Wu et al.
(2018) which were trained on noise-free training samples ( σ = 0ms ).
Noise-free Comparison In clean settings (i.e., σ = 0ms ) , the benchmark models have a clear advantage as they correctly assume noise-free labels.
Despite this, our S oftLoc model achieves stateof-the-art performance on three different metrics (KD, HH, precision) demonstrating that robustness does not come at the expense of raw localization performance (see Table 2).
5. T IME S ERIES D ETECTION
The timely detection of events in healthcare time series is a crucial challenge to improve medical decision making.
The task tackled in this section consists in the precise temporal detection of smoking episodes using wearable sensors features based on the puffMarker dataset (Saleheen et al., 2015).
Once again, in order to conduct the robustness analysis, the original annotations are artiﬁcially misaligned.
However, as each time-step in this dataset represents a full respiration cycle, the noise distributions must be applied in a discrete fashion: namely, rounded normal distribution (i.e.,
E i ∼ (cid:98)N (0 , σ 2 ) (cid:101) ) or binary constant length shifting of labels ( δ steps either to the left or the right with equal probability), denoted B ( − δ, δ ) .
This task is particularly challenging as detections have to be perfectly aligned with the ground-truth to be considered correct.
Model and Benchmark As the focus is set on robustness rather than raw performance, the model architecture is kept extremely simple: a 14-node fully connected layer followed by a 14-unit LSTM [Table] and a ﬁnal fully connected layer with softmax activation.
Both the standard cross-entropy (CE) and our L S oftLoc loss function are evaluated.
The LR-M model proposed by Adams & Marlin (2017), which was developed to achieve strong robustness to temporal misalignment of labels on this particular dataset, is also considered as benchmark.
Results
The results, produced using ten 6-fold (leave-one-patient-out) cross-validation. are summarized in Table 3.
Not only does training with the proposed L S oftLoc loss function yield a strong improvement in robustness when compared to the standard cross-entropy, but our simple recurrent model also signiﬁcantly outperforms the robust LR-M model on all metrics.
In addition, our approach displays low standard deviations, which underlines the consistency and robustness of the learning.
These observations hold for both noise distributions ( N and B ); hence, the normal smoothing ﬁlters do not require the underlying noise to be normally distributed in order for the model to be effective.
Further testing with skew normal distribution of noise conﬁrm these results even in non-symmetric settings.
5. V IDEO A CTION S EGMENTATION
Video action segmentation — a dense classiﬁcation problem where each time-step has to be mapped to one action class — differs substantially from music event localization or time series detection problems, where scattered events from multiple classes have to be precisely localized.
Nonetheless, the properties of the S oftLoc loss can still be leveraged on such a task; in this context, while the role of L SLL is unchanged, L MC acts as a count-based regularizer, rather than a means for mass convergence.
Experiments Several video segmentation experiments from Lea et al.
(2017) are replicated using either the standard cross-entropy (original loss), L SLL or L S oftLoc as training loss for the EDTCN model.
As the ED-TCN model already exhibits strong robustness properties against label misalignment
Lea et al.
(2017), these experiments will allow to measure the additional marginal gain in performance and robustness when replacing the standard cross-entropy with our the proposed L
S oftLoc loss function.
To assess robustness, each label sequence in the training set is either delayed or advanced by a ﬁxed constant δ .
( S M =7s ) .
[Table]
[Table]
[Table] [Table] [Table] Results As summarized in Table 4 and Table 5 (in Appendix B.2), replacing the standard crossentropy loss with L S oftLoc does not only signiﬁcantly increase the robustness of the ED-TCN model — which was already shown to be robust to label misalignment (Lea et al., 2017) — but also achieves competitive performance in noise-free settings.
Further experiments with different softness parameters (see Figure 10 in Appendix B.1) reveal that increasing the model softness S M as the underlying noise levels increase produces optimal performance.
For instance, in noisy settings, greater performance can be achieved (up to 25% overperformance) by simply choosing a large enough softness.
Overall, the S oftLoc loss function displays strong results on a very different task (i.e., temporal segmentation as opposed to temporal localization), highlighting once again its versatility of application.
C ONCLUSION
In this work, we have shown how relaxing annotation requirements (i.e., weakening the model’s reliance on the exact location of events) not only has the practical beneﬁt of alleviating annotation efforts but, more importantly, leads to a model that is robust to temporal noise without compromising performance on clean training data.
This contrasts with traditional approaches which attempt to strictly mimic the annotations, leading to poor predictions when training with noisy labels.
We have demonstrated these claims on a number of classical challenging tasks, in which our S oftLoc loss exhibits state-of-the-art performance.
The proposed loss function is agnostic to the underlying network and hence can be used as a loss replacement in almost any recurrent architecture.
The versatility of the model can ﬁnd applications in a wide array of tasks, even beyond temporal localization.
A. F URTHER I LLUSTRATIONS
Timeliness of S oftLoc predictions Figure 8 illustrates how consistently precise and well-centered (i.e., neither too late nor early) the predictions are regardless of the noise setting.
Indeed, there is almost no difference in prediction centering when comparing the results for σ = 0ms or σ = 200ms .
Noisy Labels and Ground-Truth Discrepancy To further illustrate the complexity of the localization task when annotations are subject to misalignment, we consider the training labels as predictions and then compare them to the clean ground-truth.
Figure 9 displays an example of the quality of the training labels.
Obviously, in the noise-free setting (i.e., σ = 0ms ), the localization is spotless as the training labels and the ground-truths are identical.
However, as the noise level increases, the proportion of labels that stay within the 50ms tolerance window decreases signiﬁcantly.
More precisely, the performance (i.e., F 1 -score) of the labels themselves is 68 .
8% and 23 .
7% for σ equal to 50ms , 100ms and 200ms respectively.
[Figure]
B V IDEO A CTION S EGMENTATION B. I MPACT OF THE S OFTNESS P ARAMETER
As depicted in Figure 10, training with the S oftLoc loss function instead of the standard cross-entropy yields improved performance (up to 25% ) in all noise settings almost regardless of the softness
S M .
The only exception occurs when selecting a softness level that is too wide while training with noise-free ( δ = 0 ) labels.
As also observed in Section 5.1.2, the model achieves optimal performance when the softness level S M is slightly larger than noise level δ .
However, although the efﬁciency of the approach is bound to decrease when the disparity between selected softness and noise level is becoming too large, a performance close to the optimal one can be achieve with a wide range of softnesses S M .
B. A DDITIONAL R ESULTS
[Table]
L OSS
F 1 @
CE L SLL
L S OFT L OC
F 1 @
CE L SLL
L S OFT L OC
[Table]