Repository: mir-evaluation/mir_eval
Issue: segment.frame*: gracefully handling end-of-track discrepancies
Root cause: Validation Gap

Given boundary time-stamps for the ground truth and prediction annotations, frame-based metrics operate by generating samples between each pair of boundaries.

Often, we have to compare two annotation sequences that don't line up exactly (due to sampling quantization, etc).  For example, the ground truth may list the final segment boundary at 243.900, while the prediction may list it at 243.85.

I'd argue that we shouldn't assume that both annotations correctly cover the track duration.  The current implementation assumes that the final timestamp in the ground truth is the end of the track.  (This may be correct enough in practice.)  But how should the prediction file be adjusted to match up?  Add a final end-of-track boundary?  This  changes the total number of segments, and may skew the metrics.


In several circumstances, I've seen cases where post-padding with silence
can completely swamp small differences in the actual data. I doubt there's
an easy fix - if you want comparable numbers, you need to make sure they're
all calculated over the same data = identical time ranges, and you don't
really want the interesting part diluted by large "easy" parts. So the eval
data (ground truth reference) should define the duration of each track. I
think it's ok simply to ignore any system outputs outside this range, but
if the system doesn't cover the full range, a default label gets assigned,
which will likely lead to a reduced score.

But this isn't really what you're asking. More responsively:

> Given boundary time-stamps for the ground truth and prediction
> annotations, frame-based metrics operate by generating samples between each
> pair of boundaries.
> 
> I'm not 100% happy with this. Time-based metrics weight errors and correct
> regions by their duration, but there's no actual need for frames or time
> quantization. The calculations can be done on time values of unlimited
> resolution, giving the same results as counting frames with an
> infinitesimal frame size.
> 
> Often, we have to compare two annotation sequences that don't line up
> exactly (due to sampling quantization, etc). For example, the ground truth
> may list the final segment boundary at 243.900, while the prediction may
> list it at 243.85.
> 
> I'd argue that we shouldn't assume that both annotations correctly cover
> the track duration. The current implementation assumes that the final
> timestamp in the ground truth is the end of the track. (This may be correct
> enough in practice.) But how should the prediction file be adjusted to
> match up? Add a final end-of-track boundary? This changes the total number
> of segments, and may skew the metrics.
> 
> All segmentations have an implicit "no sound" segment following the end of
> the track, so we're simply moving the boundary between the last non-empty
> segment and this implicit segment. In a frame-based metric, if you ended
> early, you mislabeled some frames. In a segment-based metric, you may be
> ok.

Adding a segment feels wrong, but modifying all the metrics to respect
implicit "no sound" labels before and after the explicitly labeled portions
may be some work.

  DAn.

—

> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/7
> .


I generally agree with all of DAn's points.  There is still the issue of sampling rate jitter in the timing output of different algorithms, and that's not really something we can control.  The evaluation tools should be robust (ie, not throw exceptions) when an annotation's end-of-track is off by a fractional second from the ground truth.

Here are two possible fixes:
- add a cleaning step to the annotation import function (now, `mir_eval.util.import_segment_boundaries`).  Optionally, you can specify the "true" end-of-track time, and the cleaning step will either pad out with a silence segment if the annotation is too short, or truncate the annotation if it is too long.
- round the boundary markers to the appropriate number of sig-figs (frame_size). This is easy to do for frame_sizes like (0.1, 0.01, 0.001, ...), but gets a little messy in the general case.  Probably this is the preferred route.

It is also true that some of the frame-clustering metrics can be calculated analytically by measuring segment durations and overlaps (without sampling).  However, it does obscure the code, and makes it more difficult to add new clustering-based metrics.  For this kind of thing, I think code readability trumps efficiency.