Repository: mir-evaluation/mir_eval
Issue: RFC: rounding time-stamps on import and/or adjustment to ground truth?
Root cause: Unimplemented Feature Gap

Should we add a feature to automatically round (segment boundary) timestamps upon import?

This would provide more reliable behavior in `util.adjust_segments` when trimming a prediction file against a ground truth annotation.  For example, a prediction file might list the end-of-track at 201.955, while the ground truth has it at 201.96.  In this case, we probably don't want or need to append a final marker.

Note that the frame size parameter in the various frame clustering metrics already takes care of this, but the effects are somewhat opaque and hidden from the user.  

Moreover, boundary-detection metrics currently just trim the first and last boundaries, assuming that they indicate start- and end- of track.  Rounding and uniq'ing the times upon import would make this behavior a little more reliable.


> Note that the frame size parameter in the various frame clustering metrics
> already takes care of this, but the effects are somewhat opaque and hidden
> from the user.

This talk makes me queasy.  What on earth is going on in there?  Maybe
we really need to do a profound conceptual review to shake out some
long-standing problems with these measures.

Maybe we should make a list of principles - like:
- silent padding at each end of a track doesn't affect anything (but
  what about fade outs?),
- there should be no time frames unless absolutely necessary (I can't
  really think of where it's necessary, although I guess with an
  arbitrary thing like F0, there has to be a sampling rate)
- something about invariance of metric to level of detail (sounds
  hard to achieve) or other secondary aspects of the task (maybe "where
  possible, minimize the dependence on arbitrary aspects of the test
  data" like proportion of most common frame or something).

.. then check our metrics obey them, or fix them until they do.

  DAn.

> Moreover, boundary-detection metrics currently just trim the first and last
> boundaries, assuming that they indicate start- and end- of track. Rounding
> and uniq'ing the times upon import would make this behavior a little more
> reliable.
> 
> —
> Reply to this email directly or view it on GitHub.


> This talk makes me queasy.  What on earth is going on in there?

The frame clustering metrics sample at a fixed rate (default 10Hz) between boundaries and then reduce to standard(ish) clustering evaluation.  Prior to sampling, the boundaries are rounded to 1/sr (default, 1 decimal place) to avoid fractional frame membership.

I'm not a huge fan of the method, but it is what it is.  All I was suggesting is that we may want to round times on import to make things a bit more transparent.

> Maybe we should make a list of principles 

Sure... but then we're in the game of making new metrics, and that's a separate (albeit related) issue


But we're worse than abdicating our responsibilities of leadership if
we work to cement bad metrics.

  DAn.

On Tue, Dec 3, 2013 at 3:19 PM, Brian McFee notifications@github.com wrote:

> This talk makes me queasy. What on earth is going on in there?
> 
> The frame clustering metrics sample at a fixed rate (default 10Hz) between
> boundaries and then reduce to standard(ish) clustering evaluation. Prior to
> sampling, the boundaries are rounded to 1/sr (default, 1 decimal place) to
> avoid fractional frame membership.
> 
> I'm not a huge fan of the method, but it is what it is. All I was suggesting
> is that we may want to round times on import to make things a bit more
> transparent.
> 
> Maybe we should make a list of principles
> 
> Sure... but then we're in the game of making new metrics, and that's a
> separate (albeit related) issue
> 
> —
> Reply to this email directly or view it on GitHub.


There's something to be said for having the best canonical implementation.

Worst case, we can `import warnings` and `warnings.warn('This metric is terrible')` every time it gets called...


> There's something to be said for having the best canonical implementation.

Good point.  It's much easier to sell our better ideas if we can
actually evaluate the thing we're trying to replace and show that it
performs poorly/inconsistently.  To do that right, we need to have it
properly implemented.

  DAn.

> Worst case, we can import warnings and warnings.warn('This metric is
> terrible') every time it gets called...
> 
> —
> Reply to this email directly or view it on GitHub.


I am a fan of this, and I am trusting you to be certain that this will resolve the conflicts sanely with rounding when the frame rate is not an integer multiple by the rounding factor (an edge case).