Repository: mir-evaluation/mir_eval
Issue: convention for empty intervals
Root cause: Validation Gap

I ran across this curious example in the SALAMI dataset this morning, and it got me thinking about how we should handle semantically questionable (but syntactically valid) inputs.  The example in question comes from `988/parsed/textfile1_functions.txt`, and the offending lines (events) are:

```
321.788979591   Instrumental
321.788979591   Outro
321.788979591   Solo
```

mir_eval happy loads these as events, and following the zipping convention, converts to intervals without a problem:

```
(array([ 321.78897959,  321.78897959]), 'Instrumental'),
(array([ 321.78897959,  321.78897959]), 'Outro'),
(array([ 321.78897959,  397.66927438]), 'Solo'),
```

The problem here is that the event file listed three labels for a single boundary time.  The interval converter slams these together, and makes two length-0 intervals.  On our part, this is incorrect because the assumption of sequential unique timing is not satisfied.

This causes problems in segmentation metrics in a few ways: label equivalence is no longer well defined, and there are artificial boundaries in place.

It seems like we should settle on a convention for dealing with this type of problem, as it may be reasonable to expect similar things in other tasks.  Anyone care to chime in?


To clarify: I think this annotations is syntactically valid at the level of "events".  There's no reason multiple events cannot occur simultaneously.

The difficulty lies in the conversion from events to intervals, which in this case, the data should have been in the first place.  But I think it illustrates an interesting edge case that we need to consistently handle.


Can you think of a case where this would not be considered a corrupt annotation file? It seems like we should warn/throw an error and/or just choose the first listed label for the boundary.


Seems like the consensus is to throw an exception in the interval converter if the time values are not ordered and unique.

@ejhumphrey @justinsalamon do you agree?


+1, makes perfect sense

On Mon, Apr 21, 2014 at 12:49 PM, Brian McFee notifications@github.comwrote:

> Seems like the consensus is to throw an exception in the interval
> converter if the time values are not ordered and unique.
> 
> @ejhumphrey https://github.com/ejhumphrey @justinsalamonhttps://github.com/justinsalamondo you agree?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/49#issuecomment-40951224
> .


Done as of 15ffe030a0a72d830bcc3bd2647e778c4254800d


Hello from Barcelona titans,

So, I've had some "fun" on the plane hacking on my music structure
framework. I think what I've been working on is a bit related to Brian's
problem, so here we go:

By using the latest version of the information gain in mir_eval on the
boundaries of my project, I realized that around 30 files (most of them
from the Isophonics dataset) contain wrong information: a "silence" segment
appears before the outro. E.g. (from
herehttp://isophonics.net/files/annotations/seglab/The%20Beatles/10CD2_-_The_Beatles/CD2_-_11_-_Cry_Baby_Cry.lab
):

(...)
141.638 182.491 outro
182.491 181.867 silence

I believe we should raise a ValueError in the case we find something like
this when evaluating boundaries. Note that I had to run the information
gain measure (from the beats module) in order to find the annotation error.

So, we should make sure in the validate method that no segments are empty
(or contain a negative time!). This would help spotting the problems Brian
describes. Even though it's a good idea to raise the exception in the
conversion, I think we should also have this in the validation function of
the boundaries.

Thoughts?

On Mon, Apr 21, 2014 at 4:05 PM, Brian McFee notifications@github.comwrote:

> Done as of 15ffe03https://github.com/craffel/mir_eval/commit/15ffe030a0a72d830bcc3bd2647e778c4254800d
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/49#issuecomment-40971217
> .


> 182.491 181.867 silence

Yup, I reported this bug on the isophonics website a while back, but got no response.  I ended up fixing the data manually.

You're correct though: the interval validator should raise an exception.  I'll put it in.

> I think we should also have this in the validation function of
> the boundaries.

Actually, the boundary validator already handles this case implicitly.  The validation logic is `boundaries == np.unique(boundaries)`, which handles both de-duping (obviously) and ordering, since `np.unique` sorts the data.


Derp, I was thinking of the event->interval converter, not the metric itself.  At any rate, the validator checks for strictly positive intervals now.


By the way, I believe this error arises from taking manual
annotations, then automatically adding initial silences (from time
zero through the first manual label), and final silence (from last
annotated boundary to duration of file as measured from waveform).

I wish people wouldn't do this, I believe in just giving me the data
from the annotator in the least-modified form.  because, hey, if you
modify it you might screw it up...

  DAn.

On Tue, Apr 22, 2014 at 8:02 PM, urinieto notifications@github.com wrote:

> Hello from Barcelona titans,
> 
> So, I've had some "fun" on the plane hacking on my music structure
> framework. I think what I've been working on is a bit related to Brian's
> problem, so here we go:
> 
> By using the latest version of the information gain in mir_eval on the
> boundaries of my project, I realized that around 30 files (most of them
> from the Isophonics dataset) contain wrong information: a "silence" segment
> appears before the outro. E.g. (from
> herehttp://isophonics.net/files/annotations/seglab/The%20Beatles/10CD2_-_The_Beatles/CD2_-_11_-_Cry_Baby_Cry.lab
> ):
> 
> (...)
> 141.638 182.491 outro
> 182.491 181.867 silence
> 
> I believe we should raise a ValueError in the case we find something like
> this when evaluating boundaries. Note that I had to run the information
> gain measure (from the beats module) in order to find the annotation error.
> 
> So, we should make sure in the validate method that no segments are empty
> (or contain a negative time!). This would help spotting the problems Brian
> describes. Even though it's a good idea to raise the exception in the
> conversion, I think we should also have this in the validation function of
> the boundaries.
> 
> Thoughts?
> 
> On Mon, Apr 21, 2014 at 4:05 PM, Brian McFee
> notifications@github.comwrote:
> 
> > Done as of
> > 15ffe03https://github.com/craffel/mir_eval/commit/15ffe030a0a72d830bcc3bd2647e778c4254800d
> > 
> > —
> > Reply to this email directly or view it on
> > GitHubhttps://github.com/craffel/mir_eval/issues/49#issuecomment-40971217
> > .
> 
> —
> Reply to this email directly or view it on GitHub.