Repository: mir-evaluation/mir_eval
Issue: RFC: Empty reference and estimated
Root cause: Validation Gap

In all of the metrics (I believe), a special case must be made when the reference annotations are empty and/or the estimated annotations are empty.  In the case where one is empty and the either isn't, I think we can all agree that the resulting metric should be 0 (please disagree if you can think of a good case).  If both are empty, it gets a little fuzzier.  The onset detection metric (f-measure) that I "translated" said that if both are empty (e.g. there are no onsets, and the detector says they are no onsets), then precision, recall, and f-measure are all 1.  In contrast, beat metrics and (all?) segment metrics return 0 if estimated annotations are empty.  I'm kind of comfortable with both of these things; I think I am more comfortable with "If there are no reference annotations and no annotations were estimated, you're perfect".  

Either way, I think all of the validators should throw warnings when things are empty.


In the case of melody extraction, if one of the sequences is empty it means something's wrong (even when there's no melody it should be a sequence of 0's) -- I don't think it makes sense to compute the metrics in such a case, rather the code should raise an exception..


Sure, that's what I stated above.  But what about when both sequences are empty?  E.g., "This file has no melody line, and no melody line was predicted"?  As I also said, throwing a warning at the least is needed.


If both sequences are empty, the scores for pattern discovery are 0 (I will
make sure about it, but they should be 0).

On Thu, Apr 17, 2014 at 6:02 PM, craffel notifications@github.com wrote:

> Sure, that's what I stated above. But what about when both sequences are
> empty? E.g., "This file has no melody line, and no melody line was
> predicted"? As I also said, throwing a warning at the least is needed.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/43#issuecomment-40768215
> .


Why?


Even if the file has no melody line, both the estimation and ground truth should contain sequences of 0's for the duration of the file - the only possible case where the sequences should be completely empty is when the audio file is of length 0... a fairly nonsensical case.


I don't think it makes sense to have a music piece without human annotated
patterns in a pattern discovery database. As an example, SALAMI contains a
few non-musical tracks. But even these tracks contain musical boundary
annotations. I guess the question is: do we allow empty annotations in a
human annotated dataset? If we do, then we should (or at least I should)
change the behavior of some metrics.

On Thu, Apr 17, 2014 at 6:24 PM, justinsalamon notifications@github.comwrote:

> Even if the file has no melody line, both the estimation and ground truth
> should both contain sequences of 0's for the duration of the file - the
> only possible case where the sequences should be completely empty is when
> the audio file is of length 0... a fairly nonsensical case.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/43#issuecomment-40769809
> .


I would argue that there are pieces of music with no beats, patterns, onsets, sections, etc. and that a good algorithm should be able to tell that.  

I think the solution that will make the most people comfortable is to make the metric 0 when either reference or annotated are empty, and throw a warning, not an error (most people don't put their call to eval code in a try/catch structure, and throwing errors pisses people off).


I can live with that :)


Yes, I agree with you Colin. This solution sounds good to me.

On Thu, Apr 17, 2014 at 6:44 PM, justinsalamon notifications@github.comwrote:

> I can live with that :)
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/43#issuecomment-40771158
> .


disclaimer: I don't feel super strongly about this.

My only reservation returning 0 is in the instance that you compute
averages over a collection of files (a la chord rec). Granted, in that
instance you'd probably want to return a weight value, which could also be
0 .... Not that I want to advocate None-checking all over the place, but
part of me would rather it didn't return anything numerical.

just a thought.

On Thu, Apr 17, 2014 at 6:50 PM, urinieto notifications@github.com wrote:

> Yes, I agree with you Colin. This solution sounds good to me.
> 
> On Thu, Apr 17, 2014 at 6:44 PM, justinsalamon notifications@github.comwrote:
> 
> > I can live with that :)
> > 
> > —
> > Reply to this email directly or view it on GitHub<
> > https://github.com/craffel/mir_eval/issues/43#issuecomment-40771158>
> > .
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/43#issuecomment-40771498
> .


None/NaN (because many of these metrics would actually be 0/0)-checking is worse than try-catching.  As long as there's a warning, the user knows about the problematic case and can remove it and re-compute the mean.


I don't follow the divide-by-zero bit, but I was thinking more in C++
space, where I'd want to return a null-pointer for erroneous results. but
like I said, I don't feel strongly / I think this is a good solution.

On Thu, Apr 17, 2014 at 6:59 PM, craffel notifications@github.com wrote:

> None/NaN (because many of these metrics would actually be 0/0)-checking is
> worse than try-catching. As long as there's a warning, the user knows about
> the problematic case and can remove it and re-compute the mean.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/43#issuecomment-40772136
> .


For example, if you have 0 true positives and 0 false positives, precision = 0/0


> I don't follow the divide-by-zero bit, but I was thinking more in C++
> space, where I'd want to return a null-pointer for erroneous results.

noooooooooooo this is what exceptions are for!


ya ya, I know EAFP vs LBYL ... i'm all sorts of crossed up.

On Thu, Apr 17, 2014 at 9:31 PM, Brian McFee notifications@github.comwrote:

> I don't follow the divide-by-zero bit, but I was thinking more in C++
> space, where I'd want to return a null-pointer for erroneous results.
> 
> noooooooooooo this is what exceptions are for!
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/43#issuecomment-40779892
> .


Created issue #44 for implementing this.