Repository: mir-evaluation/mir_eval
Issue: Segmentation evaluation: 'NCE Over' score delivers wrong result
Root cause: Algorithmic Error

The following code

```
import numpy as np
import mir_eval
ref_intervals = np.asarray([(0,1),(1,2)],dtype=float)
ref_labels = ['A', 'A']
est_intervals = ref_intervals
est_labels = ref_labels
print mir_eval.segment.evaluate(ref_intervals, ref_labels, est_intervals, est_labels)['NCE Over']
```

delivers 0 for the NCE over-segmentation score, although it should be 1.
A realistic example where this happens is SALAMI index 32, for the two annotations given as reference and estimate, respectively.


A similar strangeness occurs for the following (as a variation of above code)

```
# SALAMI index 118
est_intervals = ref_intervals
est_labels = ['A', 'B']
# -> ('NCE Over', 0.0), ('NCE Under', 0.0)
```

Obviously, the cause is the presence of only one kind of labels.


> delivers 0 for the NCE over-segmentation score, although it should be 1.

I agree that it's a little counter-intuitive when the reference and estimates are identical, but NCE is actually undefined in the cases you list above due to a factor of `1/log(1)` (see the original [paper](http://ismir2008.ismir.net/papers/ISMIR2008_219.pdf), section 3).  This is also covered briefly in the mir_eval [paper](http://bmcfee.github.io/papers/ismir2014_mireval.pdf), section 4.6.

We made a conscious decision to differ from MIREX and take the 0 convention in mir_eval.  I'll try to dig up the thread where we hashed this out.


I see, thanks! I am looking forward to reading the arguments. For my taste, this choice is more than a little counter-intuitive for typical segmentation evaluation purposes.


It's not coming up in my search (maybe @craffel can track it down?), but off the top of my head, the main points were as follows:
1. Whatever convention we pick should be consistent between nce_over and nce_under, so that `nce_over(ref, est) = nce_under(est, ref)` (and vice versa).
2. With the 1 convention, it's possible for an estimator to "cheat" by producing an annotation with only one label type, thus ensuring perfect nce_over independent of the reference.  Note that this does _not_ ensure a corresponding 0 score on nce_over, since nce_over would reduce to `1-H(R)/log|R| > 0`.`*` 
   If the reference is highly non-random (and most are), then a cheating estimator can achieve simultaneously large nce_over and nce_under, which is bad news for evaluation.
   
   This suggests that nce_over should take the 0 convention, which prevents that kind of cheating.  If we then adhere to point 1, nce_under should also follow the 0 convention.
3. In light of point 2, the 0 convention is also consistent with the conventions for precision/recall/f-measure in the other modules (beat, onset, melody) when the input annotation is empty.  Because structure label annotations must span the same time extent for reference and estimates, empty estimates get padded up with a 'dummy' segment.  (See the mir_eval paper for a discussion of why padding is necessary.)  If we adopted the 1 convention, then empty annotations would get a score of 1, rather than 0.  This is mainly an aesthetic preference, but I think it's better to err toward consistency unless we have a strong reason not to.
4. It's highly uncommon for the reference annotation to contain only a single label type. (I'd even say degenerate, but we can quibble about annotator disagreement elsewhere.)  It's even more uncommon for an estimator to make that "correct" prediction, so we're probably not hitting that issue very often in the wild.  Even when we are, I'd argue that those tracks should be discarded from the evaluation anyway.  (Have you ever listened to the offending salami tracks?  Some of them are just minutes of crowd noise: hardly relevant for musical structure analysis IMO.)

`*` This, to me, suggests that nce is actually improperly defined, and it would probably have been better to define it as `1 - H(R|E) / H(R)`, which I think corresponds to the `v_measure`/homogeneity/completeness scores (see [here](http://aclweb.org/anthology/D/D07/D07-1043.pdf) and [here](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.completeness_score.html#sklearn.metrics.completeness_score)).  My first cut at the segment submodule included this eval, but we dropped it when removing the dependency on sklearn.  Maybe we ought to revisit that decision.


Many thanks for that, i’ll take more time later to digest the info.
Having one type of labels is not really degenerate - it is so only for a „state approach“, as introduced by Peeters 2004, typically used with clustering algorithms.
With a „sequence approach“, having a succession of same-type labels is totally valid. The difference between the approaches is mainly that the first one sees segments as completely homogeneous (bag of frames), while the second allows segments to contain temporal structure/evolution. Thus, A, A, A would be totally legit, if it is, e.g., the repetition of a phrase. The annotation guidelines define the threshold at which points to split the temporal sequence.


> With a „sequence approach“, having a succession of same-type labels is totally valid.

Fair enough.  Most of the time when I see this in data, it's really just one long segment (eg, "crowd noise"), and not several repetitions of the same pattern (`AAA`), though that does happen.

None of this changes the fact that NCE is ill-posed in that situation, so you probably shouldn't rely on it as an evaluation no matter which `0/0` convention is adopted.


If you spot AAA to be crowd noise, it should really be "Z" and be reported as a bug ;-)


> It's not coming up in my search (maybe @craffel can track it down?)

I'm not finding it either.  This discussion may have happened IRL, so it's good that we are getting it out again here.

> My first cut at the segment submodule included this eval, but we dropped it when removing the dependency on sklearn.

I don't think dropping the sklearn dependency removed any functionality, am I wrong? The commit in question: b0cc4e6


> I don't think dropping the sklearn dependency removed any functionality, am I wrong?

No, but I think we agreed (offline) to purge the v_measure implementation first so we'd have one fewer dependency on sklearn.  (Also, nobody's ever used v_measure for music structure, so it was kind of an oddball anyway.)


It seems like this discussion died off... is there anything to actually do on this issue?

Or shall we just all agree to throw our hands up and admit "NCE considered harmful"?