Repository: mir-evaluation/mir_eval
Issue: Handle corner cases correctly
Root cause: Validation Gap

Currently, there's code for special-case handling of empty annotations (see #44), but not for some other corner cases.

`beat`:
- [x] `cemgil`
- [x] `continuity`
- [x] `f_measure`
- [x] `goto`
- [x] `information_gain`
- [x] `p_score`

``` Python
In [1]: import mir_eval
In [2]: import numpy as np
In [3]: est_events = np.array([10.])
In [4]: ref_events = np.array([10.1])
In [5]: mir_eval.beat.cemgil(ref_events, est_events)
Out[5]: (0.043936933623408392, 0.043936933623408392)

In [6]: mir_eval.beat.continuity(ref_events, est_events)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-9eab6a759553> in <module>()
----> 1 mir_eval.beat.continuity(ref_events, est_events)

mir_eval/mir_eval/beat.pyc in continuity(reference_beats, estimated_beats, continuity_phase_threshold, continuity_period_threshold)
    467             beat_differences = np.abs(estimated_beats[m] - reference_beats)
    468             # Get nearest annotation index
--> 469             nearest = np.argmin(beat_differences)
    470             min_difference = beat_differences[nearest]
    471             # Have we already used this annotation?

/usr/local/lib/python2.7/site-packages/numpy/core/fromnumeric.pyc in argmin(a, axis)
    958     except AttributeError:
    959         return _wrapit(a, 'argmin', axis)
--> 960     return argmin(axis)
    961
    962

ValueError: attempt to get argmin of an empty sequence

In [7]: mir_eval.beat.f_measure(ref_events, est_events)
Out[7]: 0.0

In [8]: mir_eval.beat.goto(ref_events, est_events)
/usr/local/lib/python2.7/site-packages/numpy/core/_methods.py:59: RuntimeWarning: Mean of empty slice.
  warnings.warn("Mean of empty slice.", RuntimeWarning)
/usr/local/lib/python2.7/site-packages/numpy/core/_methods.py:71: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
Out[8]: 0.0

In [9]: mir_eval.beat.information_gain(ref_events, est_events)
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-9-b3ffca63e7a0> in <module>()
----> 1 mir_eval.beat.information_gain(ref_events, est_events)

mir_eval/mir_eval/beat.pyc in information_gain(reference_beats, estimated_beats, bins)
    601     # Get entropy for reference beats->estimated beats
    602     # and estimated beats->reference beats
--> 603     forward_entropy = _get_entropy(reference_beats, estimated_beats, bins)
    604     backward_entropy = _get_entropy(estimated_beats, reference_beats, bins)
    605     # Pick the larger of the entropies

mir_eval/mir_eval/beat.pyc in _get_entropy(reference_beats, estimated_beats, bins)
    641         if closest_beat == 0:
    642             # Inter-annotation interval - space between first two beats
--> 643             interval = .5*(reference_beats[1] - reference_beats[0])
    644         # If last annotation is closest...
    645         if closest_beat == (reference_beats.shape[0] - 1):

IndexError: index 1 is out of bounds for axis 0 with size 1

In [10]: mir_eval.beat.p_score(ref_events, est_events)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-10-c066b3ee7e70> in <module>()
----> 1 mir_eval.beat.p_score(ref_events, est_events)

mir_eval/beat.pyc in p_score(reference_beats, estimated_beats, p_score_threshold)
    390     # defined as .2*median(inter-annotation-intervals)
    391     annotation_intervals = np.diff(np.flatnonzero(reference_train))
--> 392     win_size = int(np.round(p_score_threshold*np.median(annotation_intervals)))
    393     # Get full correlation
    394     train_correlation = np.correlate(reference_train, estimated_train, 'full')

ValueError: cannot convert float NaN to integer

In [11]: mir_eval.onset.f_measure(ref_events, est_events)
Out[11]: (0.0, 0.0, 0.0)


```

`chord`: All ok

``` Python

In [12]: ref_intervals = np.array([[10., 11.]])
In [13]: ref_labels = ['C']
In [14]: est_intervals = np.array([[10.1, 11.1]])
In [15]: est_labels = ['C']
In [20]: mir_eval.chord.evaluate(ref_intervals, ref_labels, est_intervals, est_labels)
Out[20]: OrderedDict([('thirds', 0.90000000000000036), ('thirds_inv', 0.90000000000000036), ('triads', 0.90000000000000036), ('triads_inv', 0.90000000000000036), ('tetrads', 0.90000000000000036), ('tetrads_inv', 0.90000000000000036), ('root', 0.90000000000000036), ('mirex', 0.90000000000000036), ('majmin', 0.90000000000000036), ('majmin_inv', 0.90000000000000036), ('sevenths', 0.90000000000000036), ('sevenths_inv', 0.90000000000000036)])
```

`melody`: All ok

``` Python
In [22]: ref_time = np.array([1.])

In [23]: ref_freq = np.array([440.])

In [24]: est_time = np.array([1.])

In [25]: est_freq = np.array([441.])

In [26]: mir_eval.melody.evaluate(ref_time, ref_freq, est_time, est_freq)
Out[26]: OrderedDict([('Voicing Recall', 1.0), ('Voicing False Alarm', 0.0), ('Raw Pitch Accuracy', 1.0), ('Raw Chroma Accuracy', 1.0), ('Overall Accuracy', 1.0)])
```

`onset`: All ok

``` Python
In [28]: mir_eval.onset.evaluate(ref_events, est_events)
Out[28]: OrderedDict([('F-measure', 0.0), ('Precision', 0.0), ('Recall', 0.0)])
```

`pattern`:
- [x] `establishment_FPR`
- [x] `first_n_target_proportion_R`
- [x] `first_n_three_layer_P`
- [x] `occurrence_FPR`
- [ ] `standard_FPR`
- [x] `three_layer_FPR`

``` Python
In [5]: ref_pats = [[[(0.5, 67.0)]]]
In [6]: est_pats = [[[(0.5, 68.0)]]]
In [7]: mir_eval.pattern.establishment_FPR(ref_pats, est_pats)
Out[7]: (0.0, 0.0, 0.0)

In [8]: mir_eval.pattern.first_n_target_proportion_R(ref_pats, est_pats)
Out[8]: 0.0

In [9]: mir_eval.pattern.first_n_three_layer_P(ref_pats, est_pats)
Out[9]: 0.0

In [10]: mir_eval.pattern.occurrence_FPR(ref_pats, est_pats)
Out[10]: (0.0, 0, 0)

In [12]: mir_eval.pattern.standard_FPR(ref_pats, est_pats)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-12-f7abd3d6c25b> in <module>()
----> 1 mir_eval.pattern.standard_FPR(ref_pats, est_pats)

mir_eval/mir_eval/pattern.pyc in standard_FPR(reference_patterns, estimated_patterns, tol)
    230
    231             # Check transposition given a certain tolerance
--> 232             if np.max(np.abs(np.diff(P - Q, axis=0))) < tol:
    233                 k += 1
    234                 break

numpy/core/fromnumeric.pyc in amax(a, axis, out, keepdims)
   2138     else:
   2139         return _methods._amax(a, axis=axis,
-> 2140                             out=out, keepdims=keepdims)
   2141
   2142 def amin(a, axis=None, out=None, keepdims=False):

numpy/core/_methods.pyc in _amax(a, axis, out, keepdims)
     24 # small reductions
     25 def _amax(a, axis=None, out=None, keepdims=False):
---> 26     return umr_maximum(a, axis, None, out, keepdims)
     27
     28 def _amin(a, axis=None, out=None, keepdims=False):

ValueError: zero-size array to reduction operation maximum which has no identity

In [13]: mir_eval.pattern.three_layer_FPR(ref_pats, est_pats)
Out[13]: (0.0, 0.0, 0.0)
```

`segment`: All ok

``` Python
mir_eval.segment.evaluate(ref_intervals, ref_labels, est_intervals, est_labels)
Out[32]: OrderedDict([('Precision@0.5', 1.0), ('Recall@0.5', 1.0), ('F-measure@0.5', 1.0), ('Precision@3.0', 1.0), ('Recall@3.0', 1.0), ('F-measure@3.0', 1.0), ('Ref-to-est deviation', 0.0), ('Est-to-ref deviation', 0.0), ('Pairwise Precision', 0.98033818324813216), ('Pairwise Recall', 0.99819819819819822), ('Pairwise F-measure', 0.98918758059716294), ('Rand Index', 0.98181818181818181), ('Adjusted Rand Index', 0.9321188329452534), ('Mutual Information', 0.25363470096598395), ('Adjusted Mutual Information', 0.82959507893644957), ('Normalized Mutual Information', 0.86353669596923532), ('NCE Over', 0.95736403694642902), ('NCE Under', 0.92642053835946558), ('NCE F-measure', 0.94163814497725262)])
```

`separation`: All ok (I think)

``` Python
In [34]: mir_eval.separation.evaluate(np.array([[1.]]), np.array([[1.]]))
Out[34]: OrderedDict([('Source to Distortion', [inf]), ('Source to Interference', [inf]), ('Source to Artifact', [inf]), ('Source permutation', (0,))])
```

`tempo`: All ok

``` Python
In [40]: mir_eval.tempo.evaluate(np.array([110., 120.]), .6, np.array([110., 120.]))
Out[40]: OrderedDict([('P-score', 1.0), ('One-correct', True), ('Both-correct', True)])
```

Am I missing anything?


@urinieto can you fix `pattern.standard_FPR`?

I will start looking at the beat issues.


@urinieto pinging you to fix `pattern.standard_FPR`.