Repository: mir-evaluation/mir_eval
Issue: Warn when reference or estimation are empty and return 0
Root cause: Validation Gap

The warning code should go in each submodule's validator.  Metrics should return 0 (or an otherwise appropriate "you got it wrong" value).
- [x] beat  @craffel
- [x] onset @craffel
- [x] separate @dawenl 
- [x] segment @bmcfee
- [x] chord @ejhumphrey 
- [x] melody @justinsalamon 
- [x] pattern @urinieto 


Returning an SIR/SDR/SAR of 0 doesn't make sense in separation, so empty ndarrays are returned.  Typically, (e.g.) SIR is a ndarray of size nsrc, since nsrc is zero, it's an empty ndarray.


`boundary.deviation` computes the median deviation between estimated and reference boundary times; if there are no estimated or reference boundary times, there's no deviation, so it returns NaNs.


On second thought, I'm pretty sure you can't pass this test:

``` python
    if intervals.ndim != 2 or intervals.shape[1] != 2:
        raise ValueError('Segment intervals should be n-by-2 numpy ndarray')
```

and still have .size == 0 or len() == 0.  So checking this special case in boundary is kind of pointless.


Same goes for `structure`.  @bmcfee requesting permission to delete the special case code in `boundary`:

``` Python
    # If we have no boundaries, we get no score.
    if len(reference_boundaries) == 0 or len(estimated_boundaries) == 0:
        return 0.0, 0.0, 0.0
```

and

``` Python
    # If we have no boundaries, we get no score.
    if len(reference_boundaries) == 0 or len(estimated_boundaries) == 0:
        return np.nan, np.nan
```

because it will never get run, because it's impossible that len() == 0 when .ndim == 2 and .shape[1] == 2.


I am a little concerned about the validate() method for BSS_eval. I think the following code should also be in the validate():

``` python
    # make sure the input is of shape (nsrc, nsampl)
    if estimated_sources.ndim == 1:
        estimated_sources = estimated_sources[np.newaxis, :]
    if reference_sources.ndim == 1:
        reference_sources = reference_sources[np.newaxis, :]
```

Otherwise, it's possible the `estimated_sources` is a `(nsample, )` ndarray, while `reference_sources` is a `(1, nsample)` ndarray, which is a valid input, but will fail at the validate().


OK, I've moved it into validate.  This is actually relevant to issue #31, not this one.  Feel free also to make changes like this yourself if you want.


@urinieto I added this check for pattern, but patterns are passed around in a complicated format (not just a list, a list of lists of lists of tuples..?) Since `len([[]]) = 1`, I think this will not handle all the cases we care about.  Can you come up with a better test?


@urinieto I updated the way that patterns are checked to be empty, please see the function `_n_onset_midi` which computes the number of onset_midi tuples in a pattern.
https://github.com/craffel/mir_eval/blob/master/mir_eval/pattern.py#L88
I'm going to close this for now as this is the last thing that needed to be changing.  If you see an issue with how I implemented this, please re-open and let me know.


Seems good to me. Thanks.

On Wed, Apr 23, 2014 at 10:40 PM, craffel notifications@github.com wrote:

> Closed #44 https://github.com/craffel/mir_eval/issues/44.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/44
> .


Re-opening because `boundary` and `structure` don't warn.


... which is OK for intervals because you can't have an ndarray of size 0 with ndim = 2 and shape[1] = 2, which raises an exception.  And you can't have an empty label list because you can't have an empty interval list, so an exception will be thrown because they aren't the same size. I think I figured this out before but didn't document it.  I am also going to make the error message more helpful.


The structure metrics don't seem to obey this, e.g.:

``` Python
In [12]: mir_eval.segment.ari(np.zeros((0, 2)), [], np.zeros((0, 2)), [])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-16-48c949f72419> in <module>()
----> 1 mir_eval.segment.mutual_information(np.zeros((0, 2)), [], np.zeros((0, 2)), [])

/Users/craffel/Documents/School and Research/mir_eval/mir_eval/segment.py in mutual_information(reference_intervals, reference_labels, estimated_intervals, estimated_labels, frame_size)
    846     y_ref = util.intervals_to_samples(reference_intervals,
    847                                       reference_labels,
--> 848                                       sample_size=frame_size)[-1]
    849
    850     y_ref = util.index_labels(y_ref)[0]

/Users/craffel/Documents/School and Research/mir_eval/mir_eval/util.pyc in intervals_to_samples(intervals, labels, offset, sample_size, fill_value)
    100
    101     # Round intervals to the sample size
--> 102     num_samples = int(np.floor(intervals.max() / sample_size))
    103     sample_indices = np.arange(num_samples, dtype=np.float32)
    104     sample_times = (sample_indices*sample_size + offset).tolist()

/usr/local/lib/python2.7/site-packages/numpy/core/_methods.pyc in _amax(a, axis, out, keepdims)
     15 def _amax(a, axis=None, out=None, keepdims=False):
     16     return um.maximum.reduce(a, axis=axis,
---> 17                             out=out, keepdims=keepdims)
     18
     19 def _amin(a, axis=None, out=None, keepdims=False):

ValueError: zero-size array to reduction operation maximum which has no identity
```


So the metric functions now behave properly (warn and return 0), but `segment.evaluate`, which uses `util.adjust_intervals`, will still fail:

``` Python
In [4]: mir_eval.segment.evaluate(np.zeros((0, 2)), [], np.zeros((0, 2)), [])
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-4-e17be68ab7ce> in <module>()
----> 1 mir_eval.segment.evaluate(np.zeros((0, 2)), [], np.zeros((0, 2)), [])

/Users/craffel/Documents/School and Research/mir_eval/mir_eval/segment.py in evaluate(ref_intervals, ref_labels, est_intervals, est_labels, **kwargs)
   1048     # Adjust timespan of estimations relative to ground truth
   1049     ref_intervals, ref_labels = \
-> 1050         util.adjust_intervals(ref_intervals, labels=ref_labels, t_min=0.0)
   1051
   1052     est_intervals, est_labels = \

/Users/craffel/Documents/School and Research/mir_eval/mir_eval/util.pyc in adjust_intervals(intervals, labels, t_min, t_max, start_label, end_label)
    275         intervals = np.maximum(t_min, intervals)
    276
--> 277         if intervals[0, 0] > t_min:
    278             # Lowest boundary is higher than t_min:
    279             # add a new boundary and label

IndexError: index 0 is out of bounds for axis 0 with size 0
```

@bmcfee @urinieto  what do you think the best behavior should be for `adjust_intervals` when the intervals/labels are empty?  Obviously we won't crop or remove intervals because there aren't any to crop and remove, and it's hard to define appending intervals because there are no intervals to append up to (if that makes sense).  I'd propose the following two options in order of preference when the supplied intervals/labels to adjust_intervals are empty:
- Always return the empty intervals/labels as-is
- Return the empty intervals/labels as-is, unless BOTH `t_min` and `t_max` are supplied, in which case generate a label from `t_min` to `t_max`... but not sure whether to label it `'__T_MIN'` or `'__T_MAX'`.


The defined behavior of `adjust_intervals` is to modify the input to span the range `[t_min, t_max]`.

If `t_min` is None, it is inferred from the smallest time in the input.  If the input is empty, then this is an error, since the target interval is undefined.  Similarly for `t_max`.  We should raise an exception, rather than returning the empty list as-is.

If both inputs are supplied, and the input is empty, then sure, add a single interval to cover the range.  As for it's label, I think we can default to assigning it `__T_MIN`.


Re-resolved via 05da3e9