Repository: mir-evaluation/mir_eval
Issue: sorting intervals -- OR -- interpolate_intervals bug
Root cause: Validation Gap

[porting over from https://github.com/marl/jams/issues/116 ]

I ran into the following error:

``` python
>>> import mir_eval
>>> import numpy as np
>>> ints = np.asarray([[1, 2], [0, 1]])
>>> labs = list('ab')
>>> mir_eval.segment.evaluate(ints, labs, ints, labs)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-9-dc90e2bfeebe> in <module>()
      3 labs = list('ab')
      4 
----> 5 mir_eval.segment.evaluate(ints, labs, ints, labs)

/home/bmcfee/git/mir_eval/mir_eval/segment.py in evaluate(ref_intervals, ref_labels, est_intervals, est_labels, **kwargs)
   1099                                                         ref_labels,
   1100                                                         est_intervals,
-> 1101                                                         est_labels, **kwargs)
   1102 
   1103     # Rand index

/home/bmcfee/git/mir_eval/mir_eval/util.py in filter_kwargs(_function, *args, **kwargs)
    832             filtered_kwargs[kwarg] = value
    833     # Call the function with the supplied args and the filtered kwarg dict
--> 834     return _function(*args, **filtered_kwargs)
    835 
    836 

/home/bmcfee/git/mir_eval/mir_eval/segment.py in pairwise(reference_intervals, reference_labels, estimated_intervals, estimated_labels, frame_size, beta)
    382 
    383     # Find where they agree
--> 384     matches = np.logical_and(agree_ref, agree_est)
    385     n_matches = (matches.sum() - len(y_ref)) / 2.0
    386 

ValueError: operands could not be broadcast together with shapes (20,20) (10,10) 

```

I think the error is that `util.interpolate_intervals()` assumes the intervals are sorted, but this is not stated in the docstring.

In general, I don't think we should require the intervals to be sorted, since the boundaries are all explicit.  It would be easy to add a sorting step to `interpolate_intervals` and the rest should fix itself.

@craffel what you say?


This issue actually pops up again in all the segment metrics because `validate_structure` requires `intervals[0, 0] == 0`.

Additionally, I think the [`interpolate_intervals`](https://github.com/craffel/mir_eval/blob/master/mir_eval/util.py#L157-L164) logic is incorrect if the segments don't cover all time instants in question.  (This is not documented behavior.)

So here are my proposed solutions:
1. Add a utility function `sort_interval_labels` that simply puts an interval/label tuple in order by start time.  Call this function from inside each segment metric.
2. Rewrite `interpolate_intervals` to work in more general situations (non-contiguous intervals, disordered intervals, etc.)

I think these two should pretty much cover it.


Sounds reasonable overall, but

> Add a utility function sort_interval_labels that simply puts an interval/label tuple in order by start time. Call this function from inside each segment metric.

we should probably only call this if the intervals are not sorted, because sorting things is more expensive than checking whether things are sorted.  Though I guess it's probably not slow enough to warrant a special check.


> we should probably only call this if the intervals are not sorted, because sorting things is more expensive than checking whether things are sorted. Though I guess it's probably not slow enough to warrant a special check.

Eh, we're talking about sorting on the order of tens of numbers here; I'm not worried about it.  The additional logic to check if it's sorted isn't worth the effort IMO.

Also, I've simplified the validators and metrics so that sorting is only called when needed, ie, when interpolating labels.