Repository: mir-evaluation/mir_eval
Issue: Implement multi-f0 frame level eval metrics
Root cause: Unimplemented Feature Gap

As far as I can tell, MIREX computes 14 metrics. Each of the following 7 metrics are computed first for the original values, and second for octave-wrapped values:
- Accuracy
- Precision
- Recall
- Total Error (E_tot)
- Substitution Error (E_sub)
- Miss Error (E_miss)
- False Alarm Error (E_fa)

Metrics described in part [here, page 5](http://academiccommons.columbia.edu/download/fedora_content/download/ac:144450/CONTENT/1687-6180-2007-048317.pdf).

I need this for my own work and am happy to/planning to implement it.
@craffel - what specific things would you want me to cover for a squeakly clean PR? I imagine code with tests + comparison with MIREX code on MIREX datasets? :)


Great!

> @craffel - what specific things would you want me to cover for a squeakly clean PR? I imagine code with tests + comparison with MIREX code on MIREX datasets? :)

We should document this somewhere when there are more requests like this, but the idea PR has
- metrics implemented in mir_eval/metric_name.py, plus `validate` method which makes sure annotations are in a sane format and `evaluate` method which calls all metrics and returns an ordered dict of the results (see other metrics for examples)
- loader in `io` which loads in MIREX-format data
- evaluator in the evaluators folder which loads in data, calls `evaluate`, and writes out a json results file
- Docs everywhere in numpydoc format, with an auto-module hook added to docs/index.rst
- Unit tests and regression tests for all metrics, with a regression test data ideally coming from some real algorithm's output and human-annotated examples, as well as a generating hook for regression output in tests/generate_data.py
- For the PR (but doesn't have to live permanently in the repository), ideally a comparison between the metric's output and some other implementation (e.g. MIREX); for examples, see https://github.com/craffel/mir_eval-ismir)

Hopefully that doesn't sound like an overwhelming amount - apart from implementing the metrics themselves and comparing the output to an existing implementation, most things should be pretty quick and there are examples here!


@craffel great! Seems straightforward enough ;)


@craffel: I ran into a small hitch and want to run it by you.
As far as I've seen, multi-f0 algorithms report scores as a time stamp followed by a variable number of f0 values:

```
29.52   246.94  622.25  311.13  246.94
29.53   246.94  622.25  311.13  246.94
29.54   622.25  311.13  246.94
29.55   622.25  311.13  246.94
29.56   622.25  311.13  246.94  185.00  440.00
29.57   622.25  311.13  246.94  185.00  440.00  92.50
29.58   622.25  311.13  246.94  185.00  440.00  92.50
29.59   622.25  311.13  246.94  440.00  92.50
29.60   622.25  311.13  246.94  440.00  92.50   185.00
29.61   622.25  311.13  246.94  440.00  92.50   185.00  220.00
```

`mir_eval/melody.py` is [resampling](https://github.com/craffel/mir_eval/blob/master/mir_eval/melody.py#L190) the estimate time series to ensure the time stamps are consistent with the reference time series. MIREX avoids the resampling problem entirely by requiring a particular time base.
This is all fine until you notice that the above excerpt has the following form (letters correspond to f0's belonging to a contiguous contour):

```
29.50   A   B   C
29.51   A   B   C
29.52   A   B   C   D
29.53   A   B   C   D
29.54   B   C   D
29.55   B   C   D
29.56   B   C   D   E   F
29.57   B   C   D   E   F   G
29.58   B   C   D   E   F   G   
29.59   B   C   D   F   G   
29.60   B   C   D   F   G   E
29.61   B   C   D   F   G   E   H
```

Resampling along columns would interpolate between f0 contours that are not meant to be contiguous, and I don't see a clean way to align contiguous contours.

Atm, I'm planning to resample using a nearest neighbor interpolation to avoid this problem. It's certainly not ideal, but I don't see a cleaner way. Thoughts?


> Resampling along columns would interpolate between f0 contours that are not meant to be contiguous, and I don't see a clean way to align contiguous contours.

Ooh, that is alarming.

> Atm, I'm planning to resample using a nearest neighbor interpolation to avoid this problem. It's certainly not ideal, but I don't see a cleaner way. Thoughts?

Hmm.  The nearest neighbor idea is good, but I could see it getting ugly and potentially doing the wrong thing, depending on what exactly the metrics are doing (I'm not familiar).  Can we just require that reference and estimated annotations have the same timebase like MIREX, thereby also sidestepping this issue?


Is there anything in the task definition that defines what a "contour" is in this representation?  Would it make sense to match f0 values at one row to those in the previous row if they are within some maximum cent deviation?  Then you could line up all values in one contour to a single column (padded out with nans) and resample as normal.


@craffel 

> Hmm. The nearest neighbor idea is good, but I could see it getting ugly and potentially doing the wrong thing, depending on what exactly the metrics are doing (I'm not familiar). Can we just require that reference and estimated annotations have the same timebase like MIREX, thereby also sidestepping this issue?

That's definitely an option, but as a matter of taste I personally don't like the constraint of having to have the same time steps ;). The metrics are all computed per time step and averaged across the song, and nothing. Any problems introduced by resampling would in theory be the same as those introduced when resampling the melody time series.

@bmcfee 

> Is there anything in the task definition that defines what a "contour" is in this representation? 

Nope, that's me adding terminology. afaik, multif0 metrics only care about the bundle of pitches in each time frame, not about any sort of continuity over time.

> Would it make sense to match f0 values at one row to those in the previous row if they are within some maximum cent deviation? Then you could line up all values in one contour to a single column (padded out with nans) and resample as normal.

I see what you're saying. If a "contour" is defined such that the change in pitch from one frame to the next is bounded by some value, then you could potentially stitch things back together. I think that would work most of the time, but break down if contours overlap in pitch, and the decision becomes ambiguous. In any case, multi-f0 eval doesn't care about contours for the moment (perhaps it should in the future)...
If we wanted to actually have "contours" we'd need the algorithms to explicitly output them in a form that groups pitches in a meaningful way, but this isn't how the problem is currently formulated.


Just adding my 2c into the mix: 
I also don't like the idea of enforcing an input timebase, apart from the fact that it's limited, in practice it just means people will do the resampling themselves in whichever way they find most convenient, and is likely to result in inconsistent evaluation results.

What if you state that to get results that are 100% MIREX compatible the input timebase must be fixed. However, if your data is not on this timebase, we offer interpolation method X (probably nearest neighbor being the safest) which is the best alternative we can offer and recommend that you use it?


> Any problems introduced by resampling would in theory be the same as those introduced when resampling the melody time series.

Well, not exactly, since this problem we're discussing isn't present in melody annotations.

> Atm, I'm planning to resample using a nearest neighbor interpolation to avoid this problem. It's certainly not ideal, but I don't see a cleaner way. Thoughts?

So looping back on this because I think I misunderstood - all eval is done on a temporal-independent way, right?  In that case, yes, I would suggest nearest-neighbor interpolation is the only sane solution here.  I think it would be smart, in this case, to throw a warning when resampling is done.

By the way, in order to load this type of file, we'll need to modify `load_delimited` right?


This is merged - closing this issue.