Repository: mir-evaluation/mir_eval
Issue: Transcription metrics
Root cause: Unimplemented Feature Gap

I think it'd be awesome for `mir_eval` to have metrics for evaluating automatic transcription (i.e. comparing two sets of symbolic notes). There's a relatively large number of people working on this problem in MIR, so I think such a module would very much come in handy.

Since I need this for some of my own work (ehm ehm), I'm happy to take a first stab at this.

As far as existing tools there's Emilio Molina's [matlab framework](http://www.terasoft.com.tw/conf/ismir2014/proceedings/T102_298_Paper.pdf) which has a fairly extensive set of metrics in it. For a first iteration though I'd stick to a widely used subset of metrics as defined in [Bello, 2003](https://qmro.qmul.ac.uk/xmlui/handle/123456789/3803); [Ryynänen & Klapuri, 2008](http://dl.acm.org/citation.cfm?id=1403132); and [Salamon, 2013](http://www.justinsalamon.com/phd-thesis.html) (page 186).


Please do!  I think the preferred way to prioritize different metrics is to choose those (if any) which are used in MIREX.


The metrics used in the aforementioned references are basically those used in MIREX :) Specifically, they're used in task 2 of the [multi-f0 tracking evaluation](http://www.music-ir.org/mirex/wiki/2015:Multiple_Fundamental_Frequency_Estimation_%26_Tracking#Evaluatio). It's basically recall, precision and f-measure given two sets of notes. Papers differ slightly on the thresholds used to determine whether two notes are a match, but we can use the MIREX values by default and allow the user to change them should they want to.


Sounds good.  


paging @craffel : the MIREX convention for transcription lab files is a file with 3 tab-separated columns, where each row contains the onset, offset, and pitch (in Hz) of a single note. In terms of loading this kind of data into `mir_eval`, the closest function in `io` is `load_labeled_intervals`, except that it parses the last column as a `str`, whereas I need it to be parsed as a `float`.

Does it make sense to write a new method in `io` for this scenario (e.g. `load_notes`, `load_transcription` or `load_realvalued_intervals`), modify the existing `load_labeled_intervals`, or just have the user load the data using:

``` python
load_delimited('transcription.txt', [float, float, float], '\t')
```

?


Also, do we have any preference for handling corner cases: if either the reference or estimate are empty, then the precision, recall, and f-measure (or a subset of these) will be undefined due to a division by zero. Should the metrics return None, raise an error, or a hardcoded 0 or 1 in such cases?


> Does it make sense to write a new method in io for this scenario (e.g. load_notes, load_transcription or load_realvalued_intervals)

Yes.  It should look very similar to `load_labeled_intervals`, except that it should call `load_delimited` with `[float, float, float]` as you suggested.  It should validate the resulting intervals, too, using `util.validate_intevals`.  I would call it something more generic than `load_notes` or `load_transcription`.  I like `load_valued_intervals`.

> Also, do we have any preference for handling corner cases: if either the reference or estimate are empty, then the precision, recall, and f-measure (or a subset of these) will be undefined due to a division by zero. Should the metrics return None, raise an error, or a hardcoded 0 or 1 in such cases?

It depends on the definition of the metric itself.  Given empty annotations, we always trigger a warning in the validate function.  In precision/recall/F-mesure-based metrics, we return 0s.


> I like `load_valued_intervals`

sounds good to me.

> In precision/recall/F-mesure-based metrics, we return 0s.

ok, so I'll follow that then.


Important note about the criterion for considering an offset match [for the record]:
While the MIRX [task page](http://www.music-ir.org/mirex/wiki/2015:Multiple_Fundamental_Frequency_Estimation_%26_Tracking#Evaluation) for multi-f0 tracking only specifies that the offset has to be within a window that's 20% of the reference note's duration centered on the reference offset, the MIREX [results page](http://www.music-ir.org/mirex/wiki/2015:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_-_MIREX_Dataset#Task_2:Note_Tracking_.28NT.29) actually provides a more detailed criterion:

> This subtask is evaluated in two different ways. In the first setup , a returned note is assumed correct if its onset is within +-50ms of a ref note and its F0 is within +- quarter tone of the corresponding reference note, ignoring the returned offset values. In the second setup, on top of the above requirements, a correct returned note is required to have an offset value within 20% of the ref notes duration around the ref note's offset, or within 50ms whichever is larger.

This discrepancy occurs in every year of the MIREX wiki, all the way back to 2007 which is the first year the task was run (carry over errors FTW). In light of this:
1. For the metrics that _do_ consider the offset, I plan to add the +-50ms criterion to the offset, as (a) it makes more sense (otherwise short notes have an insanely tight offset threshold) and (b) I'm 99.9% sure that's the version of the metric that's implemented for MIREX, though I couldn't find the nema code to corroborate this.
2. I will also add a second set of metrics that ignores the offset altogether, so that there will be a total of 6 metrics: (precision, recall and f-measure) x 2, one set without offsets, one with.


Hmm... Assuming that ignoring the offset values is also commonly used elsewhere, beyond the standard in MIREX, I would personally prefer having a single set of metrics with an argument which allowed the user to specify whether to evaluate offsets or not (and, ideally, with a user-settable threshold) with the default being to ignore offsets.


Yes, ignoring offsets is also commonly used elsewhere. In fact, people normally compute both sets of metrics, as they provide greater insight into the working of an algorithm than either set on its own.

I could easily merge these into a single set with an optional parameter, however I'd argue that there's value to computing both sets: as noted above they tell you different things. It's also the case that in MIREX both sets are computed (with and without offsets), and this is also commonplace in published papers. By explicitly computing and returning both sets, we avoid potential confusion where people report their results and compare them to MIREX results (or some paper) without knowing whether offsets were taken into account or not.

You can have a look at `transcription.py` (you need the latest push on the transcription branch) to see what this looks like. If you still think we should merge I'm happy to do it, but I think there's value to computing everything explicitly for the aforementioned reasons.

p.s - this is what the evaluator returns right now for the example ref00.txt and est00.txt in the test data folder:

```
                Precision : 0.600
                   Recall : 0.750
                F-measure : 0.667
    Precision_with_offset : 0.400
       Recall_with_offset : 0.500
    F-measure_with_offset : 0.444
```


I think what @craffel is suggesting (and correct me if I'm wrong) is that you can have two sets of metrics, but they should both transparently wrap a single function that accepts the offset flag and window length as a parameter.  This seems like the sanest way to implement it to me.


That's exactly how it's implemented right now


> I think what @craffel is suggesting (and correct me if I'm wrong) is that you can have two sets of metrics, but they should both transparently wrap a single function that accepts the offset flag and window length as a parameter.

Almost.  I am saying that it should follow the API of the other submodules (a good example is [segmentation](https://github.com/craffel/mir_eval/blob/master/mir_eval/segment.py#L1082)), where you have a single function which the evaluator calls with different parameters because both are used.  It's mostly an API issue.


I see, so if I understand correctly you want there to be only one function (in this case `precision_recall_f1()`, but for the evaluator to call it twice, switching the value of a parameter to obtain both sets of metrics? I guess I can do that.

I still think there's value to having two separate functions (since this makes the importance of computing the metrics with and without offsets explicit), but I'm fine consolidating.


> I see, so if I understand correctly you want there to be only one function (in this case precision_recall_f1(), but for the evaluator to call it twice, switching the value of a parameter to obtain both sets of metrics? I guess I can do that.

Yes.

> I still think there's value to having two separate functions (since this makes the importance of computing the metrics with and without offsets explicit), but I'm fine consolidating.

You can put two separate examples in the docstring.


ok, I'm on it


Alright, code has been consolidated and travis seems [happy](https://travis-ci.org/craffel/mir_eval/builds/108175644). @craffel or @bmcfee care to CR this and/or merge this PR?

Things I'm not happy about:
1. From the [MIREX description](http://www.music-ir.org/mirex/wiki/2015:Multiple_Fundamental_Frequency_Estimation_%26_Tracking_Results_-_MIREX_Dataset#Task_2:Note_Tracking_.28NT.29) it is not possible to determine whether they are using `<` or `<=` for all the threshold checks. I've opted for `<` since if I recall correctly that's what  was used in MIREX for the melody extraction pitch threshold check.
2. I only have one toy test case (ref00.txt and est00.txt), and I'm not sure there's data out there that would allow us to compare directly to MIREX results.


Ok, just made some misc. comments.  Please address them in the PR.  To your questions:

> 1. From the MIREX description it is not possible to determine whether they are using < or <= for all the threshold checks. I've opted for < since if I recall correctly that's what was used in MIREX for the melody extraction pitch threshold check.

I'd like to figure this out, if possible.  Is the evaluator code not online?  It's not [here](https://github.com/jpauwels/nemadiy/blob/ce3787191871d52196e97e6e9eb8d37caf12c911/analytics/trunk/src/main/java/org/imirsel/nema/analytics/evaluation/multif0/MultiF0EstEvaluator.java) or nearby?  If not, we should email whoever runs the task and get the code from them.

> 1. I only have one toy test case (ref00.txt and est00.txt), and I'm not sure there's data out there that would allow us to compare directly to MIREX results.

We need to try to get some real-world data to compare to MIREX with.  I don't expect the results to be the same because we are using bipartite matching, but it's a good sanity check.  Plus, I'd really like to have non-toy (and more than 1) regression tests.  So this is not a task where the ground-truth has ever been publicly available?  And no one has ever published ground-truth for it?  If not can you email someone who has run MIREX to ask for some ground-truth examples?


> Ok, just made some misc. comments. Please address them in the PR.

Thanks! I'll go through the comments and make modifications as necessary.

> I'd like to figure this out, if possible. Is the evaluator code not online? It's not here or nearby? If not, we should email whoever runs the task and get the code from them.

That code seems to compute eval metrics for continuous multi-f0 estimation only, not for note tracking (which is what we need). I guess I'll try and track down the person who ran the task in 2015 and see if they can give me a hand.

> So this is not a task where the ground-truth has ever been publicly available? And no one has ever published ground-truth for it? If not can you email someone who has run MIREX to ask for some ground-truth examples?

Correct me if I'm wrong, but in order to compare to MIREX results we need both the GT files and the output of a participating algorithm. Tbh I have no idea if that data's available or not. I guess more digging is in order... perhaps if I don't make progress contacting individuals I'll shoot an email to the ismir mailing list.


> That code seems to compute eval metrics for continuous multi-f0 estimation only, not for note tracking (which is what we need). I guess I'll try and track down the person who ran the task in 2015 and see if they can give me a hand.

I figure it must exist somewhere, either on the nemadiy repository (not necessarily where I linked!  There's a lot of code there), or on someone's server somewhere.  If you can't find it on the nemadiy repository, hopefully said someone will respond to your email.

> Correct me if I'm wrong, but in order to compare to MIREX results we need both the GT files and the output of a participating algorithm. 

Yes.  We have both for many of the metrics (all pattern discovery, actually).  Ideally, we'd have both for this task too. If the ground truth is not available on the MIREX results page (as it is for the other tasks, except for melody and pattern discovery), then maybe someone can send you the ground truth.  The output of participating algorithms is indeed available on the results page, I believe.


> The output of participating algorithms is indeed available on the results page, I believe.

Where?


Oops, nevermind, I was mistaking the results file downloads for algorithm output downloads, sorry!


Ah ok. In which case I guess I have to see if the MIREX folk would be willing to hand over at least 1 GT file and a corresponding algorithm output. Given the scarcity of multi-f0 datasets, I don't have high hopes though...


Are there literally no multi-f0 datasets for which ground truth is publicly available?  The requirement is not that we obtain exact MIREX output from some previous iteration, just that we have (preferably 10 instances of) an algorithm output, a ground-truth annotation, and the resulting scores produced by the MIREX code.  If you have algorithm output and ground-truth annotations, you just need to run them through the MIREX code, or have someone else do so.


After some snooping around and a very handy pointer from @rabitt I got to [this dataset](https://sites.google.com/site/lisupage/research/new-methodology-of-building-polyphonic-datasets-for-amt). Not only is it publicly available, it was also used for MIREX 2015. Win.


oops!


Good news, I've heard back from Li Su, who created the aforementioned dataset and was also mirex task captain for multi-f0 (and note tracking) in 2015. He replied:

> To answer your questions:
> 1) '<' or '<=': I just checked the old source codes taken for MIREX evaluation and found that we use '<' rather than '<=' for all cases including onset, offset and pitch distance. In MIREX 2015 we kept using this code so it is '<'.

Which answers that question. I'm working on getting all the data I need in order to compare to the mirex results, almost there.


Can he just send you said "old source codes"?


I can ask


The plot thickens... apparently there are 2 sets of eval code: 
- The ["old" code, which I've placed in this gist](https://gist.github.com/justinsalamon/6bb4678db587c23a41df) for reference, which uses "<" for all threshold checks. 
- The "new" code by Zhiyao Duan, which can be [downloaded here](http://www.ece.rochester.edu/~zduan/code/mpa_eval.zip). The relevant file is [`ErrorRate_MF0_NoteLevel.m`, which I've placed in this gist](https://gist.github.com/justinsalamon/a46923f9c6ab58237585) for reference. Unlike the "old" code, this code uses "<=" for all comparisons (!!!).

Right now I'm in the process of trying to figure out which code was used for which dataset in which year. My current understanding is that the "old" code was used for the "mirex" dataset (always, including 2015), whilst the "new" code was only used for the "Su" dataset in 2015 (the first year the dataset was introduced). But I'm waiting to hear back from Li Su for confirmation.

If this is indeed the case, then we'll have to use "<=" (at least for the regression tests), since the dataset we'll be comparing against is the "Su" dataset. However, we're in a bit of a kerfuffle here, since we'd have to choose whether to use "<=" by default or "<". Another option would be to use "<=" by default but include the option (`old_eval=True` or something horrible like that) to revert to the old comparison code. Yikes.


As long as you document your choice, and all of this, in the docstrings, we are covered.


> (old_eval=True or something horrible like that) to revert to the old comparison code. Yikes.

Not that I'm advocating this necessarily, but it wouldn't be too hard to do if you use the numpy ufuncs `lt` and `le` instead of the symbols.  Then you can just do something like

``` python
if strict:
    cmp = np.lt
else:
    cmp = np.le

result = cmp(difference, threshold)
```

or somesuch.


Yeah ok. I'll wait to hear back from Li Su then. I'm on the fence as to which to use... if my understanding of the situation is correct, then on the one hand "<" was used every year for the "mirex" dataset (the Su dataset and the use of "<=" were only introduced in 2015), and would also be consistent with the melody eval code that requires "<" for pitch matching. On the other hand, the beat and onset eval code in `mir_eval` use the `match_events` method which uses "<=". Joy.


How big of a difference will this make in practice?  Aren't these pitch/time values we are comparing continuously-valued?  Backwards compatibility is important, but don't go too far down the rabbit hole.  Documenting and justifying your choices is fine.


yeah the values are continuous, ultimately this won't make a huge difference. I'll just choose one once I hear back from Li Su for confirmation, and support the other via an optional argument.


Again I don't think you need to worry about supporting both as long as you are thorough about documenting which you are using and why.


I like @bmcfee's idea of giving the user a choice via a `strict` parameter. I've added this in 6978d0d. The default behavior is `strict=False` (i.e. `<=`), which should match the MIREX 2015 eval for the Su dataset. It's all documented in the transcription.py docstring.


#170 has been merged, closing.