Repository: mir-evaluation/mir_eval
Issue: Evaluation Measures for Retrieval Systems
Root cause: Unimplemented Feature Gap

Hey,

just noticed that there are no measures for retrieval systems yet.
I'm doing things like mean rank, top10 matches etc. over and over
again. Am I the only one or might this be helpful for more people out there?

What do you think? Is it worth the effort?


Mean reciprocal rank (MRR) and top-X hit rate are definitely commonplace (I've used them [myself](http://www.justinsalamon.com/uploads/4/3/9/4/4394963/salamon-mmir12.pdf) quite a bit too). Probably a few more common ones out there such as Mean Average Precision (MAP) and all the standard IR metrics.

While these metrics are not specific to MIR, I do think it would be useful to have them in `mir_eval` given their frequent use for evaluating music retrieval system (including but not limited to QBH and cover ID).

EDIT: maybe this would fit nicely in a `retrieval` module?


I think the last time this came up @bmcfee argued that all of these scores were already implemented in `scikit-learn`.  Maybe we can come up with a list of the metrics which are commonly used in MIR and then see if any are not already implemented in standard libraries?


Alright, maybe @bmcfee could give a short statement?

For me it would be very useful to have those metrics in a central place, even though it might produce redundancy with other packages. Furthermore, I think that this "retrieval" module will be a rather small one.

@justinsalamon I'd like to finish the DTW stuff in librosa first and then come back to this. Do you have any unit-tested code which could help here?

Thanks!


I still think we shouldn't reinvent the wheel here.  I don't see a strong argument for adding this to mir_eval if it already exists in a standard form in sklearn.

> Furthermore, I think that this "retrieval" module will be a rather small one.

I kinda doubt that.  These things can be annoyingly hard to get right.  Relevant bits from sklearn weigh in around 2KLOC:
- [ranking.py](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/ranking.py)
- [classification.py](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/metrics/classification.py)


> These things can be annoyingly hard to get right.

Maybe we mix up things here.
I'm talking about measures like (taken from http://www.music-ir.org/mirex/wiki/2013:Audio_Cover_Song_Identification):
- Total number of covers identified in top 10
- Mean number of covers identified in top 10 (average performance)
- Mean (arithmetic) of Avg. Precisions
- Mean rank of first correctly identified cover

I don't find these measures in sklearn.


> Total number of covers identified in top 10
> Mean number of covers identified in top 10 (average performance)

That's recall and precision at 10, respectively.  You can compute these with sklearn's [PRC](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#sklearn.metrics.precision_recall_curve) function and indexing the output at `k=10`.

> Mean (arithmetic) of Avg. Precisions

I'm not sure what you mean exactly by "average precision".  Usually in IR, that's computed as the average precision at all positive recall points.  [Mean AP](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.average_precision_score.html#sklearn.metrics.average_precision_score) is the average over all queries.

> Mean rank of first correctly identified cover

IMO, there's no good reason to use this metric, and I've never seen it outside of a couple of MIR papers.  Mean reciprocal rank (MRR) is the standard way to measure this kind of quantity, and is much better behaved numerically.


If you're willing @stefan-balke and @justinsalamon it may be good to enumerate _all_ of the IR metrics used in MIREX and MIR papers and then try to recreate them using `scikit-learn`.  If it's infeasible for any of them, we can discuss further.  I think this has already been done for cover songs, any other metrics of interest?


> it may be good to enumerate all of the IR metrics used in MIREX and MIR papers and then try to recreate them using scikit-learn

My thoughts on this went in the same direction. We could then discuss if it's meaningful to have some wrapper-functions in `mir_eval` or just put the necessary function calls for sklearn into the docs.


We can compile a list, though my guess would be that all the IR metrics used in MIR can probably be computed using `sklearn`. That said, one potential benefit of having these re-implemented in `mir_eval` (or providing wrappers in `mir_eval` around sklearn, though that would be a pretty large new dependency) is that we can then provide wrappers around those in `jams` which would make evaluation of (M)IR tasks on datasets with `jams` annotations very straight forward.


>  (or providing wrappers in mir_eval around sklearn, though that would be a pretty large new dependency)

It used to be a dependency, and now it's not #20.  If there's justification for it being called in a few different places, I could be convinced to add it back in the mix (partially because we are not trying to create evaluator binaries #65, which was one of the reasons, apart from it being a heavy and not-as-standard dependency).

> we can then provide wrappers around those in jams which would make evaluation of (M)IR tasks on datasets with jams annotations very straight forward.

Is it an absolute requirement that metrics be implemented in `mir_eval` in order to be called from the `jams` library?  If not, if the translation is simple enough, couldn't you just call appropriate `scikit-learn` functions from `jams` directly?  If so, I'm curious why, but we don't need to discuss that here.


> Is it an absolute requirement that metrics be implemented in mir_eval in order to be called from the jams library? If not, if the translation is simple enough, couldn't you just call appropriate scikit-learn functions from jams directly? If so, I'm curious why, but we don't need to discuss that here.

In principle no, we could write wrappers around a different library. That said, as you can see from the [wrapper code](https://github.com/marl/jams/blob/master/jams/eval.py) all the wrappers are very simple because they just need to get the data in the right format and then they let `mir_eval` take if from there. If we were to wrap around `sklearn` it would involve mimicking the `mir_eval` output structure etc., and I wonder whether that's in/outside the scope of `jams`.

@bmcfee a penny for your thoughts?


> If we were to wrap around sklearn it would involve mimicking the mir_eval output structure etc., and I wonder whether that's in/outside the scope of jams.

I think it's outside the scope of jams.

FWIW, our previous solution in #20 was to copy-paste the relevant bits of code into mir_eval.  Hurray for BSD licensing! (Also, the install problems referenced in that thread should no longer be a problem since all the C dependencies have moved over to shipping wheels.)

I'm warming a bit to the idea of mir_eval including tagging and retrieval metrics, but it will take some serious thought.
- What's the scope?  Within-track annotations (eg time-varying tagging) or across-track (eg cover retrieval)?  The current mir_eval conventions are really geared toward comparing annotations on a track-by-track basis, so it makes sense to start there.
- What does the output format look like?  In general, we might have multi-label data and want scores on a per-label basis.  In that case, the output dictionary from `evaluate` might have an unspecified key set, or at least a level of indirection, eg, `{'F1_micro': {'guitar': 0.7, 'bass': 0.3, ...}, 'Precision': {...}}`.
- What does the input format look like?  I'm a a fan of `(interval, label)` tuples (maybe with a confidence/score attached) but that might be insufficient for certain applications.  I think it makes more sense to do it this way and then parameterize the metrics by sampling rate (eg 10hz as in segmentation), rather than accept point-wise annotations, which makes comparison between annotations at different frame rates more difficult.
- What would an api look like for corpus-level evaluation, like cover retrieval or full-track tagging?  Would we even need to define such a thing, or just expose the ranking and classification metrics in such a way as to make it easy?


Is the consensus that the required metrics are best left to scikit-learn?  If so, can you close this issue?

For now it's fine but some day :)

I'm doing this atm with scikit-learn plus pandas, when that is stable I might present it here. But for cleaning up the issues this can be closed imho.

Ok, feel free to re-open if there should be movement on this.