Repository: mir-evaluation/mir_eval
Issue: Add window to melody.voicing_measures()

What about adding a time tolerance to the voicing measures for melody extraction?

Couldn't we use this from the onset module?

```
# Compute the best-case matching between reference and estimated onset
# locations
matching = util.match_events(reference_onsets, estimated_onsets, window)
```

I'm not sure if this is not too smart for a simple lookup in a region of events.
I also have to admit that I'm not too familiar with _bipartite_match() :)

Any recommendations here? Otherwise I would go for the simplest solution but don't want to create overhead.


@stefan-balke Interesting idea. Could you elaborate more on the motivation for adding this tolerance?


> What about adding a time tolerance to the voicing measures for melody extraction?

I could see this being useful, but as far as I know no-one is doing it.  We usually try to stick to standardized metrics

> Couldn't we use this from the onset module?

Not really; matching events is intended for the case where you have a list of event times, e.g. `[1., 1.1., 1.5, 10., 13.]` and voicing is represented as a boolean array, e.g. `[0, 0, 0, 1, 1, 0, 0, 1, 1, 1]`.


> Not really; matching events is intended for the case where you have a list of event times, e.g. [1., 1.1., 1.5, 10., 13.] and voicing is represented as a boolean array, e.g. [0, 0, 0, 1, 1, 0, 0, 1, 1, 1].

Well, you could convert from the boolean array to event times with a diff (and knowledge of the sample rate). That said, it's indeed not a standardized metric. It's also problematic since all the metrics are per-frame - if you add a tolerance, conceptually the metric becomes "per-note" (onset), even though melody extraction algorithms have no notion of notes. Makes more sense for evaluating transcription algorithms (i.e. algs that produce a piano-roll style transcription), where an onset (and offset) tolerance is indeed used.

Slightly off-topic, but we should add metrics for evaluating this kind of transcription. I might have cycles for doing this some time in February.


p.s. also, unlike onset eval, if an AME algorithm is slightly off, it will only be marginally penalized (since only a few frames will be incorrect). Adding a tolerance would also be problematic since e.g. imagine an f0 sequence returned by an algorithm that has a few unvoiced frames in the middle, whilst the reference does not. For any sensible tolerance value this would still crop-up as a completely incorrect onset, possibly over-penalizing the algorithm. Not sure it makes sense to evaluate the algorithms on a task they weren't designed to do.


Hey there,

first, my motivation comes from work I'm currently doing on annotator agreement:
I have several f0 trajectories for the same tracks and want to see if the different annotators are agreeing or not. Since MIREX is splitting the melody extraction task into voicing and frequency related subtasks, I do this as well.

So, when calculating the figures for the melody extraction, e.g. for frequency I can set the tolerance for the agreement. In my case, I'm dealing with jazz solos and annotations which are either annotations on a "musical pitch" or on "frequency bins" - so two different granularities of resolution.
With that in mind, I could now perform a pairwise comparison for all annotator-pairs with sweeping over the frequency tolerance, the "pitch accuracy" looks like this:

![wayneshorter_footprints_solo](https://cloud.githubusercontent.com/assets/7326732/11869371/a0b695a4-a4bf-11e5-90d9-ee57c6fe16d4.png)

I'm now thinking of doing the same for the voicing. My hypothesis for this is that different human annotators chose different note onsets. With allowing now some tolerance, I might see a similar behaviour as with the frequency tolerance shown above.

Maybe my scenario is special and my assumptions are not valid for every AME alogrithm, since the mentioned intra-note problems @justinsalamon mentioned.
Question to this:
Wouldn't it be the other way around? I think of the window as a kind of "dilation" around a window,
so "into-note unvoicing" would be "filtered" away since I'm looking to a region around the current frame?

To give some basis for discussion, first I would do something as simple as this:

```
# no window - so do the old thing
if window == 0:
    TP = (ref_voicing*est_voicing).sum()
    FP = ((ref_voicing == 0)*est_voicing).sum()
    FN = (ref_voicing*(est_voicing == 0)).sum()
    TN = ((ref_voicing == 0)*(est_voicing == 0)).sum()

# Oh, I have some tolerance to take into account...
if window > 0:
    TP = 0
    FP = 0
    FN = 0
    TN = 0

    window = int(window / 2)

    # do it for every frame...
    for cur_est_idx, cur_est_vx in enumerate(est_voicing):
        window_start_idx = np.max([0, cur_est_idx-window])
        window_end_idx = np.min([cur_est_idx+window, len(ref_voicing)])

       # TP, FP... calculations by not just comparing with the ref_voicing[cur_est_idx],
       # but also looking in the window around this...
```

So far, thanks a lot for discussing this...

P.S.: Also, to give some more background: I'm following Arthur Flexer's analysis on Music similarity and inter-annotator agreement - but applying this to F0 annotations:
Arthur Flexer - On inter-rater agreement in audio music similarity, ISMIR 2014
Maybe the whole "Fleiss' Kappa" thing could also be interesting for mir_eval, as people are starting to think about inter-annotator agreement as a problematic point in many annotations - but that's another story.


@stefan-balke before digging further I must ask - how were your f0 annotations generated? To the best of my knowledge no human ever annotates an f0 curve manually, it is always obtained by means of an automatic pitch tracker that is then manually corrected. Does it makes sense to compare annotations under this scenario?

In any case, I think we're starting to veer off-topic (assuming the topic is extending `mir_eval`). At this point I should echo @craffel in that this is probably not standard enough to be included in the library at this stage. You can, of course, extend your local copy for the purpose of running your experiments. I'm happy to continue discussing metrics offline (well, online but off-hub) over email.


Okay, got this. My intention was more like "Oh, it's there for frequency and there might be the same thing for time in the package - so let's just spend another function parameter".
But that's fine, I'll add it in my local branch...

Coming back to your question @justinsalamon:
Yes, the annotations are from a tool which is based on the same salience function as MELODIA but with user input.
Rest is coming via mail!


> You can, of course, extend your local copy for the purpose of running your experiments. I'm happy to continue discussing metrics offline (well, online but off-hub) over email.

It's worth mentioning that if/when you suss out this technique and publish these results, it's the kind of thing we would be interested in having in a PR!


@craffel should we continue the discussion here then (for posterity's sake)?


Up to you!