Repository: mir-evaluation/mir_eval
Issue: "incomplete" files passed to melody.evaluate should pass a warning.
Root cause: Documentation Deficiency

f0 annotations are often output by algorithms with the "unvoiced" frames omitted, resulting in an incomplete set of time stamps. I was evaluating one of these types of melody output against a ground truth (with all time stamps, f0=0 for unvoiced frames), and melody.evaluate runs without problems.

After a brief talk with @justinsalamon, I learned that melody.evaluate is actually interpolating the missing f0 values. I can see use cases where you would want this behavior, but I might not have caught this problem. So, maybe melody.evaluate should pass a warning if a file with unevenly spaced time stamps is inputted, or, better yet, have a parameter to "fill missing frames".

Example files can be found [here](https://gist.github.com/rabitt/c45b969adf927cbe14b2)


So I would consider the interpolation that it's doing to be "fill in missing frames", do you have something else in mind?  Either way, looking over the documentation it's not totally clear that this is what's going on by default (lots of mention to resampling, as opposed to interpolation).  I can add this and/or a warning.


The documentation _does_ explicitly describe the convention for encoding silence: https://github.com/craffel/mir_eval/blob/master/mir_eval/melody.py#L14

So, in essence, a file where the silences are skipped is against the expected convention and not currently supported. That said, I totally agree that it can't hurt to add a warning, something a la - "This file seems to contain non-uniform timestamps. Is silence encoded properly (0 or negative Hz values)? Files that don't adhere to the expected convention may result in incorrect evaluation".

The alternative would be to fill-in the missing values, but that (as @rabitt and I have found) is a non-trivial thing to do.


Right, but it doesn't describe what it does given missing frames (other than "resampling"), which is somewhat vague.  That warning sounds reasonable.  


@craffel  I see what you mean about "fill missing frames" being ambiguous. Perhaps I should call it "fill missing unvoiced frames". Unfortunately, there are a lot of pitch tracking algorithms that only return the voiced frames. As @justinsalamon mentioned, it's nontrivial to convert the output to one that is mir-eval friendly in general (I have a [scripts](https://github.com/rabitt/medleydb/blob/master/scripts/fill_tony_file.py) to do it for the output of [Tony](https://code.soundsoftware.ac.uk/projects/tony), but it makes some assumptions about the format of the data -- namely the hopsize). It could be a nice thing to add to mir-eval, which is the only use case where I've absolutely needed it in the format with 0's in the unvoiced frames. 


In that case, it sounds like there is a commonly used annotation format which `mir_eval` does not correctly handle.  To summarize my understanding:
- `mir_eval` expects melody input in the form of two 1-d `np.ndarray`s of the same length, one representing frequency values and one representing the time at which the frequency values occur.  A frequency of 0 or a negative value indicates "unvoiced"
- Some melody estimation algorithms don't output time/frequency values when they believe the audio is unvoiced
- As part of the data pre-processing, `mir_eval` resamples the estimated melody annotation to match the timebase of the reference annotation, which is correct.  However, an algorithm denoting unvoiced frames as skipped values in the annotation would manifest as large gaps between successive time/frequency values.  In the resampling process, this would mean there would end up being linearly interpolated frequency values between the last observed frequency and the next observed frequency, all treated as voiced.
- @rabitt wrote some code which adds unvoiced frames to an annotation which is missing then, given an assumption about the timebase the annotation was sampled on.

If the above is correct, I'd suggest that @rabitt and @justinsalamon converge on an agreement about conventions, and that the code for inferring missing unvoiced frame is added to `mir_eval.util`, and that notes are added to the documentation about all of this.  If this is the case, please submit a pull request (to the develop branch).


@craffel yes, the above is correct, with the exception that I wasn't aware that this (skipping silent frames) was a common format - since mirex requires the zero/neg format, all the algorithms that have been submitted (afaik) follow this format. @rabitt, out of curiosity, other than Tony what other algorithms/tools have you encountered that skip silent frames?

In any case, I think it is a good idea to support both formats. @rabitt, since you already have the code for imputing missing frames, do you want to take point on this? In addition to the imputation itself we'd have to write a few lines to actually detect _when_ there is a time gap larger than the hop size (which we'd have to infer), but that should be relatively straight forward to do. Note however - if for some reason someone decides to use a time-series with non-uniform time hops this imputation will completely destroy their data, so we should also add a warning whenever the imputation is applied and provide an option for disabling it.

@rabitt I'm off the grid until the first week of January but if you wanna look at his together I'm happy to do so then.


> Note however - if for some reason someone decides to use a time-series with non-uniform time hops this imputation will completely destroy their data, so we should also add a warning whenever the imputation is applied and provide an option for disabling it.

I'd suggest that this take the form of a separate function which can be optionally called between `mir_eval.io.load_time_series` and `mir_eval.melody.evaluate` (or when not using `evaluate`, `mir_eval.melody.to_cent_voicing`).  So, it will only ever be applied if the user explicitly calls it.  This, coupled with the warning telling them that their time series is non-uniform, should cover all our bases.


@rabitt @justinsalamon any progress/plan on adding this functionality?


I'll work on this later on this week. I agree with @craffel's last suggestion - by default we should only issue a warning when the time series is non-uniform, and provide the option to explicitly activate an intermediate "silence imputation" step.


@justinsalamon Do you think you'll work on this today or tomorrow?  Trying to get a release out by tomorrow night.  If not, I can look into it.


@craffel I'm in the middle of a shitstorm of work, topped by the icing of having to move apartments. Right now the only "builds" I'm working on is disassembling my ikea furniture :hammer:  Wasn't aware there was a release pending, sorry. In this case, you might want to take a quick look.


How would you (and @rabitt) feel for now just issuing a warning when `not np.allclose(np.diff(times), np.diff(times).mean())`?  And then we can add the functionality for files with gaps at a later date, if @rabitt wants to port her code?


I think that could work for now, yes. Something along the lines of "Warning: non-uniform timestamps detected. Pitch will be linearly interpolated between every pair of points. Silences should be indicated by nonpositive frequency values (as per MIREX)." (just a suggestion).


PR: #142