Repository: mir-evaluation/mir_eval
Issue: melody.evaluate does not give perfect scores when estimate and reference are identical
Root cause: Algorithmic Error

Just for kicks, I fed melody.evaluate the same file for the reference and the estimate. 

``` python
file_path = "'Phoenix_ScotchMorris_MELODY2.csv'"
ref_time, ref_freq = mir_eval.io.load_time_series(file_path, ',')
est_time, est_freq = mir_eval.io.load_time_series(file_path, ',')
scores = mir_eval.melody.evaluate(ref_time, ref_freq, est_time, est_freq)
```

I was surprised to see that the scores are high, but not 100%:

``` python
scores
OrderedDict([('Voicing Recall', 0.99310953321289019), ('Voicing False Alarm', 0.025912137006701415), ('Raw Pitch Accuracy', 0.99310953321289019), ('Raw Chroma Accuracy', 0.9937397588336625), ('Overall Accuracy', 0.98892384322978111)])
```

Maybe there's a bug buried somewhere?


Interesting. I just tried to replicate this experiment, but I got perfect scores:

``` python
scores
Out[7]: OrderedDict([('Voicing Recall', 1.0), ('Voicing False Alarm', 0.0), ('Raw Pitch Accuracy', 1.0), ('Raw Chroma Accuracy', 1.0), ('Overall Accuracy', 1.0)])
```

Can you send me the file you were using?

Off the top of my head - when you don't provide a hop size only one sequence is resampled, i.e. the estimate is resampled to the time-grid of the reference (even if they have the same time grid originally). Maybe what you're seeing is an artefact of the interpolation+floating point precision. But I haven't looked into it.


Looks like github doesn't let me attach files like that, so Justin I'm
emailing it to you.

On Thu, Dec 4, 2014 at 4:30 PM, justinsalamon notifications@github.com
wrote:

> Interesting. I just tried to replicate this experiment, but I got perfect
> scores:
> 
> scores
> Out[7]: OrderedDict([('Voicing Recall', 1.0), ('Voicing False Alarm', 0.0), ('Raw Pitch Accuracy', 1.0), ('Raw Chroma Accuracy', 1.0), ('Overall Accuracy', 1.0)])
> 
> Can you send me the file you were using?
> 
> Off the top of my head - when you don't provide a hop size only one
> sequence is resampled, i.e. the estimate is resampled to the time-grid of
> the reference (even if they have the same time grid originally). Maybe what
> you're seeing is an artefact of the interpolation+floating point precision.
> But I haven't looked into it.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/craffel/mir_eval/issues/101#issuecomment-65648651.


Please put it on http://gist.github.com if you don't mind.


[Example file uploaded here](https://gist.github.com/rabitt/4a7a3349f28f2a331f9d)


I also don't get perfect scores, but I am getting different scores than you:

```
OrderedDict([('Voicing Recall', 0.99310953321289019), ('Voicing False Alarm', 0.025912137006701415), ('Raw Pitch Accuracy', 0.99310953321289019), ('Raw Chroma Accuracy', 0.9937397588336625), ('Overall Accuracy', 0.98892384322978111)])
```

Are you using the latest of `mir_eval`?  I noticed that you're not supplying a delimiter to `load-time_series`, which makes me think you have some older version which uses a comma by default.  It should be 

``` Python
ref_time, ref_freq = mir_eval.io.load_time_series(file_path, ',')
est_time, est_freq = mir_eval.io.load_time_series(file_path, ',')
```

I'll look into why the scores are not perfect.


@craffel notice rachel's output values have been updated and match yours as well as mine:

``` python
In [13]: scores
Out[13]: OrderedDict([('Voicing Recall', 0.99310953321289019), ('Voicing False Alarm', 0.025912137006701415), ('Raw Pitch Accuracy', 0.99310953321289019), ('Raw Chroma Accuracy', 0.9937397588336625), ('Overall Accuracy', 0.98892384322978111)])
```


Sorry @craffel  - the scores in the first comment were for a different file. I've updated it to match the uploaded file.


It's cased by `mir_eval.melody.resample_melody_series`.  By default, the estimated melody is resampled using linear interpolation to match the reference.  Intuitively, it seems like if you resample something onto the timebase it's already sampled on, it shouldn't change, but it does.  I'll figure out why.


Looks like this is caused by the fact that the original times are rounded to 10 decimal places, but the new times are not:
https://github.com/craffel/mir_eval/blob/master/mir_eval/melody.py#L213
Looks like it's supposed to round both but at this commit I failed to translate both over:
https://github.com/craffel/mir_eval/commit/bcf9bf22d99262e2483b5eea61db28ba3aed0bf4#diff-75a0b5004723bc936d0ad974d7b2cc2dL155
Making this change fixes the issue.


Thanks for this @rabitt .


You got it.