Repository: mir-evaluation/mir_eval
Issue: Unit tests
Root cause: Unimplemented Feature Gap

Create unit tests, which should verify that the output is the same as reference implementations.
- [x] beat should match the beat evaluation toolbox http://code.soundsoftware.ac.uk/projects/beat-evaluation/ @craffel
- [x] onset should match https://github.com/CPJKU/onset_detection/blob/master/onset_evaluation.py @craffel
- [x] separate should match http://bass-db.gforge.inria.fr/bss_eval/ @dawenl 
- [x] segment should match MIREX output @bmcfee
- [x] chord should match Pauwel's findings https://github.com/jpauwels/mirex-tools @ejhumphrey 
- [ ] melody should match MIREX 2005 MATLAB code @justinsalamon 
- [x] pattern should match MIREX 2013 MATLAB code @urinieto 


Currently, I'm including the example data/test scores in the repository directly.  The directory layout is such that there is a subfolder in tests/data for each submodule.


We don't know of a pre-existing segment eval codebase to compare against aside from the MIREX codebase, which will come in 0.0.2


@ejhumphrey can you list the codebases you are unittesting the chord stuff against?  


so, I'm porting / refactoring Taemin's code, which AFAIK isn't published
and builds upon Chris Harte / C4DM's chordtools:
http://code.soundsoftware.ac.uk/projects/c4dm-chord-transcriptions/repository

so far, my unittests are my own and based on what "should" be correct;
there is no guarantee that anyone else's code behaves this way though.. yet
at least? I'd rather start over with a test-ably correct implementation and
proceed from there.

e

On Sun, Mar 30, 2014 at 2:05 PM, craffel notifications@github.com wrote:

> @ejhumphrey https://github.com/ejhumphrey can you list the codebases
> you are unittesting the chord stuff against?
> 
> ## 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/16#issuecomment-39033555
> .


For sure.  In general, we are unit testing modules against any pre-existing implementation as a starting point, and will be deviating once we are confident that everything matches and works.  It would be great if we could get Taemin or Chris's code to output a mapping, and unit test the chord mappings against that.


Actually I just tried to run all the unittests and some segment data is missing (specifically, 'data/segment/reference.lab')? Is this a known issue? 


Sorry, I just added the unit test yesterday at the end of HAMR and neglected to check in the data.  It's there now.


cool, I pulled and now it fails an assertion:

```
======================================================================
FAIL: test_segment.test_segment_functions
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/Users/ejhumphrey/Dropbox/NYU/marldev/src/mir_eval_git/tests/test_segment.py", line 22, in test_segment_functions
    assert np.allclose(score, reference_metrics[metric])
AssertionError
```

...just me?

also, is anyone else getting a DeprecationWarning from the separation tests? cc @dawenl 

```
/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/scipy/fftpack/basic.py:133: DeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  z = zeros(s,x.dtype.char)
```


I'm not failing anything in test_segment_functions, but I am getting AttributeErrors/TypeErrors in util_test.py:

```
======================================================================
ERROR: Check that an interval set is interpolated properly, with boundaries
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/craffel/Documents/School and Research/mir-evaluate/tests/util_test.py", line 28, in test_interpolate_intervals
    util.interpolate_intervals(intervals, labels, time_points, 'N'),
AttributeError: 'module' object has no attribute 'interpolate_intervals'

======================================================================
ERROR: Check that two non-identical yield correct results.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/craffel/Documents/School and Research/mir-evaluate/tests/util_test.py", line 57, in test_intersect_files
    sublist1, sublist2 = util.intersect_files(flist1, flist2)
AttributeError: 'module' object has no attribute 'intersect_files'

======================================================================
ERROR: Check that an interval set is sampled properly, with boundaries
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/craffel/Documents/School and Research/mir-evaluate/tests/util_test.py", line 41, in test_intervals_to_samples
    intervals, labels, offset=0, sample_size=0.5, fill_value='N')
TypeError: intervals_to_samples() got an unexpected keyword argument 'offset'

----------------------------------------------------------------------
```


Looks to me like it doesn't see the changes I've pushed to util.py ... are you on master?


Yes, I am on master. All clear now. I didn't get the Warning on separation test, but am getting the same AssertionError as you did.


I just pulled your commits from a second ago and am getting more errors in util_test --- I will just wait until you are done coding and re-try.  The error in segment.py is on Dawen's system too, and not mine, so I will hack on it with him.


wait, _more_ errors? any idea why I'm not seeing any?

```
ejhumphrey ~/Dropbox/NYU/marldev/src/mir_eval_git/tests $ nosetests ./test_util.py
...
----------------------------------------------------------------------
Ran 3 tests in 0.002s

OK
```

I even deleted every *.pyc I could find in the tree, just to be sure.


OK, I am now totally up to date and am not seeing any errors in test util.  The segment assertion failure is also now happening on my computer, which means that the segment module is outputting different metrics than it was yesterday, which I will look into now.


@urinieto `pattern` has the values from MATLAB hard-coded in the test file.  Can you put them into an ordered dict or something and save them as a pickle in tests/data/pattern?


So I'm assuming y'all know / knew about NEMA's SVN repository floating
around the internets? http://nemadiy.googlecode.com/svn/

in particular, I found this, which seems kinda ... relevant:
http://nemadiy.googlecode.com/svn/analytics/trunk/src/main/java/org/imirsel/nema/analytics/evaluation/

On Mon, Apr 14, 2014 at 11:02 AM, craffel notifications@github.com wrote:

> @urinieto https://github.com/urinieto pattern has the values from
> MATLAB hard-coded in the test file. Can you put them into an ordered dict
> or something and save them as a pickle in tests/data/pattern?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/16#issuecomment-40375731
> .


Yeah, see this discussion here:
https://github.com/craffel/mir_eval/issues/19
The general consensus is that getting the java codebase running is a huge time commitment and not worth roadblocking for 0.0.1.  And, that it would be hard to interpret if we wanted to try to trace down any output mismatch, so it would be better to unit test against published MIREX scores as a black box (for tasks where the test data is available) and other reference implementations.  We are planning to unit test against the NEMA code for MIREX compatibility later on.


@craffel done adding the results into a pickle file, and modified the unit tests accordingly.


so, while I think this might warrant a separate thread / issue, I'll keep it here for the time being. 

I'm getting in the right ballpark on past published MIREX scores, but I'm beginning to doubt hitting these numbers is sane without a reference implementation to port directly. I keep finding matters of interpretation and could see various label resolution / comparison decisions going either way. There are plenty of methodological issues I could go on about at length, but I'm losing my mind trying to get a grip on a reasonable target.

In the meantime, I'm going to keep doing musically rational things as transparently as possible. I'm hoping that if nothing else, we can put an open chord eval framework in the hands of the community and say "alright folks, here's a good starting point, now let's actually discuss the methodological nuances of chord recognition in the light of day."

I've pushed everything I've got so far (will finish the handful of outstanding things tonight), keen to discuss whenever. 

Thoughts?


...sorry for the million posts, chrome and github do _not_ git along.


I haven't had the chance to compare the results of my eval code to mirex yet (tomorrow will be the day), but I suspect there'll be some variation as well, though I'm hoping it'll be within 1-2%...


Validating against MIREX (or any other codebase) is without a doubt the most tedious part of this project, but my main reason for being a stickler about it is that I think it is one of the concerns people will raise almost immediately.  "If your new library doesn't give the same results as MIREX, then how can we compare its results to results of the past?"  I don't think we should go insane making things exactly the same, but I do think backwards compatibility is an important feature as far as eval goes.  I 100% agree that many of these things come down to implementation, but being able to state that our implementation can be parametrized to reproduce someone else's implementation is an extremely powerful statement.  

To be more concrete, I think if we are off by a percent, we have a lot to be worried about.  If we're off by a small fraction of a percent, I think we're fine.  If we're exactly the same, we may have gone too far.  And, regardless of what we achieve, I think the paper will certainly need to include a direct comparison against existing implementations (including MIREX), just to make it clear what we are claiming.


One more thought - there is some point at which a discrepancy in the output of two pieces of code which are (from a high level) supposed to do the same thing indicates a bug.  That threshold varies somewhat (and is probably subjective), but it's a pretty vital exercise to determine why the output is different when it's anywhere near this threshold.  For example, in the beat submodule, my information gain score was off by a percent or so from the BET toolbox; after an afternoon of debugging it turned out to be due to the egregious way that MATLAB automatically sets histogram bins.  Knowing this, I'd argue that using MATLAB's histogram routine is "wrong", but I wouldn't be able to make the argument had I not traced down this discrepancy. 


If it's any consolation, Eric, I also got a headache trying to do this for
the MIREX chord scores (you were talking about chords, right?) during my
thesis and the early stages of this project.

Chord evaluation is broken anyway, so I think the correct thing to do
(which indeed you are doing) is to fix it, remaining as open, commented and
flexible as possible. MIR-eval will be such a useful tool when complete
that I'm sure it will be embraced within the community, and after this we
can use this code to steer people towards more sensible evaluation
techniques. Wish I had more time to help out on this project!

Matt

On Tue, Apr 15, 2014 at 8:14 AM, Eric J. Humphrey
notifications@github.comwrote:

> so, while I think this might warrant a separate thread / issue, I'll keep
> it here for the time being.
> 
> I'm getting in the right ballpark on past published MIREX scores, but I'm
> beginning to doubt hitting these numbers is sane without a reference
> implementation to port directly. I keep finding matters of interpretation
> and could see various label resolution / comparison decisions going either
> way. There are plenty of methodological issues I could go on about at
> length, but I'm losing my mind trying to get a grip on a reasonable target.
> 
> In the meantime, I'm going to keep doing musically rational things as
> transparently as possible. I'm hoping that if nothing else, we can put an
> open chord eval framework in the hands of the community and say "alright
> folks, here's a good starting point, now let's actually discuss the
> methodological nuances of chord recognition in the light of day."
> 
> I've pushed everything I've got so far (will finish the handful of
> outstanding things tonight), keen to discuss whenever.
> 
> Thoughts?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/16#issuecomment-40429339
> .

## 

Matt McVicar
Postdoctoral Researcher, National Institute of Advanced Industrial Science
and Technology
mattjamesmcvicar@gmail.com (personal), matt.mcvicar@aist.go.jp(professional)
blog/website: www.mattmcvicar.com


My quick response is (1) I agree, at least in principle, (2) I'll continue to look into the MIREX chord eval stuff further and try to figure it out any sources of discrepancy, and (3) I am concerned that the challenges I am facing are indicative of fundamental methodological issues in chord recognition. I'll sideline that last bit until I have a better handle of what's going on though.

Matt, I am talking about chords, and good gracious, do I appreciate the words of support. 

more to follow...


Hah, for what it's worth I only agree with myself in principle on this matter too.


Okay, good news / no more whining out of me ... I've got all but one of the MIREX metrics well under a 1% margin, with the handful of files I've compared mir_eval.chords with. The only problem comparison is majmin-inv, hovering right about ±1%, and I can't for the life of me figure out why. Could use for something more comprehensive (I think it'd be possible to compare against every file from 2013), but I think this will do for now.


MIREX segmentation metrics are an absolute mess.

Looking around the 2013 data, almost none of the reference annotations start at 0 and span the entire track duration; it looks like someone's gone in and trimmed out silence buffers.

AGH.


Maybe we should start our own MIREX? :D

On Tue, Apr 15, 2014 at 3:37 PM, Brian McFee notifications@github.comwrote:

> MIREX segmentation metrics are an absolute mess.
> 
> Looking around the 2013 data, almost none of the reference annotations
> start at 0 and span the entire track duration; it looks like someone's gone
> in and trimmed out silence buffers.
> 
> AGH.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/16#issuecomment-40524449
> .


well not to be too off-topic, but posting JSON to a server calling mir_eval
in the background sounds like a sweet demo...

On Tue, Apr 15, 2014 at 3:38 PM, urinieto notifications@github.com wrote:

> Maybe we should start our own MIREX? :D
> 
> On Tue, Apr 15, 2014 at 3:37 PM, Brian McFee notifications@github.comwrote:
> 
> > MIREX segmentation metrics are an absolute mess.
> > 
> > Looking around the 2013 data, almost none of the reference annotations
> > start at 0 and span the entire track duration; it looks like someone's
> > gone
> > in and trimmed out silence buffers.
> > 
> > AGH.
> > 
> > —
> > Reply to this email directly or view it on GitHub<
> > https://github.com/craffel/mir_eval/issues/16#issuecomment-40524449>
> > .
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/16#issuecomment-40524648
> .


Well, I'm now doing the thing we all don't want to do: crawling through the mirex/nema code.

The boundary detection code is here: https://code.google.com/p/nemadiy/source/browse/analytics/trunk/src/main/resources/org/imirsel/nema/analytics/evaluation/structure/resources/segmentRetrievalEval2.m
... and differs pretty dramatically from mir_eval's implementation.  

Specifically, it applies a greedy matching between estimated and reference boundaries, so that each boundary can only be counted once.  Our current implementation simply counts whether there was a hit for each estimate (precision) or a prediction for each reference (recall).  Of course, this gets dicey when you have something like `Ref = [1.0], Est=[0.75, 1.25]`, which will get `precision=1` (each prediction hit) and `recall=1` (each reference hit).  Whether the precision score here is correct is a matter of debate, as there's nothing about "uniqueness" in the definition.

On the other hand, the greedy matching heuristic used by mirex is guaranteed to be correct. Here's a counter-example: `Ref = [0.0, 0.5, 1.0], Est=[0.4, 0.9, 1.4], W=0.5`.  Here, the greedy matching will assign `Est[0]` to `Ref[1]`, `Est[1]` to `Ref[2]`, and leave `Ref[0]` and `Est[2]` unmatched; but this is clearly incorrect.

What we actually need is the size of a maximal matching between predicted and true boundaries, where matchings are only allowed within the specified window.  Dividing the match size `M` by `|estimate|` or `|reference|` will get precision and recall, respectively.

So, here's the question: do we allow for redundant hits and keep the simple implementation?  Or implement the much more expensive matching method?  In either case, mirex is provably wrong, and we should not support backwards compatibility here.


> On the other hand, the greedy matching heuristic used by mirex is [not] guaranteed
> to be correct. Here's a counter-example: Ref = [0.0, 0.5, 1.0], Est=[0.4,
> 0.9, 1.4], W=0.5. Here, the greedy matching will assign Est[0] to Ref[1],
> Est[1] to Ref[2], and leave Ref[0] and Est[2] unmatched; but this is clearly
> incorrect.

So the alignment is

Ref  0.0    0.5   1.0    INS
Est  DEL  0.4    0.9    1.4

which you consider "clearly incorrect"; you're thinking it should be

Ref  0.0  0.5  1.0
Est  0.4  0.9  1.4

?

I think the "greedy" result is pretty good.  I think it's the right
thing to do - you definitely want to penalize both insertions and
deletions, so you want each boundary to be able to be used only once.

I don't think it's that hard to implement the greedy scheme (find
nearest ref for each hyp, go through matching them and removing them
from the lists, see how much you have left at the end (none within
threshold)).

  DAn.

> What we actually need is the size of a maximal matching between predicted
> and true boundaries, where matchings are only allowed within the specified
> window. Dividing the match size M by |estimate| or |reference| will get
> precision and recall, respectively.
> 
> So, here's the question: do we allow for redundant hits and keep the simple
> implementation? Or implement the much more expensive matching method? In
> either case, mirex is provably wrong, and we should not support backwards
> compatibility here.
> 
> —
> Reply to this email directly or view it on GitHub.


Well, for what it's worth, I found a provably wrong result in the mirex melody results too. I haven't found the bit of code that caused it to be wrong (rummaging through nema code too...), but I know it's wrong. I'm at the point were I'm <1% on _most_ files, but not for files with deep vibrato where I suspect the difference in interpolation methodology means the mirex results are actually lower than they should be. I know we want backwards compatibility, but does it make sense to be backwards compatible to something that's not 100% correct?


p.s.

I think I see your point - that the "correct" assessment in your
example is that Est is right but shifted by 0.4 sec.  But that's not
an issue with the scoring, it's above that; if you want the scoring to
allow global shifts ... that becomes a harder scoring problem.
Easiest way is to score after incorporating a range of shifts on a
grid and take the min, but I'm not sure it's the scoring routine's
place to be trying to fix that.

  DAn.

On Tue, Apr 15, 2014 at 4:31 PM, Dan Ellis dpwe@ee.columbia.edu wrote:

> > On the other hand, the greedy matching heuristic used by mirex is [not] guaranteed
> > to be correct. Here's a counter-example: Ref = [0.0, 0.5, 1.0], Est=[0.4,
> > 0.9, 1.4], W=0.5. Here, the greedy matching will assign Est[0] to Ref[1],
> > Est[1] to Ref[2], and leave Ref[0] and Est[2] unmatched; but this is clearly
> > incorrect.
> 
> So the alignment is
> 
> Ref  0.0    0.5   1.0    INS
> Est  DEL  0.4    0.9    1.4
> 
> which you consider "clearly incorrect"; you're thinking it should be
> 
> Ref  0.0  0.5  1.0
> Est  0.4  0.9  1.4
> 
> ?
> 
> I think the "greedy" result is pretty good.  I think it's the right
> thing to do - you definitely want to penalize both insertions and
> deletions, so you want each boundary to be able to be used only once.
> 
> I don't think it's that hard to implement the greedy scheme (find
> nearest ref for each hyp, go through matching them and removing them
> from the lists, see how much you have left at the end (none within
> threshold)).
> 
>   DAn.
> 
> > What we actually need is the size of a maximal matching between predicted
> > and true boundaries, where matchings are only allowed within the specified
> > window. Dividing the match size M by |estimate| or |reference| will get
> > precision and recall, respectively.
> > 
> > So, here's the question: do we allow for redundant hits and keep the simple
> > implementation? Or implement the much more expensive matching method? In
> > either case, mirex is provably wrong, and we should not support backwards
> > compatibility here.
> > 
> > —
> > Reply to this email directly or view it on GitHub.


> which you consider "clearly incorrect"; you're thinking it should be
> 
> Ref  0.0  0.5  1.0
> Est  0.4  0.9  1.4
> 
> ?

Yes.  The point here is that there are many feasible matchings subject to the W<=0.5 constraint, and they do not all produce the same score.  If we're going to define boundary detection to require uniqueness, then we should define the scores in terms of the maximum feasible matching.  Since the greedy algorithm does not find this -- yes, it's a degenerate case, but the algorithm is still wrong -- we should not use it.

> I think the "greedy" result is pretty good.  I think it's the right
> thing to do - you definitely want to penalize both insertions and
> deletions, so you want each boundary to be able to be used only once.

Those are two separate ideas.  Yes, each boundary should probably only be used once -- and if this is the case, it should be written down somewhere.  However, the greedy algorithm does not produce stable/well-characterized scores, and the "right" algorithm does exist: http://en.wikipedia.org/wiki/Matching_(graph_theory)#In_unweighted_bipartite_graphs


> I think I see your point - that the "correct" assessment in your
> example is that Est is right but shifted by 0.4 sec.  But that's not
> an issue with the scoring, it's above that; if you want the scoring to
> allow global shifts ... that becomes a harder scoring problem.
> Easiest way is to score after incorporating a range of shifts on a
> grid and take the min, but I'm not sure it's the scoring routine's
> place to be trying to fix that.

The "global shift" isn't really the issue; it was just a simple way to construct a counter-example for the greedy algorithm's correctness.  With a larger window size (3s?), one could imagine less-regular counter-examples that don't look like global shifts.  I think this absolutely is an issue with the implementation of the scoring function.

I don't like the idea of trying to hack around it by manipulating the data.  There already exists a perfectly good algorithm to solve this problem the way we (should) want it.  It's just a question of whether now is the time to start making things more complicated in the name of correctness.  At any rate, I'm quite uncomfortable with propagating incorrect implementations for the sake of backwards compatibility.


My two cents: I used to evaluate my boundary algorithms using a "heuristic"
approach (similar to the one in MIREX). Simply because I wasn't aware that
there was any issue with that. A few months ago, after reading Brian's
boundary evaluation implementation in OLDA (and comparing it with my own
code) I realized that it makes more sense to check "whether there is a hit
for each estimate (precision) or a prediction for each reference (recall)",
instead of counting each boundary just once (which, as Brian just showed,
can produce erroneous results).

I never actually looked at the code in MIREX til now, and I'm actually
surprised that they are using a "heuristic" approach.

So, I would go for your implementation Brian. But I also think that we
should talk to the guys at MIREX about that, since I believe their
implementation is not accurate enough.

On Tue, Apr 15, 2014 at 4:49 PM, Brian McFee notifications@github.comwrote:

> I think I see your point - that the "correct" assessment in your
> example is that Est is right but shifted by 0.4 sec. But that's not
> an issue with the scoring, it's above that; if you want the scoring to
> allow global shifts ... that becomes a harder scoring problem.
> Easiest way is to score after incorporating a range of shifts on a
> grid and take the min, but I'm not sure it's the scoring routine's
> place to be trying to fix that.
> 
> The "global shift" isn't really the issue; it was just a simple way to
> construct a counter-example for the greedy algorithm's correctness. With a
> larger window size (3s?), one could imagine less-regular counter-examples
> that don't look like global shifts. I think this absolutely is an issue
> with the implementation of the scoring function.
> 
> I don't like the idea of trying to hack around it by manipulating the
> data. There already exists a perfectly good algorithm to solve this problem
> the way we (should) want it. It's just a question of whether now is the
> time to start making things more complicated in the name of correctness. At
> any rate, I'm quite uncomfortable with propagating incorrect
> implementations for the sake of backwards compatibility.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/16#issuecomment-40532450
> .


mmmhh maybe we could include both implementations? If we add the proper
documentation justifying this, it might even raise awareness why the MIREX
metric is actually incorrect (but if you still want to use it, go ahead and
do it).

On Tue, Apr 15, 2014 at 5:06 PM, Oriol Nieto oriol.nieto@gmail.com wrote:

> My two cents: I used to evaluate my boundary algorithms using a
> "heuristic" approach (similar to the one in MIREX). Simply because I wasn't
> aware that there was any issue with that. A few months ago, after reading
> Brian's boundary evaluation implementation in OLDA (and comparing it with
> my own code) I realized that it makes more sense to check "whether there
> is a hit for each estimate (precision) or a prediction for each reference
> (recall)", instead of counting each boundary just once (which, as Brian
> just showed, can produce erroneous results).
> 
> I never actually looked at the code in MIREX til now, and I'm actually
> surprised that they are using a "heuristic" approach.
> 
> So, I would go for your implementation Brian. But I also think that we
> should talk to the guys at MIREX about that, since I believe their
> implementation is not accurate enough.
> 
> On Tue, Apr 15, 2014 at 4:49 PM, Brian McFee notifications@github.comwrote:
> 
> > I think I see your point - that the "correct" assessment in your
> > example is that Est is right but shifted by 0.4 sec. But that's not
> > an issue with the scoring, it's above that; if you want the scoring to
> > allow global shifts ... that becomes a harder scoring problem.
> > Easiest way is to score after incorporating a range of shifts on a
> > grid and take the min, but I'm not sure it's the scoring routine's
> > place to be trying to fix that.
> > 
> > The "global shift" isn't really the issue; it was just a simple way to
> > construct a counter-example for the greedy algorithm's correctness. With a
> > larger window size (3s?), one could imagine less-regular counter-examples
> > that don't look like global shifts. I think this absolutely is an issue
> > with the implementation of the scoring function.
> > 
> > I don't like the idea of trying to hack around it by manipulating the
> > data. There already exists a perfectly good algorithm to solve this problem
> > the way we (should) want it. It's just a question of whether now is the
> > time to start making things more complicated in the name of correctness. At
> > any rate, I'm quite uncomfortable with propagating incorrect
> > implementations for the sake of backwards compatibility.
> > 
> > —
> > Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/16#issuecomment-40532450
> > .


My approach here: If MIREX is doing something which we see as incorrect, we should still provide a way to reproduce it, but make it non-default behavior and possibly throw a deprecation warning. Backwards compatibility is highly important for shared tasks and telling someone they're wrong is a good way to make them defensive and pissed off. 

Either way, this seems like something we should have a discussion about in order to define our plan of action. Tomorrow would be good as I don't think this should wait until Friday. 


FWIW, here's a quick implementation of a correct version (without boundary trimming or any of that):
https://gist.github.com/bmcfee/10782851


I pushed the matching-based algorithm, which happens to validate on the example data I pulled from mirex2013.


Segment is almost there.  

One general question though: how much numerical precision are we shooting for in these unit tests?

For the structure metrics (nce, pairwise), it's pretty easy to get into the right ballpark (2 digits), but that last digit is tricky.  I've tracked it down to an (implied) inconsistency/ambiguity between the mirex wiki and the implementation, where rather than sampling frames at a fixed rate (10Hz, as claimed in the pairwise section here: http://www.music-ir.org/mirex/wiki/2013:Structural_Segmentation#Frame_clustering ), they fix the number of samples to 1000 over the entire track duration.  Our numbers are close, but not quite exact.

And, because I just can't get enough of starting fights about backwards compatibility, I think the fixed-sample version is wrong because its accuracy changes from one track to the next.  Specifically, it under-represents short segments in long tracks.


So I'm operating on the assumption that two-digit consistency is money.

And, FWIW, I still contend that one of the most compelling arguments for
mir_eval is that _no one really knows what MIREX is_. Ya, we read the wiki,
we think we know, but you don't really know / get it until you dig into the
implementation. If anything, mir_eval is an opportunity to have a public
ledger, do our accounting in the town square, as it were, and I'm primarily
excited for the methodological conversations that should occur as a result.

On Wed, Apr 16, 2014 at 10:49 AM, Brian McFee notifications@github.comwrote:

> Segment is almost there.
> 
> One general question though: how much numerical precision are we shooting
> for in these unit tests?
> 
> For the structure metrics (nce, pairwise), it's pretty easy to get into
> the right ballpark (2 digits), but that last digit is tricky. I've tracked
> it down to an (implied) inconsistency/ambiguity between the mirex wiki and
> the implementation, where rather than sampling frames at a fixed rate
> (10Hz, as claimed in the pairwise section here:
> http://www.music-ir.org/mirex/wiki/2013:Structural_Segmentation#Frame_clustering), they fix the number of samples to 1000 over the entire track duration.
> Our numbers are close, but not quite exact.
> 
> And, because I just can't get enough of starting fights about backwards
> compatibility, I think the fixed-sample version is wrong because its
> accuracy changes from one track to the next. Specifically, it
> under-represents short segments in long tracks.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/16#issuecomment-40607914
> .


>  they fix the number of samples to 1000 over the entire track duration.
> Our numbers are close, but not quite exact.
> 
> I think the fixed-sample version is wrong because its
> accuracy changes from one track to the next. Specifically, it
> under-represents short segments in long tracks.

Agree 100%.

  DAn.


Aaaaaaand the saga continues, this time, in pairwise classification.  Quoting from the mirex wiki, which is consistent with the text in Levy & Sandler, 2008:

> Both the result and the ground truth are handled in short frames (e.g., beat or fixed 100ms). All frame pairs in a structure description are handled. The pairs in which both frames are assigned to the same cluster (i.e., have the same label) form the sets P_E (for the system result) and P_A (for the ground truth).

The Levy & Sandler explanation is similar:

> We evaluate segment labels with pairwise f-value, one of the standard metrics for quality of clustering. This compares pairs of beats which are labeled with the same segment-type (i.e., assigned to the same cluster)

The term "segment type" here denotes -- unambiguously -- that the labels are structure annotations (verse/chorus/verse) and not segment identifiers (segment 1, segment 2, segment 3).

The relevant mirex code for this metric can be found here: https://nemadiy.googlecode.com/svn/analytics/trunk/src/main/resources/org/imirsel/nema/analytics/evaluation/structure/resources/pairwiseF.m

`pairwiseF.m` does not even look at the segment labels, but instead, assigns a unique id to each segment.  This clearly deviates from both the wiki and the original paper, and again, I don't think we should try to reproduce their behavior.  Rather, we should fix it (already done) and point out the error.


Brian and I just finished discussing this pretty thoroughly.  Here's what we've settled on:  Each task submodule should have its output compared to at least one reference implementation, be it MIREX or something else.  The comparison should be done over at least 10 files. For any metrics which are not exactly the same (here, defined as deviating by more than 1e-4 relative), we should figure out why they are different.  Always assume that the original reference (the paper) describing the metric is correct.  Beyond that, if it's a difference in implementation, make sure we can back up why our code does it the way we do, and that we know exactly how our methodology is different.  If our code has a bug, we need to fix it (this is possible and this exercise is a good way to find bugs!).  If their code has a bug, be very careful to document what we believe the bug is.

This changes what we're doing here a little bit - we're not unit testing against existing implementations, we're just comparing and being very thorough.  In the end, we will only make statements of the following types:
- "mir_eval and (reference implementation) produce exactly the same output" 
- "mir_eval and (reference implementation) produce different outputs, because although they both follow (reference defining the metric) they differ in implementation in the following ways: ..."
- "mir_eval and (reference implementation) produce different outputs, because (reference implementation) does not exactly follow the definition of the metric in (reference defining the metric) in the following ways: ..."

Is this OK with everyone?


what??

Is this really MIREX implementation of the pair-wise clustering measure? I
can't believe they're not looking at the assigned labels at all. I assume
that the function "desc2seq" is the one responsible to assign unique
identifiers to the segments, but how does it do it? Maybe refDesc.times
includes the labels somehow?

This is pretty ridiculous...

On Wed, Apr 16, 2014 at 2:09 PM, Brian McFee notifications@github.comwrote:

> Aaaaaaand the saga continues, this time, in pairwise classification.
> Quoting from the mirex wiki, which is consistent with the text in Levy &
> Sandler, 2008:
> 
> Both the result and the ground truth are handled in short frames (e.g.,
> beat or fixed 100ms). All frame pairs in a structure description are
> handled. The pairs in which both frames are assigned to the same cluster
> (i.e., have the same label) form the sets P_E (for the system result) and
> P_A (for the ground truth).
> 
> The Levy & Sandler explanation is similar:
> 
> We evaluate segment labels with pairwise f-value, one of the standard
> metrics for quality of clustering. This compares pairs of beats which are
> labeled with the same segment-type (i.e., assigned to the same cluster)
> 
> The term "segment type" here denotes -- unambiguously -- that the labels
> are structure annotations (verse/chorus/verse) and not segment identifiers
> (segment 1, segment 2, segment 3).
> 
> The relevant mirex code for this metric can be found here:
> https://nemadiy.googlecode.com/svn/analytics/trunk/src/main/resources/org/imirsel/nema/analytics/evaluation/structure/resources/pairwiseF.m
> 
> pairwiseF.m does not even look at the segment labels, but instead,
> assigns a unique id to each segment. This clearly deviates from both the
> wiki and the original paper, and again, I don't think we should try to
> reproduce their behavior. Rather, we should fix it (already done) and point
> out the error.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/16#issuecomment-40632334
> .


This sounds good to me. Where in the code should we include this? Also, I
think this important enough to talk about it in a full section of the paper.

On Wed, Apr 16, 2014 at 2:33 PM, Oriol Nieto oriol.nieto@gmail.com wrote:

> what??
> 
> Is this really MIREX implementation of the pair-wise clustering measure? I
> can't believe they're not looking at the assigned labels at all. I assume
> that the function "desc2seq" is the one responsible to assign unique
> identifiers to the segments, but how does it do it? Maybe refDesc.times
> includes the labels somehow?
> 
> This is pretty ridiculous...
> 
> On Wed, Apr 16, 2014 at 2:09 PM, Brian McFee notifications@github.comwrote:
> 
> > Aaaaaaand the saga continues, this time, in pairwise classification.
> > Quoting from the mirex wiki, which is consistent with the text in Levy &
> > Sandler, 2008:
> > 
> > Both the result and the ground truth are handled in short frames (e.g.,
> > beat or fixed 100ms). All frame pairs in a structure description are
> > handled. The pairs in which both frames are assigned to the same cluster
> > (i.e., have the same label) form the sets P_E (for the system result) and
> > P_A (for the ground truth).
> > 
> > The Levy & Sandler explanation is similar:
> > 
> > We evaluate segment labels with pairwise f-value, one of the standard
> > metrics for quality of clustering. This compares pairs of beats which are
> > labeled with the same segment-type (i.e., assigned to the same cluster)
> > 
> > The term "segment type" here denotes -- unambiguously -- that the labels
> > are structure annotations (verse/chorus/verse) and not segment identifiers
> > (segment 1, segment 2, segment 3).
> > 
> > The relevant mirex code for this metric can be found here:
> > https://nemadiy.googlecode.com/svn/analytics/trunk/src/main/resources/org/imirsel/nema/analytics/evaluation/structure/resources/pairwiseF.m
> > 
> > pairwiseF.m does not even look at the segment labels, but instead,
> > assigns a unique id to each segment. This clearly deviates from both the
> > wiki and the original paper, and again, I don't think we should try to
> > reproduce their behavior. Rather, we should fix it (already done) and point
> > out the error.
> > 
> > —
> > Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/16#issuecomment-40632334
> > .


That's a good question; I think that comparison against MIREX should not be included in the mir_eval repository.  I will create a new repo for that stuff.  Re: paper section, yes, that's exactly the reason we need to be thorough about documenting differences.


> s this really MIREX implementation of the pair-wise clustering measure? I
> can't believe they're not looking at the assigned labels at all. I assume
> that the function "desc2seq" is the one responsible to assign unique
> identifiers to the segments, but how does it do it? Maybe refDesc.times
> includes the labels somehow?

Yup.  `desc2seq` is roughly analogous to our frame sampler.  It looks to me like they just forgot to map segment numbers to labels before computing agreement.  They do use the labels properly in NCE (search for labelMask): https://nemadiy.googlecode.com/svn/analytics/trunk/src/main/resources/org/imirsel/nema/analytics/evaluation/structure/resources/condEntropyEval.m


I see... since when has MIREX been using this implementation? This is
pretty terrible.

On Wed, Apr 16, 2014 at 2:39 PM, Brian McFee notifications@github.comwrote:

> s this really MIREX implementation of the pair-wise clustering measure? I
> can't believe they're not looking at the assigned labels at all. I assume
> that the function "desc2seq" is the one responsible to assign unique
> identifiers to the segments, but how does it do it? Maybe refDesc.times
> includes the labels somehow?
> 
> Yup. desc2seq is roughly analogous to our frame sampler. It looks to me
> like they just forgot to map segment numbers to labels before computing
> agreement. They do use the labels properly in NCE (search for labelMask):
> https://nemadiy.googlecode.com/svn/analytics/trunk/src/main/resources/org/imirsel/nema/analytics/evaluation/structure/resources/condEntropyEval.m
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/16#issuecomment-40635803
> .


> I see... since when has MIREX been using this implementation? This is
> pretty terrible.

At least 2010... 

https://code.google.com/p/nemadiy/source/list?path=/analytics/trunk/src/main/resources/org/imirsel/nema/analytics/evaluation/structure/resources/pairwiseF.m&start=2199


This is a response about the length of the discretization units used
in segment evaluation, not the broader question of what to validate
mir_eval against.

I believe it is unnecessary to have any time quantization in
segment-labeling evaluation (relevant to structure segmentation, chord
labeling, etc).  You can get a value that is based on continuous time
instead, which removes the dependence on quantization size, and which
seems more appealing to me.

The trick is to divide the time axis into regions where all time
points will be treated the same, calculate the relevant scores for
each of these, then weight by their duration.

In particular, you take the sorted union of all the boundary times in
both reference and hypothesized.  Then, the intervals between
successive elements in this boundary list have a single {ref_label,
hyp_label} pair.  At a very high frame rate, they would contribute
(frame_rate x segment_duration) individual quantized frames.  So treat
each one as a single point, then weight its contribution to the final
metric by segment_duration.  Assuming your metrics cancel out time at
the end, you get the asymptote you'd get with infinite frame rate, but
typically with less memory use than any reasonable frame rate.

See the core of
http://labrosa.ee.columbia.edu/projects/chords/labrosachordrecog-2010/score_chord_id.m
I'm sorry that code is so confounded by the details of the particular system.

For the "all possible pairs" metric, I think you weight by the product
of the durations of each of the segments in the pair, then somewhere
keep track of the normalization as the sum of the product of all pairs
of durations, corresponding to the N^2 (or N(N-1)/2) normalization
denominator that shows up in counting-based exhaustive-pairwise
measures.

  DAn.

On Wed, Apr 16, 2014 at 10:49 AM, Brian McFee notifications@github.com wrote:

> Segment is almost there.
> 
> One general question though: how much numerical precision are we shooting
> for in these unit tests?
> 
> For the structure metrics (nce, pairwise), it's pretty easy to get into the
> right ballpark (2 digits), but that last digit is tricky. I've tracked it
> down to an (implied) inconsistency/ambiguity between the mirex wiki and the
> implementation, where rather than sampling frames at a fixed rate (10Hz, as
> claimed in the pairwise section here:
> http://www.music-ir.org/mirex/wiki/2013:Structural_Segmentation#Frame_clustering
> ), they fix the number of samples to 1000 over the entire track duration.
> Our numbers are close, but not quite exact.
> 
> And, because I just can't get enough of starting fights about backwards
> compatibility, I think the fixed-sample version is wrong because its
> accuracy changes from one track to the next. Specifically, it
> under-represents short segments in long tracks.
> 
> —
> Reply to this email directly or view it on GitHub.


re: DAn's response ... this is what MIREX has shifted to for chords, and
what I've implemented in mir_eval (uri, this might sound familiar to you
from at least one of our previous conversations).

The trick is to divide the time axis into regions where all time

> points will be treated the same, calculate the relevant scores for
> each of these, then weight by their duration.

see this function:
https://github.com/craffel/mir_eval/blob/master/mir_eval/util.py#L350

It finds the union of all intervals, and may prove useful / relevant. it's
not bulletproof, but should be fine so long as interval[i, 1] ==
interval[i+1, 0]. I believe this is the only (primary?) assumption that
I've not robustly verified. sorry dudes.

DAn's next paragraph says as much, and I've nothing to offer on pairwise
stuff. WRT the weighting though, you can reference this method:
https://github.com/craffel/mir_eval/blob/master/mir_eval/chord.py#L950

obvious, but a reference nonetheless.

hth,
ejh

On Wed, Apr 16, 2014 at 2:55 PM, Dan Ellis notifications@github.com wrote:

> This is a response about the length of the discretization units used
> in segment evaluation, not the broader question of what to validate
> mir_eval against.
> 
> I believe it is unnecessary to have any time quantization in
> segment-labeling evaluation (relevant to structure segmentation, chord
> labeling, etc). You can get a value that is based on continuous time
> instead, which removes the dependence on quantization size, and which
> seems more appealing to me.
> 
> The trick is to divide the time axis into regions where all time
> points will be treated the same, calculate the relevant scores for
> each of these, then weight by their duration.
> 
> In particular, you take the sorted union of all the boundary times in
> both reference and hypothesized. Then, the intervals between
> successive elements in this boundary list have a single {ref_label,
> hyp_label} pair. At a very high frame rate, they would contribute
> (frame_rate x segment_duration) individual quantized frames. So treat
> each one as a single point, then weight its contribution to the final
> metric by segment_duration. Assuming your metrics cancel out time at
> the end, you get the asymptote you'd get with infinite frame rate, but
> typically with less memory use than any reasonable frame rate.
> 
> See the core of
> 
> http://labrosa.ee.columbia.edu/projects/chords/labrosachordrecog-2010/score_chord_id.m
> I'm sorry that code is so confounded by the details of the particular
> system.
> 
> For the "all possible pairs" metric, I think you weight by the product
> of the durations of each of the segments in the pair, then somewhere
> keep track of the normalization as the sum of the product of all pairs
> of durations, corresponding to the N^2 (or N(N-1)/2) normalization
> denominator that shows up in counting-based exhaustive-pairwise
> measures.
> 
> DAn.
> 
> On Wed, Apr 16, 2014 at 10:49 AM, Brian McFee notifications@github.com
> wrote:
> 
> > Segment is almost there.
> > 
> > One general question though: how much numerical precision are we
> > shooting
> > for in these unit tests?
> > 
> > For the structure metrics (nce, pairwise), it's pretty easy to get into
> > the
> > right ballpark (2 digits), but that last digit is tricky. I've tracked
> > it
> > down to an (implied) inconsistency/ambiguity between the mirex wiki and
> > the
> > implementation, where rather than sampling frames at a fixed rate (10Hz,
> > as
> > claimed in the pairwise section here:
> > 
> > http://www.music-ir.org/mirex/wiki/2013:Structural_Segmentation#Frame_clustering
> > ), they fix the number of samples to 1000 over the entire track
> > duration.
> > Our numbers are close, but not quite exact.
> > 
> > And, because I just can't get enough of starting fights about backwards
> > compatibility, I think the fixed-sample version is wrong because its
> > accuracy changes from one track to the next. Specifically, it
> > under-represents short segments in long tracks.
> > 
> > —
> > Reply to this email directly or view it on GitHub.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/16#issuecomment-40637703
> .


Yes Eric, it's quite interesting that Dan's explanation is basically what
you told me about the other day :)

On Wed, Apr 16, 2014 at 8:35 PM, Eric J. Humphrey
notifications@github.comwrote:

> re: DAn's response ... this is what MIREX has shifted to for chords, and
> what I've implemented in mir_eval (uri, this might sound familiar to you
> from at least one of our previous conversations).
> 
> The trick is to divide the time axis into regions where all time
> 
> > points will be treated the same, calculate the relevant scores for
> > each of these, then weight by their duration.
> 
> see this function:
> https://github.com/craffel/mir_eval/blob/master/mir_eval/util.py#L350
> 
> It finds the union of all intervals, and may prove useful / relevant. it's
> not bulletproof, but should be fine so long as interval[i, 1] ==
> interval[i+1, 0]. I believe this is the only (primary?) assumption that
> I've not robustly verified. sorry dudes.
> 
> DAn's next paragraph says as much, and I've nothing to offer on pairwise
> stuff. WRT the weighting though, you can reference this method:
> https://github.com/craffel/mir_eval/blob/master/mir_eval/chord.py#L950
> 
> obvious, but a reference nonetheless.
> 
> hth,
> ejh
> 
> On Wed, Apr 16, 2014 at 2:55 PM, Dan Ellis notifications@github.com
> wrote:
> 
> > This is a response about the length of the discretization units used
> > in segment evaluation, not the broader question of what to validate
> > mir_eval against.
> > 
> > I believe it is unnecessary to have any time quantization in
> > segment-labeling evaluation (relevant to structure segmentation, chord
> > labeling, etc). You can get a value that is based on continuous time
> > instead, which removes the dependence on quantization size, and which
> > seems more appealing to me.
> > 
> > The trick is to divide the time axis into regions where all time
> > points will be treated the same, calculate the relevant scores for
> > each of these, then weight by their duration.
> > 
> > In particular, you take the sorted union of all the boundary times in
> > both reference and hypothesized. Then, the intervals between
> > successive elements in this boundary list have a single {ref_label,
> > hyp_label} pair. At a very high frame rate, they would contribute
> > (frame_rate x segment_duration) individual quantized frames. So treat
> > each one as a single point, then weight its contribution to the final
> > metric by segment_duration. Assuming your metrics cancel out time at
> > the end, you get the asymptote you'd get with infinite frame rate, but
> > typically with less memory use than any reasonable frame rate.
> > 
> > See the core of
> > 
> > http://labrosa.ee.columbia.edu/projects/chords/labrosachordrecog-2010/score_chord_id.m
> > I'm sorry that code is so confounded by the details of the particular
> > system.
> > 
> > For the "all possible pairs" metric, I think you weight by the product
> > of the durations of each of the segments in the pair, then somewhere
> > keep track of the normalization as the sum of the product of all pairs
> > of durations, corresponding to the N^2 (or N(N-1)/2) normalization
> > denominator that shows up in counting-based exhaustive-pairwise
> > measures.
> > 
> > DAn.
> > 
> > On Wed, Apr 16, 2014 at 10:49 AM, Brian McFee notifications@github.com
> > 
> > wrote:
> > 
> > > Segment is almost there.
> > > 
> > > One general question though: how much numerical precision are we
> > > shooting
> > > for in these unit tests?
> > > 
> > > For the structure metrics (nce, pairwise), it's pretty easy to get
> > > into
> > > the
> > > right ballpark (2 digits), but that last digit is tricky. I've tracked
> > > it
> > > down to an (implied) inconsistency/ambiguity between the mirex wiki
> > > and
> > > the
> > > implementation, where rather than sampling frames at a fixed rate
> > > (10Hz,
> > > as
> > > claimed in the pairwise section here:
> > 
> > http://www.music-ir.org/mirex/wiki/2013:Structural_Segmentation#Frame_clustering
> > 
> > > ), they fix the number of samples to 1000 over the entire track
> > > duration.
> > > Our numbers are close, but not quite exact.
> > > 
> > > And, because I just can't get enough of starting fights about
> > > backwards
> > > compatibility, I think the fixed-sample version is wrong because its
> > > accuracy changes from one track to the next. Specifically, it
> > > under-represents short segments in long tracks.
> > > 
> > > —
> > > Reply to this email directly or view it on GitHub.
> > 
> > —
> > Reply to this email directly or view it on GitHub<
> > https://github.com/craffel/mir_eval/issues/16#issuecomment-40637703>
> > .
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/16#issuecomment-40668921
> .


It would actually be pretty easy to rework the structure metrics to use interval intersections (see example 2 at http://nbviewer.ipython.org/github/craffel/crucialpython/blob/master/week5/outer.ipynb?create=1 ) to directly compute the contingency tables without sampling.  This would have the added benefit of being much more memory-efficient for long tracks.

Shall we make this a to-do for 0.0.2?


As per discussion above, we won't be testing against reference implementations anymore; instead we will be doing regression testing to make sure our metrics are fixed (unless we're absolutely positive we want them to change) and actual unit testing (sanity checking on individual functions from each submodule).  We will be _comparing_ against existing implementations, but this will not be a part of this repository/package.  New issue is #48