zhiminy/EvalEng huggingface dataset:
Repository: mir-evaluation/mir_eval
Issue: Docstrings in pattern functions need more detail

The documentation for `patterns.evaluate` and `io.load_patterns` does not actually explain the data structure produced by the loader, or required by the evaluator.

In `load_patterns`, you get

```
Returns
-------
pattern_list : list
    The list of patterns, containing all their occurrences,
    using the following format::

        pattern_list = [pattern1, ..., patternN]
        pattern = [occurrence1, ..., occurrenceM]
        occurrence = [onset_midi1, ..., onset_midiO]
        onset_midi = (onset_time, midi_number)

    E.g.::
        P = [[[(77.0, 67.0), (77.5, 77.0), ... ]]]
```

but this does not explain what `N`, `M`, and `O` are.


What do you want it to say, "where `N` is the number of patterns, `M` is the number of occurrences, and `O` is the number of onsets"?  Or just remove the indexes, e.g. `pattern_list = [patterns ...]`?  Is the description "Loads the patters contained in the filename and puts them into a list of patterns, each pattern being a list of occurrence, and each occurrence being a list of (onset, midi) pairs." not enough?  @urinieto would need to confirm the actual semantics here, I am just guessing how the pattern stuff works.


> What do you want it to say, "where N is the number of patterns, M is the number of occurrences, and O is the number of onsets"?

Better yet:

```
where N is the number of patterns, M[i] is the number of occurrences of the i'th pattern, and O[j] is the number of onsets in the j'th occurrence
```

As it currently reads, it's not clear that there is a separate M for each pattern, and a separate O for each occurrence.

> Is the description "Loads the patters contained in the filename and puts them into a list of patterns, each pattern being a list of occurrence, and each occurrence being a list of (onset, midi) pairs." not enough?

A small, but complete example would help a lot here.


> Better yet:
> 
> ```
> where N is the number of patterns, M[i] is the number of occurrences of the i'th pattern, and O[j] is the number of onsets in the j'th occurrence
> ```

That's exactly it. I can come up with a little example and update it myself if you want.


Ok, feel free to make these changes, or get @urinieto to do it.  If neither of you do it, I can try to hack at it eventually, but I have virtually no experience/understanding of the pattern discovery task.


> I have virtually no experience/understanding of the pattern discovery task.

Lucky you.


Done it. I added the 5th symphony canonical motive as an example. Let me know if you need anything else.


Should we close this?


Looks good!
Repository: mir-evaluation/mir_eval
Issue: beat.information_gain fails when <2 beats
Root cause: Validation Gap

This comes through from `beat._get_entropy`, where it looks at the inter-onset interval without checking that the beat array is large enough.

There should probably be:
- an exception in entropy if there are fewer than two beats
- a try-except in information_gain which sets the score to 0 (or nan, but i think 0 is appropriate here).  This way, a call to `beat.evaluate()` will still work, even when one of the metrics fails.  Single-beat annotations are legal for something like f-measure, so kicking the exception all the way upstream doesn't entirely make sense.


Hm, I guess this should have been covered when we were thinking about #44.  I'm surprised that other metrics don't also fail when there's only when annotation - is information gain the only one you checked?  I'd like to avoid an exception or NaN; see discussion starting here https://github.com/craffel/mir_eval/issues/43#issuecomment-40770763.  I need to investigate, but I think we actually just need some special logic - the beat error histogram, when there is only one estimated beat, should just have a single non-zero bin at the error corresponding to the difference between the one estimated beat and the closest reference beat; when there is only one reference beat, we should compare all estimated beats to the single reference beat, right?


> I'm surprised that other metrics don't also fail when there's only when annotation - is information gain the only one you checked?

No, but I haven't done a thorough test either.  This one just happened to come up in some corner cases of a parameter optimization I'm running.

> I need to investigate, but I think we actually just need some special logic

Sure.  I was thinking the exception would be caught within mir_eval and handled as a special case, not punted up to the user.  You can hard-wire the special case without exceptions also, depending on which ends up being cleaner.


Moving this issue to a bigger issue #122
Repository: mir-evaluation/mir_eval
Issue: Don't require that tempi be provided in ascending order
Root cause: Documentation Deficiency

Currently, the docstring for `tempo.detection` states that `reference_tempi` and `estimated_tempi` be provided as two-element `np.ndarray`s, such that their first entry is smaller than their second.  Why not just require that two tempi are passed for both, and simply sort them?  I believe it's just a matter of comparing them in the correct order, unless I'm mistaken.


Sure, we can do that.  It's mainly there to follow the mirex convention.  More generally, `weight` is the relative weight of the first tempo vs the second, and the actual ordering doesn't matter at all.  By convention, it applies to the slower of the two, but that's not necessary for the metric to work.

Maybe we should just clarify the docstring instead?


Either is fine.


Fixed in #131
Repository: mir-evaluation/mir_eval
Issue: Travis fails, rake not found
Root cause: Configuration Error

Travis is currently reporting:

```
$ rake
rake aborted!
No Rakefile found (looking for: rakefile, Rakefile, rakefile.rb, Rakefile.rb)
/home/travis/.rvm/gems/ruby-1.9.3-p551/bin/ruby_executable_hooks:15:in `eval'
/home/travis/.rvm/gems/ruby-1.9.3-p551/bin/ruby_executable_hooks:15:in `<main>'
(See full trace by running task with --trace)
The command "rake" exited with 1.
```


Looks like this is because .travis.yml is only in develop, not master.


Indeed.  We should just merge develop into master and handle things in the normal way.


`gem install rake` On my server solved the problem. You could also include this into a `pre-install.sh` script

I had the same issue, in my case I had not setup a build script and apparently by default travis tries to test common configurations to the language of choice > ruby via rake: no rake = no test, then build failed.

It could be an issue with travis file name. Please ensure its .travis.yml

hello, can be tabs in the file with extension yml
Repository: mir-evaluation/mir_eval
Issue: travis build on 3.4, 3.5?
Root cause: Configuration Error

The current config is set for 2.7 and 3.3; we should at least bump to 3.4 since [nobody really uses 3.3](http://www.randalolson.com/2015/01/30/python-usage-survey-2014/).  3.5 would be nice as well.


Sounds good, go for it!


Any objection to adding coveralls support, while i'm at it?


No, please do!  It was on some to-do list somewhere.


Roger.

Also, any thoughts about replacing this wheels download with a miniconda environment?  Conda's scipy packages get updated much more regularly, apparently.


> Also, any thoughts about replacing this wheels download with a miniconda environment? Conda's scipy packages get updated much more regularly, apparently.

Yeah, I'd prefer miniconda.  I actually thought that's what we were using, but I didn't add the travis support, so what do I know.


Alright.  Maybe i'll rename this issue to "generally fix up the CI configs"


:+1:


Got it.  Miniconda doesn't yet have 3.5 support http://continuum.io/blog/python35 , so let's hold off on that one for now.


Closed via #130.
Repository: mir-evaluation/mir_eval
Issue: event_validation limits onset times to <=30000
Root cause: Architectural Constraint

It looks like the maximum allowable onset time is [hardcoded](https://github.com/craffel/mir_eval/blob/master/mir_eval/onset.py#L32) to 30000. We're using mir_eval for some environmental recordings that are longer than 8 hours, and so the code spits a value error and dies.

```
File "/usr/local/lib/python2.7/site-packages/mir_eval/onset.py", line 99, in f_measure
    validate(reference_onsets, estimated_onsets)
  File "/usr/local/lib/python2.7/site-packages/mir_eval/onset.py", line 56, in validate
    util.validate_events(onsets, MAX_TIME)
  File "/usr/local/lib/python2.7/site-packages/mir_eval/util.py", line 634, in validate_events
    'seconds?)'.format(events.max(), max_time))
ValueError: An event at time 35336.67 was found which is greater than the maximum allowable time of max_time = 30000.0 (did you supply event times in seconds?)
```

Shouldn't this threshold be an optional input parameter?


It's not hardcoded, just set `mir_eval.onset.MAX_TIME = 1e100`before calling any functions in `mir_eval.onset`.  This was discussed briefly here https://github.com/craffel/mir_eval/issues/87


righty! cheers
Repository: mir-evaluation/mir_eval
Issue: filter_kwargs and python 3.5 compatibility
Root cause: External Dependency Breakage

`util.filter_kwargs` uses the inspect module's `getargspec` function [here](https://github.com/craffel/mir_eval/blob/master/mir_eval/util.py#L715), which is deprecated in python 3.5.

In python 3, this has been replaced with the `signature` function (see [here](https://docs.python.org/3/library/inspect.html#inspect.signature)), which sadly doesn't work on py2.

You might be able to side-step this by using `six.get_function_defaults()`.

Alternatievly, our buddies at sklearn have a fix for this [here](https://github.com/scikit-learn/scikit-learn/commit/6966881328c256bbef37752cce23bbac8e23fb27).


> You might be able to side-step this by using six.get_function_defaults().

I don't think so, it looks like that function just retrieves default values for arguments, rather than the name of each argument (what we need).

So, as far as you know, none of the python 2 backport libraries include code for this?  It looks like sklearn has just copied this https://pypi.python.org/pypi/funcsigs into their library.  I'd be ok with adding funcsigs as a dependency, as it's pure Python.


> it looks like that function just retrieves default values for arguments,

Yeah, you're right.  I realized that after suggesting it, but forgot to edit the comment.  Meanwhile: paging @amueller for any additional insight on this.


Here's a little function which re-implements the part of `getargspec` we need:

``` Python
def get_arg_names(function):
    func_code = function.func_code
    return func_code.co_varnames[:func_code.co_argcount]
```

Usage:

``` Python
>>> def foo(a, b, c=100):
...     d = 30
...     return a + b, c + d
...
>>> print get_arg_names(foo)
('a', 'b', 'c')
```

Spot anything wrong with that?  If not, I'll submit a PR to change the call to `getargspec` to the `func_code` method above.


Seems legit, I guess?  We'll probably need to set up a test environment for py3.5, but afaik travis/miniconda don't have this yet.  (There is a full anaconda distribution with 3.5, but that's a big heavy for test environments.)


We us miniconda I think: https://travis-ci.org/scikit-learn/scikit-learn/jobs/85767336#L1012


Huh.  Well shut my mouth!  Last I checked, miniconda didn't have 3.5.


Ok.  I will try to do this all in one go then.
Repository: mir-evaluation/mir_eval
Issue: util.boundaries_to_intervals with labels is weird
Root cause: Architectural Constraint

When `labels=None` (the default), `None` is returned.  We don't have any tests or code anywhere that doesn't use `labels=None`.  When `labels != None`, `labels[-1]` is returned.  There's no test that `len(labels) == len(boundaries)`, so this can lead to weird behavior.  It also pre-supposes that the final label should be discarded.  Is this actually a convention?  Is there any utility to cutting off the last label for the user?  Depending on the answer to these questions, I'm in favor of removing the `label` argument from `util.boundaries_to_intervals`.


This is another artifact of trying to support salami annotations.

I'm okay with dropping it, since the behavior is generally ambiguous.


PR #152
Repository: mir-evaluation/mir_eval
Issue: Unit test for util.bipartite_match

Currently, the `recurse` function in `util.bipartite_match` is not covered by tests.  This is probably because none of the test annotations have any annotations which are close enough to warrant recursion, or something.  We don't have a unit test for it in general.  @bmcfee, can you add one which covers this case?


I think you should be able to trigger this by doing an event match with very large window size. I'll try to work out a concrete example.


Resolved in 5b8fdecc1703863208838a29d4207d7b75fb78ca
Repository: mir-evaluation/mir_eval
Issue: Docstring description for chord.tetrads is wrong?

@ejhumphrey should this https://github.com/craffel/mir_eval/blob/master/mir_eval/chord.py#L933 read something like "Compare chords along tetrad (root, full quality, & bass) relationships." instead?


Yep, good catch!


Fixed in 5655a8dce3a498b355038aaff490b96c1e3e91a0
Repository: mir-evaluation/mir_eval
Issue: frame_size in mir_eval.util.intervals_to_samples docstring?

@bmcfee I think you wrote the docstring for `intervals_to_samples`.  https://github.com/craffel/mir_eval/blob/master/mir_eval/util.py#L112
Is `frame_size` supposed to be `sample_size`?


Yup, good catch!


Fixed in 5655a8dce3a498b355038aaff490b96c1e3e91a0
Repository: mir-evaluation/mir_eval
Issue: filter_kwargs parameter name collision
Root cause: Architectural Constraint

I'm trying to use `util.filter_kwargs` on sonify functions, and hitting a collision with the argument `function`.

This would be fixed by renaming `filter_kwargs`'s `function` argument (not a keyword argument) to something unique, like `wrapped_function`.

This should be a trivial change, and I can PR if it sounds reasonable.


.. sorry, just realized that report wasn't terribly precise.  The issue is that you can't use `filter_kwargs` on parameters whose names coincide with those of `filter_kwargs` itself, since you would have to have multiple values for the colliding names.


> I'm trying to use util.filter_kwargs on sonify functions

I'm curious what the context is.

> This would be fixed by renaming filter_kwargs's function argument (not a keyword argument) to something unique, like wrapped_function.

I see.  That's no problem, of course.  Though maybe we should be extra sure that a collision with the arg name doesn't happen, and call it something like `wrapped_function_87h12ry3b7926fyubun38796tyiuh2ig5uwte87v6fyvub2in5` (/s but only sort of).


> I'm curious what the context is.

`time_frequency` uses `function` as the name of its synthesis generator.

> Though maybe we should be extra sure that a collision with the arg name doesn't happen, and call it something like

The real solution here is to not use **, and keep the target kwargs as a dict.  But that breaks backward-compatibility.

Otherwise, I'd recommend something that connotes a private name, eg, `__function`.  But it's your call.


> time_frequency uses function as the name of its synthesis generator.

Of course, what I meant was I am curious why you are using `filter_kwargs` on `time_frequency`.

> Otherwise, I'd recommend something that connotes a private name, eg, __function. But it's your call.

I like your one-underscore solution.


> Of course, what I meant was I am curious why you are using filter_kwargs on time_frequency.

Oh! The upcoming jams.sonify module does dynamic dispatch to map annotation objects to mir_eval.sonify functions.  Since we don't (can't) know the sonification type up front, all sonify parameters live in a kwargs, which needs to be filtered because the sonify functions don't have a common signature.

For instance, I have this fragment for making a full-track sonification of all isophonics annotations in one go:

``` python
y_annotated = y.copy()
for ann in jam.annotations:
    try:
        print('Trying to sonify {:s}'.format(ann.namespace))
        y_ann = jams.sonify.sonify(ann)
        y_ann = librosa.util.fix_length(y_ann, len(y_annotated))
        y_annotated += y_ann
    except jams.NamespaceError:
        pass
```

And I'd like to be able to say `jams.sonify.sonify(ann, function=scipy.signal.square)` without it barfing when it hits a call to `click` instead of `chords` (or what-have-you).


Oh, nice.
Repository: mir-evaluation/mir_eval
Issue: Tests taking too long - travis timeouts

Outstanding PRs are failing because travis is timing out on our tests after 30 minutes.

This might be a temporary issue with travis, though I'm not seeing it on my other repos.  More likely, something is taking way longer than it should in the test suite.  It's currently impossible to tell what the hold up is, since the tests don't run with verbose mode.

@craffel want to look into it?


> @craffel want to look into it?

Haha, no, not really, but I will.  I would guess it was the separation metrics, which take a very long time in slow/non-tuned settings.
Repository: mir-evaluation/mir_eval
Issue: index order in match_events output

@bmcfee  I'm basing `mir_eval.util.match_notes` in on `match_events`, and I think I have it down. One thing I noticed however is that calling `matching = sorted(_bipartite_match(G).items())` will return a list of tuples, where the value in each tuple is the index of the estimate node, and the second is the index of the reference node. Given that the order of the input parameters to the method is reference then estimate, I thought it would be more consistent for the returned tuples to be (ref_ind, est_ind) rather than (est_ind, ref_ind), as it is right now. This can be accomplished by flipping the graph construction:

``` python
G = {}
    for ref_i, est_i in zip(*hits):
        if est_i not in G:
            G[est_i] = []
        G[est_i].append(ref_i)
```

(where I just swapped every occurrence of `ref_i` with `est_i` and vice-versa). 
I have two questions though:
1. Is _bipartite_match() symmetric? (I'd think so)
2. Does the internal structure of the returned `matching` list matter for anything else downstream? I know that for matching beats you just check for the size of the list, but not sure if it's used anywhere else.

Basically this is a tiny thing, but since I'm implementing `match_notes` now, I thought I'd ping you.


Yeah, that sounds reasonable.
1. Yes, since the edges are undirected, everything here is symmetric, so swapping ref and est here ought to be fine.
2. I don't think we use it anywhere else (it also gets called in segments, and conceivably anywhere else you have a matching problem), but always in terms of size.  I could imagine use-cases in which you'd want to preserve the matching structure -- eg, visualization/debugging/etc -- as opposed to just the count summary.


ok cool. So I'll keep the flipped version in `match_notes`, you wanna flip it in `match_events`?
p.s - there's no unit test for `match_events` :ghost: 


> p.s - there's no unit test for match_events :ghost:

There's a [unit test for _bipartite_match](https://github.com/craffel/mir_eval/blob/master/tests/test_util.py#L126), and the rest of `match_events` (which isn't much) gets hit pretty thoroughly by the rest of the tests.


Fair enough. Btw, I just noticed that the docstring in `match_events` describes `matching` as:
"`matching[i] == (i, j)` where `ref[i]` matches `est[j]`"
which will only be correct when the graph construction is flipped as discussed above.


> Fair enough. Btw, I just noticed that the docstring in match_events describes matching as:
> "matching[i] == (i, j) where ref[i] matches est[j]"
> which will only be correct when the graph construction is flipped as discussed above.

Hah, derp.  If you want to correct all of that in this PR, that would be okay by me.


roger that


Fixed by #170
Repository: mir-evaluation/mir_eval
Issue: sonify.chords, unique times strikes again
GitHub labels: bug

There appears to be a bug in sonify.chords stemming from running unique on time events.  It's difficult to reproduce, since it boils down to numerical precision issues, but the resulting artifact is that some chords do not get sonified.

I've gisted a [notebook](https://gist.github.com/829701891958b7f9ae41) to illustrate the problem, but it depends on having the jams-sonify branch and beatles isophonics data, so it may take some effort to reproduce outside of my environment.  Still, I'm pretty sure this is caused by our old buddy [unique](https://github.com/craffel/mir_eval/blob/master/mir_eval/sonify.py#L213).


Yes, that looks wrong, and yes, unique would be my best guess...  Does rounding before calling unique fix this example for you?


Yup, sticking a `times = np.around(times, 3)` fixes it.

I'm curious as to why all this is necessary though; why not just take the first column of intervals as label start times?


By "all this" do you mean [this](https://github.com/craffel/mir_eval/blob/master/mir_eval/sonify.py#L211-L218)?  I believe it's because you coud have something like

```
Start | End  | Label
0.0   | 1.0  | C
1.5   | 2.0  | D
```

which you needed to convert to

```
Start | End  | Label
0.0   | 1.0  | C
1.0   | 1.5  | N
1.5   | 2.0  | D
```

Though I think we can avoid this now that you can pass intervals to `sonify.time_frequency`.


> Though I think we can avoid this now that you can pass intervals to sonify.time_frequency.

Yup. I'd forgotten that the implicit-end-time version existed. :)

I can open a PR that does this via intervals if that sounds good to you.


> I can open a PR that does this via intervals if that sounds good to you.

:+1:
Repository: mir-evaluation/mir_eval
Issue: filter_kwargs drops parameters when the target has **kwargs
Root cause: Algorithmic Error

For example, take

```
def myfunc(**kwargs):
    return kwargs
```

Feeding this into `filter_kwargs` does the following:

``` python
In [9]: mir_eval.util.filter_kwargs(myfunc, name='boo')
Out[9]: {}
```

when I would instead expect

``` python
In [10]: myfunc(name='boo')
Out[10]: {'name': 'boo'}
```

That is, I think `filter_kwargs` should retain parameters if the target already has a slop dict.  Is there a reason not to do this?

This came up again in sonify, where `chords` and `chroma` both have slop dicts that should accept `time_frequency` parameters.


I don't see why not.  I guess the reason this isn't supported is that it is outside of the original intended scope of `filter_kwargs` (remove any keyword arguments that aren't explicitly defined in the provided function), because if a function has a kwarg slop dict you don't need to use `filter_kwargs` at all.  If you want to do this, do you know how to check programmatically whether a function has a kwarg slop dict or not?


> because if a function has a kwarg slop dict you don't need to use filter_kwargs at all.

Yup, but in my case, i'm applying this to functions that may or may not have kwargs, so it would be handy to detect it automatically.

> do you know how to check programmatically whether a function has a kwarg slop dict or not?

`inspect.getargspec(myfunc).keywords is not None` does the trick, though it seems to be deprecated.  I'll check to see if there's a six wrapper for this.


> I'll check to see if there's a six wrapper for this.

There isn't, so I just hacked up a `has_kwargs` function that uses six to determine which python we're on, and does the right thing accordingly.