zhiminy/EvalEng huggingface dataset:
Repository: mir-evaluation/mir_eval
Issue: hierarchy.evaluate(), forward-compatibility
GitHub labels: question

Dragging back up some discussion from #125...

The current `evaluate` function calls only one metric (tmeasures), which in turn only takes boundaries and not labels.  Consequently, `evaluate` only exposes positional arguments for boundaries.

I can imagine the not-too-distant future`*` in which we may have metrics for label hierarchies, which will require a change to the `evaluate` API.  The resulting api should look similar to `segment.evaluate()`, excepting that each of the inputs is a list of annotations (intervals or labels) instead of a single flat interval/label array.

The question here is: when should we change the API?
- Option 1: ASAP, to prevent too much breakage down the road
- Option 2: wait until a metric using the labels is introduced.

I'd advocate for the former, since it costs us nothing and will make the transition easier later on.  What do others think?

`*` I have an implementation of one such metric already, but it's nowhere near ready for PR/inclusion.


I'm fine with changing the API now.


I vote option 1 too.

On Mon, Feb 15, 2016 at 1:19 PM, Colin Raffel notifications@github.com
wrote:

> Closed #177 https://github.com/craffel/mir_eval/issues/177 via #178
> https://github.com/craffel/mir_eval/pull/178.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/craffel/mir_eval/issues/177#event-551226560.
Repository: mir-evaluation/mir_eval
Issue: beat.information_gain should not replicate beat evaluation toolbox exactly
Root cause: Algorithmic Error

Currently, beat.information_gain matches the unusual histogram binning used in the beat evaluation toolbox.  It should use uniform bins instead.  It was changed from uniform bins to the beat evaluation toolbox in commit 9aa7b701f9d967751852b853fb39f3c4f16bc004 and should be changed back once we have a reference release which matches MIREX exactly.


Unit test now fails.


Didn't finish fixing this


This was also changed in 962bcbf771215ad70a71075d8b4f1ddc91fef0f0 but the regression tests were not updated.
Repository: mir-evaluation/mir_eval
Issue: Create loader for tempo data
Root cause: Unimplemented Feature Gap

There's currently no loader for tempo annotations in the MIREX format.  You can load them in with `io.load_delimited`, but it's cumbersome:

``` Python
data = mir_eval.io.load_delimited(filename, [float, float, float])
tempi = np.array([data[0][0], data[1][0]])
weight = data[2][0]
```

It should be very simple to create this and will look a lot like `io.key`.
Repository: mir-evaluation/mir_eval
Issue: Load delimited skips the first line?
Root cause: Documentation Deficiency

Here: https://github.com/craffel/mir_eval/blob/master/mir_eval/io.py#L73

Does anyone know why it does this? I assume it is because some annotation files have a header, but I would not assume all annotation files do; this should be a function argument, I think.  Looks like it showed up [here](https://github.com/craffel/mir_eval/commit/08db16f586ec7d813e53fe38451e2557de94931c), maybe.  @bmcfee can you confirm?

@rabitt noticed this [here](https://github.com/craffel/mir_eval/pull/186#discussion-diff-56299780)


It doesn't skip the first line.  It starts the enumeration at 1 instead of 0.  This is so that errors reported below correctly report the offending line number in the input file.


Derp, thanks.
Repository: mir-evaluation/mir_eval
Issue: Verify against MIREX

We are currently unit testing against various codebases.  We should really be verifying against MIREX.  Code is here: https://code.google.com/p/nemadiy/source/browse/analytics/trunk/src/main/java/org/imirsel/nema/analytics/evaluation/


Even with the code, my understanding is that it will be a royal pain to get running.

Instead, why not black-box test it with existing prediction files and scores?  This should be possible at least for beats (using SMC) and segments (using RWC or mirex09) where the ground truth is public.  We should just be able to scrape the prediction files from 2013.


That's a much better idea, although it limits us to four decimal places of accuracy... which is probably sufficient.


This is handled in the mir_eval-ismir repo:
https://github.com/craffel/mir_eval-ismir
Repository: mir-evaluation/mir_eval
Issue: Fix `sonify.time_frequency` docstring

https://github.com/craffel/mir_eval/blob/master/mir_eval/sonify.py#L67 should read "from `times[m]` to `times[m + 1]`"
https://github.com/craffel/mir_eval/blob/master/mir_eval/sonify.py#L71 should read "shape=`(gram.shape[1],)` or `(gram.shape[1], 2)`"

(I think)

Not sure how I missed these in multiple docstring audits.
Repository: mir-evaluation/mir_eval
Issue: use context managers for open file handles

In `io.py` `load_delimited` and `load_ragged_delimited`:

[This line](https://github.com/craffel/mir_eval/blob/master/mir_eval/io.py#L59) could leak file descriptors if there's an error anywhere below (before the call to close). Replace with `with my_open(buf_or_str, 'r') as fdisc: ....`
Repository: mir-evaluation/mir_eval
Issue: Chroma transcription metrics
Root cause: Unimplemented Feature Gap

To cover all evaluation done in MIREX, we also need to add the ability to evaluate transcription annotations after mapping the pitch values to a chroma (single octave) scale, as discussed in #180.


Punting to 0.5.


I'd like to take a stab at this. Seems like the way to do this is add a flag in precision_recall_f1_overlap and match_notes?

not sure if it'll be of any help, but there's machinery in [mir_eval.chord](https://github.com/craffel/mir_eval/blob/master/mir_eval/chord.py) that _might_ be useful??

> I'd like to take a stab at this. 

Cool, contributions welcome.  @justinsalamon will know best what is necessary.  

> Seems like the way to do this is add a flag in precision_recall_f1_overlap and match_notes?

Seems that way to me too.

This should be relatively straight forward. The metrics (excluding ones that only consider onsets or offsets) rely on [`match_notes`](https://github.com/craffel/mir_eval/blob/master/mir_eval/transcription.py#L336), which computes pitch distances [here](https://github.com/craffel/mir_eval/blob/master/mir_eval/transcription.py#L426) and checks for pitch matches [here](https://github.com/craffel/mir_eval/blob/master/mir_eval/transcription.py#L428). You'd have to add a flag which, if set, calls an alternative version of these lines which checks for matches in (cyclic) chroma space.

I've got a 3 line solution that makes sense to me [here](https://github.com/chf2117/mir_eval/blob/master/mir_eval/transcription.py#L430). Still need to test.

Don't know a good way to generate test data. Any advice @justinsalamon?

@chf2117 that looks right to me (but should be tested of course). To test that it really does what you expect it to you need to add unit tests to [`test_transcrpition.py`](https://github.com/craffel/mir_eval/blob/master/tests/test_transcription.py). The toy data for the unit tests is hard coded [here](https://github.com/craffel/mir_eval/blob/master/tests/test_transcription.py#L17), if it's sufficient for covering all the chroma cases great, if not you should create more hard coded data (but don't add to / change the existing data as that will break the existing tests!).

Data for regression tests lives [here](https://github.com/craffel/mir_eval/tree/master/tests/data/transcription), no need to touch est* and ref* but you'll have to update the output* files.

Finally, to check your output against the MIREX results see [this comment](https://github.com/craffel/mir_eval/pull/170#issuecomment-187262131) in the original transcription pull request.
Repository: mir-evaluation/mir_eval
Issue: Add chord detection evaluation
Root cause: Unimplemented Feature Gap

Add a submodule chord.py which implements common chord detection metrics.  I think you have code for this already right?  All the sampling and overlap kind of stuff.


On second thought, isn't this basically the same as what people to do evaluate transcription algorithms?  Eg, take in two n_features x n_frames matrices, resample to the same sampling rate, and compute overlap?  In which case, doesn't need to be just chord.py... maybe harmony.py or something....


Not really, no.

Mostly because the chord alphabets in the prediction and GT need to be
simplified to an agreed alphabet. Evaluation is indeed then just overlap.

I should be able to port my code over from the chromaluma project, but
probably not today as I'm quite busy.

On 6 August 2013 22:36, craffel notifications@github.com wrote:

> On second thought, isn't this basically the same as what people to do
> evaluate transcription algorithms? Eg, take in two n_features x n_frames
> matrices, resample to the same sampling rate, and compute overlap? In which
> case, doesn't need to be just chord.py... maybe harmony.py or something....
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir-evaluate/issues/2#issuecomment-22213448
> .


Sure, that makes sense.  No rush on any of this.  Is reduce_chords.py all set to go?  That is mostly what I'm scared of.


yeah it's pretty scary.

I know this shouldn't matter for the quality of my code, but what is this
repo for?

On 7 August 2013 15:36, craffel notifications@github.com wrote:

> Sure, that makes sense. No rush on any of this. Is reduce_chords.py all
> set to go? That is mostly what I'm scared of.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir-evaluate/issues/2#issuecomment-22255664
> .


It's meant to be a distributable module for most of the common MIR evaluation metrics.  So you can track your beats in librosa or wherever, then

``` python
import mir_evaluate
print mir_evaluate.beat.f_measure( annotated_beats, detected_beats )
```

Including submodules for beat, onset, chord, segmentation, source separation, etc.

-Colin


I'm closing this for now, as the initial commit has been made. May be a fair bit to go still, but it exists now.
Repository: mir-evaluation/mir_eval
Issue: Remove sklearn dependency
Root cause: Unimplemented Feature Gap

Ideally, we'd like to minimize dependencies as much as possible.  sklearn is not unreasonable but we want to minimize the barrier of entry to its absolute lowest (numpy/scipy).  This is low priority, but it would be nice eventually.


Just to fully document the existing discussion behind this, sklearn is currently used in the `segment` submodule to implement various clustering metrics (rand index, mutual information, etc.)

The arguments for and against keeping the sklearn dependency go roughly as follows:
- _PRO_
  - Minimizes our own code footprint and relies upon known, tested tools
  - sklearn is a relatively standard package, and ships with python distributions (eg anaconda). A good chunk of people are likely to have it already, or won't mind installing it.
  - Some of the metrics we borrow from sklearn are fairly complex to implement, so there shouldn't be multiple implementations kicking around.
  - If we expand mir_eval into other tasks (eg, classification or retrieval), there may be other portions of sklearn that we would want to use anyway.
- _CON_
  - It's an added dependency
  - We only use a very small portion of sklearn


Just wanted to upvote this issue.... this dependency is causing hair pulling for me at the moment with a system I can't personally admin. 


Can't we do something like simply making the few dependent functions raise
an exception if someone tries to use them without sklearn, but otherwise
continue as normal?  Is this kind of "partial" installation frowned upon?

 DAn.

On Friday, August 15, 2014, Eric J. Humphrey notifications@github.com
wrote:

> Just wanted to upvote this issue.... this dependency is causing hair
> pulling for me at the moment with a system I can't personally admin.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/craffel/mir_eval/issues/20#issuecomment-52345146.


Yes, we can do that, but yes, it's frowned upon.  It also doesn't resolve the huge headaches sklearn is currently causing with pyinstaller -- pyinstaller will still see it as a dependency.


It's worth noting that while the unit tests pass, the adjusted mutual info score differs by about 1e-7.  This is almost certainly due either
- sklearn uses a compiled C version, while we are using the Python code which the C is compiled from.
- sklearn uses (without an apparent pattern) its own log gamma function and scipy's log gamma function; we always use scipy's log gamma function
Repository: mir-evaluation/mir_eval
Issue: interval validation: is empty okay?
Root cause: Validation Gap

I'm running into validation failures in some segment data due to [this check](https://github.com/craffel/mir_eval/blob/master/mir_eval/util.py#L708-L710).  I can't remember why we require non-empty intervals, but I don't think there's a technical reason for it in terms of div-by-zero errors in any of the metrics.  I suspect this came out of some of the weirder conversion errors in parsing the salami dataset.

Could we relax this check to instead fail only if `intervals[:, 1] < intervals[:, 0]` (negative duration) and throw a warning if they're equal?


Just to be clear, you're saying you want to change `<=` to `<` so that intervals with effectively "length zero" are considered valid?  I don't see why not, but playing devil's advocate, why would that ever be considered valid?  It doesn't make a lot of conceptual sense to me, but maybe there's a use-case for it I don't know of.


Right.  Thinking back about it, this was probably introduced to prevent multiple hits for duplicated boundaries.

You're right though; I think we should leave it as is.  I'll close this one out.
Repository: mir-evaluation/mir_eval
Issue: Code review integration?
GitHub labels: question

@craffel any thoughts about adding a code review integration tool?  Some of the CR here is getting difficult to keep track of across revisions, and something like [reviewable](https://github.com/integrations/reviewable), [review ninja](https://github.com/integrations/reviewninja), or [pull approve](https://github.com/integrations/pullapprove) could help.

(FWIW, I've been using reviewable and it's mostly pretty good.)


I'm not opposed to it, but I've never used any of them, so I don't really know what we're missing.  I don't have the bandwidth to learn/integrate now, but am open to it soon.


tldr: none of them are perfect, but the benefits are:
- CR status integration (ala travis or coveralls) 
- threading / comment resolution. This is the big one for me, since it's all too easy for CR comments to get buried on github when the file in question is changed (comments on old diffs)


It sounds great.  I assume it will be messy to do while we are in the midst of CR/PRs though, so maybe we can do this once the outstanding PRs are merged?


Nah, it's all pretty non-invasive; everything shows up on github as comments.  It's more a matter of adopting procedure than anything else.  But it's your call, check it out whenever.

If you want to see the process, check out [this open PR](https://github.com/librosa/librosa/pull/323) on librosa.


While at this, how about adding a https://dependencyci.com/ integration?


+1 here for CR tool.


@craffel now that 0.4 is out, any objection to me adding this?


Sure, go for it. 


derp, it looks like only the repo owner can do it. :cry: 


Ok, I can do it.  Which one did you end up choosing?


I like reviewable.

On Wed, Sep 14, 2016, 11:25 Colin Raffel notifications@github.com wrote:

> Ok, I can do it. Which one did you end up choosing?
> 
> —
> You are receiving this because you were assigned.
> Reply to this email directly, view it on GitHub
> https://github.com/craffel/mir_eval/issues/206#issuecomment-247051036,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/ABIqjI5ClszXYvLcXNQZvHiL34z1CY1sks5qqBIAgaJpZM4I62aO
> .


How about using [GitHub's new built-in code review features](https://github.com/blog/2256-a-whole-new-github-universe-announcing-new-tools-forums-and-features)?


Oh snap.


> How about using GitHub's new built-in code review features?

After poking around github's new CR stuff, I'm not sure it's the best way to go.  I think I prefer reviewable's status management and comment thread system, but I'll leave the call to @craffel.


Having never used either, I'd rather start with the simple, built-in solution than try something external, especially if the external one has more features.  If we are finding GitHub's approach is awkward/not enough, we can consider switching.
Repository: mir-evaluation/mir_eval
Issue: mir_eval.display.piano_roll should raise a ValueError when both pitches and midi are None
Root cause: Validation Gap

As specified [here](https://github.com/craffel/mir_eval/blob/master/mir_eval/display.py#L659).  There is no such check; it just throws a confusing AtributeError [when `None` is passed to `hz_to_midi`](https://github.com/craffel/mir_eval/blob/master/mir_eval/display.py#L675).  I think this is the only place this is true in `display`.
Repository: mir-evaluation/mir_eval
Issue: Make validate_chord_label more strict
Root cause: Validation Gap

I think it would be usful if `validate_chord_label` would raise an exception for any invalid chord, rather than the specific things it looks for.  For example, `mir_eval.chord.validate_chord_label('asdf')` passes without an exception.  Can we formalize what exactly a valid chord label is, and put it in this function?  @ejhumphrey do you have bandwidth for this?


Feel free to use the [jams chord regexp](https://github.com/marl/jams/blob/master/jams/schemata/namespaces/chord/chord.re) for this.


... it might also make sense, if anyone's so inclined, to use the regexp to redo `chord.split` using back-references.  There's nothing wrong with the current implementation (AFAIK), but if we already have the regexp, it would be much simpler to use it for parsing chord strings, and would eliminate a good amount of code.
Repository: mir-evaluation/mir_eval
Issue: melody.freq_to_voicing returns single boolean if not a numpy array
Root cause: Validation Gap

[`melody.freq_to_voicing`](https://github.com/craffel/mir_eval/blob/master/mir_eval/melody.py#L144) is meant to take a numpy array. I accidentally discovered that if you instead give it a list of floats instead of a numpy array, it returns `True`, rather than an array of booleans.

```
>>> np.array([440.0, 0.0, 0.0]) > 0
[True, False, False]
>>> [440.0, 0.0, 0.0] > 0
True
```

I definitely just used the function wrong, but it is a bit of a sinister error. I don't have any ideas for an ideal solution... maybe the function could first check that `frequencies` is indeed a numpy array and throw a `ValueError` if not? Any ideas? Happy to make a PR.


This arguably falls within the scope of #143.  In general, at the moment, we don't make an effort to type-check arguments, with the policy that if the docstring asks for, say, an `np.ndarray` and the user provides a `list`, they are on their own.  This isn't the best policy, but if we are going to fix it we should fix it everywhere... and I'm guessing you don't want to make a PR for that right now :)


Fair enough. In that case I will hold off ;).


Ok, closing this in favor of #143.
Repository: mir-evaluation/mir_eval
Issue: Better docstrings for separation
GitHub labels: wontfix

Currently, the docstrings for the metrics in `mir_eval.separation` are not super helpful - they basically say "Computes xyz function from bss_eval".  Specifically, this makes it unclear what the difference between `bss_eval_sources` and `bss_eval_images` is.  @ErikJohnsonCU @faroit @carlthome when one of you have a minute, can you make a PR to make them a bit more informative?  Thanks!


@craffel, I don't feel I have a solid enough grasp of the single-channel and multi-channel difference aside from `bss_eval_sources` pretty much only being a fair estimate for point sources and instantaneous mixes (let's say dry vocals recorded in a dampened booth, or line-in recorded Moogs), so I don't dare to add anything to the docstring. @faroit is the guru here. :hamster: 


Ping @aliutkus
Repository: mir-evaluation/mir_eval
Issue: Warning in sonify.clicks
Root cause: Validation Gap

I get this a lot:

```
/home/bmcfee/git/mir_eval/mir_eval/sonify.py:45: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  click_signal = np.zeros(length)
```

whenever `length` is a float type.  I can cast it myself, but I think it's better to handle that within `clicks`.


The docstring specifies supplying an `int`, so the current policy of `mir_eval` is that it's on you:
https://github.com/craffel/mir_eval/blob/master/mir_eval/sonify.py#L25
Otherwise, this falls under #143 - we don't type check most inputs in `mir_eval`; if we're going to check some, we should check them all (e.g. that when a np.ndarray is asked for, a list is not provided #219, etc).


Actually I don't think that's correct in this case.

At [these lines](https://github.com/craffel/mir_eval/blob/master/mir_eval/sonify.py#L41-L43), the length is inferred from times, but not explicitly cast to int.  Since `times` need not be int, this warning can happen anyway.


> Actually I don't think that's correct in this case.
> 
> At these lines, the length is inferred from times, but not explicitly cast to int. Since times need not be int, this warning can happen anyway.

Well, it's definitely correct (we need `length` to be an `int`, user-supplied or not), it's just a different issue,.  We should definitely be doing `length = int(times.max()*fs + click.shape[0] + 1)`.


Yes, sorry if my initial post was unclear, I was writing in a hurry.

I think this does have to be fixed within mir_eval.