zhiminy/EvalEng huggingface dataset:
Repository: mir-evaluation/mir_eval
Issue: Inconsistency between mir_eval.beat.evaluate and its individual components
Root cause: Documentation Deficiency

There is a slight inconsistency between mir_eval.beat.evaluate and its individual components, such as mir_eval.beat.f_measure. E.g, the score will be different if it is computed by default through the general evaluate method or with the individual component.

```
import mir_eval
import numpy as np

b1 = np.array([1.0, 2.0, 4.0, 6.0, 7.0, 8.1, 10.0])
b2 = np.array([2.0, 4.2, 6.01, 7.02, 8.0, 9.5])

print(mir_eval.beat.f_measure(b1, b2))
print(mir_eval.beat.evaluate(b1, b2)['F-measure'])
```
Outputs:
```
>>> 0.461538461538
>>> 0.5
```
The reason is that the beats appearing before the 5th second are removed by default on the evaluate method (`reference_beats = util.filter_kwargs(trim_beats, reference_beats, **kwargs)`) but not on the individual components. It is not a major issue as it can be changed manually, but it seems to me this inconsistency might be confusing to people not paying too much attention on what is going on.

Is there a specific reason to have a different default behaviour between the different methods? Otherwise I would suggest keeping the same default behaviour.

Hi, this is intended.  Note that in all of the example usage docstrings we show calling the trim_beats function before the beat metric function http://craffel.github.io/mir_eval/#mir_eval.beat.f_measure We don't want to trim out the first 5 seconds in the individual beat functions, because they may already be trimmed before being passed to them, and then we'd trim twice.  We *do* want to trim them in `evaluate` because we actually have just loaded in the beats, so we know they aren't trimmed.
Repository: mir-evaluation/mir_eval
Issue: Travis fails with cannot import name 'KnownFailureTest'
Root cause: External Dependency Breakage

https://travis-ci.org/rabitt/mir_eval/jobs/298700718#L2022

Looks like KnowFailureTest doesn't exist in Matplotlib anymore (maybe because of a new release), which is used by `mpl_ic.py` [here](https://github.com/craffel/mir_eval/blob/master/tests/mpl_ic.py#L26).  @bmcfee if you have a quick fix a PR is welcome otherwise I will investigate later.

Ill try to get back to this matter in the week.

Yup, mpl 2.1 moved from nose to pytest, which is something we should also do.  I'll see about hacking `mpl_ic.py` back into working shape for the time being.
Repository: mir-evaluation/mir_eval
Issue: computation of the projection variables G,D, C should have their own function
Root cause: Architectural Constraint

right now, the code in _project and _project_images is very redundant for no reason,

furthermore, there are cases where we want to compute these variables without doing the actual filtering, as I believe we should do for the frame_wise eval


IIRC the reason for this was to keep the translation from the matlab implementation as direct as possible.  Maybe @dawenl can comment?

I am in the process of working on these two recent issues I opened, I will keep you posted !
Repository: mir-evaluation/mir_eval
Issue: display module needs to be updated for matplotlib deprecations
Root cause: External Dependency Breakage

MPL has deprecated string literals for flag controls (`'on'` => `True`, `'off'` => 'False'), so we get warnings now in our tests.

I think this only comes up in places using `grid`, but we should do a thorough audit.

Also, `segment.py` uses `scipy.misc.comb` which has been deprecated toward `scipy.special.comb` in 1.0.

In the doc build, `sphinx.ext.pngmath` has been deprecated to `scipy.ext.imgmath`.

I think that covers the deprecation warnings in the current doc build.
Repository: mir-evaluation/mir_eval
Issue: isolate pyplot import and side-effects
Root cause: Architectural Constraint

(Note: we recently fixed this over in librosa [here](https://github.com/librosa/librosa/pull/704).)

The issue is that importing `matplotlib.pyplot` triggers the backend selector, which has side-effects (including occasional memory leaks).  But we don't always need to import `pyplot` in the display module.  If the user is working with the object API (as opposed to the stateful pyplot API), then we can avoid this import entirely.

In fact, the only place the `pyplot` import is used is within our axes getter, and even then, only when the axes are not provided by the user.  It's an easy fix to move the import to within that conditional block, and avoid any side-effects until they are absolutely necessary.

Since the only call to `plt.anything` is `plt.gcf`, could we just reproduce the contents of `plt.gcf` instead of importing pyplot at all?  Would look like this:
https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/pyplot.py#L565
It would involve importing matplotlib._pylab_helpers (or figuring some equivalent way to do this), which I assume doesn't have the same side effects.

> Since the only call to plt.anything is plt.gcf, could we just reproduce the contents of plt.gcf instead of importing pyplot at all?

No, I think we want the import here.  The idea is that if a user does not provide axes or a figure object, then they must be using the pyplot API.  If the call into our display code is the first pyplot thing the user has done, then no backend will have yet been selected.  However, we really want that to happen before executing any draw commands so that the state is updated appropriately.  (This solution was arrived at after conferring with the mpl devs in the context of librosa, but I think the general idea carries over.)

It's also bad form to access other packages' internals, so there's that.

Fair enough.
Repository: mir-evaluation/mir_eval
Issue: Separation Evaluation Error: All sources should be non-silent
Root cause: Validation Gap

Hi!
I'm trying to evaluate signal separation algorithms, but I'm unable to do it as in `mir_eval.separation.validate(reference, estimated)` I always get the error 

> ValueError: All the reference sources should be non-silent (not all-zeros), but at least one of the reference sources is all 0s, which introduces ambiguity to the evaluation...

My reference is `voc_array` (here I send some stats about this numpy array -dtype= float32):
```
np.mean(voc_array)
Out[17]: 8.973399e-06
np.max(voc_array)
Out[18]: 0.2822876
np.std(voc_array)
Out[19]: 0.059242096
np.shape(voc_array)
Out[20]: (2094534, 1)
```

Can you help me?
Thanks in advance


Your `voc_array` is transposed. It should be of shape `(n_sources, n_samples)`, so `(2094534, 1)`.  The docstrings all state this.

The docstrings do explain this, but this seems like the wrong exception to throw.  Or, at least, the message being given is not as helpful as it could be.

> The docstrings do explain this, but this seems like the wrong exception to throw. Or, at least, the message being given is not as helpful as it could be.

indeed. I remember I was running into this myself a couple of times in the past. Maybe it would make sense to move up the [number of sources check](https://github.com/craffel/mir_eval/blob/master/mir_eval/separation.py#L111) in the validation function so that is triggered before the silence validation.

Thanks. Sorry for not getting it from the documentation, but I was focusing on the error message.
Repository: mir-evaluation/mir_eval
Issue: display functionality: scape plots from segmentations
Root cause: Unimplemented Feature Gap

[Scape plots are pretty](https://ai2-s2-public.s3.amazonaws.com/figures/2017-08-08/62ea16ed458ffbc140dd9761a274a7e0e72c201e/4-Figure4-1.png), but I don't know of a simple way to generate them in python.

Would there be interest in adding this to mir_eval's display module?

That looks cool but I have no idea what I am looking at. Is this a common visualization format for some task in `mir_eval`?

(didn't mean to close)

https://www.semanticscholar.org/paper/A-Scape-Plot-Representation-for-Visualizing-of-M%C3%BCller-Jiang/62ea16ed458ffbc140dd9761a274a7e0e72c201e gives a good overview of the method.

They're useful for multi level segment viz, which can include things like local->global key estimation.

Sure, if you want to code them in a way which is compatible with `mir_eval` annotation format, I am down.
Repository: mir-evaluation/mir_eval
Issue: why is SDR not infinity even though reference and estimate are eqaul?
Root cause: Algorithmic Error

I put the same audio file in both inputs of the  'bss_eval_sources' function.
but SDR is not infinity but around 290 dB...
Why does this happen?

This is probably due to some numerical instabilities. Are you running it on 32bit floats? Also have you tried [BSSeval v4](https://sigsep.github.io/sigsep-mus-eval/museval.metrics.html) for comparison?

Can you provide a full example code snippet that reproduces this behavior?  (Synthetic signals ought to be fine for this.)

My hunch is that it's just a numerical precision / stability quirk.  290 dB may as well be infinite, but the check for returning `np.Inf` requires an exact 0: https://github.com/craffel/mir_eval/blob/master/mir_eval/separation.py#L827-L834 .

@faroit 
Thank you for answer me! 
The variable type is 32bit float. And the BSSeval v4 ('.metrics.bss_eval_source's) also return around 290 dB. 
i just accept that 290dB is infinity.


did you try with dtype 64bit float?
Repository: mir-evaluation/mir_eval
Issue: Is there an acceptable SDR level in speech separation?
GitHub labels: wontfix

In the several speech separation papers, SDR is around 9 ~ 10 dB.
But when i search the google, the minimum acceptable SNR for speech is around 20dB.
And in the image processing, as far as i know, 30dB is acceptable SNR.

what is the acceptable SDR?

And additional i have an additional question.
Does the language affect SDR performance?

Hello @chosw93, this sort of questions is best directed to a user group rather than an issue tracker. You are welcome to post them on the auditory@mcgill list
http://www.auditory.org/
Repository: mir-evaluation/mir_eval
Issue: strict validate_chord_label in 0.4 release
Root cause: External Dependency Breakage

Although issue #217 was solved by bmcfee in #242 , it is still present in the current 0.4 release, which results in undesired behavior.
For example
`mir_eval.chord.validate_chord_label('C:maj')` 
and 
`mir_eval.chord.validate_chord_label('iwufhiweuhf')`
both pass without an exception.



#242 was merged after 0.4 was released. If you want to use the code committed in #242, you'll need to run from master, not from 0.4. There's no plans to make a release in the immediate future, but on the other hand we could do one now if there's demand.

We do seem to have a lot of merged PRs since the last release in 2016. I'd be in favor of a 0.5 release just to get those things out there.

Cool, I can make one in the not-too-distant-future.

A 0.5 release would be great! Thanks for the awesome work. If there is anything I can do, please let me know.

Ok, [0.5 release created](https://github.com/craffel/mir_eval/releases/tag/0.5) and [on PyPI](https://pypi.org/project/mir_eval/0.5/), and [docs are updated](https://craffel.github.io/mir_eval/)! Closing this issue.
Repository: mir-evaluation/mir_eval
Issue: tempo: why not allow 0.0 as reference tempo?
Root cause: Validation Gap

In the docs to `tempo` it says:

> Reference tempi should be strictly positive, and provided in ascending order
as a numpy array of length 2.  Estimated tempi are allowed to be 0, but
otherwise are subject to the same constraints as reference.

I'm wondering why we are dictating that reference tempi must be greater than zero? How do we represent a non-ambiguous track with `T1, T2, ST1 = None, 100.0, 0.0`? The easiest format would actually be `(0.0, 100.0, 0.0)`, but that fails in [validate_tempi](https://github.com/craffel/mir_eval/blob/master/mir_eval/tempo.py#L41).

That aside, It seems to me that contrary to the docs the estimated tempi are not allowed to be 0, either.

Good point.  In fact, the docstrings all say "non-negative" (not "positive"), which indicates that 0 should be allowed.

However, the reason for this comes down to the definition of the tempo detection metrics [here](https://github.com/craffel/mir_eval/blob/master/mir_eval/tempo.py#L122-L123), which requires a division by `ref_t`.

I think what we actually want is for at least one strictly positive value in `reference_tempi` (estimated can both be 0), and proper handling of divide-by-zero cases in the metric.  This complicates the validation logic a bit, but I'd support this change if someone wants to make it.

Gee, Brian—you're quick!

> I think what we actually want is for at least one strictly positive value in reference_tempi (estimated can both be 0)

Agreed.

divide-by-zero logic change—like this?
```
for ref_t in reference_tempi:
        if ref_t == 0:
            continue
        # Compute the relative error for this reference tempo
        relative_errors.append(np.min(
            np.abs(ref_t - estimated_tempi) / float(ref_t)))

        # Count the hits
        hits.append(bool(relative_errors[-1] <= tol))

```

> but I'd support this change if someone wants to make it.

I'm interested in "fixing this" and will send a PR some time tomorrow.

You could also stick an `else` clause on that for loop to catch the case where there are no non-zero reference tempi.
Repository: mir-evaluation/mir_eval
Issue: Deprecation warning in hierarchy
Root cause: External Dependency Breakage

The tests for hierarchy generate so many of the following warning that Travis quits:
https://travis-ci.org/craffel/mir_eval/jobs/477467833#L1241
Apparently somewhere in scipy's sparse matrix implementation, they use the numpy matrix class, and numpy now warns whenever the matrix class is used. See e.g. https://github.com/scipy/scipy/issues/9093
https://github.com/scikit-learn/scikit-learn/issues/12327
@bmcfee do you know where in the hierarchy code this would be triggered?

not off the bat -- I can take a look at this later in the week.

In general, mir_eval is due for an audit of upstream deprecation warnings, so it might be better to handle that all in one go.  I just did the same for librosa, and it was a bit of a pain, but not too bad.

Worst case, you can set certain warnings / modules to only display once during testing, which should "fix" this in the short term.

Fixed in #305
Repository: mir-evaluation/mir_eval
Issue: Chord encoding always adds root note even if not annotated
Root cause: Algorithmic Error

Currently, the root of the chord is *always* added to the semitone bitmap. The encoding thus does not distinguish between, e.g., `C:maj` (or, `C:(1,3,5)`) and `C:(3,5)`:

```python
>>> mir_eval.chord.encode('C:maj')
(0, array([1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0]), 0)
>>> mir_eval.chord.encode('C:(3,5)')
(0, array([1, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 0]), 0)
```
According to Harte's format definition (see his dissertation, page 100), the two should be treated differently:

> This syntax also allows for the case where we have a chord with an
implied root note (such as the C major chord in the example from the
second bar in Figure 4.6). In the example of the C major chord where the
root is not voiced (Figure 4.7d) we can represent it in this way:
C:(3,5)

@ejhumphrey @bmcfee 

I see what you mean, but I think the example that Harte gives is inconsistent with the general semantics of added / suppressed notes.  My understanding, which may diverge from Harte's, is that notes in parenthesis are added or suppressed relative to the stated quality, and that the default quality (absent an explicit name) is a major triad.  In that sense, a rootless C:maj could be encoded as `C:(*1)`, where the 3 and 5 are left in by default.  I would expect `C:(3,5)` to be a no-op on `C:maj` since those notes are already present.

If we do not take the above interpretation, then the consistent alternative is that when parentheses are used, all notes must be explicitly coded.  I don't think that's how most people use this notation, but I'm willing to be convinced otherwise.

In Harte's definition, parentheses are not used to denote added/suspended notes. Instead, they directly define the chord intervals. According to Harte, the fundamental notation is `root : (interval1, interval2, ...) / bass`. There is no such thing as "default quality"  (see 4.2.3 in his diss). Shorthand notations like `maj`, `min`, `sus2` etc. are only introduced afterwards. These "map directly to pre-defined interval lists for the most common chordtypes" (4.2.4). One can then change these pre-defined interval lists with additional intervals in parentheses, and remove intervals using the `*` symbol. As shown in his thesis, the syntax becomes `root : shorthand (additional intervals) / bass`.

I believe his definition is consistent, and I think we should stick to it until some better alternative is found. This probably does not affect many use cases, as annotations as `G:(3,5)` are rare, but I think they still should be parsed correctly (i.e., according to Harte's syntax definition).

I think that makes sense -- I'd like to get @ejhumphrey and @mattmcvicar 's take on things as well, since they've been at chords longer than I.  

Having thought about your interpretation a bit more, I think it all comes down to the default quality assumption.  When a quality shorthand is explicitly coded, I think both of our interpretations are equivalent, and perhaps I was incorrectly generalizing the add/suppress notion to the case without shorthand.
Repository: mir-evaluation/mir_eval
Issue: Test failure in chord on Python 3.5
Root cause: Validation Gap

Our chord test is failing on Python 3.5:
https://travis-ci.org/craffel/mir_eval/jobs/480136121#L9494
I think this is because there are actually two warnings being triggered, one of which is 
`mir_eval/chord.py:573: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use arr[tuple(seq)] instead of arr[seq]. In the future this will be interpreted as an array index, arr[np.array(seq)], which will result either in an error or a different result.`
Repository: mir-evaluation/mir_eval
Issue: Test failure: "No module named 'matplotlib.testing.noseclasses'"
Root cause: External Dependency Breakage

Matplotlib v3+ does not have the matplotlib.testing.noseclasses, which causes a test failure. For now, let's force using matplotlib <3 and migrate to pytest in a separate issue.
https://travis-ci.org/craffel/mir_eval/jobs/480152073#L9273

Issue for migrating to pytest: #269
Repository: mir-evaluation/mir_eval
Issue: melody.raw_chroma_accuracy wrong scores on unvoiced estimates
Root cause: Algorithmic Error

If the input estimate is unvoiced, the chroma accuracy metric should return 0. Instead it counts reference frames that are the multiples of 1200 cents as correctly estimated frames. Consider this example:
```
import mir_eval
import numpy as np

ref_time = np.array([0, 1, 2])
est_time = np.array([0, 1, 2])
ref_freq = np.array([40, 80, 160])
est_freq = np.array([0, 0, 0])

arrays = mir_eval.melody.to_cent_voicing(ref_time, ref_freq, est_time, est_freq)
# ref_cent = [2400., 3600., 4800.]
chroma = mir_eval.melody.raw_chroma_accuracy(*arrays)

print(chroma)
# should be 0.0 but actually is 1.0
```

Just for fun I averaged the raw chroma score of unvoiced estimates on MedleyDB, the result is 7% accuracy.

```
results = []
for t in mdb.load_melody_multitracks():
    ref_time, ref_freq = np.array(t.melody2_annotation).T
    est_time = ref_time
    est_freq = np.zeros_like(ref_freq)
    arrays = mir_eval.melody.to_cent_voicing(ref_time, ref_freq, est_time, est_freq)
    results.append(mir_eval.melody.raw_chroma_accuracy(*arrays))
print(np.mean(results))
# 0.07011304858972267
```

I created a pull request to fix this issue: #312 

Wow, that seems.. not great!  Thanks for catching it.

Tagging @rabitt and @justinsalamon for input.

Just had a quick look, seems like there's indeed a bug (thanks @kukas for catching this!). Will give it (and the PR) a second look hopefully tomorrow to confirm fix.
Repository: mir-evaluation/mir_eval
Issue: Inaccurate pitch_tolerance in transcription.precision_recall_f1_overlap
Root cause: Documentation Deficiency

Computing `transcription.precision_recall_f1_overlap([...], pitch_tolerance = 30)` gives lower statistical outputs than with the default value of `pitch_tolerance`, `transcription.precision_recall_f1_overlap([...], pitch_tolerance = 50)` in my examples. (For example, F-measure is equal to 0.379 with a pitch_tolerance set to 30 and to 0.396 when pitch_tolerance is set to 50).
However, my estimated pitches are midi-scale integers, so as my ground truth. In that sense, the minimal positive gap between an estimated pitch and the ground truth is a semi-tone, or 100 cents.
Hence, a tolerance smaller than 100 cents shouldn't affect the statistical outputs.
I highly suspect a rounding operation misleading the pitch comparison.

Thanks for reporting this. Can you provide a MWE or example files which reproduce the issue?

Sure, here are two .txt files [reference.txt](https://github.com/craffel/mir_eval/files/3079315/reference.txt) and [estimation.txt](https://github.com/craffel/mir_eval/files/3079331/estimation.txt) reproducing the issue. The first and second columns of the files contain respectively the onset and offset times, and the third ones the pitches. In my tests, offset times were ignored.


Sorry, I missed an important detail in your first message. You wrote 

> my estimated pitches are midi-scale integers

That's the wrong format for `transcription.precision_recall_f1_overlap`. The docstring clearly says

> Array of estimated pitch values in Hertz

You can convert from your MIDI pitches to Hz using either `pretty_midi.note_number_to_hz` or
`librosa.midi_to_hz` or just `440.0*(2.0**((note_number - 69)/12.0))`.
Repository: mir-evaluation/mir_eval
Issue: How to control the number of processes mir_eval used.


When calculating SDR, it will use all CPU cores, taking up too much resources. Can this be controlled?

This is not related to mir_eval. You can control the number of threads being used by numpy using the 'OMP_NUM_THREADS' flag. 

> This is not related to mir_eval. You can control the number of threads being used by numpy using the 'OMP_NUM_THREADS' flag.

good，‘OMP_NUM_THREADS‘ is set to 1 as fast as when set to 64 (there are 64 virtual cores).  awkward.
Repository: mir-evaluation/mir_eval
Issue: Couldn't convert value
Root cause: Validation Gap

When I've tried to use mir_eval onset or beat evaluation code, I got this error.

`ValueError: Couldn't convert value  using float found at models/wa4/ah_development_guitar_Guitar_Licks_07-06.onsets:3:
`
What should I do to solve the problem?

It looks like you are trying to load in the file 
`models/wa4/ah_development_guitar_Guitar_Licks_07-06.onsets`
using `mir_eval.io.load_events` (or some other `io` function), but the third line of that file has a value that cannot be converted to a float using the `float()` built-in function. You need to fix your file.