zhiminy/EvalEng huggingface dataset:
Repository: mir-evaluation/mir_eval
Issue: New fingerprinting module
Root cause: Unimplemented Feature Gap

Hi everyone,

We want to add a new evaluation module for audio fingerprinting. We want to display IR metrics related to the correct identification/retrieval of songs but we are also interested in segment-based metrics. The latter should measure the overlapping between the annotated boundaries for a piece of music in a real-life recording/tv/radio broadcast and the music track. Thus, we may have song-based metrics and segment-based metrics related to true-positives, false-positives, true-negatives, false-negatives. 
In addition, we want to account for multiple annotators, so the same metric may be computed for unanimity (all annotators agree) or majority (the majority of the annotators agree). 

The song-based metrics are traditional IR metrics related to precision and recall. 
The closest example to the segment-based metrics are here: https://github.com/craffel/mir_eval/blob/master/mir_eval/segment.py 
So where should we start? 

Best,

Marius Miron 


Hi Marius, are these metrics already published/widely used? Are there any existing implementations?

I don't have a full grasp of the spec from your message, but the IR metrics sound like they would not be different from any standard IR metrics, so I'm not sure there would be a compelling reason to add them to `mir_eval`. If the segment/song-based metrics are different from the current segment-based metrics, I could see a reason to add them.

Hi everyone, @protyposis's [Aurio](https://github.com/protyposis/Aurio) library implements most common/used fingerprint algos (and not only).

If you need more/others we've collected some for our [HyMPS project](https://forart.it/HyMPS/) under the [AUDIO category](https://github.com/forart/HyMPS#-) \ [Visualization/analisys page](https://github.com/forart/HyMPS/blob/main/Audio/Visualysis.md#--) \ [Fingerprinting subsection](https://github.com/forart/HyMPS/blob/main/Audio/Visualysis.md#fingerprinting-)).

Hope that inspires !
Repository: mir-evaluation/mir_eval
Issue: Evaluation of SDR and SIR
Root cause: Documentation Deficiency

Hello.
I would like to evaluate SDR and SIR.
I am running the following sample code.
```
import mir_eval

reference_beats = mir_eval.io.load_events('reference_beats.txt')
estimated_beats = mir_eval.io.load_events('estimated_beats.txt')

scores = mir_eval.beat.evaluate(reference_beats, estimated_beats)

print(scores)
```

I have some questions.
What do I put in `reference_beats.txt` and `estimated_beats.txt`?
Is it the path to the wav file?
When I put the path to the wav file in the text file, I get the following error.

> (eval_test) C:\Users\4020737\Documents\git\MirEval\mir_eval>python test.py
Traceback (most recent call last):
  File "C:\Users\4020737\Documents\git\MirEval\mir_eval\mir_eval\io.py", line 108, in load_delimited
    converted_value = converter(value)
ValueError: could not convert string to float: 'C:\\Users\\4020737\\Documents\\git\\MirEval\\mir_eval\\data\\ref.wav'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "test.py", line 3, in <module>
    reference_beats = mir_eval.io.load_events('reference_beats.txt')
  File "C:\Users\4020737\Documents\git\MirEval\mir_eval\mir_eval\io.py", line 149, in load_events
    events = load_delimited(filename, [float],
  File "C:\Users\4020737\Documents\git\MirEval\mir_eval\mir_eval\io.py", line 110, in load_delimited
    raise ValueError("Couldn't convert value {} using {} "
ValueError: Couldn't convert value C:\Users\4020737\Documents\git\MirEval\mir_eval\data\ref.wav using float found at ref
erence_beats.txt:1:
        C:\Users\4020737\Documents\git\MirEval\mir_eval\data\ref.wav

Could you please tell me the procedure for evaluating SDR and SIR?

Best regards.

My environment is as follows.
I installed using `>python setup.py install`
> ca-certificates           2022.4.26            haa95532_0
certifi                   2022.6.15        py38haa95532_0
future                    0.18.2                   pypi_0    pypi
numpy                     1.23.0                   pypi_0    pypi
openssl                   1.1.1o               h2bbff1b_0
pip                       21.2.2           py38haa95532_0
python                    3.8.13               h6244533_0
scipy                     1.8.1                    pypi_0    pypi
setuptools                61.2.0           py38haa95532_0
six                       1.16.0                   pypi_0    pypi
sqlite                    3.38.3               h2bbff1b_0
vc                        14.2                 h21ff451_1
vs2015_runtime            14.27.29016          h5e58377_2
wheel                     0.37.1             pyhd3eb1b0_0
wincertstore              0.2              py38haa95532_2


Your example code seems to be for beat evaluation, not SDR/SIR evaluation. You can use https://craffel.github.io/mir_eval/#mir_eval.separation.bss_eval_sources
Repository: mir-evaluation/mir_eval
Issue: How to evaluate SIR and SDR for mono wav files
Root cause: Documentation Deficiency

Hello.

I was taught how to evaluate SDR and SIR in Issue `#347`. Thank you very much.

I have a question about how to evaluate SIR and SDR for mono wav files.

For example, I want to evaluate ref.wav and est.wav.
The length of the wav file is 4 seconds. The sampling frequency is 16k Hz.

When I evaluated them with the following code, the SIR was Inf.

```
import mir_eval
import numpy as np
from scipy.io import wavfile

Fs = 16000
file_len_sec = 4
file_ch = 1

ref_sources = np.zeros((file_ch, Fs*file_len_sec))
est_sources = np.zeros((file_ch, Fs*file_len_sec))

fs, ref1 = wavfile.read("./data/ref.wav")
_,  est1 = wavfile.read("./data/est.wav")

ref_sources[0] = ref1
est_sources[0] = est1

(sdr, sir, sar, perm) = mir_eval.separation.bss_eval_sources(ref_sources, est_sources, compute_permutation=False)

print('sdr:', sdr)
print('sir:', sir)
print('sar:', sar)
print('perm:', perm)
```

```
sdr: [14.18888428]
sir: [inf]
sar: [14.18888428]
perm: [0]
```

To avoid the SIR to be Inf, I divided the wav file into 4 parts. Is the following code able to evaluate SIR and SDR correctly?
```
import mir_eval
import numpy as np
from scipy.io import wavfile

Fs = 16000
file_len_sec = 1
file_ch = 4

ref_sources = np.zeros((file_ch, Fs*file_len_sec))
est_sources = np.zeros((file_ch, Fs*file_len_sec))

fs, ref1 = wavfile.read("./data/ref1.wav")
_,  est1 = wavfile.read("./data/est1.wav")
fs, ref2 = wavfile.read("./data/ref2.wav")
_,  est2 = wavfile.read("./data/est2.wav")
fs, ref3 = wavfile.read("./data/ref3.wav")
_,  est3 = wavfile.read("./data/est3.wav")
fs, ref4 = wavfile.read("./data/ref4.wav")
_,  est4 = wavfile.read("./data/est4.wav")

ref_sources[0] = ref1
ref_sources[1] = ref2
ref_sources[2] = ref3
ref_sources[3] = ref4
est_sources[0] = est1
est_sources[1] = est2
est_sources[2] = est3
est_sources[3] = est4

(sdr, sir, sar, perm) = mir_eval.separation.bss_eval_sources(ref_sources, est_sources, compute_permutation=False)

print('sdr:', sdr.mean())
print('sir:', sir.mean())
print('sar:', sar.mean())
print('perm:', perm)
```

```
sdr: 16.156123610321224
sir: 28.957842593287353
sar: 16.444840346137298
perm: [0 1 2 3]
```

How do I evaluate the SIR and SDR for a mono wav file?I am attaching the zip data.
Best regards.
[data.zip](https://github.com/craffel/mir_eval/files/8988886/data.zip)

See https://github.com/craffel/mir_eval/issues/260 - sorry, can't provide any additional support of this kind.
Repository: mir-evaluation/mir_eval
Issue: Are there any methods for transposing chord notations from one key to another?

I am working with data whose chord notations / symbols are in the format mir_eval works with (e.g C:min, C:7, etc). In order to model the transitions between chords, it's more convenient to transpose all the chords to C Major or A minor. This is a step that I have seen mentioned in many papers regarding chord detection / harmonic analysis.

Q: **Does mir_eval have any methods to do the same, on the chord notations / symbols in the format it accepts?** I know a few python libraries like chords-parser, mingus which work with notation like this (Cm, C7, Cmaj7), however, I would need to translate between the 2 types of notations ( I am able to do it for easier chords). The difference can get a lot more complicated for chords with inversions, augmented chords, suspended chords, etc.

mir_eval doesn't do this directly, but I have implemented this kind of thing over in my data augmentation library [muda](https://github.com/bmcfee/muda/), specifically here:

https://github.com/bmcfee/muda/blob/9e4ef5fabce1f35e53829940aff51aa6b0a30893/muda/deformers/pitch.py#L18-L50

This operates on chord annotations that would validate against an extended version of Harte's grammar, which is essentially what mir_eval uses for chord parsing.  This would cover all the inversion, added/suppressed notes, and extended quality information automatically.  It's mainly used to transpose a chord annotation by a fixed number of semitones, which occurs during training of chord models to achieve key invariance.

The one thing it won't do for you is spelling the root note of the chords - it goes through midi numbers for convenience, and converts back assuming C:maj.  The implementation above pre-dates our implementation of note spelling in librosa, but it would be easy to add if you need it.  OTOH, if you're only ever transposing into C:maj or A:min, it shouldn't be necessary.
Repository: mir-evaluation/mir_eval
Issue: Test fails on 32 bit x86
GitHub labels: bug

```
ERROR: test_sonify.test_time_frequency
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/builds/debian/mir-eval/debian/output/source_dir/tests/test_sonify.py", line 24, in test_time_frequency
    signal = mir_eval.sonify.time_frequency(
  File "/builds/debian/mir-eval/debian/output/source_dir/.pybuild/cpython3_3.10_mir-eval/build/mir_eval/sonify.py", line 177, in time_frequency
    wave[start:end] * gram_interpolator(np.arange(start, end)))
ValueError: array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size.
======================================================================
ERROR: test_sonify.test_chroma
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/builds/debian/mir-eval/debian/output/source_dir/tests/test_sonify.py", line 36, in test_chroma
    signal = mir_eval.sonify.chroma(
  File "/builds/debian/mir-eval/debian/output/source_dir/.pybuild/cpython3_3.10_mir-eval/build/mir_eval/sonify.py", line 297, in chroma
    return time_frequency(gram, frequencies, times, fs, **kwargs)
  File "/builds/debian/mir-eval/debian/output/source_dir/.pybuild/cpython3_3.10_mir-eval/build/mir_eval/sonify.py", line 177, in time_frequency
    wave[start:end] * gram_interpolator(np.arange(start, end)))
ValueError: array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size.
----------------------------------------------------------------------
```

I think this should have been addressed by #355 - we had a units conversion error between samples and seconds.  For reasonable input values, this should no longer hit an integer overflow.

I tested this out on a 32bit environment and it seems fixed now.  I'll close this out, but reopen if the problem recurs.
Repository: mir-evaluation/mir_eval
Issue: Migrate pull request automated testing from Travis CI to GitHub Actions
Root cause: Unimplemented Feature Gap

To help make sure mir_eval stays in good shape as its dependencies moves, it would be convenient to have test automation for PRs to make reviews easier. Time flies, Travis CI seems broken (?), and the world has moved to GitHub Actions.

I'm wondering if you'd like to have automated testing by a .github/workflows/ci.yaml - if so, I could draft a proposal in a PR (since I happen to have professional work hours to spend on helping out currently).

👍 / 👎 ❓

This would be great, and long overdue.

In setting this up, I would also advocate for breaking up some of the CI functionality across different workflows, so that there are separate actions for unit tests, linting, and doc testing.

That would be great.
Repository: mir-evaluation/mir_eval
Issue: segment unit test fails
Root cause: Algorithmic Error

(posted by Colin)

On my computer, the segment unit test matches exactly.  On Dawen and Eric's (and maybe other's) computers, the segmentation unit test fails on the following metrics:

Pair-P 0.760466079298 0.759487707538 False
Pair-R 0.439344606659 0.439064910407 False
Pair-F 0.556932313401 0.55644516106 False
ARI 0.36752433328 0.366763678811 False
MI 0.69019595974 0.690003644248 False
AMI 0.414313462999 0.414259923205 False
NMI 0.509383741119 0.509459218087 False
S_Over 0.765096115476 0.765422031141 False
S_Under 0.505541891353 0.505574473705 False
S_F 0.608809330754 0.608936120723 False


It doesn't fail as of commit 3ca9534c25b3e014bde72e648e361e31294fca7e -- @ejhumphrey is there anything you would have done which would have changed the output of these metrics in the past 24 hours?


ahhhhhh, yes. this one's on me, and is due to a discrepancy in quantization
strategies. In reworking intervals_to_samples such that it can return the
time points used to interpolate the intervals, I modified the part where
@bmcfee was rounding off the interval times, opting for a strictly uniform
time grid.

After probably too much thought, I _think_ rounding the intervals in that
previous manner introduces a teeny bit of non-linear time warping and
weighs the contributions of each observation differently as a function of
time (and is the source of the discrepancy we're seeing now), but I could
certainly be wrong. Not sure how you guys want to handle this, but I'm not
terribly opposed to rolling back to the previous strategy.

sorry for the goose chase...

On Mon, Mar 31, 2014 at 4:52 PM, craffel notifications@github.com wrote:

> It doesn't fail as of commit 3ca9534https://github.com/craffel/mir_eval/commit/3ca9534c25b3e014bde72e648e361e31294fca7e--
> @ejhumphrey https://github.com/ejhumphrey is there anything you would
> have done which would have changed the output of these metrics in the past
> 24 hours?
> 
> ## 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/36#issuecomment-39140331
> .


No worries - I think once consensus is reached in terms of how to sample, we can just update the cached scores as we are only regression testing the segment module at this point.  @bmcfee maybe give your two cents once you have jumped the interspeech hurdle?


resolved in 87f1ee9577cb659ee202071542eb1a14bfcf4ed4
Repository: mir-evaluation/mir_eval
Issue: Entropy Based Evaluation of unlabelled sections
Root cause: Unimplemented Feature Gap

Dear MIR community,

I've been revisiting the reference Professor McFee gave in https://github.com/craffel/mir_eval/issues/363#issuecomment-1715848028, and it's still not clear that either Onset or Hit Rate are the right metric for our problem (which is N unlabeled estimated boundaries vs M human boundaries), primarily around the lack of a smoothness function (due to the tolerance factor).

I've actually just started using Entropy measure to help fit estimated boundaries to 16-bar boundaries, and realized that this approach might be useful for measuring the estimated boundaries to reference boundaries as well. I'm not very mathematically equipped, but the algorithm would go something like this:

Each boundary contains beat aligned intervals. The Entropy H(p,n) = p/p+n log p/p+n) is calculated by taking the first estimated boundary, making the intervals inside that "p", and all the rest "n". We then project those into the reference boundary, and compute the Entropy. EG   0 0 0 0 1 1 1 1 2 2 2 2, vs  0 0 0 0 0 0 1 1 1 1 1 1, would give  H(4, 6) + H(0, 6), + H(2, 6) +  H(2,6) + H(0, 6) + H(4, 6). However, we'd want both directions, and so would also have H(4,4) + H(2,4) + H(0,4) + H(0,4) + H(2,4) + H(4,4), and we'd combine them using some aggregate (sum, min, average). 

I don't want to reinvent the wheel here, so if anyone has any ideas why this is a bad idea or if it's been done before, I'd love feedback. To reiterate, the goal is an objective function measuring closeness of sectioning, not the precision/recall of hit/not a hit. In the tolerance world, the precision and recall of the above section is zero (IIUC).
Repository: mir-evaluation/mir_eval
Issue: Release a patched version to PyPI for NumPy compatibility
Root cause: External Dependency Breakage

Noticed that people have started employing the work around of upper bounding NumPy in their usage of mir_eval now ([example](https://gitlab.inria.fr/amarmore/autosimilarity_segmentation/-/blob/main/setup.py?ref_type=heads#L32)) which tells me that it's high time to make a new release.

What steps are remaining to take to make mir_eval compatible with the latest NumPy again?

IIRC it's only a matter of running the GitHub Actions release workflow after bumping the version number in the code but I'm also not fully aware of relevant changes since 0.7 was built, so is it an incoming 0.8 or 0.7.1?

Something I could help with to expedite this? I'm available to help.

> What steps are remaining to take to make mir_eval compatible with the latest NumPy again?

AFAIK it's really just down to replacing uses of deprecated dtype aliases (np.complex, np.float) for python builtins (complex, float).  And of course, updating our CI environments to make sure that it's all tested appropriately.

When I use `mir_eval.chord.encode`, I got:
```
  File "/home/lzq/anaconda3/lib/python3.12/site-packages/mir_eval/chord.py", line 513, in encode
    semitone_bitmap = (semitone_bitmap > 0).astype(np.int)
                                                   ^^^^^^
  File "/home/lzq/anaconda3/lib/python3.12/site-packages/numpy/__init__.py", line 324, in __getattr__
    raise AttributeError(__former_attrs__[attr])
AttributeError: module 'numpy' has no attribute 'int'.
`np.int` was a deprecated alias for the builtin `int`. To avoid this error in existing code, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
The aliases was originally deprecated in NumPy 1.20; for more details and guidance see the original release note at:
    https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations. Did you mean: 'inf'?
```
It' s an unfortunate limitation that `mir_eval` is only compatiable with `numpy<=1.20`.

Release candidate 1 is now up.  Barring any major problems, I'll cut a final release within the next couple of days.
Repository: mir-evaluation/mir_eval
Issue: Docs: multipitch.evaluation macro scores
Root cause: Documentation Deficiency

I'm confused by the docs: https://github.com/craffel/mir_eval/blob/4d52264c90ec6f69d5dcfa013eba98d1479ae7af/mir_eval/multipitch.py#L348

in what sense is this the "macro" level? I thought this is what is called the 'micro-average', i.e. average across every single instance, whereas "macro" means per-class scores that are then averaged. Are the docs wrong here?

@rabitt
Repository: mir-evaluation/mir_eval
Issue: 0.8 meta issue
Root cause: Unimplemented Feature Gap

:wave: Seems like we're overdue for a new release here, if for no reason other than #352 (mir_eval currently doesn't work on the most recent numpy).

I've gone ahead and tagged a few issues that seemed particularly important / relevant to include here, and am happy to take the lead on knocking some of this out.  In particular, I'd like to modernize the CI / test / packaging environments.

@craffel any thoughts on this?  I know it's been a while, but I think we should aim for a quick turnaround here, and I have a few spare cycles right now to get things moving.

Yes, happy to have a new release! I'm on parental leave, so happy to have you do it if you have spare cycles. Thanks.

Congrats!  Seems like there's a lot of that going around these days :grin:  (my leave is just winding down now, hence the spare cycles before everyone else notices...)

I can manage most of this myself, but we will need to coordinate a little on setting up the release.  The best practices now using GH actions to automate pypi push on release make it pretty easy to deploy, but you'll need to do a little configuring on the mir_eval pypi settings to enable it.

See https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/

I just set this up in pescador yesterday, so we can go off that as a template: https://github.com/pescadores/pescador/blob/main/.github/workflows/publish.yml

Great - just bug me with the steps I need to take. There may be some latency (right now the baby is napping so latency is low).

Release candidate made its way out - I had to modernize some workflow stuff for the pypi upload, but it seems to all be working now.  I'll leave it in rc status for a day or so, but if nobody finds any glaring problems, we can finally get this off the stack!

Last step upon release will be to set up a redirect on the gh-pages to point to the new rtd build.
Repository: mir-evaluation/mir_eval
Issue: Submodules should include ordered dictionaries listing all metrics
Root cause: Unimplemented Feature Gap

Each task submodule should have a member `METRICS` which is an ordered dict.  The keys should be strings which are the names of each metric, and the values should be the functions which implement each metric.
- [x] beat  @craffel
- [x] onset @craffel
- [x] separate @dawenl 
- [x] segment @bmcfee
- [x] chord @ejhumphrey 
- [x] melody @justinsalamon 
- [x] pattern @urinieto 


After implementing this for boundary and structure, it seems to not line up exactly with the original intent from `segment_eval.py`.  Most of the metrics return multiple values (precision, recall, f-measure), and don't map cleanly into a single keyword.

Is this per-module dict still a useful concept?


It's not a problem that most metrics return a tuple.  The point is to have an ordered dict which points to functions.  Yes, it's still a useful concept.


"Should be uppercase." -PEP8/Brian


Melody now has a metric dictionary.  Unfortunately, the `voicing_measures` metric only takes two voicing arguments, so you can't do

``` Python
for name, metric in mir_eval.melody.METRICS.items():
    print name, metric(ref_v, est_v, ref_c, est_c)
```

The options would be to not actually advise that people do this (a sad option because it's kind of one of the nice things about having the METRICS object), make `voicing_measures` take four arguments and use only two (an appalling option), just forget about the METRICS object altogether because it's not as general purpose as we might like (a defeatist option), or just ignore it for now (an ostrich solution).