zhiminy/EvalEng huggingface dataset:
Repository: mir-evaluation/mir_eval
Issue: sonify.time_frequency regression
Root cause: Algorithmic Error

@leighsmith looks like modernizing the tests #370 surfaced a bug in the optimization of time_frequency sonification #355 

The bug only seems to show up in the chord sonification, which goes through chord→chroma and chroma→time_frequency.  It works fine when `length=None` is used, but when `length` is explicitly passed and different from the end of the last time interval, we end up with a length mismatch at
https://github.com/craffel/mir_eval/blob/57f7c31b120f6135c31207295372e3b67848126d/mir_eval/sonify.py#L190-L191 .

We didn't have this problem before #355 because the length was always precisely determined to using matching `start:end` indices for each time interval.

Since I'm in the midst of revamping all the tests, we're not really in a good place to handle this bugfix as a separate PR yet.  If there's an easy fix, I'd like to roll it into #370 so we can merge things in a passing state.  Otherwise I might mark that test as a known fail for now and we can come back to it after merging 370.

Thought this over a bit, and I think I have a not-too-intrusive fix that I can PR after #370 merges.

The basic idea is to decouple the output array allocation length from the input / time grid length.  Right now these things are implicitly tangled, and that's causing us trouble.

Unit tests will need to be expanded to cover a few cases:

- Time indices vs time intervals
- length inferred (None, should not be a problem)
- length < max time (in either interval or index case)
- length > max time (ditto)




Yes, good call, totally agree that decoupling the allocation length from the time grid length is the right approach. Sorry, I'm snowed under with other work and can't get to it until the weekend.

I found a cleaner solution here: pull out the sample indices for the interval in question, and index directly into the output array.  (This is how it used to work, pre-#355.)

So instead of 

```python
output += wave[:length] * interpolator(...)
```

we do

```python
t_min, t_max = ...
signal = interpolator(np.arange(t_min, t_max))  # len(signal) = t_max - t_min
output[t_min:t_max] += wave[:len(signal)] * signal
```

Tested out and it works, and we retain the speedups from #355.

This is implemented in #374 and should be good to go.
Repository: mir-evaluation/mir_eval
Issue: Set warning stacklevels for validation
Root cause: Validation Gap

Validator functions often raise warnings when something isn't quite right, eg:

https://github.com/craffel/mir_eval/blob/57f7c31b120f6135c31207295372e3b67848126d/mir_eval/chord.py#L640-L644

However, the warning shows up to the user as coming from the `validate` function, which is not all that helpful since they never call this directly.  We can make this more useful by setting  `stacklevel=2`.  

This way the warning will point to the evaluator that called validate, which is a little more helpful for debugging purposes, especially when the user is calling a module-level `evaluate()` function.

Good idea! Didn't know you could specify where in the stack the warning should come up.
Repository: mir-evaluation/mir_eval
Issue: Replace scipy.fftpack with scipy.fft
Root cause: Unimplemented Feature Gap

This came up in #370, but the source separation metrics are presently implemented using the old / legacy scipy.fftpack API.  As the modernization PR bumps the minimum scipy version requirement to 1.4, we can now update this to use the newer scipy.fft API.  See https://docs.scipy.org/doc/scipy/release/1.4.0-notes.html#scipy-fft-added

The main benefits of doing so are that we can get some speed and memory improvements by using rffts instead of ffts (big benefit for long signals!).  We can also squeak out a bit more performance by using the [next_fast_len](https://docs.scipy.org/doc/scipy/reference/generated/scipy.fft.next_fast_len.html#scipy-fft-next-fast-len) function instead of assuming `2**k`.

Probably we can also vectorize some of the transform implementations.

Just a quick benchmark on my laptop using our fixtures, TLDR is we can cut FFT costs about in half, just by using rfft and a smarter length calculation:

```
In [32]: ref_sources = __load_and_stack_wavs('data/separation/ref06/')

In [33]: ref_sources.shape
Out[33]: (2, 16000)

In [35]: n = ref_sources.shape[-1]

In [36]: n_fft = int(2**np.ceil(np.log2(n + flen - 1)))

In [37]: n_fft2 = scipy.fft.next_fast_len(n + flen - 1)

In [38]: n_fft
Out[38]: 32768

In [39]: n_fft2
Out[39]: 16632

In [40]: %timeit scipy.fftpack.fft(ref_sources, n=n_fft, axis=1)
382 µs ± 3.72 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [41]: %timeit scipy.fft.fft(ref_sources, n=n_fft, axis=1)
383 µs ± 3.4 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [42]: %timeit scipy.fft.rfft(ref_sources, n=n_fft, axis=1)
285 µs ± 20.5 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each)

In [43]: %timeit scipy.fft.rfft(ref_sources, n=n_fft2, axis=1)
196 µs ± 124 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each)
```

This would allow us to simplify some downstream code (ie remove some np.real casts) and should not cost us anything in precision.
Repository: mir-evaluation/mir_eval
Issue: Migrate to organization

This was discussed in #370, but we agreed to migrate this repo to a dedicated organization.  (`mir_eval`?)

There are a couple of other things that will need to be squared away when we do this:

- [x] update author in README and setup.cfg
- [x] update copyright in sphinx docs
- [x] update all URLs to point to the new home.  (GitHub will redirect this automatically, and we can use sphinx linkcheck to identify this automatically in a CI action.)
- [x] update badges in README
- [x] manage permissions for integrated services, codecov, pypa, zenodo, readthedocs, etc

Feel free to add to this list as needed.

please tag a release to enable `nose` removal from distributions. Greetings

Nudging @craffel - any chance we could get the org for this setup this week?  We're looking to do a bit of in-house code maintenance over here, and AFAICT this is the only thing blocking the next mir_eval release (which in turn blocks other downstream updates for numpy 2 compatibility).

Frankly will likely not get to this until the semester is done, sorry.

What if I was to create a new org and all you have to do is transfer the repo?  (Assuming you're still okay with the entire idea, that is.)

Yes, of course, happy to do that.

All the good names were already taken :\ but I just made https://github.com/mir-evaluation and sent you an invite.

Transferred, thanks!

Thanks!  I think the only other thing that will need doing from your side before a new release is updating the credentials on pypi to allow the publish action from the new repo address.

Added: 
<img width="660" alt="image" src="https://github.com/user-attachments/assets/2cf2aa79-4837-4566-b306-ba66fbce067f">
Let me know if that does the trick. Pypi was also yelling at me that I was the sole maintainer of mir_eval so I added Brian.McFee as a maintainer too.

Err, it's telling me Brian.McFee is not your username even though that's the owner of librosa, crema, etc. What's your pypi username?

> Let me know if that does the trick.

LGTM, thanks!



> Pypi was also yelling at me that I was the sole maintainer of mir_eval so I added Brian.McFee as a maintainer too.  Err, it's telling me Brian.McFee is not your username even though that's the owner of librosa, crema, etc. What's your pypi username?

No, that's correct.  Weird.

I don't think it's critical for getting pub action to work though.

@craffel can you take another crack at adding me as a maintainer on pypi?  I see no reason why it shouldn't work - perhaps that was a momentary glitch back in october?

Now it says "Invite pending" so I think it may have worked this round?

I received one for pretty midi, but not mir_eval.

Oh, well, I suppose that's because I clicked on the wrong project. I've done the same for mir_eval...

Thanks!  Looks like we're all set.
Repository: mir-evaluation/mir_eval
Issue: Separation test precision on OSX
Root cause: Environment Incompatibility

Looks like we're hitting numerical precision issues again, sometimes, in the source separation tests on OSX.  See https://github.com/craffel/mir_eval/actions/runs/8426164125/job/23106646394?pr=374

The tests passed previously (eg in #370), and since we're now seeing the RNG before every test function execution, the deviation must be coming from the unerlying blas implementation and/or some interaction with the hardware on which it's deploying.

I've been seeing similar weirdness in other packages (pescador, librosa) lately, and there are some strange things happening with openblas and xsimd that expose the non-associativity of floating point arithmetic in cases like this.

I propose to "fixing" this by detecting the execution platform and raising the `atol` parameter on OSX deployments in `test_separation.py`.  Since the separation metrics are in decibels, I don't think we should be too concerned about raising the tolerance from 0.01dB to 0.05dB, and keeping the stricter tolerance on better-behaved platforms should keep us safe.

Meanwhile, I don't think separation test failures on osx should be a blocker to merging unrelated PRs (eg #374).

Punting this from the milestone since #382 renders it irrelevant.
Repository: mir-evaluation/mir_eval
Issue: Split segment submodule into two submodules

One for boundary, and one for structure


Let's split the unit tests too.


The unit tests are functionally split (and test independently in nose), but live in the same source file because of shared functionality in the fixture generator (loading inputs).  I don't see any compelling reason to split the tests into separate files.


Agreed.


@bmcfee I think the reason we did this was so that a user could iterate over the METRICS dictionary in each and call each metric using the same call structure.  Is that what you remember?  If so, I advocate undoing this split once #54 is resolved.
Repository: mir-evaluation/mir_eval
Issue: Fix display tests for matplotlib 3.9
Root cause: External Dependency Breakage

This came up in #382 , but matplotlib 3.9 is causing some regressions on display tests.  From a quick read-through of the changelog, I don't see any obvious culprits.

It may have something to do with legend positioning or tick markers. :shrug: 

Had a chance to test this locally, it's a grab bag.  Most failures are indeed due to legend location, and we can fix this by setting the loc explicitly.

Segment displays are somehow back to not setting limits properly though, so the viewport is completely wrong.  These will take some work to fix.

Ok, I think this issue with segments is due to this change: https://matplotlib.org/stable/api/prev_api_changes/api_changes_3.9.0.html#boxplots-now-ignore-masked-data-points - axvspan now creates Rectangle instead of Polygon patches, and for some reason that means they are now not participating in data limit calculations.

Issue punted upstream to https://github.com/matplotlib/matplotlib/issues/28383

Looks like the fix is merged and will be in the 3.9.1 release.
Repository: mir-evaluation/mir_eval
Issue: Separation module not compatible with NumPy 2.0
Root cause: External Dependency Breakage

`np.Inf` was replaced with `np.inf` in NumPy 2.0 so the [`_safe_db` ](https://github.com/craffel/mir_eval/blob/485a42551f449517567313f9871f58e9c99fbd8a/mir_eval/separation.py#L834) function breaks.

Here's the traceback of a dummy example.
```
>>> import numpy as np
>>> import mir_eval
>>> y_ref = np.random.randn(1, 48000)
>>> y_est = np.random.randn(1, 48000)
>>> mir_eval.separation.bss_eval_sources(y_ref, y_est)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/richanamballa/miniconda3/envs/mss/lib/python3.12/site-packages/mir_eval/separation.py", line 214, in bss_eval_sources
    _bss_source_crit(s_true, e_spat, e_interf, e_artif)
  File "/Users/richanamballa/miniconda3/envs/mss/lib/python3.12/site-packages/mir_eval/separation.py", line 810, in _bss_source_crit
    sir = _safe_db(np.sum(s_filt**2), np.sum(e_interf**2))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/richanamballa/miniconda3/envs/mss/lib/python3.12/site-packages/mir_eval/separation.py", line 833, in _safe_db
    return np.Inf
           ^^^^^^
  File "/Users/richanamballa/miniconda3/envs/mss/lib/python3.12/site-packages/numpy/__init__.py", line 397, in __getattr__
    raise AttributeError(
AttributeError: `np.Inf` was removed in the NumPy 2.0 release. Use `np.inf` instead.. Did you mean: 'inf'?
```

It should be an easy fix to replace `np.Inf` with `np.inf`.

For reference, I'm using `numpy 2.0.1` and `mir_eval 0.7`. As a temporary fix, I downgraded to `numpy 1.26.4`.

Never mind! I see that the issue is being fixed in #385.

Yeah, trying to get this fixed ASAP. ETA this week. 🤞
Repository: mir-evaluation/mir_eval
Issue: Link to online docs broken

The mir_eval [README](https://github.com/mir-evaluation/mir_eval/blob/main/README.rst) links to the documentation for this project at [https://craffel.github.io/mir_eval/](https://craffel.github.io/mir_eval/). However, when trying to visit that page I am told "File not found” (404 error). Has the documentation perhaps been moved so that this link should be updated?

Thanks a lot

Best wishes,
Torsten Anders


Thanks for noting this.  We moved the repo to a new organization last week, and it looks like github does not automatically forward old addresses.

@craffel If you can add a redirect from your craffel.github.io repo to the new URL https://mir-evaluation.github.io/mir_eval/ - then we can transition this gracefully.

Ye olde chatbot says this should work in lieu of proper server-side redirects (which gh pages apparently doesn't do):

- `mir_eval/index.html`
```html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <script type="text/javascript">
        // Get the current URL path after /mir_eval/
        var path = window.location.pathname.replace('/mir_eval/', '');
        // Redirect to the new GitHub Pages site, maintaining the same path
        window.location.href = 'https://mir-evaluation.github.io/mir_eval/' + path;
    </script>
    <title>Page Moved</title>
</head>
<body>
    <p>If you are not redirected, click <a href="https://mir-evaluation.github.io/mir_eval/">here</a>.</p>
</body>
</html>
```

Done!

Thanks a lot for the quick update!
Repository: mir-evaluation/mir_eval
Issue: Metrics should include example usage
Root cause: Documentation Deficiency

Each metric in each task submodule should include an example usage in the docstring, which includes loading and preprocessing data from a file and calling the metric, e.g.

```
:usage:
    >>> reference_beats = mir_eval.beat.trim_beats(mir_eval.io.load_events('reference.txt'))
    >>> estimated_beats = mir_eval.beat.trim_beats(mir_eval.io.load_events('estimated.txt'))
    >>> f_measure = mir_eval.beat.f_measure(reference_beats, estimated_beats)
```
- [x] beat  @craffel
- [x] onset @craffel
- [x] separate @dawenl 
- [x] segment @bmcfee
- [x] chord @ejhumphrey 
- [x] melody @justinsalamon 
- [x] pattern @urinieto 


The beat examples are no longer valid because load_events returns the event times and synthetic labels.
Repository: mir-evaluation/mir_eval
Issue: BrokenBarHCollection has been removed from matplotlib in 3.7
Root cause: External Dependency Breakage

mir_eval.display.piano_roll does not work for matplotlib>=3.7 because it uses BrokenBarHCollection that has been remove in  matplotlib 3.7 ( see https://matplotlib.org/stable/api/prev_api_changes/api_changes_3.7.0.html) 

Should have been fixed in #380, display no longer used BrokenBarHCollection
Repository: mir-evaluation/mir_eval
Issue: Segment: make it possible to provide boundaries, not only intervals
Root cause: Unimplemented Feature Gap

In some use-cases we have boundaries as list of points like [0, 10, 21, 55, 365] (peak positions, breakpoints, etc.), to make mir_eval.segment applicable we have to turn those points into intervals. But one can notice, that some of the evaluation methods actually convert intervals back to boundaries.

I suppose, that would be convenient to have api allowing users to avoid unnecessary conversions.
Repository: mir-evaluation/mir_eval
Issue: Codecov uploads failing (again)

Title says it all.  The error we're seeing now is "Repository not found" - it appears that this is hitting several codecov users, and I don't think it is due to a configuration error on our side.  I'm opening an issue here anyway to keep tabs on it.

This appears to be working again, after I purged the config and set up from scratch.  Go figure.
Repository: mir-evaluation/mir_eval
Issue: Change "label_prefix" to "fill_value" in adjust_intervals?
GitHub labels: enhancement

question

While I can see how a prefix might help (filtering and whatnot), it'd be super helpful in chord-rec to just specify what the out-of-boundary label is / should be. I think this would be more elegant than writing chord parsing code to clean up the result after the fact. 

Any major objections or justifications I'm not seeing? @bmcfee I think this is relevant to you?


I like it.


Done.
Repository: mir-evaluation/mir_eval
Issue: Display docs not building on RTD
Root cause: Configuration Error

I've noticed in the most recent doc build that the display submodule is not populating correctly in the RTD-built docs, probably because matplotlib is not a build requirement for docs.  From the build logs on RTD:

```
[autosummary] generating autosummary for: api/alignment.rst, api/beat.rst, api/chord.rst, api/display.rst, api/hierarchy.rst, api/index.rst, api/io.rst, api/key.rst, api/melody.rst, api/multipitch.rst, ..., api/pattern.rst, api/segment.rst, api/separation.rst, api/sonify.rst, api/tempo.rst, api/transcription.rst, api/transcription_velocity.rst, api/util.rst, changes.rst, index.rst
WARNING: Failed to import mir_eval.display.
Possible hints:
* ModuleNotFoundError: No module named 'matplotlib'
* AttributeError: module 'mir_eval' has no attribute 'display'
```

Fix here would be to make mpl a requirement for doc building.  This is a bit overkill perhaps, but definitely the simplest fix.
Repository: mir-evaluation/mir_eval
Issue: `sonify.chords` does not work with a single interval
Root cause: Validation Gap

if we try to run `sonify.chords` with a single interval, it throws a TypeError.

upon inspection, the problem seems to be in the [`_const_interpolator`](https://github.com/mir-evaluation/mir_eval/blob/a0a8672d11ab66c3a89900d0dd35d8cdfcf8a1f9/mir_eval/sonify.py#L159-L167) (inside `time_frequency`) that returns a single float. my understanding is that it should instead return an array filled with the correct value. if that is the case, i can work on a fix. 


--- 

steps to reproduce:

```python
import mir_eval 
import numpy as np

intervals = np.array([[3,6]])
mir_eval.sonify.chords(["C"], intervals, 8000) 
```

> Traceback (most recent call last):
  File ".../mir_eval/tmp_sonify_test.py", line 7, in <module>
    mir_eval.sonify.chords(["C"], intervals, 8000)
  File ".../mir_eval/mir_eval/sonify.py", line 374, in chords
    return chroma(chromagram, intervals, fs, **kwargs)
  File ".../mir_eval/mir_eval/sonify.py", line 339, in chroma
    return time_frequency(gram, frequencies, times, fs, **kwargs)
  File ".../mir_eval/mir_eval/sonify.py", line 212, in time_frequency
    output[t_in:t_out] += wave[: len(signal)] * signal
TypeError: object of type 'numpy.float64' has no len()

---

@bmcfee ?


After looking at this a bit with @giovana-morais , we also discovered that the timing of sonified chords is not correct in the example given.

I suspect a regression due to #355, though at this point I fear that I don't understand the implementation anymore and it might just be simpler to rewrite the thing from scratch.

I think the first problem here comes from this line:

https://github.com/mir-evaluation/mir_eval/blob/a0a8672d11ab66c3a89900d0dd35d8cdfcf8a1f9/mir_eval/sonify.py#L116-L117

If the input time intervals is the array `[[3, 6]]`, then interval adjustment will pad this out with a new interval so it becomes  `[[0, 3], [3, 6]]`.  However, we're not propagating the `labels` (ie `gram` matrix) through here, so the resulting timing for the chroma vector generated for the chord `C` gets shifted off by one.

There may be other errors after this, but this is behind the timing error noted above.
Repository: mir-evaluation/mir_eval
Issue: [Feature Request] Handle Monotonicity in Hierarchical Segmentation Metrics
Root cause: Unimplemented Feature Gap

#### **Description**
The hierarchical segmentation metrics in `mir_eval` are defined with the assumption that input annotations has monotonic labels and boundaries. However, in practice, hierarchies often don't have montonic labels, or even boundaries (outputs of spectral clustering for example).
While the original papers only define the case for fully monotonic hierarchies, this issue proposes a additional option for another sensible way of handling cases where hierarchies are not monotonic.

To extend flexibility regarding monotonicity, I propose:
1. **Adding a monotonicity check**: Before computing metrics, we should validate whether the hierarchical annotation has monotonic labels/boundaries.
2. **Handling different monotonicity cases**: The `_meet` function in `hierarchy.py` currently records the deepest level where two frames has the same label (meet), however, they don't require all the previous levels to meet (garunteed by monotonic labels, but not otherwise). Giving `_meet` function a option to record the deepest **monotonic** level that two frames meet in labels will enforce monotonicity in evaluation that way. This would not change the metric for the monotonic case but provide a different definition for annotation meet matrix that takes monotonicity into account.
3. **Exposing an option to control this behavior**: This option of how to build the annotation meet matrix should be elevated to the metric functions so users can specify how to handle monotonicity constraints.

#### **Proposed Changes**
- Implement checks for monotonicity violation.
- Introduce a new argument (e.g., `monotonicity_mode`) in `_meet` to determine how to construct the annotation meet matrix.
- Update hierarchical segmentation metric functions to expose `monotonicity_mode` as a user-configurable option.
- Update documentation and add tests for different monotonicity scenarios.

#### **Next Steps**
I will follow up with a pull request implementing these changes. Feedback on the approach and naming conventions would be appreciated! @bmcfee 


The monotonicity business is subtle and deserves some detailed treatment. I'll leave mir_eval metrics the way they are.
Repository: mir-evaluation/mir_eval
Issue: Support Path objects in data loaders
Root cause: Validation Gap

In modernizing some tests, I'm finding myself wanting to use Path objects rather than strings to specify file locations.  This revealed that our file loaders (eg io.load_events and the like) only support strings as input.  It would be easy enough to generalize this to support Paths in the `_open` context manager, and I see no reason not to do it.

Implemented this in 416, though upon some reflection, I'm now thinking it might be best to just try throwing whatever the user gave as input to `open` and letting it fail (or not).

Reasoning here is that `open` technically supports "path-like objects" https://docs.python.org/3/glossary.html#term-path-like-object which include (but are not limited to), str, bytes, or Path.  (IE anything that implements the Pathlike protocol wlil work.)  Rather than trying to force a type here:
https://github.com/mir-evaluation/mir_eval/blob/caa6ff3844892f3f05fc5a63656559996a67949e/mir_eval/io.py#L26-L30

we can just try to open the thing and let any exceptions propagate back up to the user if it fails.  The else clause here is an IOError anyway, so there really will be no functional difference except the content of the exception string.