zhiminy/EvalEng huggingface dataset:
Repository: mir-evaluation/mir_eval
Issue: Deprecate io.load_wav
GitHub labels: question

I should have noticed this when we deprecated separation #382 , but as far as I can tell, the `io.load_wav` function no longer serves any purpose.  I think we should deprecate it in 0.8.1 (and should have deprecated it in 0.8.0), and remove it in 0.9.

Any objections?
Repository: mir-evaluation/mir_eval
Issue: efficiency of structure.pairwise
Root cause: Algorithmic Error

`structure.pairwise` currently uses `np.triu_indices` to find all unique pairs of frames and evaluate their agreement.  It turns out that this is woefully inefficient in memory, as `np.triu_indices` constructs an intermediate (int64) masking matrix.

This should be replaced with a more direct implementation with leaner memory requirements.
Repository: mir-evaluation/mir_eval
Issue: Cover / version ID task

I think there's a good case to be made that we should add a module for evaluating cover detection / version identification.

We have not done this in the past under the assumption that it's a fairly straightforward information retrieval problem, for which off-the-shelf tools (ie sklearn) provide the necessary functionality already.

While this is true in theory, I think in practice we often end up with custom implementations anyway, and there's actually more room for interpretation in the eval definition than we might initially have thought.  Things like how empty positive sets are handled, or even transitivity / symmetry assumptions in version clusters, are not generally well defined, and have outsized influence on the resulting evaluation.  This can lead to some discrepancies and/or bugs in evaluating version ID systems, the kind of which mir_eval was meant to resolve in general.  (See, eg, https://github.com/furkanyesiler/re-move/issues/4 )

This coupled with a general lack of a standard (and *well maintained*) tool for evaluating cover ID leaves things in a rather unsatisfying state.  (I know acoss exists, but it hasn't been touched in 5 years, and it does force the user into a particularly narrow way of working.)

All that said, this is a fairly different kind of task from what we typically cover in mir_eval, since it's inherently collection-based rather than example-based.  (This was partially, but not entirely, behind my posting over in #422 )  I don't think it's as radically different as something like bsseval though, and I think it can be done well in a clean and maintainable way.

What do folks think?  Am I missing some widely adopted tool that's already out there?  Or should we just do it?

I think if there's a way to do it in a mir_eval-y way and there's value over off-the-shelf generic solutions, I'm in support.

> in a mir_eval-y way

I think we can basically cook down all the relevant metrics to a ranking retrieval interface. E.g., the following example should basically cover it:

```python
results = mir_eval.cover.evaluate(ref_ids=[7, 105, 31], 
                                  est_ranking=[99, 105, 22, 7, ...]  # ordered (probably truncated) list of track ids
)
```

This would give a score per query, which would then need to be aggregated for compatibility with how these are usually reported.  (Hence #422 )



> there's value over off-the-shelf generic solutions

The main values I see here would be A) consistency, B) streamlined interface, and C) in-community maintenance.

SGTM
Repository: mir-evaluation/mir_eval
Issue: invalid escape sequence SyntaxWarnings in docstrings
Root cause: Validation Gap

As of Python 3.12, some of the docstrings contain escape sequences that are considered invalid and yield SyntaxWarnings:

- `2\pi` in [sonify.py](https://github.com/mir-evaluation/mir_eval/blob/8db0b3812e2032544c1fc00d02d4256cab043f3d/mir_eval/sonify.py)
- `\*` in [hierarchy.py](https://github.com/mir-evaluation/mir_eval/blob/8db0b3812e2032544c1fc00d02d4256cab043f3d/mir_eval/hierarchy.py) and [util.py](https://github.com/mir-evaluation/mir_eval/blob/8db0b3812e2032544c1fc00d02d4256cab043f3d/mir_eval/util.py)
- `\%` in [beat.py](https://github.com/mir-evaluation/mir_eval/blob/8db0b3812e2032544c1fc00d02d4256cab043f3d/mir_eval/beat.py)

Not sure if worth fixing yet, but apparently these will become SyntaxErrors in the future...

In sonify.py I would just say `2π-periodic` without any special maths markup. It seems clear enough.

Definitely worth fixing!

I'd second @tkoeppe 's suggestion of dropping the sphinx/reST cruft in favor of unicode.  That doesn't necessarily help with the `\*` or `\%`, but I think that's probably unnecessary in the first place.

@iansimon do you know if it's possible to add an action to the docs testing CI that would check for this sort of thing?

I'm not sure if these `\%` and `\*` were in support of some other doc processing system, but very naively thinking I'd just remove the backslashes and that would look like what I'd expect?
Repository: mir-evaluation/mir_eval
Issue: How to convert the chord symbol(such as Csus7/G) to the format that suitable for the mir_eval
Root cause: Unimplemented Feature Gap

I have some data whose  chord symbols(such as Fmaj7/C, E/G#, E(add2), Cm(add2)(add4)) are not suitable for the mir_eval, so happen error when use mir_eval works. Is there any library can convert the chord symbols to the format with colon like mir_eval?

Not that I know, of, sorry! @bmcfee might know. Closing this because it's not an issue with mir_eval.

There is no such library to my knowledge.  It's much more involved than just adding a colon; you'll need to convert roots to scale degrees (eg "E/G#" -> "E/3"), and convert added notes into Harte's notation (eg "Cm(add2)(add4)" -> "C:min(2,4)").

It sounds like the notation of your data is somewhat idiosyncratic.  Probably the best thing to do is collect a dictionary of all unique chord symbols and then write a regexp-based converter to cover all the cases identified.
Repository: mir-evaluation/mir_eval
Issue: aug7 chord cannot be encoded
Root cause: Validation Gap

The regular expression [in the code](https://github.com/mir-evaluation/mir_eval/blob/fa55a92be6a0e9b6499eb746e4f17517587d5084/mir_eval/chord.py#L343) includes `aug7` as a valid quality, but the quality map [here](https://github.com/mir-evaluation/mir_eval/blob/fa55a92be6a0e9b6499eb746e4f17517587d5084/mir_eval/chord.py#L245) does not include `aug7`, which will lead to an error when trying to encode aug7 chord.

Could the `aug7` chord be added into the quality map to make sure that all the valid chords matched successfully with the harte regular expression can be encoded by the encode function?

A temporary solution for me is to replace `aug7` with `aug(b7)` in my chord string.

Thanks for catching that!  A little surprising that this hasn't come up before, but I guess that just means that none of the data we commonly train or test on has an aug7 chord in it. 🙃 

But yes, I would agree that adding this to the quality map is the correct solution here.
Repository: mir-evaluation/mir_eval
Issue: chord.score should be a decorator
GitHub labels: enhancement

... which decorate the comparator functions so that each comparator is callable as a metric.


The following things still need to be changed:
- `validate` still operates on comparator functions, before they are decorated with `score`.  It should operate on the actual metrics because, for example, attributes of `intervals` need to be validated against `reference_labels` and `estimated_labels`.
- The comparators should have their docstrings updated to include the `intervals` parameter.
- Unit tests need to be updated because the call signature is different now.


Maybe we should remove the `compare_` prefix too...


`validate` operates on comparators after they have been decorated into metrics, so that `intervals` can be validated, but I'm not 100% sure what an appropriate `intervals` actually is so I have not implemented validation for that yet.  


Tests updated.  Should be good to go now.
Repository: mir-evaluation/mir_eval
Issue: numpy error when intervals is length 2
Root cause: Validation Gap

A bogus edge case but when calling `score`, if `len(reference_labels) == len(estimated_labels) == 1` and `len(intervals) == 2`, then

``` Python
durations = np.abs(np.diff(intervals, axis=-1)).squeeze()
```

returns an array with shape `()`, and the line

``` Python
durations = durations[valid_idx]
```

which indexes it with a boolean array of size 1 gives the following numpy error:

```
IndexError: 0-dimensional arrays can't be indexed
```


I also might not be using `intervals` with the correct format... An example is

``` Python
reference_labels = ['C']
estimated_labels = ['D']
intervals = [0.0, 1.0]
```


Intervals should always be `shape=(n, 2)`.


OK, is that also the intention in `chord`, @ejhumphrey?  Either way, you still get this error:

``` Python
reference_labels = ['C']
estimated_labels = ['D']
intervals = np.array([[0., 1.]])
```
Repository: mir-evaluation/mir_eval
Issue: Regression tests and (actual) unit tests
Root cause: Unimplemented Feature Gap

Each task submodule should have a testing script in `tests` which performs unit/sanity tests and regression tests against the current stable implementation of the code.  This is all primarily to prevent bugs from creeping in the code.  We're not trying to unit test against an existing implementation anymore.

**Regression tests**
- [x] beat  @craffel
- [x] onset @craffel
- [x] separate @dawenl 
- [x] segment @bmcfee
- [x] chord @ejhumphrey 
- [x] melody @justinsalamon 
- [x] pattern @urinieto 

**Unit tests**
- [x] beat  @craffel
- [x] onset @craffel
- [x] separate @dawenl 
- [x] segment @bmcfee
- [x] chord @ejhumphrey 
- [x] melody @justinsalamon 
- [x] pattern @urinieto 


Melody has unit tests but the to_cent_voicing unit test is just a placeholder right now. 


Punting `util` and `io` tests to next release; they are semi-implicitly tested currently but should be actually tested at some point.
Repository: mir-evaluation/mir_eval
Issue: rename the repository/package?
Root cause: Architectural Constraint

Hyphens are bad mojo for package names:

`import mir-evaluate`

will cause a syntax error.

Can we either:
- Move actual code into a sub-directory `mir_evaluate`, or
- Rename the repository to `mir_evaluate`?
- Or both?

Either way, the code should definitely be moved into a sub-directory to facilitate having an install script down the line.


Definitely, I hastily created the repo. It's not meant to be installed as is. Let's not rename the repo though-feel free to move into a folder. 

-Colin

On Aug 13, 2013, at 11:39 PM, Brian McFee notifications@github.com wrote:

> Hyphens are bad mojo for package names:
> 
> import mir-evaluate
> 
> will cause a syntax error.
> 
> Can we either:
> 
> Move actual code into a sub-directory mir_evaluate, or
> Rename the repository to mir_evaluate?
> Or both?
> Either way, the code should definitely be moved into a sub-directory to facilitate having an install script down the line.
> 
> —
> Reply to this email directly or view it on GitHub.


This was fixed in 68e86ac.
Repository: mir-evaluation/mir_eval
Issue: Watch out for arange
Root cause: Algorithmic Error

`np.arange` may not give consistent results across calls.  This may be an issue when resampling the timebases in `melody`.


Found where this is mentioned:
http://stackoverflow.com/a/10011517/2007700

The numpy documentation for arange used to say: 

```
When using a non-integer step, such as 0.1, the results will often not be consistent. It is better to use linspace for these cases.
```

This comment has since been changed in the official document to:

```
For floating point arguments, the length of the result is ceil((stop - start)/step). Because of floating point overflow, this rule may result in the last element of out being greater than stop.
```

I guess it's just a clearer version of the same problem. Compare:

```
In [20]: np.arange(0.1,0.4,0.1)
Out[20]: array([ 0.1,  0.2,  0.3,  0.4])
```

and

```
In [21]: np.arange(0.1,0.3,0.1)
Out[21]: array([ 0.1,  0.2])
```

As far as melody eval is concerned, since you'd only be dropping / adding 1 frame, it's not significant. But if you want consistent behavior always, you need to revert to the linspace implementation.
Repository: mir-evaluation/mir_eval
Issue: nans when the reference melody has no voiced frames
Root cause: Validation Gap

When the reference has no voiced frames (i.e. all 0 Hz), the following lines in melody.py return nans:

```
279: vx_recall = TP/float(TP + FN)
326: raw_pitch = (cent_diff[ref_voicing] <= 50).sum()/float(ref_voicing.sum())
370: raw_chroma = (cent_diff_chroma[ref_voicing] <= 50).sum()/float(ref_voicing.sum())
```

A simple fix would be to add epsilon to the denominator, but in theory there should never be a reference file with no voiced frames (melody extraction on a file without any melody...), meaning this case is more likely to indicate a faulty reference file. So I think it would probably make more sense to raise a validation exception - @craffel ?


I've added a warning when either ref_voicing or est_voicing has no voiced frames and added code to make the effected metrics return 0 in these cases.  Ask @bmcfee if you want an explanation as to why precision/false alarm/etc should be 0 when there are no "relevant documents".
Repository: mir-evaluation/mir_eval
Issue: RFC: Do negative event locations make sense?
Root cause: Validation Gap

 `util.validate_events` raises a `ValueError` when events are less than 0, but MIREX and the beat evaluation toolbox seem to allow them.  Some MIREX submissions in, e.g., beat and onset tasks do actually produce negative events (although the beat evaluation toolbox trims all beats before 5 seconds, which includes negative beat locations).  If you have a reference annotation
`[0.0, 1.0, 2.0]`
and an estimation
`[-.1, 1.1, 2.0]`
then pruning negative event locations will effectively result in a false negative (which will affect different metrics in different ways), while leaving them in tact will mean it's a pretty good estimation.

Any thoughts?


Obviously negative time values do have a meaning, so it seems obnoxious to
refuse to handle them at all. But they probably indicate something
unfortunate has happened - something like a printed warning message seems
appropriate.

  DAn.

On Tuesday, April 29, 2014, craffel notifications@github.com wrote:

> util.validate_events raises a ValueError when events are less than 0, but
> MIREX and the beat evaluation toolbox seem to allow them. Some MIREX
> submissions in, e.g., beat and onset tasks do actually produce negative
> events. If you have a reference annotation
> [0.0, 1.0, 2.0]
> and an estimation
> [-.1, 1.1, 2.0]
> then pruning negative event locations will effectively result in a false
> negative (which will affect different metrics in different ways), while
> leaving them in tact will mean it's a pretty good estimation.
> 
> Any thoughts?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/56
> .


If we don't throw a ValueError, then at least `beat.p_score` will need to be changed, due to the following logic when creating an impulse train which has impulses at beat locations:

``` Python
    annotations_train = np.zeros(end_point*sampling_rate + 1)
    beat_indices = np.ceil(reference_beats*sampling_rate).astype(np.int)
    annotations_train[beat_indices] = 1.0
```

The same is done for `estimated_beats`.  If any beats are negative, `beat_indices` may have negative indices, which will cause some point near the end of `annotations_train` to be 1 when it really shouldn't.  A simple fix would be to subtract `min(estimated_beats.min(), reference_beats.min())` from `estimated_beats` and `reference_beats`; this would move the first beat in either sequence to 0 and move the other sequence appropriately, which should not effect the metric.  As far as I can tell, this is the only metric for which this would matter.
Repository: mir-evaluation/mir_eval
Issue: beat.p_score is sensitive to offsets
Root cause: Algorithmic Error

Probably due to the fact that the sequences are turned into sampled impulse trains.  This is true in the beat evaluation toolbox too.  Seems like it shouldn't be.

``` Python
import mir_eval
import numpy as np
ref = np.random.random_sample(10)*.3 + np.arange(10)
est = np.random.random_sample(10)*.3 + np.arange(10)
for n in xrange(10):
    offset = np.random.random_sample()
    print mir_eval.beat.p_score(ref + offset, est + offset)
```

```
0.9
0.8
0.9
0.8
0.8
0.9
0.8
0.8
0.9
0.8
```


If the beat sequences are shifted so that the first beat in either sequence is at time 0, then this problem goes away, but the metrics will not always be the same as when this preprocessing step is not carried out (e.g., it will deviate from the beat evaluation toolbox).

``` Python
import mir_eval
import numpy as np
ref = np.random.random_sample(10)*.3 + np.arange(10)
est = np.random.random_sample(10)*.3 + np.arange(10)
for n in xrange(1000):
    offset = np.random.random_sample()
    assert np.equal(mir_eval.beat.p_score(ref, est),
                    mir_eval.beat.p_score(ref + offset, est + offset))
```

Passes.
Repository: mir-evaluation/mir_eval
Issue: beat.continuity can raise an IndexError when beat sequences are length 2
Root cause: Validation Gap

``` Python
mir_eval.beat.continuity(np.array([1, 2]), np.array([2, 3]))
```

raises an `IndexError` on line 403 because when `m = 0` (first estimated beat) then `nearest = 1` and `nearest + 1 = 2 > reference_beats.shape[0] - 1`


Same thing happens in beat evaluation toolbox.

``` Matlab
>> be_continuityBased([5, 6], [6, 7])
Index exceeds matrix dimensions.

Error in be_continuityBased>ContinuityEval (line 112)
            if (and((abs(a/(anns(b+1)-anns(b)))<p), ...

Error in be_continuityBased (line 79)
  [totAcc(j),contAcc(j)] = ContinuityEval(testAnns{j},beats,t,p);
```
Repository: mir-evaluation/mir_eval
Issue: Refactor time series sampling in melody
Root cause: Unimplemented Feature Gap

Per discussion elsewhere, `mir_eval.melody.resample_melody_series` should take in a new_times argument instead of a hop arg.  The input frequencies and voicing will be resampled to this new time grid.  There should also be a function which generates a constant-hop time grid. 

Additionally, `resample_melody_series` should take in an interpolation type arg which is passed to `scipy.interpolate.interp1d` (e.g. 'zero', 'nearest', 'linear', 'quadratic', etc.)  When it's not 'zero' or 'nearest', the zero transitions should be corrected as they currently are.  The voicing series should always be resampled using 'zero' unless the interpolation for frequencies is 'nearest', in which case it should be 'nearest'.

Finally, `to_cent_voicing` should be updated so that by default the estimated time series is resampled to the same time base as the reference time base, unless a hop parameter is supplied, in which case both will be resampled to a constant-hop time grid.  It should also take in the interpolation type arg to pass to `resample_melody_series`.


OK, code has been updated.  To replicate the behavior before I made these changes, add the hop parameter, set to 0.01, to your `to_cent_voicing` call like so:

``` Python
mir_eval.melody.to_cent_voicing(ref_time, ref_freq, est_time, est_freq, hop=0.01)
```
Repository: mir-evaluation/mir_eval
Issue: empty beat predictions throw exceptions
Root cause: Validation Gap

The beat eval helper method `_clean_beats` throws a `ValueError` exception when there are no detected beats outside the boundaries.  This is probably not the correct behavior, as many of the metrics are well-defined for empty predictions.


Which metrics would you consider well-defined for empty detected (or annotated) beat lists?


Shouldn't they all get a score of 0 for empty detections? Certainly f-measure and p-score do.. I'd have to look into the others to know for sure.

Empty annotations are a separate issue, and I'm ok with not handling those explicitly.


F-measure, P score and maybe Goto are I think the only ones that make sense to have 0 accuracy.  I think the rest rely on the error between an annotated beat and the closest generated beat, which is not well defined if there are no generated beats.


What if you take a limiting argument as the closest generated beat gets infinitely far away?  That would push cemgil to 0.

CMLc should be 0, since the largest segment of contiguous beats is empty.

CMLt should be 0 because all segments of contiguous beats are empty.

AML{c,t} should be 0 accordingly.

IGain should also be 0, as a constant function has 0 mutual information with the annotation.


I would argue in all of these cases you're adding a special case to the definition of the metrics which is not in the original papers, but it's a pointless argument to make.  If you'd rather the metrics return 0 feel free to change it to do to so.


Not adding a special case, just inferring the consistent behavior from the original definitions. (Ok, the definitions as stated in Davies et al. '09, but you get my point.)  F-measure is probably the only tricky one because it requires the `0/0=0` convention.


"Special case" to me means having to write
if generated_beats.size == 0:
  return 0
which you would have to do in all of these cases I think (if you can do without it, I'm impressed).  "Inferring consistent behavior" is adding a special case to the original definition.
Repository: mir-evaluation/mir_eval
Issue: Move chord.evaluate_file_pair to the evaluator
Root cause: Architectural Constraint

We can have a data preprocessing function (filtering and merging intervals) in .chord, like melody.to_cent_voicing:
https://github.com/craffel/mir_eval/blob/master/mir_eval/melody.py#L218
but we shouldn't have a function in the task submodule which loads in data, and also we shouldn't have a function in the task submodule which runs all of the metrics - those are things which should be moved to the evaluator


Related: It looks like chord_eval prints per-file statistics, and doesn't aggregate over files.  I think this is functionality that @ejhumphrey mentioned adding.  Once this has been added, it can match the structure of the other evaluators -- with a `evaluate` function which takes in a file (or in this case a list of files), computes the scores for them, and returns a metric dictionary (here the keys would be the vocabs -- probably best to default to all vocabs) which summarizes the scores for all of the included files.


Merging with other issues into #81
Repository: mir-evaluation/mir_eval
Issue: evaluator binaries
Root cause: Unimplemented Feature Gap

Create command-line binaries for the evaluators.


With the current development version of `pyinstaller` (cloned from their github), I can create binaries for all of the evaluators on my Mac using 

```
pyinstaller --onefile beat_eval.py
```

They're huge though (34mb each).  `sklearn` accounts for about 28mb of that.  `scipy` and its dependencies probably accounts for the rest.


As of sklearn 0.15 and above, pyinstaller doesn't work anymore, as reported by me here:
https://groups.google.com/forum/#!topic/pyinstaller/mjAfmuGUCSQ


#20 is fixed, so this resolves the problem with sklearn and 

```
pyinstaller --onefile beat_eval.py
```

works fine again.


If you run `pyinstaller` from within a virtualenv which only has `numpy`, `scipy`, `pyinstaller`, and `mir_eval` installed, it cuts the size down by about 20mb, which I think is the baseline for something which uses numpy and the parts of scipy we're using.


Once 0.1 is otherwise finished, evaluator binaries should be built for Mac and Windows and .tar.gz'd into a big fat package and put up at http://librosa.ee.columbia.edu/mir_eval/mir_eval_mac.tar.gz and http://librosa.ee.columbia.edu/mir_eval/mir_eval_win.tar.gz 


The binaries are causing segmentation faults and other crashes on some systems.  Punting this to after 0.1.


Now that `mir_eval`-as-a-service works, there isn't nearly enough motivation to have binaries.  Both allow no-dependency command-line calls to be made.  The advantage of the binaries are that they don't require internet access, and would probably be much faster.  But, I think it's fair to ask people to set up a Python environment if they need those things.