zhiminy/EvalEng huggingface dataset:
Repository: mir-evaluation/mir_eval
Issue: Always getting negative SDR values when using BSS Eval matlab toolbox
Root cause: Validation Gap

@faroit 
Hello! 
When I use "bss_eval_sources.m" function to evaluate performance of recovering clean speeches from mixed speeches, I always get negative SDR values. It's so strange because the results are quite good by just listening to them.

My code is as follows:
nsamp = fs *  duration; fs = 22050; duration = 8;
gt_1 = audioread(['C:\Users\samsung\Desktop\gt_100_L.mp4'], [1, nsamp]); 
gt_1 = gt_1';
gt_2 = audioread(['C:\Users\samsung\Desktop\gt_100_R.mp4'], [1, nsamp]); 
gt_2 = gt_2';
sep_1 = audioread(['C:\Users\samsung\Desktop\sep_100_L.mp4'], [1, nsamp]);
sep_1 = sep_1';
sep_2 = audioread(['C:\Users\samsung\Desktop\sep_100_R.mp4'], [1, nsamp]);
sep_2 = sep_2';
se = [sep_1; sep_2]; 
s = [gt_1; gt_2];
[SDR, SIR, SAR, perm] = bss_eval_sources(se, s);

sep_1 and sep_2 refer to two separated speeches. gt_1 and gt_2 refer to two clean speeches.
I get results like this: SDR=[-15.1; -18.3], SAR=[-16.1, -19.6], SIR=[6.4, 4.6]
I have no idea why I get negative and low SDR and SAR values. 

What a coincidence. I'm currently having the same problem but with the python implementation mir_eval. I'm pretty sure its not the fault from the mir_eval toolkit. Somehow we probably use the toolkit in a wrong way.



> What a coincidence. I'm currently having the same problem but with the python implementation mir_eval. I'm pretty sure its not the fault from the mir_eval toolkit. Somehow we probably use the toolkit in a wrong way.

After I changed the audio format to wav, the problem still exists. I think maybe it's necessary to do some audio pre-processing before evaluating SDR, but I don't know how to do. 

In my case the audio signal was the reason. So the evaluation works.

I finally figured out the problem. In my case, the separated speech signals are advanced of clean signals in phase, so the SDR values are always negative. After fixing this, the eval code works.

> I finally figured out the problem. In my case, the separated speech signals are advanced of clean signals in phase, so the SDR values are always negative. After fixing this, the eval code works.

@YiyuLuo 
I have the same problem. May I ask you how did you fixing this? Thank you.

@xin-h963 this seems like a "simple" programming error. In the case of @YiyuLuo the reason probably was that some offsets were wrong. This means that for time `t` the wrong phase information from `t=t-2` was taken.

This is probably not audible with speech. You can try to run your processing on music data and see whether the audio sounds good.

I'm facing the same issue but the signals seem very similar. Could anyone advise me how to fix it? full details can be found [here](https://stackoverflow.com/q/71861816/12828249)
Repository: mir-evaluation/mir_eval
Issue: Generalized melody metrics
Root cause: Unimplemented Feature Gap

Myself and @juanjobosch have a recently accepted ISMIR paper proposing an extension of the existing melody metrics (allows for continuous, rather than binary, voicing estimations). The new metrics are equivalent to the existing ones when binary voicings are given.

Can we add these new metrics to mir_eval? They remove the need for reporting "unvoiced" frames as negative frequency values, which would clean up the underlying code a lot. But, it would change the existing API - right now the input to `mir_eval.melody` is `ref_times, ref_freqs, est_times, est_freqs`. To use the new metrics, it would need `ref_times, ref_freqs, ref_voicing, est_times, est_freqs, est_voicing`.

Alternatively, we could keep `mir_eval.melody` as-is and instead create `mir_eval.f0` with the new metrics. The metrics are already used for more tasks than melody (e.g. pitch tracking, bass tracking), so `f0` could be a nice generalization.

@craffel , what do you think?

Quick comment - @juanjobosch also proposed some melody metrics that factor into account pitch continuity back in 2016. Would it be worth considering adding those in as well?

In both cases this diverges from the "standard" metrics used in MIREX, but I think these metrics are not sufficient, so if we think this can encourage better eval in this area I'd be all for adding both the soft voicing metrics and pitch continuity metrics.

I might however, be worth supporting backwards compatibility (neg freqs for voicing) for the sake of keeping the API compatible with MIREX and older algs?

Some additional (ugly) options:

1. Make the args  `ref_times, ref_freqs, est_times, est_freqs, ref_voicing=None, est_voicing=None` so that these new inputs are optional. 
1. Make `ref_freq` optionally be 2D. When 1D, assume the format where a negative value indicates unvoiced. When 2D, Row 1 is (strictly positive) frequency estimate and row 2 is the voicing.

I generally prefer not to change the API of existing functions, but if you expect your new metrics to be a very common use-case then I could be convinced. Note that you are not actually changing the API for any of the metric functions, only for `to_cent_voicing` which is a preprocessing function.

If your new metrics are significantly different from the existing ones (so that minimal code could be shared), `mir_eval.f0` is an OK idea. Otherwise, I am not inclined to add an additional module.

To @justinsalamon's point, I agree that either way we must maintain backward compatibility.

Between options 1 and 2 perhaps 1 is better?

closed by #324 ?
Repository: mir-evaluation/mir_eval
Issue: IO functions do not skip comment lines
Root cause: Validation Gap

Title says it all -- io functions are unable to parse files with commented lines (by convention, those starting with # in .lab format).

Fixing this violates my general rule against writing our own *sv parser, but i think there's enough data out there to warrant an exception in this case.  @craffel if this sounds good to you, I'll prep a PR.

Sounds totally reasonable to me!
Repository: mir-evaluation/mir_eval
Issue: Add pattern discovery metrics

Code is available here:
http://www.music-ir.org/mirex/wiki/2013:Discovery_of_Repeated_Themes_%26_Sections#Data


Should I include a new function in the input_output module to read the MIREX file format for this task?

An example of this format can be seen [here](http://www.music-ir.org/mirex/wiki/2013:Discovery_of_Repeated_Themes_%26_Sections#Example_Algorithm_Output_for_a_Ground-Truth_Piece).


Yes. 


Ok, this is done.
Repository: mir-evaluation/mir_eval
Issue: structural segmentation

Do i get the structural segmentation of music like intro, chorus, verse, bridge and outro?

Thanks!

This package evaluates structural segmentation algorithms, it does not compute a structural segmentation of a given song. For that you might want to try MSAF
https://github.com/urinieto/msaf

MSAF provides structural segmentation, but the segment labels that it yields are not _functional_ (i.e., they're not semantically meaningful like chorus, verse, etc)

okay no problem!  but which section of code i can try in MSAF to get the structual segmenation. now labelling is not important for me.
Thanks

This notebook should be pretty self explanatory: https://github.com/urinieto/msaf/blob/master/examples/Run%20MSAF.ipynb
Repository: mir-evaluation/mir_eval
Issue: `separation.bss_eval_sources()` and `separation.bss_eval_sources_framewise()`

Although the docs describe the differences between these two functions, it's still unclear, even the returned results from these two functions have different shapes. Could someone elaborate the differences and the usage of `separation.bss_eval_sources()` and `separation.bss_eval_sources_framewise()`?

@faroit Any thoughts?

`bss_eval_sources_framewise` computes several `bss_eval_sources` as set by the `window` and `hop` parameter. if your signals are short: use non-framewise, if your signals are long >10s, use the framewise metric.

See the main loop in the code to understand whats going on:
https://github.com/craffel/mir_eval/blob/19ed4dfbc193f9de48e56fc34198556fd3de3766/mir_eval/separation.py#L591-L601
Repository: mir-evaluation/mir_eval
Issue: Docs website missing styling (css/js)

Looks like almost all of the js and css references are 404'ing on the published version of the website (https://craffel.github.io/mir_eval/). Tested on Firefox, Safari, and Chrome on OS X.

Thanks, this was broken in the recent docs update and just fixed it in https://github.com/craffel/mir_eval/commit/01920c9105a46d2121f4918b719fcdd239ea4d79

Yup, looks good! Thanks!
Repository: mir-evaluation/mir_eval
Issue: [QUESTION] About BASS and PEAQ, PESQ

I run several MSS algorithms, [open-unmix](https://github.com/sigsep/open-unmix-pytorch),  [spleeter](https://github.com/deezer/spleeter), etc. for which we have evaluation metrics on the test sets with `mir_eval` scripts. Clearly this evaluation is based on a comparison of the extracted sources from reference sources and "_...attempt to measure the perceptual quality of the separation_" as stated in https://github.com/craffel/mir_eval/blob/master/mir_eval/separation.py
Assumed that in general perceptual evaluation of audio quality (`PEAQ`, `PESQ`, `PEQV`) in somewhat a standard, that is based on Psychoacoustics/perceptual features for for _objectively measuring perceived audio quality_ [(1)](https://github.com/stephencwelch/Perceptual-Coding-In-Python/blob/master/PEAQ.md), while for the [BASS](https://hal.inria.fr/inria-00544230/document) problem we consider the distortions between the estimated source and the reference source to compute the `SDR`, `SAR`, `SIR`, `SNR` metrics, can we define a relation among these different evaluations?
Specifically, if we would consider [PESQ](https://en.wikipedia.org/wiki/Perceptual_Evaluation_of_Speech_Quality) (Perceptual Evaluation of Speech Quality) for separated vocals, and [PEAQ](https://en.wikipedia.org/wiki/Perceptual_Evaluation_of_Audio_Quality) for mixed and accompaniment separated tracks, which considerations can be done of these two kind of evaluations?

Thank you.


we typically didn't use perceptual metrics such as PEAQ because the distortions from source separation methods are still huge compared to what we have in audio coding (for what PESQ and PEAQ was designed for). Maybe we can already run these these metrics on current SOTA methods but in any cases this would have to be validated against human listening tests which is lot of work and requires professional equipment. Furthermore the questions to be asked to distinguish between quality, interference and artifacts are important ones. Have a look at [this more recent paper](https://ieeexplore.ieee.org/abstract/document/8462325/).

Then I think this conversation is out-of-scope for mir-eval, except for you would propose to implement peaq here (which is lot of work) ;-)

@faroit thanks a lot Fabian, yes infact I was thinking to do a specific research on that with a researcher (and musician) in the psychoacoustics field supported by the right equipment :) And maybe this could lead to an implementation in `mir-eval` :) So I will start from your references and now looking to PEASS.
Thank you very much for the indications, closing for now.
Repository: mir-evaluation/mir_eval
Issue: sonify.pitch_contour should support nans

Minor inconvenience, but for f0 estimation, it's nice to be able to use `nan` to encode missing/unvoiced regions (rather than 0, or egad, negative frequency).  Doing so breaks the sonifier, but it would be an easy enough fix to support it.

Is there a convention to use `nan` to mean "missing/unvoiced"?

Not specifically, but it's common to use `nan` for missing data in general, and I think this qualifies.  (FWIW, librosa.pyin uses nan for unvoiced frames by default.)

"PRs welcome", as they say
Repository: mir-evaluation/mir_eval
Issue: mir_eval.key.validate(reference_key, estimated_key)¶ when the key or the mode are unknown
Root cause: Validation Gap

Hello,

First of all, thank you very much for the library. It is fantastic!

At mir_eval.key:

When you don't know the key or the mode (in the reference or in the estimated) you should be able to compute the metric.

see the examples :
```
x minor
C
```
Well, x minor can be computed but I think that if your algorithm is performing the mode, the score should be more than 0.

Thank you so much!

On the other hand, why do you only allow to evaluate (key, modes) that have major or minor modes? I believe that there are key datasets with much more extended vocabularies.

Thank you so much!!

The key module was created to match the conventions used by MIREX. If there are other benchmarks that use a richer set of annotations, we welcome PRs to implement that kind of evaluation.

Hey @craffel!
![angel faraldo](https://user-images.githubusercontent.com/16807496/96749353-fe8e6b00-13ca-11eb-8edb-414732b434a4.png)
As is seen in this Angel Faraldo phd Thesis figure there are some datasets with extended vocabularies. (I think that all the vocabularies can be reduced to diatonic modes).

![image](https://user-images.githubusercontent.com/16807496/96749912-b4f25000-13cb-11eb-858b-f3dbbc4ee658.png)

There are more datasets with diatonic modes annotations. As:
https://zenodo.org/record/1290737#.X5BjPC3FRhE
https://zenodo.org/record/1095691#.X5BjUi3FRhE
https://zenodo.org/record/1101082#.X5BjaC3FRhE
Repository: mir-evaluation/mir_eval
Issue: More information on multipitch evaluation?

I'm trying to understand the `multipitch`, the documentation redirects to two papers but I couldn't find anything that explains the metrics:

````
OrderedDict([('Precision', 0.5),
             ('Recall', 0.5),
             ('Accuracy', 0.3333333333333333),
             ('Substitution Error', 0.5),
             ('Miss Error', 0.0),
             ('False Alarm Error', 0.0),
             ('Total Error', 0.5),
             ('Chroma Precision', 0.5),
             ('Chroma Recall', 0.5),
             ('Chroma Accuracy', 0.3333333333333333),
             ('Chroma Substitution Error', 0.5),
             ('Chroma Miss Error', 0.0),
             ('Chroma False Alarm Error', 0.0),
             ('Chroma Total Error', 0.5)])
````

In [the first paper from 2007](https://www.ee.columbia.edu/~graham/papers/piano.pdf) there are two sections in the Transcription Results section: Frame-level transcription (5.1) and Note onset detection (5.2). Due to the format of the input for `multipitch.evaluate` (frequencies associated with an onset) I suppose the 5.2 was mentioned. There's literally nothing in it that explains the metrics. 

What am I missing? It seems unnecessary obscure to me.

@rabitt 

Hey @maxpv 

>In the first paper from 2007 there are two sections in the Transcription Results section: Frame-level transcription (5.1) and Note onset detection (5.2).

In the [documentation](https://craffel.github.io/mir_eval/#id11)
![image](https://user-images.githubusercontent.com/8169843/107972232-d5b26700-6fb3-11eb-9138-9dbcb4417ad9.png)

The paper you mentioned, and a second are cited. The equations for all the metrics are there - Equations 3-6 in the first paper, and equations 1-8 in the second. Both papers give pretty lengthly explanations of the metrics if you want more details.

>Due to the format of the input for multipitch.evaluate (frequencies associated with an onset) I suppose the 5.2 was mentioned.

There's no notion of onsets in [`mir_eval.multipitch.evaluate`](https://craffel.github.io/mir_eval/#mir_eval.multipitch.evaluate). Note-level metrics are implemented separately in [`mir_eval.transcription`](https://craffel.github.io/mir_eval/#module-mir_eval.transcription).

Hope that helps clarify things. 

Thanks for your detailed answer.

I think this should be added to the documentation, it is not obvious that the `5.1 Frame-Level Transcription` is related to the metrics we try to compute with `multipitch` or `transcription`. Partly because of the input format  -frequencies and timestamps, as opposed to an NxT matrix.

Example from the 5.1 section: `TP (“true positives”) is the number of correctly transcribed voiced frames (over all notes) `.
Ok, but what is a frame when the input is a list of intervals and pitches? To do that we need to set the `offset_min_tolerance` but this parameter isn't exposed in the doc for the `mir_eval.transcription.evaluate`.
Repository: mir-evaluation/mir_eval
Issue: util.intervals_to_samples() -- return time array?
GitHub labels: enhancement

I'm thinking it might it be more consistent to return an array of time points with this function (especially in light of rounding)? Granted it can be easily computed / created, but it (1) prevents users from screwing it up and (2) enables a pass-through loop that would make converting intervals -> samples -> intervals -> samples trivial, both for practice and testing purposes.


Sure, sounds good, change at will.  Any calls to intervals_to_samples will also need to be changed in order to deal with additional output arguments.  Also cc @bmcfee on any changes in case he uses it outside of mir_eval.


sounds great. I actually just finished all of this, and updated all the calls I could find. 

@bmcfee, could you give this a gander and see how I did? I opted for slicing the return arguments (via [-1], like https://github.com/craffel/mir_eval/blob/master/mir_eval/segment.py#L254) rather than return data that goes unused (better lint score). 


I read through the changes, and this all looks fine to me.

The name `interpolate_intervals` seems inaccurate though.  The intervals have already been interpolated (sampled, actually) at pre-specified points.  All this function does is propagate the interval-level labels to the sample times.  Maybe a better name for this would be `sample_interval_labels`?


yep, which is kind of an interpolation? (something like numpy's interp
function (docstringhttp://docs.scipy.org/doc/numpy/reference/generated/numpy.interp.html),
except the function is piecewise-constant and 'y' is nominal, rather than
cartesian / whathaveyou) I don't feel strongly though, and
sample_interval_labels works for me.

On Tue, Apr 1, 2014 at 2:05 PM, Brian McFee notifications@github.comwrote:

> I read through the changes, and this all looks fine to me.
> 
> The name '''interpolate_intervals''' seems inaccurate though. The
> intervals have already been interpolated (sampled, actually) at
> pre-specified points. All this function does is propagate the
> interval-level labels to the sample times. Maybe a better name for this
> would be '''sample_interval_labels'''?
> 
> ## 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/34#issuecomment-39237980
> .
Repository: mir-evaluation/mir_eval
Issue: Hierarchy measures: speed vs memory?
Root cause: Algorithmic Error

Most of the hierarchy measures are defined in terms of a *meet matrix*.  This is a square matrix where each row and column corresponds to a frame (default, sampled at 10Hz per the segment metrics), and the value is a small integer corresponding to the maximum depth within the hierarchy at which frames i,j receive the same label.  We do this twice, once for the reference and once for the estimate.

For a typical ~4min track, this object is of size 2400x2400, so not huge, but not tiny either.  In most cases, most of the matrix is empty, so we store it as a CSR (compressed sparse rows) matrix.  However, we don't know the sparsity pattern in advance, so it is first built as a LIL matrix and then converted to CSR, as seen here (`mir_eval.hierarchy._meet`):
https://github.com/craffel/mir_eval/blob/576aad4e0b5931e7c697c078a1153c99b885c64f/mir_eval/hierarchy.py#L215-L236

Rather than iterating over rows or columns, we use slice notation to assign blocks at a time.  It turns out that this is horrendously slow in LIL matrices, and comprises the major bottleneck in the evaluation.  The subsequent processing mainly involves sorting and inversion counting, and I believe is about as fast as can be (cf #201 and #225).

If we're willing to sacrifice memory for speed, it's simple enough to replace the initial LIL matrix by an ndarray.  (I've done this in a local branch, and it is indeed significantly faster and sufficient for my current needs.)  This hardly seems like an ideal solution though, as it completely defeats the purpose of constraining memory by going through a sparse representation.  I expect that with a bit of cleverness, we could probably bypass this initial lil/dense issue and go directly to CSR format.

Thanks Brian. Maybe as a preliminary step we could add an arg that specifies whether to use a LIL matrix or an ndarray?

> we could add an arg that specifies whether to use a LIL matrix or an ndarray?

I thought about that.  It's doable, but kind of ugly from an API perspective since it would have to thread through the entire module.
Repository: mir-evaluation/mir_eval
Issue: Docs don't list the new alignment metrics
Root cause: Documentation Deficiency

Hey Colin!

Hope you're doing well. I noticed that the alignment metrics that I contributed (#343) are not yet showing up in the mir_eval documentation - do you need to manually regenerate the docs so they become visible? Since it's a new module people would otherwise not notice it at all, I suppose.

https://craffel.github.io/mir_eval/#

Hm, looks like I attempted to update the docs to 0.7 (https://github.com/craffel/mir_eval/commit/6906a2f9717e6cdff972b8924dca4977f6f7c547) but failed. Let me try again.

The docs updated, but your alignment metrics still aren't there. I think you need to add it to https://github.com/craffel/mir_eval/blob/master/docs/index.rst

Got it, please have a look at PR #345 and see if that does it

Looks good. The tricky thing is that I would normally wait until the next release before re-generating the docs, but I assume you want them online before then?

I remember you mentioned that there was quite a lot of time between the current version release and the last one, so it would be good to not have to wait long (like more than a 1-2 weeks). So if this generation doesn't have any annoying side effects when doing it independent of a release, it would be good to regenerate soon. But if you're very busy right now, we can also delay for a while of course 👍 

K, updated them manually: https://craffel.github.io/mir_eval/#mir-eval-alignment !