zhiminy/EvalEng huggingface dataset:
Repository: mir-evaluation/mir_eval
Issue: Difference between transcription.precision_recall_f1_overlap and multipitch.metrics?

Considering that transcription.py's symbolic representation is piano rolls with non-quantized pitches, what's the difference between transcription.py and multipitch.py really? I'm extra confused by how transcription.py [references the multi-f0 mirex estimation task](https://github.com/craffel/mir_eval/blob/master/mir_eval/transcription.py#L20). Could anyone help clarify why both modules exist? When should which be used? 


cc @rabitt @justinsalamon 

Well, the difference starts with the fact that their annotation format is quite different.  multipitch annotations specify, on a fixed timing grid, "what fundamental frequencies are present at this point in time?"  transcription is more like MIDI, specifying individual notes, with start times, end times, and pitches.  This means that evaluating them ends up looking a good bit different.  I think the confusion about transcription linking to MIREX multi-f0 is that MIREX lumps these two arguably different tasks into one "heading". (@justinsalamon please correct me if I'm wrong and there is in fact an error in the docs).  See [the annotation format](http://www.music-ir.org/mirex/wiki/2014:Multiple_Fundamental_Frequency_Estimation_%26_Tracking#I.2FO_format) section of the MIREX task description page - there are clearly two different formats/tasks here.

Maybe we can make a PR to clarify this in the documentation?


>  (@justinsalamon please correct me if I'm wrong and there is in fact an error in the docs).

Nope, that's correct.


Aha, so multipitch.py is for frame-based pitch estimation while transcription.py is for event-based pitch estimation? Thanks, @craffel!

It could be clarified in the docs, IMHO. Especially pros/cons of the respective formats, with some example use cases. Does it make sense to use multipitch.py for a solo piano? Does it make sense to use transcription.py for a string quartet? I'm confused.


> It could be clarified in the docs, IMHO.

I agree. If you or @justinsalamon or @rabitt want to make a PR, it would be welcome.

> Especially pros/cons of the respective formats, with some example use cases.

This is outside the scope of `mir_eval`.


Another difference - multipitch is intended for time varying pitch
curves/sequences
whereas transcription is events with a single pitch throughout the note.

On Wednesday, September 21, 2016, Colin Raffel notifications@github.com
wrote:

> It could be clarified in the docs, IMHO.
> 
> I agree. If you or @justinsalamon https://github.com/justinsalamon or
> @rabitt https://github.com/rabitt want to make a PR, it would be
> welcome.
> 
> Especially pros/cons of the respective formats, with some example use
> cases.
> 
> This is outside the scope of mir_eval.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> https://github.com/craffel/mir_eval/issues/223#issuecomment-248763237,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/AHypc1Oe0QOIF_REPkV_LWU-IGuNvY2xks5qsbIAgaJpZM4KCzDL
> .
Repository: mir-evaluation/mir_eval
Issue: Question interval util function

Hi, is there a function opposite to what `mir_eval.util.intervals_to_samples` does?
i.o.w.  I give it a list of equally-spaced sample points and it creates intervals
for input 
ts   label
0.1 A
0.2 A
0.3 A
0.4 B
0.5 B
0.6 A
0.7 A
it would return 
0.1 0.4 A
0.4 0.6 B
0.6 0.7 A
Thanks!


No, sorry, there isn't.
Repository: mir-evaluation/mir_eval
Issue: return indices of adjusted intervals 

I request an additional output of `mir_eval.util.adjust_intervals():`
`idx_start:` int
 the idx in the parameter  **intervals** of the returned **new_intervals**[0]
`idx_end:` int
 the idx in the parameter  **intervals** of the returned **new_intervals**[-1]

This could be useful when marking intervals of interest in a large file for later usage.

Seems reasonable, but changing the output pattern of the function will break backwards compatibility, so can't be taken lightly.  In particular, if there is scarce interest in this, I would suggest instead having users implement a custom function based on `adjust_intervals` as needed.  @bmcfee any thoughts?

> This could be useful when marking intervals of interest in a large file for later usage.

Can you give me a more specific example?


I don't think I understand the suggested modification.  Can you provide an example input and output of the proposed behavior?


@georgid 
> Can you provide an example input and output of the proposed behavior?

If not or all is ok, we will close the issue.

very sorry for the slow responses. 
Better idea for the return format  is to return one additional entry in the tuple:

```
indices_intervals: nd.array shape(new_intervals.shape[0],1)
  the indices of the intervals as ordered in the input intervals, which are returned 
```
Not to break backwards compatibility, one can do boolean parameter, return_indices and set it to false by default, no? 

Example:
```
intervals = np.array([[0,10], [10,20], [20,30], [30,40]])
(new_intervals, labels, indices_intervals) = mir_eval.util.adjust_intervals(intervals, labels=None, t_min=5,t_max=15, return_indices=True)
indices_intervals = (array([[ 5, 10], [10, 15]]), None, [0,1])

```

I have a file with annotations of lyrics lines. I need to work with particular segment of the song, for which I have the t_min and t_max. I would like to pick up only the relevant lines for that segment. The only way to take their content (lyrics) is by their ordered indices in the original file. 

Or maybe there is a better function, which could do that?
Repository: mir-evaluation/mir_eval
Issue: Fix on naming conventions
GitHub labels: requirement

Yeah


- Annotation to compare to: `reference`
- Annotation to compare: `estimated`
- Something that spans an entire object (track/song): `observation`
- Something that happens at a specific point in time (beat, onset): `event`
- Something that covers a span of time (chords, segments): `range`
- A dense grid of time stamps with corresponding values: `time series`


i love this, and will be using consistent terminology in the JAMS paper.

On Sat, Mar 29, 2014 at 3:04 PM, craffel notifications@github.com wrote:

> - Annotation to compare to: reference
> - Annotation to compare: estimation
> - Something that spans an entire object (track/song): observation
> - Something that happens at a specific point in time (beat, onset):
>   event
> - Something that covers a span of time (chords, segments): range
> - A dense grid of time stamps with corresponding values: melody
> 
> ## 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/23#issuecomment-39005370
> .


alright!


- [x] beat  @craffel
- [x] onset @craffel
- [x] separate @dawenl 
- [x] segment @bmcfee
- [x] chord @ejhumphrey 
- [x] melody @justinsalamon 
- [x] pattern @urinieto 
- [x] util @craffel
- [x] io @craffel


`util` calls things intervals, boundaries, samples; shouldn't these be range and event, range and time series respectively?  I think this was brought up somewhere.  Let's try to come to a consensus and make the changes as needed.


`beat_eval` and `segment_eval` need to be updated.


`util.load_annotation` needs to be changed @bmcfee 


> util.load_annotation needs to be changed @bmcfee

Done. 4232370286e038497983e8507a6ecb3a2663c02c


Intervals seem to be called intervals instead of ranges, but I think the semantics are OK here.  Closing this now.


> Intervals seem to be called intervals instead of ranges, but I think the semantics are OK here. Closing this now.

The problem with 'range' was that it's a reserved word in python; interval is much better.
Repository: mir-evaluation/mir_eval
Issue: Web service not available

Hi,

since the webservice is not available we should either delete it from the README.md or setup some other resources for that.

Opinions?

> we should either delete it from the README.md or setup some other resources for that.

Or we could, you know, fix the webservice!  This was caused by the labrosa webserver crashing and losing the appropriate apache configuration to successfully deploy the proxy to http://mireval.herokuapp.com/ (where the service actually lives).  Now fixed.
Repository: mir-evaluation/mir_eval
Issue: matplotlib 2.0 may break regression tests on display module
Root cause: External Dependency Breakage

Just putting this issue here, since the new release might change some of the default settings in ways that cause the regression tests to fail.

My advice is to update the build requirements for testing to `matplotlib >= 2.0.0` and re-generate the test cases accordingly.  People can still run code on 1.5, but testing should stay up to date.

Ok, thanks.

> re-generate the test cases accordingly

How did you generate the test cases in the first place?

Ran the tests functions, checked the output manually, and then copied over to baseline images.

I can take care of this in a few days.

Ok, cool.
Repository: mir-evaluation/mir_eval
Issue: float-valued indexing will break everything on numpy >= 1.12
Root cause: External Dependency Breakage

This warning is popping up in the current tests on python 3:
```
test_beat.test_beat_functions(<function p_score at 0x7f9f3449a8c8>,) ... /home/travis/build/craffel/mir_eval/mir_eval/beat.py:409: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
```
The [offending code](https://github.com/craffel/mir_eval/blob/master/mir_eval/beat.py#L405-L409) is due to a float-divison `/` rather than integer division `//` in computing `middle_lag`.  This is why the warning does not show up in py2.7.

This gives a warning in numpy <= 1.11, but in 1.12 and above, it will throw TypeErrors.  Note that 1.12 has not made it into conda yet, so this isn't affecting our test builds, but we should get out ahead of it.  Probably there are other points where this comes up, and we should do a thorough audit.

Will look into this in a few weeks.  If I am blocking something after then, don't hesitate to ping me.

This one is causing build errors on jams, and holding up a release.  Any objection to me fixing it directly and merging?  Ought to be a one-liner for beats.

note: the np 1.12 issues are gonna stack up quickly, so it'd be good to get a mir_eval release out ASAP.

> This one is causing build errors on jams, and holding up a release. Any objection to me fixing it directly and merging? Ought to be a one-liner for beats.

Fine with me, otherwise I have a paper deadline at the end of the month; I will be available to work on it after then.

> note: the np 1.12 issues are gonna stack up quickly, so it'd be good to get a mir_eval release out ASAP.

You mean beyond what's mentioned in this thread?
Repository: mir-evaluation/mir_eval
Issue: Readthedocs?
GitHub labels: question

Any interest in setting mir_eval up on rtd?  It would be nice to have automatic doc builds for PRs, if nothing else.

Sounds like a good idea to me.

Good project for [docathon](https://bids.github.io/docathon/) this coming week.  I'll take a crack at it probably around wednesday.

Hrm, looks like only the project owner can set up the integration. :cry: 

Blocked by #375 

RTD integration is now working, but the sphinx configs need some work to get autodoc running again.

Is this one-page layout up for debate or could we organize it in topics? IMHO, would be easier to get to the information needed. However, CTRL+F one a single page is also charming in a way.

What you think?

I would absolutely be in favor of moving to a modern layout and RTD theme.  I think our docs are verbose enough that a one-pager is pretty unwieldy.

Ok, I think we're in business: https://mir-eval.readthedocs.io/latest/

Closing this out.  Final step will be to modify our gh-pages to point to the rtd build instead.

Looks good, thanks for picking this up!
Repository: mir-evaluation/mir_eval
Issue: Exception classes?
GitHub labels: enhancement

question

In implementing #242 , I noticed that the chord submodule defines a derived exception class, but it's the only place where we do this.  Everywhere else, primitive exception types (valueerror, typeerror, etc) are used.

This isn't good practice because it makes it difficult for an end-user to encapsulate exceptions based on what level of the call stack they come from.

Instead, we should have a base exception class `MIRException` from which we can derive more specific exception classes in a coherent hierarchy.  That way, a user can catch any `MIRException` on parse/eval without having to worry about all the various types of ways that an eval can go wrong.

What do yall think?

I'm not opposed to this at all, but I don't think we have intricate enough exceptions to warrant this, by which I mean there aren't many functions (OTOH, would be happy to be corrected) that raise enough different exceptions that the end-user would want to handle separately.

It's less about having a deep/wide hierarchy of exceptions than making it easy to catch all mir_eval-generated exceptions in one go, if the caller wants to.  Right now, we can't distinguish mir_eval errors from numpy errors, or anything else.

My point (as usual) is basically that this is worth implementing once people start complaining because they don't have enough fine-grained exception control when they use `mir_eval`, and that is stopping them from doing what they want to.
Repository: mir-evaluation/mir_eval
Issue: use proper ticker locators for display.labeled_intervals
GitHub labels: enhancement

This is a minor peeve that I'll probably fix this afternoon:

In interactive mode, the `labeled_intervals` plots do not show the correct vertical axis information in the mouse coordinate display (lower-right of the display window).  This is because tick labels are set directly to the axis, and not run through a `Locator` object from matplotlib.

It should be trivial to swap the tick labeling code out for a `FixedLocator` object to fix this.  `librosa.display` does this for tonnetz plots, so we have a model to work from.

This change should have no perceivable difference in static plots.

Cool, thank you.
Repository: mir-evaluation/mir_eval
Issue: sonify.chord, X should be silent but sounds like garbage
Root cause: Validation Gap

Title says it all.

Background: the chord module distinguishes two "non-chord" labels:

- `N` (no-chord) used to indicate things like silence, crowd noise, or sounds that have no harmony
- `X` (unknown) used to indicate, as far as I can tell, out-of-gamut chords.

The `X` label mainly comes up when a predictor can only operate in a reduced label space (eg, is not allowed to predict 9ths), but you still want to model those observations as distinct from `N`.

When these are encoded via [`chord.encode`](https://github.com/craffel/mir_eval/blob/master/mir_eval/chord.py#L99-L102), the `N` label maps to an empty set of pitch classes.  The `X` label maps to a vector of all `-1`, which gets sonified as button-mashing the keyboard.

Proposed change: `sonify.time_frequency` thresholds `gram` at 0 so that negative amplitudes are treated as silence.  This would solve the problem indirectly, and prevent similar mishaps from popping up elsewhere.

Good catch.

> Proposed change: sonify.time_frequency thresholds gram at 0 so that negative amplitudes are treated as silence. This would solve the problem indirectly, and prevent similar mishaps from popping up elsewhere.

Seems reasonable to me, provided there's a note in the docstring to this effect.
Repository: mir-evaluation/mir_eval
Issue: mir_eval.util.intervals_to_samples expected behavior
GitHub labels: enhancement

What's the intended behavior for `mir_eval.util.intervals_to_samples`?

In the following example:
```python
import numpy as np
import mir_eval

intervals = np.array(
    [[0.2, 0.49],
    [0.8, 0.9]]
)
labels = ['a', 'b']
sample_times, sample_labels = mir_eval.util.intervals_to_samples(intervals, labels, offset=0, sample_size=0.1, fill_value=None)
```
I expected something close to:
```python
sample_times = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8])
sample_labels = np.array([None, None, 'a', 'a', 'a', None, None, None, 'b'])
```
(i.e. time periods where there is no labeled interval get filled in with `None`)
but instead get:
```python
sample_times = np.array([0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8])
sample_labels = np.array([None, None, 'a', 'a', 'a', 'a', 'a', 'a', 'b'])
```
Am I missing something?

I think there is an implicit assumption that the intervals are contiguous.  The `fill_value` argument I believe refers to "out of range" meaning before `intervals[0, 0]` or after `intervals[-1, 1]`.  Your expected behavior would still work fine for contiguous intervals and seems reasonable to me though.  @bmcfee knows best here.

@craffel is correct; the current implementation does not handle gaps.

Was also interested in your input as to whether we should change the implementation so that it does handle gaps.

@craffel @bmcfee makes sense. That assumption should probably be in the docs. (Happy to make a pr.) Especially because the description for the variable `intervals` says "An array of time intervals, as returned by `mir_eval.io.load_intervals()`" which could be non-contiguous.


yeah, fair point.  It would be easy to support this behavior; feel free to tag this for the next release, i can hack it on friday.

I have a prototype working locally for this.

The change is entirely localized to `interpolate_intervals`, since that's where the heavy lifting is done.

One thing I noticed is that it only works if `time_points` is in sorted order.  The only place where `interpolate_intervals` is called in mir_eval is from within `intervals_to_samples`, which only generates sorted time grids.  However, the documentation does not stipulate ordering as a requirement.

Question for @craffel -- do we add ordering as a requirement to this function, or extend the function to support unordered time points?  I'm having a hard time imagining a use case where that would be necessary, but one never knows.

We should warn/throw an error when the time points are not ordered, and put a note in the docstring mentioning this.  Thanks.

> We should warn/throw an error when the time points are not ordered, and put a note in the docstring mentioning this. Thanks.

Done.  I also rewrote it once more using `np.searchsorted`, so it should be a smidgen faster.
Repository: mir-evaluation/mir_eval
Issue: Improve efficiency for bipartite matching?
Root cause: Algorithmic Error

Right now we rely on an [all-pairs distance calculation](https://github.com/craffel/mir_eval/blob/master/mir_eval/util.py#L718) between reference and estimate events.  If you have a large set on either end (say, 10^5), this is not a great idea.  [Tagging @lostanlen for bringing this up.]

Since these are all comparing 1-dimensional objects, we could accelerate this from `O(n*m)` (for `n` reference and `m` estimates) to `O(m * log(n))` by sorting both arrays and doing bisection-search (`np.searchsorted`) for each event.  For each estimate event `t`, we can bisect-left the reference for `t-window` and bisect-right for `t+window` to get a slice `u:v`, constituting all hits.  This can be done quickly in numpy as follows:

```python
ref_sort = np.sort(refs)
left_hits = np.searchsorted(ref_sort, est - window, side='left')
right_hits = np.searchsorted(ref_sort, est + window, side='right')

# hits for est[i] are ref_sort[left_hits[i]:right_hits[i]]
```

Looked into this more closely; supporting alternative distance functions is a bit of a pain.  I think for now the best approach is to modify the signature so that `distance=None` implies `|x-y|`, for which we can do the fast bisection search.  Any non-None metric supplied (eg mod-n distance used in multipitch) will revert to the current outer-distance method.
Repository: mir-evaluation/mir_eval
Issue: SIR=inf
Root cause: Documentation Deficiency

This may be a basic question but I'm struggling to get sensible results from bss_eval_sources. I 
am using it to assess a blind source separation system of musical mix -> vocals. To use bss_eval_sources, I feed in two signals, one being the original clean vocal, the other my recreated estimated vocal. I get seemingly sensible results for SDR (that increase and decrease as I'd expect), however SAR always returns an identical answer, and SIR is always inf. I suppose e_`interf `and e_`noise `are always returned as zero but I'm not sure why. Am I doing something wrong?

code:

```
estimated_sources.shape # recreated acapella
(8322560,)

reference_sources.shape # clean vocal
(8322560,)

sdr, sir, sar, perm = bss_eval_sources(reference_sources, estimated_sources, compute_permutation=False)
```

result:
```
sdr [ 3.75990655] 
sir [ inf] 
sar [ 3.75990655] 
perm  [0]
```
I got my understanding of what's going on from [this](https://hal.inria.fr/inria-00564760)

bsseval assumes linear mixtures, therefore there is no point in comparing just reference and estimate for a single source. To correctly compute the  `SIR` you would need to pass the interfering sources as well, e.g. [vocals, accompaniment].

If you are only interested in the quality of a single source compared to the reference, you are better off using something like [peaq](https://en.wikipedia.org/wiki/PEAQ).
Repository: mir-evaluation/mir_eval
Issue: Segmentation Measures missing for Chord Recognition
Root cause: Unimplemented Feature Gap

The segmentation measures (directed hamming distance) as defined by Harte 2010 (Sec. 8.3) and used in MIREX (MeanSeg, UnderSeg, OverSeg) are missing. I have Python code lying around for these, and am happy to contribute it. Before I start, however, what are the thoughts on this from the maintainers?

@ejhumphrey @mattmcvicar @bmcfee What do you think?

I think additional metrics can only be a good thing 👍

On Fri, Sep 29, 2017 at 4:35 PM, Colin Raffel <notifications@github.com>
wrote:

> @ejhumphrey <https://github.com/ejhumphrey> @mattmcvicar
> <https://github.com/mattmcvicar> @bmcfee <https://github.com/bmcfee> What
> do you think?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/craffel/mir_eval/issues/262#issuecomment-333160289>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ACfGsmoxW--qUUkhGWY3II494b7xq91rks5snQ5IgaJpZM4Poalc>
> .
>



-- 
Matt McVicar
Machine Learning Researcher
Jukedeck
contact: mattjamesmcvicar@gmail.com
blog/website: www.mattmcvicar.com


SGTM. :+1: