zhiminy/EvalEng huggingface dataset:
Repository: mir-evaluation/mir_eval
Issue: util.generate_labels
Root cause: Unimplemented Feature Gap

As part of the refactoring for #35, `io.load_events` and `io.load_intervals` will not generate synthetic labels if no labels are present.  So, in the case that someone wants to create synthetic labels, we need a separate function, `util.generate_labels`, which should do the same thing that `io.load_events` and `io.load_intervals` do when no labels are present in the annotation file: Take in a label prefix (default '__') and generate labels '(label prefix)0', '(label prefix)1', '(label prefix)2'... for each event/interval.
Repository: mir-evaluation/mir_eval
Issue: Docstring audit
GitHub labels: requirement

Before releasing 0.1, check the dosctrings for valid usage examples and that the doc represents how the function currently works.
- [x] index
- [x] beat
- [ ] chord
- [x] io
- [x] melody
- [x] onset
- [x] pattern
- [x] segment
- [x] separation
- [x] sonify
- [x] util


Need to add references, `:raises: ValueError` and short introductions.


Seems like there might still be inconsistencies in docstrings.  For example, `mir_eval.chord.encode_many` has the signature

```
mir_eval.chord.encode_many(chord_labels, reduce_extended_chords=False)
```

and docstring fragment:

```
 - reduce_extended_chords : bool, default=True
```

Probably worth an extra sweep for default parameter consistency, since there may be others.


Note above:
https://github.com/craffel/mir_eval/issues/69#issue-38775725
that chord did not have its docstring audited, see #81
Repository: mir-evaluation/mir_eval
Issue: Move evaluate method to task submodules
Root cause: Unimplemented Feature Gap

As part of moving away from the METRICS dictionary, each submodule will have an `evaluate` method as is seen in the evaluators.  It will take in data in the appropriate format and call all of the metric functions in the standard way.  The `evaluate` functions should then be removed from the evaluators.
- [x] beat
- [x] segment
- [ ] chord
- [x] melody
- [x] onset
- [x] pattern
- [x] separation
Repository: mir-evaluation/mir_eval
Issue: Change back melody call signature order
Root cause: Architectural Constraint

In https://github.com/craffel/mir_eval/commit/7a2cacc34a51a4675dfca42d14b1791bb58a59fc the order of the arguments for melody metrics was changed from 
`ref_cent, ref_voicing, est_cent, est_voicing`
to
`ref_voicing, est_voicing, ref_cent, est_cent`
so that the validate_voicing decorator could be used on metrics which use both cent and voicing.  We aren't using decorators anymore, so we can change back to the old format which matches the convention elsewhere in `mir_eval`.  @justinsalamon this will break all of your code which calls these functions but it will be a quick fix.


roger that :)
Repository: mir-evaluation/mir_eval
Issue: Merge boundary/structure back into segment

Now that we aren't using the METRICS dictionary or validate decorators, we don't need to have these submodules split.


It's debatable.  Boundary detection and structural annotation are two distinct tasks, and the form of annotation differs between the two.

Even though our original motivation for splitting them was automation and consistency, I think it still might make sense to keep them split.  What's the argument for merging them again?


Mostly the evaluate function.  Change your code @bmcfee and @urinieto


If you feel very strongly about it, feel free to re-split them, but then also to split out the evaluate function, the evaluator, and the test scripts/data.


Segmentation is a task that can easily be split into two. However, in many
papers, and even in MIREX, this task is treated as one. However, more
recent papers clearly separates the two problems, and it wouldn't surprise
me that MIREX splits the task in the near future. So, if we want to be
exactly like MIREX, it makes sense to merge the task right now.

On Wed, Aug 27, 2014 at 1:31 PM, craffel notifications@github.com wrote:

> If you feel very strongly about it, feel free to re-split them, but then
> also to split out the evaluate function, the evaluator, and the test
> scripts/data.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/craffel/mir_eval/issues/72#issuecomment-53609682.


If you guys want to re-split, I'm fine with that; I remember when I originally proposed splitting the two it was met by some resistance, and the tests and evaluators were never split so I figured the split was begrudged.  If you do re-split them, make sure to split the tests and evaluators too.
Repository: mir-evaluation/mir_eval
Issue: Python3 compatibility
Root cause: Environment Incompatibility

Apparently in python 3 the print statement was replaced with a print function, meaning that instead of 
`print x` you need to call `print(x)`. Consequently, this breaks mir_eval on Python 3 (brought to my attention by someone trying to use mir_eval on Windows 7 with Python 3.4). 

Perhaps we should replace all calls to `print x` with e.g. `sys.stdout.write(x)` which should work both on python 2 and 3?


There are probably more things than just that which break in Python 3.  Currently, `mir_eval` is not designed to be used with Python 3.  If you'd like to go through and resolve all of the issues that come up when one uses 3 instead of 2.7, feel free to do so.


Fair point. In which case I'd rather not open this can of worms before we really have to :)


I don't have any plans to switch to Python 3 in the foreseeable future.  If someone else does, they're welcome to do the conversion!


Here's a handy reference, if you actually want to futurize things: http://python-future.org/compatible_idioms.html

See also: http://python-future.org/ for futurize and pasteurize


I love these modules' names. In any case, I also wouldn't worry about
Python 3 compatibility for now.

On Wed, Aug 27, 2014 at 4:30 PM, Brian McFee notifications@github.com
wrote:

> Here's a handy reference, if you actually want to futurize things:
> http://python-future.org/compatible_idioms.html
> 
> See also: http://python-future.org/ for futurize and pasteurize
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/craffel/mir_eval/issues/74#issuecomment-53635747.


> In any case, I also wouldn't worry about
> Python 3 compatibility for now.

It might be worth prioritizing py3 compatibility just to make unicode less of a headache.  I can imagine this being a common problem for using mir_eval with annotation filenames keyed by track titles with extended character sets.  (This problem comes up frequently for some of my database projects, though not in eval yet.)  See: http://python3porting.com/problems.html#binary-section


This should be handled by #113.
Repository: mir-evaluation/mir_eval
Issue: Melody evaluation: output the matching of each individual value of the input time series
Root cause: Unimplemented Feature Gap

This is quite useful to know which values have been matched to the reference or ground truth and which not.

This cannot be done if the input time series is resampled to match the reference signal's sample rate, but if the sample rates match or if the reference is resampled to match the input time series, then it would be very useful.


As you noted in #78, this kind of diagnostic is currently outside the scope of `mir_eval`.  The same functionality is absent in the other modules simply because the metric functions are meant to report metrics.  Creating this functionality based on the current code ouside of `mir_eval` should be pretty straightforward if you're so inclined.
Repository: mir-evaluation/mir_eval
Issue: Melody evaluation: allow modification of the accepted frequency deviation
Root cause: Unimplemented Feature Gap

Currently the tolerance for an estimated frequency value to be matched to a reference frequency value is fixed at 50 cents.

https://github.com/craffel/mir_eval/blob/master/mir_eval/melody.py#L419
https://github.com/craffel/mir_eval/blob/master/mir_eval/melody.py#L477

In some tasks it is interesting to accept a lower tolerance, in those cases having this value as a keyword argument will be very practical.
Repository: mir-evaluation/mir_eval
Issue: Melody evaluation: allow resampling reference to match estimated
Root cause: Unimplemented Feature Gap

Currently the only resampling choices available are to a custom hop step and to match the reference.

https://github.com/craffel/mir_eval/blob/master/mir_eval/melody.py#L283-L286

In some cases it may be interesting to resample the reference to match the estimated.  For instance when trying to figure out which estimated values are incorrect and which not.


Thou shalt not modify the reference! (Unless you really need to, of course.)

In the scenario you described, why not just swap the roles of estimated and reference?


Yes, probably swapping roles is a good workaround for those scenarios. Another workaround is to resample the reference prior to using mir_eval for the given task.

In any case, it is true that those scenarios are simply out of the scope of mir_eval, since they are not for evaluation but rather for exploration, visualisation or optimisation of a given algorithm.
Repository: mir-evaluation/mir_eval
Issue: Change segmentation metric name from detection to hit rate
GitHub labels: enhancement

wontfix

The "detection" segmentation metric is most commonly called "hit rate". I also think that "hit rate" is more meaningful, especially now that boundaries and structure are merged.

Should we do this change?


If we're merging, I think we need to put the word `boundary` back into the function names, regardless of whether it's `detection` or `hit_rate`.

My preference isn't too strong either way, though I think `hit rate` connotes precision, while `detection` is more inclusive to recall and f-score.  Just my $2e-2.


Up to you guys.  If you change anything make sure to change the `evaluate` method and unit tests, etc.


Deferring to @urinieto 


Will do.

On Fri, Sep 5, 2014 at 11:55 AM, Brian McFee notifications@github.com
wrote:

> Deferring to @urinieto https://github.com/urinieto
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/craffel/mir_eval/issues/79#issuecomment-54643886.


Here is a (hopefully exhaustive) list of places you need to update if you change the metric name:

```
./mir_eval/segment.py:29:* :func:`mir_eval.segment.detection`: An estimated boundary is considered
./mir_eval/segment.py:152:def detection(reference_intervals, estimated_intervals,
./mir_eval/segment.py:165:        >>> P05, R05, F05 = mir_eval.boundary.detection(ref_intervals,
./mir_eval/segment.py:169:        >>> P3, R3, F3 = mir_eval.boundary.detection(ref_intervals,
./mir_eval/segment.py:173:        >>> P, R, F = mir_eval.boundary.detection(ref_intervals,
./mir_eval/segment.py:1155:        util.filter_kwargs(detection, ref_intervals, est_intervals, **kwargs)
./mir_eval/segment.py:1159:        util.filter_kwargs(detection, ref_intervals, est_intervals, **kwargs)
./tests/test_segment.py:35:        if metric == mir_eval.segment.detection:
./tests/test_segment.py:51:    if metric == mir_eval.segment.detection:
./tests/test_segment.py:120:    for metric in [mir_eval.segment.detection,
```

If you split the segment module into two there's a bunch of other things you need to change, but I'm not sure if you're doing that.


@urinieto @bmcfee do this by tomorrow or forever hold your peace on the matter 


No time to do it, so let's leave it like that forever.

On Thu, Oct 9, 2014 at 6:34 PM, Colin Raffel notifications@github.com
wrote:

> @urinieto https://github.com/urinieto @bmcfee
> https://github.com/bmcfee do this by tomorrow or forever hold your
> peace on the matter
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/craffel/mir_eval/issues/79#issuecomment-58588582.
Repository: mir-evaluation/mir_eval
Issue: output methods: csv, json, pickle
Root cause: Unimplemented Feature Gap

It would be useful to be able to output reports in either flat format (csv), or structured objects (json, pickle).


Are we gonna do this for milestone 0.0.1?


No, 0.0.2 makes more sense. Unless someone is extremely motivated. 


We'll be outputting json only from the evaluators.  See here for an example:
https://github.com/craffel/mir_eval/blob/master/evaluators/segment_eval.py#L81
https://github.com/craffel/mir_eval/blob/master/evaluators/segment_eval.py#L110
https://github.com/craffel/mir_eval/blob/master/evaluators/segment_eval.py#L138
- [x] beat
- [ ] chord
- [x] melody
- [x] onset
- [x] pattern
- [x] segment
- [x] separation
Repository: mir-evaluation/mir_eval
Issue: Update chord
Root cause: Unimplemented Feature Gap

`mir_eval.chord` needs the following changes to be up-to-date with the structure/conventions of the other modules, before 0.1 gets released:
- [x] Separate out scoring function #63 
- [x] Make `evaluate` function #70
- [x] Move `evaluate_file_pair` to the evaluator #62 
- [x] Docstring audit/reformatting #69 
- [x] Evaluator should output json and match other evaluator's structure #8 
- [x] Regression/unit tests should match other tests' structure and should use `evaluate` and not `METRICS` #73 
- [x] Remove `METRICS` dictionary #54

~~chord.validate should use util.validate_intervals~~


The scoring function (`mir_eval.chord.weighted_accuracy`) now takes in durations instead of intervals, so the last checkpoint is no longer relevant.
Repository: mir-evaluation/mir_eval
Issue: Sphinx :member-order: bysource doesn't work
GitHub labels: enhancement

We use `:member-order: bysource` (see docs/index.rst) ostensibly so that the order in which things show up in the compiled docs is the order in which they are defined in the source.  But, they don't; they show up in alphabetic order which is the default.  This is kind of annoying and I don't know why it's not working.  @bmcfee @dpwe any ideas?


Fixed!

Problem was that in order to figure out bysource, it has to find the
source, which it does by inspecting the path it gets when it actually
imports the module.  But although the sphinx make was able to import
the module, it was getting it via the centrally-installed egg (at
least on my machine), not from the local source.  The fix was easy -
sys.path.insert(0, os.path.abspath('..')) (committed). But it took a
lot of inserting prints into the sphinx source for me to figure it
out!

On Mon, Oct 6, 2014 at 3:24 PM, Colin Raffel notifications@github.com wrote:

> We use :member-order: bysource (see docs/index.rst) ostensibly so that the
> order in which things show up in the compiled docs is the order in which
> they are defined in the source. But, they don't; they show up in alphabetic
> order which is the default. This is kind of annoying and I don't know why
> it's not working. @bmcfee @dpwe any ideas?
> 
> —
> Reply to this email directly or view it on GitHub.


Excellent! Thanks.
Repository: mir-evaluation/mir_eval
Issue: Get rid of magic value in util.validate_events
Root cause: Architectural Constraint

When validating events, a maximum event time is enforced via a `ValueError`:
https://github.com/craffel/mir_eval/blob/master/mir_eval/util.py#L619
This is basically to ensure that the supplied event times are in seconds, but a magic value should not be used.  Instead, `MAX_TIME` should be defined in the submodules which use `validate_events` (`beat` and `onset`) and passed to validate_events; this way, if for some strange reason and end-user actually does have extremely large event times they can set the value of `beat.MAX_TIME` or `onset.MAX_TIME` to avoid this `ValueError`.
Repository: mir-evaluation/mir_eval
Issue: Check that input to separate.bss_eval_sources is (nsrc, nsamples)
Root cause: Validation Gap

If the input matrix is transposed, the result would be a ridiculously huge and incorrect computation.  This should be enforced by ensuring that `reference_sources.shape[0] < MAX_SRC` and `estimated_sources.shape[0] < MAX_SRC`, where `MAX_SRC` is a constant defined in `mir_eval.separation`.
Repository: mir-evaluation/mir_eval
Issue: boundary import to include labels
Root cause: Unimplemented Feature Gap

`util.import_segment_boundaries` currently loads in data of the flavor:

```
0.00\t21.7\tINTRO
21.7\t32.8\tVERSE
...
```

Right now, it only processes the time information.  We should extend this to return segment labels as well.

Maybe we should generify the loader to support this kind of i/o for chord (or arbitrary) annotations as well?
Repository: mir-evaluation/mir_eval
Issue: Tests use glob, which has different order on Mac/Linux
Root cause: Validation Gap

They should use sorted globs instead.


Possibly better form to extract file loading/sorting from the tests and set them up as proper fixtures instead.

http://nose.readthedocs.org/en/latest/writing_tests.html


Probably a good idea, but we'd still have to use sorted globs, just in a different place. @bmcfee should be fixed now, please re-run tests to confirm they all pass.


All pass, but one has a runtime warning:

```
test_separation.test_separation_functions('data/separation/output08.json', 'Source permutation', (0, 1, 2), [0, 1, 2]) ... ok
/usr/local/lib/python2.7/dist-packages/mir_eval-0.0.0-py2.7.egg/mir_eval/separation.py:258: RuntimeWarning: divide by zero encountered in double_scalars
test_separation.test_separation_functions('data/separation/output09.json', 'Source to Distortion', [4.413822820998936], [4.413822820999018]) ... ok
```


That test will give Inf SIR (by computing log(a/b) where a > 0 and b = 0) because it's only single-channel, so this warning is expected. So I think we are all clear now. 


If the warning is expected, shouldn't it be handled directly as a special case?  The semantics of log(0/0) aren't always obvious, so it'd be best to have this fully (and clearly) specified in the code.


Now all the separation tests should pass without throwing a RuntimeWarning.
Repository: mir-evaluation/mir_eval
Issue: Potential +/-Inf dB SXR in separation
Root cause: Validation Gap

Edge case: Under certain unusual cases, we might have +/-Inf dB SXR (for example, one true source being all 0, or one estimated source being all 0) but currently `bss_eval_sources` will return `nan` or incorrect number.


Currently `bss_eval_sources` does not allow any silent sources (in either reference or estimated) because these are not well-defined test conditions -- If we allow all-zero reference source, then potentially we could add infinitely many all-zero reference sources, which only adds ambiguity.
Repository: mir-evaluation/mir_eval
Issue: mir_eval.beat.goto throws error when track_start is an array
Root cause: Algorithmic Error

In fe8eb5ac2028840c82818ebc70874ab78b63ac2f , https://github.com/craffel/mir_eval/blame/master/mir_eval/beat.py#L346 was changed from 

``` Python
track_start = np.nonzero(np.diff(incorrect_beats) == track_length)[0][0]
```

to

``` Python
track_start = np.flatnonzero(np.diff(incorrect_beats) == track_length)
```

This means that track_start is sometimes an array with more than one element (when there are multiple tracks with the maximal track length).  This causes an error.
Repository: mir-evaluation/mir_eval
Issue: consistency between metrics (specifically F-measure)
Root cause: Interface Contract Mismatch

Hi, thanks for this very useful library.

I have noticed a small inconsistency between some metric function outputs that are shared between some of the submodules. F-measure is one of them.

When evaluating based of F-measure it is often useful to know the precision and recall values. Calculating F-measure in the "onset" submodule returns this information, whereas the "beat" submodule does not.


`mir_eval.beat.f_measure` only returns F-measure because only F-measure is used by the beat evaluation toolbox and MIREX; `mir_eval.onset.f_measure` returns all three because MIREX uses all three.  I'd potentially support changing `mir_eval.beat.f_measure` to return all three, but FWIW `mir_eval.beat.f_measure` and `mir_eval.onset.f_measure` are functionally identical (except that the threshold/window is .05 and .07 in `onset` and `beat` respectively) so you can safely use `mir_eval.onset.f_measure` in place of `mir_eval.beat.f_measure` to get all three if you set the `window` kwarg to .07.  At any rate, to implement this change I'd like to get some sort of community consensus first.


Yes that's what I've been doing as a work around.

I see your point, but the counter argument is that by open sourcing this library it is now not necessarily tied to the one MIREX implementation, but a community resource.

Either way it's a small issue that will come down to the ethos of the maintainers.
Repository: mir-evaluation/mir_eval
Issue: Do we need to require numpy >= 1.7.0?
Root cause: Configuration Error

Or is 1.6.1 sufficient?  `apt-get install python-numpy` installs 1.6.1 still..!


I had problems with librosa and np 1.6.1.  Forget exactly what.  I was
mainly using librosa.load and librosa.specgram.

  DAn.

On Tuesday, October 28, 2014, Colin Raffel notifications@github.com wrote:

> Or is 1.6.1 sufficient? apt-get install python-numpy installs 1.6.1
> still..!
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/craffel/mir_eval/issues/96.


> apt-get install python-numpy installs 1.6.1 still..!

On which distribution?  The current ubuntu has 1.8 by default.


12.04.  The larger requirement, I think, is just to require the lowest version possible for all dependencies.  I will try some things out in a virtualenv and make the appropriate changes.


Just tested with numpy 1.5.1, the earliest version in conda.  PR for updating the dependency lists: #137


Oops, nevermind, conda doesn't have numpy <= 1.7 for  Python 3, so let's skip this.
Repository: mir-evaluation/mir_eval
Issue: All chord comparators should ignore 'X' labels.
Root cause: Validation Gap

The "X" chord convention is used in some chord transcriptions (e.g. Billboard) to signify that the musical content is not harmonic / poorly described by a different chord label. As a result, any "X" chords in a reference annotation should (arguably) be ignored in evaluation.

As of c8867b6e4669, the following comparison functions return non-negative scores for reference "X" chords.
- thirds
- thirds_inv
- triads
- triads_inv
- tetrads
- tetrads_inv
- majmin
- majmin_inv
- mirex
Repository: mir-evaluation/mir_eval
Issue: tempo estimation validation
GitHub labels: enhancement

I just finished validating mir_eval's tempo estimation metrics against (private) MIREX 2014 data.  Everything validates to machine precision.

I think we're safe merging the develop branch back into master.

TO-DO:
- [x] add regression tests for tempo evaluation


This one should be closed as well via #131, right?


Check that damn box first!


:white_check_mark:
Repository: mlcommons/inference
Issue: MLPerf - Inference Reference Implementations
Root cause: Unimplemented Feature Gap

When will the reference implementations for the inference of the model be added?

An alpha release of the reference implementations is slated for Q1 next year. 

Okay Thanks

@ArunkumarRamanan You may be interested to know that the first reference implementations are in (MobileNets, SSD-MobileNet, SSD-ResNet50, DeepSpeech2). You are very welcome to try them out and provide your feedback.

please check the lates code under v0.5 dir.