Repository: mir-evaluation/mir_eval
Issue: Demo evaluator scripts for all submodules
Root cause: Unimplemented Feature Gap

Each submodule should have a driver program that can be operated from the command line, and output each metric's score in a machine-readable format.  These should follow the general formula of

```
./SUBMODULE_eval.py TRUTH.TXT PREDICTION.DATA
```

e.g.

```
./segment_eval.py ~/data/SALAMI/data/1000/parsed/textfile1_functions.txt predictions/1000.lab
```
- [x] @craffel beats
- [x] @craffel onsets
- [x] @ejhumphrey chords
- [x] @dawenl bss
- [x] @bmcfee segments
- [x] @justinsalamon melody
- [x] @urinieto pattern

I'll push the segment driver asap so you can use it as a template for your own submodule drivers.


I'm a fan of fixing the boundaries as proposed. But just as an FYI, this
means the numbers generated probably will no longer match up to the MIREX
scores.

On Saturday, January 18, 2014, Brian McFee notifications@github.com wrote:

> Each submodule should have a driver program that can be operated from the
> command line, and output each metric's score in a machine-readable format.
> These should follow the general formula of
> 
> ./SUBMODULE_eval.py TRUTH.TXT PREDICTION.DATA
> 
> e.g.
> 
> ./segment_eval.py ~/data/SALAMI/data/1000/parsed/textfile1_functions.txt predictions/1000.lab
> -  @craffel https://github.com/craffel beats, onsets
> -  @mattmcvicar https://github.com/mattmcvicar chords
> -  @dawenl https://github.com/dawenl bss
> -  @bmcfee https://github.com/bmcfee segments
> 
> I'll push the segment driver asap so you can use it as a template for your
> own submodule drivers.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/14
> .

## 

Matt McVicar
Postdoctoral Researcher, National Institute of Advanced Industrial Science
and Technology
mattjamesmcvicar@gmail.com (personal), matt.mcvicar@aist.go.jp(professional)
blog/website: www.mattmcvicar.com


Yup.. we discussed this a bunch at ismir and in the late-breaking session on segment eval, seems like everyone was for the trimming.


A quick survey: how would you prefer to use bss_eval from command line? I would like to save the data in a mat file (from scipy.io), but there might be other preference.


Two questions:
1. Why .mat? What would go in there?
2. The use-case I have in mind for the demo scripts is just score reporting.  For this, printing to the screen should suffice.  The script can have an option to save off extra stuff if you want, but this seems secondary to me.


2 matrices go in there: one for true sources and one for estimated sources, both of shape (n_sources, n_samples), so they are essentially actual audio waveforms. Since the estimated sources normally come from some algorithms implemented in Python, it's straightforward to save them in a mat file, instead of actual wav files (which will require dependency to librosa in order to read them, and even more problematic if n_sources is not very small).


Wait.. are you talking about the input data, or the output? I'm confused.


Input to bss_eval, output of some source separation algorithms. 


Oh, I see.

A few things to keep in mind:
- The goal here is to make it easy to use. If there's a standard format for source separation output, use that.  If not, go with whatever's easiest to save to.
- Truth and predictions should never live in the same file. Have one (or more) files for each.
- .mat files are a bit opaque, so I'd generally prefer to avoid them if possible
- Here are a couple of pure python options:
  - Save .wav files on disk. Basically anything dealing with audio should be able to handle this, and it's obvious what the data is.  Note: `scipy.io.wavfile` is your friend here, though you might have to do some int<->float conversion to make it work.  You don't need librosa for this.  You can use argparse with `nargs='+'` to handle variable-number inputs for each type (truth and estimation).
  - Save the truth matrix as one file and the prediction as another.  Use .npy or .mat.  I generally like this less, since as I said above, it's not obvious what the files are unless you have code to accompany them.


I had always envisioned wav files as input to the bss command line script. .mat would be ok but I'd want it to either accept mat or npy. 


If the data fits easily into a wav file (I.e., it's a time-domain
waveform), that's surely the format to use - it's not tied to a particular
language.

If there's structural information that needs to saved in addition, I'd go
for a human-editable text file that points to the wav files.

I'm motivated by maximizing compatibility with as-yet unenvisioned tools
and applications.

  DAn.

On Monday, January 20, 2014, craffel notifications@github.com wrote:

> I had always envisioned wav files as input to the bss command line script.
> .mat would be ok but I'd want it to either accept mat or npy.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/14#issuecomment-32773812
> .


Just finished the pattern discovery evaluator.


@dawenl can you look over the separation evaluator?  It works a little different from the others, as it takes in a folder and reads in .wav's, each of which corresponds to a source...  Seems to do the right thing.


> @dawenl can you look over the separation evaluator? It works a little different from the others, as it takes in a folder and reads in .wav's, each of which corresponds to a source... Seems to do the right thing.

Maybe I'm being a bit pedantic here, but this seems overly restrictive.  We shouldn't require a specific directory structure for running the evaluators.

Why not accept variable-length arrays of input files using `argparse` and `nargs='+'`?  They would have to be keyword arguments (non-positional), but I think it's a better route all around.


It was a hasty design choice, I'd be fine with it changing.  The reason I chose folders instead of variable arguments is that sometimes you have 5 or 8 each of reference and estimation files, and it would kind of be a pain to enter all of them in.  I think I'd be most comfortable with it taking a folder or a list of .wav names.