Repository: mir-evaluation/mir_eval
Issue: Data validation functions
GitHub labels: enhancement

In mir_eval.beat, there's a function _clean_beats (which I'm in the process of removing/refactoring) which essentially takes as input an array of beat locations and runs a series of assertions about its content - e.g., that it is a 1d np.ndarray, no beat times are negative, all beat times are less than 30,000 (largely to make sure the beat times are in seconds).  It also sorts the beat times.  

Given that the design doc https://github.com/craffel/mir_eval/wiki/Design requires that each metric function just takes in raw annotations and does not load files or provide error checking, it seems like it would be beneficial to have functions like this one which provide sanity checks for the input data, and which fail in a helpful way when the data is poorly formed.  It seems also like all the tasks would need something like this.  If that's the case, can we agree on some kind of convention as to where this functionality should go?  Maybe in a function mir_eval.task.clean or something?  


bmcfee says "all tasks should have a validate function, called validate, and each metric should call it before doing anything else.  It should not return a value and should not modify the input.  It could be a decorator to avoid redundancy."


We might want to relax validation, and maybe support different validators for different metrics within a submodule.  For example, `segment.boundary_detection` does not require exact time matching, but the frame clustering metrics do.


See https://github.com/craffel/mir_eval/blob/master/mir_eval/beat.py for an example of how this could be done.


I think we may want to revert the validation for segment boundary detection metrics.  It currently requires that input annotations have the same duration, and start at 0.  This has been fine so far, but I could imagine a developer wanting to experiment with different trimming strategies, eg, suppress first/last five seconds, rather than end-point boundaries.  With the current implementation, this would be impossible.

Thoughts?


This only makes sense in terms of experimenting. When evaluating boundaries
you always assume that both annotations have the same length. Maybe we
could allow that, but throwing a warning, or having a flag (symmetric), or
something similar, in order to make clear that this is not the standard way.

On Mon, Apr 7, 2014 at 10:40 AM, Brian McFee notifications@github.comwrote:

> I think we may want to revert the validation for segment boundary
> detection metrics. It currently requires that input annotations have the
> same duration, and start at 0. This has been fine so far, but I could
> imagine a developer wanting to experiment with different trimming
> strategies, eg, suppress first/last five seconds, rather than end-point
> boundaries. With the current implementation, this would be impossible.
> 
> Thoughts?
> 
> ## 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/31#issuecomment-39738210
> .


> This only makes sense in terms of experimenting. When evaluating boundaries
> you always assume that both annotations have the same length. Maybe we
> could allow that, but throwing a warning, or having a flag (symmetric), or
> something similar, in order to make clear that this is not the standard way.

You can assume that in your evaluation scheme, but there's nothing inherent to the metric that requires it.  Note that this differs from frame clustering metrics, which have a strong alignment requirement.

Of course, the evaluator scripts will enforce this anyway, so this is really a point toward developers, not end-users (such as they are).


I see. Then yeah, it should be possible to change the time of the first
boundary so that it is different than 0. As long as it's transparent to
end-users, go for it.

On Mon, Apr 7, 2014 at 10:52 AM, Brian McFee notifications@github.comwrote:

>  This only makes sense in terms of experimenting. When evaluating
> boundaries
> you always assume that both annotations have the same length. Maybe we
> could allow that, but throwing a warning, or having a flag (symmetric), or
> something similar, in order to make clear that this is not the standard
> way.
> 
> You can assume that in your evaluation scheme, but there's nothing
> inherent to the metric that requires it. Note that this differs from frame
> clustering metrics, which have a strong alignment requirement.
> 
> Of course, the evaluator scripts will enforce this anyway, so this is
> really a point toward developers, not end-users (such as they are).
> 
> ## 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/craffel/mir_eval/issues/31#issuecomment-39739724
> .


- [x] beat  @craffel
- [x] onset @craffel
- [x] separate @dawenl 
- [x] segment @bmcfee
- [x] chord @ejhumphrey 
- [x] melody @justinsalamon 
- [x] pattern @urinieto