Repository: mir-evaluation/mir_eval
Issue: Framewise separation evaluation where some frames are silent
Root cause: Validation Gap

While evaluating the framewise separation methods that have been added #199 and are almost ready in #212 , I've figured out, the validation routines are a bit too strict: Let's assume there is silence in one of the frames being evaluated:

``` python
nsrc = 2
nchan = 2

empty_source = np.zeros((nsrc, 20, nchan))
estimates = np.random.random_sample((nsrc, 200, nchan))

estimated_sources = np.concatenate(
    (empty_source, estimates),
    axis=1
)

mir_eval.separation.bss_eval_images_framewise(
    estimated_sources,
    estimated_sources,
    window=10,
    hop=5
)
```

this results in 

`ValueError: All the reference sources should be non-silent (not all-zeros), but at least one of the reference sources is all 0s, which introduces ambiguity to the evaluation. (Otherwise we can add infinitely many all-zero sources.)`

and therefore would not continue evaluating the other (non-silent) frames. I think it would make sense to return `np.nan` or `None` for those (2) frames in the framewise evaluation. I don't know how to handle these results later when it comes to compute aggregated measures, but thats more of a discussion for SISEC (@aliutukus ?) 

@craffel @ErikJohnsonCU @carlthome what do you think?


Yes, I agree. I don't think an exception should be raised. It should continue (but maybe output the same text as a warning). It should be left to the user to think about what material they're evaluating and whether it makes sense.


> I think it would make sense to return np.nan or None for those (2) frames in the framewise evaluation.

`np.nan` is better than `None` because a `np.ndarray` can't store a `None` without being of dtype `object`.  I will leave it up to you all to decide what the best validation/metric conventions are!


@craffel okay, cool. Shall we wait for #212 and then add the `nan` routine in a separate PR that changes all framewise calculations? 


Sounds good to me!


This issue is corrected in PR #216. I'd invite further discussion on that webzone. I used the `np.nan` approach since it seemed that was unanimously decided to be a better approach.


🎉 @craffel @bmcfee @carlthome @dpwe 

While #216 looks like almost to be ready to merge, I would like to take a moment to thank @ErikJohnsonCU for his excellent work. Today was his last day of his summer internship here at @audiolabs. I hope that his contributions to `mir_eval` (like #199 #207 #212) are useful for the community. We are so glad we had him here... so lets hope he enjoyed working in audio and we meet him again at some point in the future!


:tada: :balloon: :dancers: 


Did not know that! Totally excellent work, truly appreciate your contributions. Very happy to have all of the bss_eval metrics in `mir_eval` updated and on lock. Sorry again for any delays or nitpicking from my end :) 


@craffel @craffel @bmcfee @dpwe 

It's been a pleasure being able to contribute and help `mir_eval` grow a little bit! Thank you so much to @faroit for hosting me; I really enjoyed the experience of working @audiolabs! I've learned a lot and I really appreciate all the nitpicks that have helped me refine my python style and understanding of open source software contributing.


Resolved in 8a1cb7210354ebff5d776a034f6df83132c85fe5.  Thanks!