Repository: mir-evaluation/mir_eval
Issue: Question about the np.allclose() in utils.boundaries_to_intervals()
Root cause: Validation Gap

I was porting SALAMI annotation files to .lab format and there are more than a few cases where the annotations look like this:

0.0 silence
0.0 Z 

and the np.allclose() just can not handle arrays with different sizes (between boundaries and np.unique(boundaries) in this case). Although it still throws an error anyway... 

And I am wondering how everyone deals with this issue? Having a 0-length interval event or manually editing the annotation file getting rid of one of the redundant event?


Deferring to @bmcfee and @urinieto, as I'm sure they have experience with this.


[paging @jblsmith ]

This was originally discussed in #49 , and I would consider it to be an error in the salami annotations / the salami parser.  (BTW, which version of the annotations are you using?  And can you point to a specific annotation?)

To summarize, there are a few things happening here:
1. Converting from events to intervals is only possible when intervals are disjoint; otherwise, end-boundaries have to be specified somehow.
2. I don't think the salmai parser should have put that "silence" in, and it may be a mistake.  (Or maybe the Z, depending on whether you're looking at large or function.)
3. We have SALAMI ported to JAMS format [here](https://github.com/marl/jams-data/tree/master/datasets/SALAMI), and using these instead of labs may save you a bunch of headaches.


Thanks for the reply! 

The annotations are from the latest version on salami Github. The annotations have duplicate time stamps (meaning same time for two labels) are pasted below (a pretty long list). 

14 textfile2_lowercase
14 textfile2_uppercase
28 textfile1_uppercase
28 textfile2_uppercase
32 textfile2_lowercase
32 textfile2_uppercase
34 textfile2_lowercase
34 textfile2_uppercase
42 textfile2_uppercase
44 textfile2_lowercase
44 textfile2_uppercase
60 textfile2_lowercase
60 textfile2_uppercase
68 textfile2_lowercase
68 textfile2_uppercase
116 textfile2_lowercase
116 textfile2_uppercase
118 textfile2_lowercase
118 textfile2_uppercase
160 textfile2_uppercase
180 textfile2_lowercase
180 textfile2_uppercase
266 textfile2_lowercase
266 textfile2_uppercase
294 textfile2_lowercase
294 textfile2_uppercase
298 textfile2_lowercase
298 textfile2_uppercase
308 textfile1_lowercase
308 textfile2_uppercase
310 textfile2_lowercase
310 textfile2_uppercase
320 textfile2_lowercase
320 textfile2_uppercase
338 textfile1_uppercase
338 textfile2_uppercase
340 textfile2_lowercase
340 textfile2_uppercase
346 textfile2_lowercase
346 textfile2_uppercase
350 textfile2_lowercase
350 textfile2_uppercase
358 textfile1_lowercase
358 textfile1_uppercase
358 textfile2_lowercase
358 textfile2_uppercase
368 textfile2_lowercase
368 textfile2_uppercase
374 textfile2_lowercase
390 textfile2_lowercase
394 textfile2_lowercase
394 textfile2_uppercase
406 textfile2_lowercase
406 textfile2_uppercase
418 textfile2_lowercase
418 textfile2_uppercase
444 textfile2_uppercase
480 textfile2_uppercase
492 textfile2_lowercase
492 textfile2_uppercase
530 textfile2_lowercase
530 textfile2_uppercase
546 textfile2_lowercase
546 textfile2_uppercase
552 textfile2_lowercase
552 textfile2_uppercase
558 textfile2_lowercase
558 textfile2_uppercase
562 textfile2_lowercase
562 textfile2_uppercase
570 textfile2_lowercase
570 textfile2_uppercase
574 textfile2_lowercase
574 textfile2_uppercase
586 textfile2_lowercase
586 textfile2_uppercase
590 textfile2_lowercase
590 textfile2_uppercase
592 textfile2_lowercase
592 textfile2_uppercase
598 textfile2_lowercase
598 textfile2_uppercase
602 textfile2_lowercase
602 textfile2_uppercase
610 textfile2_lowercase
610 textfile2_uppercase
620 textfile2_lowercase
620 textfile2_uppercase
638 textfile2_lowercase
638 textfile2_uppercase
640 textfile2_lowercase
644 textfile1_uppercase
650 textfile1_lowercase
650 textfile1_uppercase
650 textfile2_lowercase
650 textfile2_uppercase
666 textfile2_uppercase
670 textfile2_lowercase
670 textfile2_uppercase
678 textfile1_lowercase
678 textfile1_uppercase
678 textfile2_uppercase
686 textfile2_lowercase
686 textfile2_uppercase
694 textfile2_lowercase
694 textfile2_uppercase
700 textfile1_uppercase
704 textfile2_lowercase
704 textfile2_uppercase
706 textfile2_lowercase
706 textfile2_uppercase
732 textfile2_lowercase
732 textfile2_uppercase
734 textfile1_lowercase
734 textfile1_uppercase
734 textfile2_lowercase
734 textfile2_uppercase
736 textfile1_lowercase
736 textfile1_uppercase
776 textfile2_lowercase
776 textfile2_uppercase
778 textfile2_lowercase
778 textfile2_uppercase
790 textfile1_lowercase
790 textfile1_uppercase
816 textfile2_lowercase
816 textfile2_uppercase
818 textfile2_lowercase
818 textfile2_uppercase
822 textfile2_lowercase
822 textfile2_uppercase
842 textfile1_lowercase
842 textfile1_uppercase
842 textfile2_lowercase
842 textfile2_uppercase
870 textfile1_lowercase
870 textfile1_uppercase
874 textfile1_lowercase
874 textfile1_uppercase
874 textfile2_lowercase
874 textfile2_uppercase
886 textfile1_lowercase
902 textfile1_lowercase
902 textfile1_uppercase
902 textfile2_lowercase
902 textfile2_uppercase
908 textfile1_lowercase
908 textfile1_uppercase
956 textfile1_uppercase
960 textfile2_uppercase
972 textfile2_lowercase
972 textfile2_uppercase
974 textfile2_lowercase
974 textfile2_uppercase
976 textfile1_lowercase
976 textfile1_uppercase
976 textfile2_lowercase
976 textfile2_uppercase
978 textfile2_uppercase
980 textfile1_lowercase
980 textfile1_uppercase
982 textfile1_uppercase
988 textfile1_lowercase
988 textfile1_uppercase
988 textfile2_uppercase
994 textfile2_uppercase
1000 textfile2_uppercase
1002 textfile1_uppercase
1006 textfile1_lowercase
1006 textfile1_uppercase
1006 textfile2_uppercase
1016 textfile2_lowercase
1016 textfile2_uppercase
1022 textfile2_lowercase
1022 textfile2_uppercase
1034 textfile2_uppercase
1038 textfile1_uppercase
1044 textfile2_lowercase
1044 textfile2_uppercase
1050 textfile2_uppercase
1058 textfile1_uppercase
1066 textfile1_uppercase
1078 textfile2_uppercase
1090 textfile1_lowercase
1090 textfile1_uppercase
1090 textfile2_uppercase
1100 textfile2_uppercase
1106 textfile2_uppercase
1114 textfile1_uppercase
1114 textfile2_lowercase
1114 textfile2_uppercase
1118 textfile2_uppercase
1122 textfile1_uppercase
1122 textfile2_lowercase
1122 textfile2_uppercase
1128 textfile2_lowercase
1128 textfile2_uppercase
1134 textfile2_uppercase
1142 textfile1_lowercase
1142 textfile1_uppercase
1146 textfile2_uppercase
1148 textfile1_lowercase
1148 textfile1_uppercase
1150 textfile1_uppercase
1150 textfile2_lowercase
1150 textfile2_uppercase
1156 textfile2_uppercase
1162 textfile2_uppercase
1170 textfile1_lowercase
1170 textfile1_uppercase
1174 textfile2_uppercase
1184 textfile2_uppercase
1190 textfile2_uppercase
1198 textfile1_uppercase
1202 textfile2_uppercase
1206 textfile1_uppercase
1206 textfile2_lowercase
1206 textfile2_uppercase
1212 textfile2_uppercase
1218 textfile2_uppercase
1222 textfile2_lowercase
1222 textfile2_uppercase
1226 textfile1_uppercase
1230 textfile2_uppercase
1234 textfile1_lowercase
1234 textfile1_uppercase
1234 textfile2_lowercase
1234 textfile2_uppercase
1240 textfile2_uppercase
1246 textfile2_uppercase
1258 textfile2_uppercase
1262 textfile1_lowercase
1262 textfile1_uppercase
1262 textfile2_lowercase
1262 textfile2_uppercase
1268 textfile2_uppercase
1274 textfile2_uppercase
1282 textfile1_uppercase
1286 textfile2_uppercase
1290 textfile1_lowercase
1290 textfile1_uppercase
1290 textfile2_lowercase
1290 textfile2_uppercase
1296 textfile2_lowercase
1296 textfile2_uppercase
1302 textfile2_uppercase
1310 textfile1_uppercase
1318 textfile1_uppercase
1318 textfile2_lowercase
1318 textfile2_uppercase
1324 textfile2_uppercase
1330 textfile2_lowercase
1330 textfile2_uppercase
1342 textfile2_uppercase
1346 textfile1_uppercase
1346 textfile2_lowercase
1346 textfile2_uppercase
1350 textfile2_uppercase
1352 textfile2_uppercase
1358 textfile2_uppercase
1370 textfile2_lowercase
1370 textfile2_uppercase
1374 textfile1_uppercase
1374 textfile2_lowercase
1374 textfile2_uppercase
1380 textfile2_uppercase
1386 textfile2_uppercase
1392 textfile2_lowercase
1392 textfile2_uppercase
1402 textfile1_uppercase
1402 textfile2_lowercase
1402 textfile2_uppercase
1408 textfile2_uppercase
1414 textfile2_uppercase
1436 textfile2_uppercase
1438 textfile2_lowercase
1438 textfile2_uppercase
1442 textfile2_uppercase
1454 textfile2_uppercase
1458 textfile1_lowercase
1458 textfile1_uppercase
1458 textfile2_lowercase
1458 textfile2_uppercase
1464 textfile2_uppercase
1470 textfile2_uppercase
1480 textfile1_lowercase
1480 textfile1_uppercase
1480 textfile2_lowercase
1480 textfile2_uppercase
1482 textfile2_lowercase
1482 textfile2_uppercase
1490 textfile2_uppercase
1492 textfile2_lowercase
1492 textfile2_uppercase
1496 textfile2_lowercase
1496 textfile2_uppercase
1498 textfile2_lowercase
1498 textfile2_uppercase
1512 textfile1_lowercase
1512 textfile1_uppercase
1522 textfile1_uppercase
1532 textfile1_lowercase
1532 textfile1_uppercase
1542 textfile1_lowercase
1542 textfile1_uppercase
1562 textfile1_lowercase
1562 textfile1_uppercase
1572 textfile1_lowercase
1572 textfile1_uppercase
1582 textfile1_uppercase
1588 textfile1_lowercase
1588 textfile1_uppercase
1592 textfile1_uppercase
1598 textfile1_lowercase
1598 textfile1_uppercase
1652 textfile1_lowercase
1652 textfile1_uppercase
35 textfile2_uppercase
47 textfile1_lowercase
47 textfile1_uppercase
59 textfile2_lowercase
59 textfile2_uppercase
75 textfile2_uppercase
115 textfile2_lowercase
115 textfile2_uppercase
139 textfile1_lowercase
139 textfile1_uppercase
171 textfile1_lowercase
171 textfile1_uppercase
315 textfile2_lowercase
315 textfile2_uppercase
343 textfile2_lowercase
343 textfile2_uppercase
355 textfile1_lowercase
355 textfile1_uppercase
363 textfile1_lowercase
363 textfile1_uppercase
479 textfile1_lowercase
479 textfile1_uppercase
479 textfile2_lowercase
479 textfile2_uppercase
619 textfile1_lowercase
619 textfile1_uppercase
619 textfile2_lowercase
619 textfile2_uppercase
623 textfile2_lowercase
623 textfile2_uppercase
643 textfile1_lowercase
643 textfile1_uppercase
647 textfile2_lowercase
647 textfile2_uppercase
651 textfile2_uppercase
663 textfile1_lowercase
663 textfile1_uppercase
671 textfile1_uppercase
775 textfile1_lowercase
775 textfile1_uppercase
775 textfile2_lowercase
775 textfile2_uppercase
783 textfile1_lowercase
783 textfile1_uppercase
783 textfile2_lowercase
783 textfile2_uppercase
831 textfile1_lowercase
831 textfile1_uppercase
867 textfile1_uppercase
871 textfile1_lowercase
871 textfile1_uppercase
871 textfile2_lowercase
871 textfile2_uppercase
959 textfile1_uppercase
959 textfile2_lowercase
959 textfile2_uppercase
963 textfile1_lowercase
963 textfile1_uppercase
967 textfile2_uppercase
971 textfile2_lowercase
971 textfile2_uppercase
975 textfile1_lowercase
975 textfile1_uppercase
979 textfile1_lowercase
979 textfile1_uppercase
979 textfile2_lowercase
979 textfile2_uppercase
983 textfile1_lowercase
983 textfile1_uppercase
991 textfile1_lowercase
991 textfile1_uppercase
999 textfile2_uppercase
1003 textfile1_lowercase
1003 textfile1_uppercase
1003 textfile2_uppercase
1019 textfile1_lowercase
1019 textfile1_uppercase
1027 textfile2_uppercase
1035 textfile2_lowercase
1035 textfile2_uppercase
1039 textfile1_lowercase
1039 textfile1_uppercase
1043 textfile2_lowercase
1043 textfile2_uppercase
1051 textfile2_lowercase
1051 textfile2_uppercase
1055 textfile2_uppercase
1059 textfile2_uppercase
1063 textfile2_lowercase
1063 textfile2_uppercase
1083 textfile2_uppercase
1119 textfile2_lowercase
1119 textfile2_uppercase
1127 textfile1_uppercase
1139 textfile2_uppercase
1143 textfile2_uppercase
1155 textfile1_uppercase
1175 textfile1_lowercase
1175 textfile1_uppercase
1195 textfile2_lowercase
1195 textfile2_uppercase
1199 textfile2_uppercase
1223 textfile2_uppercase
1251 textfile2_uppercase
1255 textfile2_uppercase
1279 textfile2_lowercase
1279 textfile2_uppercase
1283 textfile2_lowercase
1283 textfile2_uppercase
1307 textfile2_uppercase
1311 textfile2_uppercase
1335 textfile2_lowercase
1335 textfile2_uppercase
1339 textfile2_uppercase
1343 textfile1_lowercase
1343 textfile1_uppercase
1363 textfile2_uppercase
1367 textfile2_uppercase
1391 textfile2_uppercase
1395 textfile2_uppercase
1399 textfile2_lowercase
1399 textfile2_uppercase
1415 textfile2_lowercase
1415 textfile2_uppercase
1419 textfile2_uppercase
1423 textfile2_uppercase
1431 textfile2_lowercase
1431 textfile2_uppercase
1447 textfile2_lowercase
1447 textfile2_uppercase
1471 textfile2_lowercase
1471 textfile2_uppercase
1475 textfile2_uppercase
1487 textfile1_lowercase
1487 textfile1_uppercase
1487 textfile2_lowercase
1487 textfile2_uppercase
1527 textfile1_lowercase
1527 textfile1_uppercase
1567 textfile1_lowercase
1567 textfile1_uppercase
1587 textfile1_uppercase
1631 textfile1_lowercase
1631 textfile1_uppercase


> The annotations are from the latest version on salami Github. The annotations have duplicate time stamps (meaning same time for two labels) are pasted below (a pretty long list).

Yeah, those look like parse/semantic errors to me, mostly stemming from the `End` marker.

The JAMS-formatted data that I linked above might be on a slightly older version of SALAMI, but you could always re-run our parse script on the most recent checkout of the data.  There's explicit logic in there to drop empty segments.  Once it's in JAMS format, you can always dump out with the `jams_to_lab.py` script.

Overall, I think this discussion is specific to the SALAMI dataset, and not a problem with mir_eval per se.  Maybe we should migrate this over to an issue on the salami repo?


Hi all,

Just coming into the middle of this...

Glancing at issue #49, where the problem was durationless intervals: that
looks like a problem in SALAMI's own parser (conversion from textfile1
https://github.com/DDMAL/salami-data-public/blob/master/annotations/988/textfile1.txt
to textfile1_functions
https://github.com/DDMAL/salami-data-public/blob/master/annotations/988/parsed/textfile1_functions.txt,
rather than an error in textfile1 per se), which ignores the possibility of
multiple function labels being applied to a single interval. If JAMS
converts from textfile1 rather than from textfile1_uppercase/etc., and does
not have this issue, then you could use the JAMS file. I see this issue as
more serious, since it's information about the type of segments that isn't
being properly assigned to the segment.

Do I understand that the general issue here is duplicate timestamps?
Looking at a few of the results, I see things like "0: silence; 0: a", and
similar things at the ends of tracks. Sometimes the "silence" tag was
considered mandatory—i.e., it was annotated even if there was just a
fraction of a second of silence at the beginning, or a silence marker was
added at the beginning even if it was effectively 'overwritten' by another
tag. Ditto for silences at the end.

But I see this second issue (about silences, as opposed to multiple
function labels) as less serious, since the timespans in question are so
tiny—fractions of a second near the beginning and end of tracks—and there
would be virtually no impact, as far as I can tell, on boundary retrieval
grades—the evaluation algorithm shouldn't be treating two boundaries at the
0 second as two separate items to retrieve, and in any case it shouldn't
care about boundaries that occur within the first or final few seconds of a
track. But maybe I haven't seen more serious examples of inconsistencies,
or don't understand the problem?

In any case, it seems like a problem that could be corrected at either the
SALAMI level or JAMS level—or, safely ignored at the user level!

Thanks for your close attention to the annotations, and help in curating
them!

By the way, it's that time of year for another 50% of the remaining private
data to go public again, so this conversation is timely. Thanks!

Jordan

On 3 February 2016 at 04:43, Brian McFee notifications@github.com wrote:

> The annotations are from the latest version on salami Github. The
> annotations have duplicate time stamps (meaning same time for two labels)
> are pasted below (a pretty long list).
> 
> Yeah, those look like parse/semantic errors to me, mostly stemming from
> the End marker.
> 
> The JAMS-formatted data that I linked above might be on a slightly older
> version of SALAMI, but you could always re-run our parse script on the most
> recent checkout of the data. There's explicit logic in there to drop empty
> segments. Once it's in JAMS format, you can always dump out with the
> jams_to_lab.py script.
> 
> Overall, I think this discussion is specific to the SALAMI dataset, and
> not a problem with mir_eval per se. Maybe we should migrate this over to an
> issue on the salami repo?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/craffel/mir_eval/issues/162#issuecomment-178782175.


Thanks for reporting this @wangsix . My .02c:
- Whatever you do, make sure you report it correctly when you present results (e.g., if you decide to ignore some redundant boundaries, include a link to a file containing the exact boundaries that are ignored). Also, in case you decide to go ahead and modify the SALAMI set, maybe it would be a good idea to also report results using the original version, in order to better compare your method against existing ones.
- The JAMS parser uses the "parsed" version of the SALAMI annotations, so I guess the problem persists in the SALAMI JAMS.
- We had to modify [some labels](https://github.com/marl/jams-data/blob/master/parsers/salami_parser.py#L25-L54) when parsing the SALAMI dataset into JAMS in order to comply with the actual SALAMI guidelines. That might be helpful for @jblsmith when producing a new SALAMI version.
- This conversation should be moved to the SALAMI repo.