Repository: mir-evaluation/mir_eval
Issue: load_delimited and friends fail with quoted fields
Root cause: Validation Gap

I ran into this while testing against the mirex tempo estimation ground truth data.  CSV inputs often have quoted fields, so something like the following is not uncommon:

```
"120.0","60.0","0.5"
```

Using `float` as the column interpreter fails, since it cannot parse the " symbols.

I don't think we should rely on the end-user to specify the column interpreter here; it should work out of the box.

PS: once more for good measure, https://tburette.github.io/blog/2014/05/25/so-you-want-to-write-your-own-CSV-code/


Hm, OK.  If you have a good idea as to how to do this in a non-hacky way, feel free... maybe allow the user to specify a quote character? Because who knows, maybe there are files with the format

```
'120.0','60.0','0.5'
```


> maybe allow the user to specify a quote character?

I don't think we should be in the business of directly parsing *SV files at all, as mentioned in the discussion for #75 and in my snarkily quoted link above.

One option is to use `np.recfromtxt` as a first pass to get the data into reasonably well-formed columns, and then apply some stripping/casting afterwards.  It doesn't entirely solve all of our problems, but it at least removes the need to directly deal with encoding, delimiters, missing values, data shape, comments, etc.


If we want to have an `io` module, we need to be able to parse *sv files, because .lab files are *sv files, for example.  I don't think we need to re-write a feature-complete *sv parser, but I do think we need an *sv parser which is capable of handling file formats used in the tasks which `mir_eval` implements.  If one of the tasks commonly uses a *sv format with quote characters, we should support that.  If not, and beyond that, I'd rather the user be told that their file is invalid than try to handle all cases of *sv files.


> I don't think we need to re-write a feature-complete *sv parser, but I do think we need an *sv parser which is capable of handling file formats used in the tasks which mir_eval implements.

Yes, this is all I meant.  We need to support loading data, we do not need to reinvent the wheel.


OK, then as I said, if one of the tasks commonly uses a *sv format with quote characters, we should support that.  Is this the case?


> OK, then as I said, if one of the tasks commonly uses a *sv format with quote characters, we should support that. Is this the case?

picking this one up again..

In general, *sv files do use quote characters, and no, they're not generally consistent.  This is part of the enormous headache of writing a csv parser.  I wouldn't say that it's exactly "common" for any particular task, but I'd expect it to happen more often for tasks like segmentation, where the label space can be open and have strings with spaces.

The way I see it, we have two reasonable options here:
- add a dependency to handle parsing sanely (eg, pandas)
- do a simple best-effort implementation (what we currently have) and otherwise punt upstream to the user.

My personal preference is to offload to pandas, since it's a pretty standard package at this point.  But I understand if others don't want to add the dependency.


> In general, *sv files do use quote characters, and no, they're not generally consistent.

Well, as we discussed above `mir_eval` shouldn't include a general-purpose CSV parser, it should include a parser for the formats used for each of the tasks included, which is a subset of CSV-land.  I am comfortable telling someone that their tempo annotation file is not valid because it has quotes in it and then letting them either fix their files or use the parser of their choice instead of `mir_eval`'s parser.  If there are a few CSV edge-cases that are part of the imaginary "lab file spec" that we don't cover, then we should add functionality for them.  I think adding pandas (or another fully-featured library that has a nice CSV parser) as a dependency is over the top.  I'm going to continue to stand by `mir_eval`'s goal of being lightweight and requiring as few dependencies as possible (I, for one, have never used or installed pandas.  Not that I have anything against it, I just never have.)


Ok.  Well in that case, I think this issue is a no-op.

(PS: you should install pandas. and the evaluators should return dataframes instead of ordereddicts. :))


> Ok. Well in that case, I think this issue is a no-op.

Ok, closing.

> (PS: you should install pandas. and the evaluators should return dataframes instead of ordereddicts. :))

Not using pandas is the only thing maintaining the fragile boundary between the work I do and statistics.


> Not using pandas is the only thing maintaining the fragile boundary between the work I do and statistics.

What's the line... the opposite of "statistics" isn't "no statistics" but "bad statistics"?  (I kid, i kid...)


Well, like I said, it's a fragile margin - so if anything, it's "marginally bad statistics".  I can live with that.