Repository: beir-cellar/beir
Issue: FIQA Empty Rows
Root cause: Documentation Deficiency

I'm not sure if this is expected but in FIQA, I'm noticing some rows are empty in the `corpus.jsonl` file

If I load it with pandas, I see the following rows as empty:

```
>>> import pandas as pd
>>> df = pd.read_json("corpus.jsonl", lines=True)
>>> df[df.text.str.len() < 1]                                                                                                                                                  
          _id title text metadata                                                                                                                                              
741      7915                  {}                                                                                                                                              
1158    12229                  {}                                                                                                                                              
1329    14135                  {}                                                                                                                                              
3208    33445                  {}                                                                                                                                              
3937    40982                  {}                                                                                                                                              
5266    54960                  {}                                                                                                                                              
6399    66248                  {}
11237  115636                  {}
11407  117276                  {}
12328  126502                  {}
13459  138418                  {}
14869  153104                  {}
16143  167128                  {}
16326  169220                  {}
18297  189754                  {}
19839  206368                  {}
19944  207473                  {}
20739  215635                  {}
22923  237392                  {}
23949  248226                  {}
24603  254541                  {}
28093  290110                  {}
30966  319894                  {}
31492  325407                  {}
34386  356535                  {}
34539  358213                  {}
34697  360125                  {}
38382  399083                  {}
39802  414219                  {}
42403  441462                  {}
43008  447488                  {}
45178  470232                  {}
46794  486245                  {}
50845  528558                  {}
53111  552533                  {}
54963  572451                  {}
56460  587667                  {}
57452  597929                  {}
```

This also seems to be prevalent in TRECCOVID in about 25% of the lines 
```
>>> len(df[df.text.str.len() < 1]) / len(df)
0.245955221441412
```

Hi @zanussbaum, 

Yes, we know a few documents are present empty in FiQA and almost 25% of documents in TREC-COVID are title only. The reason why they haven't been filtered out is because we use the original corpora for each dataset which contains these documents. 

Regards, 
Nandan

Ah okay, I misunderstood. I didn't realize the corpus got concatenated with the title as well. I was seeing odd behavior in MTEB though when we were only passing the text (or at the very least the text to be embedded was an empty string. I'll open an issue there