t20062443/AT1ds huggingface dataset:
YipingNUS (NONE) commented on 2020-12-17 03:05:09:
Thanks for your quick response! I'm fine with specifying the split as @lhoestq suggested. My only concern is when I'm loading from python dict or pandas, the library returns a dataset instead of a dictionary of datasets when no split is specified. I know that they use a different function `Dataset.from_dict` or `Dataset.from_pandas` but the text/csv files use `load_dataset()`. However, to the user, they do the same task and we probably expect them to have the same behavior.
Narsil (CONTRIBUTOR) commented on 2020-12-17 08:58:52:
@yjernite Ready for review !
@thomwolf 

So I tried to follow closely the original format that means I still had to drop information (namely tags on elements are simply discarded for now but they don't seem to carry critical information).
Some elements are also discarded they tend to not come up often:
 - The most notable is Author affiliation, which seems to be all over the place in terms of what it look meaning it's hard to actually get a consistent format.
 - Journal is the same, all the elements in there can be wildly different so I decided to drop it for now instead of trying to figure out a way to have a common presentation. (the DOI and medline ID are present so it can be recovered).

I think this PR could go as it but we probably should add a way to get easier information to use with a config.
For instance `{"title": "string", "abstract": "string", "authors": List[str], "substances": List[str]}` maybe ? (substances for instance is a tricky one, some substances only have an international identifier, some have simply a common name, some both)

It's relatively easy to do I think it's mostly discarding other fields and renaming some deep structure into a flat one.
jramapuram (NONE) commented on 2020-12-17 10:36:57:
I was thinking about using something like [importlib](https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly) to over-ride the collision. 

**Reason requested**: I use the [following template](https://github.com/jramapuram/ml_base/) repo where I house all my datasets as a submodule.
davidefiocco (NONE) commented on 2020-12-17 18:13:43:
I can't speak for the HF team @jramapuram, but as member of the community it looks to me that HF wanted to avoid the past path of changing names as scope broadened over time:

Remember
https://github.com/huggingface/pytorch-openai-transformer-lm
https://github.com/huggingface/pytorch-pretrained-BERT
https://github.com/huggingface/pytorch-transformers
and now
https://github.com/huggingface/transformers

;) 

Jokes aside, seems that the library is growing in a multi-modal direction (https://github.com/huggingface/datasets/pull/363) so the current name is not that implausible. Possibly HF ambition is really to grow its community and bring here a large chunk of datasets of the world (including tabular / vision / audio?).
SBrandeis (CONTRIBUTOR) commented on 2020-12-18 08:03:02:
Hi @david-waterworth!

As indicated in the error message, `load_dataset("csv")` returns a `DatasetDict` object, which is mapping of `str` to `Dataset` objects. I believe in this case the behavior is to return a `train` split with all the data.
`train_test_split` is a method of the `Dataset` object, so you will need to do something like this:
```python
dataset_dict = load_dataset(`'csv', data_files='data.txt')
dataset = dataset_dict['split name, eg train']
dataset.train_test_split(test_size=0.1)
```

Please let me know if this helps. 🙂
yavuzKomecoglu (CONTRIBUTOR) commented on 2020-12-18 09:32:14:
> I think you accidentally pushed the readme of another dataset (name_to_nation).
> I removed it so you have to `git pull`
> 
> Because of that I guess your changes about the ttc4900 was not included.
> Feel free to ping me once they're added

I did `git pull` and updated readme **ttc4900**.
lhoestq (MEMBER) commented on 2020-12-18 15:27:14:
I see ! This is because the library did some verification to make sure it downloads the same files as in the first time you ran the `datasets-cli test` command with `--save_infos`. Since we're now downloading files, the verification fails. 

To fix that you just need to regenerate the dataset_infos.json file:
```
datasets-cli test ./datasets/wiki_summary --save_infos --all_configs --ignore_verifications
```
lhoestq (MEMBER) commented on 2020-12-18 16:27:02:
Hi @abdulelahsm ! Thanks for adding this one :) 
you don't actually have to add builder configurations if you don't need them. It's fine as it is now

And as @skyprince999 noticed, the current URLs don't work. to download files.
You can use this one for example for the first batch instead:
https://github.com/parallelfold/SaudiNewsNet/raw/master/dataset/2015-07-21.zip

Feel free to ping me if you have questions or if you're ready for a review :)
rabeehkarimimahabadi (NONE) commented on 2020-12-20 00:52:16:
Hi Thomas
thanks for reponse, yes, I did checked it, but this does not work for me please see 

```
(internship) rkarimi@italix17:/idiap/user/rkarimi/dev$ python 
Python 3.7.9 (default, Aug 31 2020, 12:42:55) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import datasets 
2020-12-20 01:48:50.766004: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2020-12-20 01:48:50.766029: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
>>> data = datasets.load_dataset("scitail", "snli_format")
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
cahce dir  /idiap/temp/rkarimi/cache_home_1/datasets
Reusing dataset scitail (/idiap/temp/rkarimi/cache_home_1/datasets/scitail/snli_format/1.1.0/fd8ccdfc3134ce86eb4ef10ba7f21ee2a125c946e26bb1dd3625fe74f48d3b90)
>>> data.shuffle(seed=2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: shuffle() got an unexpected keyword argument 'seed'

```

datasets version
`datasets                  1.1.2                     <pip>
`
katnoria (CONTRIBUTOR) commented on 2020-12-20 08:47:37:
> Awesome thank you !
> 
> I just left one comment.
> 
> Also it looks like the dummy_data.zip files are quite big (around 500KB each)
> Can you try to reduce their sizes please ? Ideally they should be <20KB each
> 
> To do so feel free to take a look inside them and in the jsonl files only keep 1 or 2 samples instead of 5 and also remove big chunks of text to only keep a few passages.

Thanks, I have updated the dummy data to keep each domain <20/30KB.
katnoria (CONTRIBUTOR) commented on 2020-12-20 08:51:26:
> > Awesome thank you !
> > I just left one comment.
> > Also it looks like the dummy_data.zip files are quite big (around 500KB each)
> > Can you try to reduce their sizes please ? Ideally they should be <20KB each
> > To do so feel free to take a look inside them and in the jsonl files only keep 1 or 2 samples instead of 5 and also remove big chunks of text to only keep a few passages.
> 
> Thanks, I have updated the dummy data to keep each domain <20/30KB.

Looks like this branch has other commits. I will open a new PR with suggested changes.
ncoop57 (NONE) commented on 2020-12-21 01:25:10:
> Ok I found the issue with the dummy data.
> It's currently failing because it's not generating a single example using the dummy csv file.
> That's because there's only only line in the dummy csv file, and this line is skipped using the `next()` call used to ignore the headers of the csv.
> 
> To fix the dummy data you must add headers to the dummy csv files.

Oh man, I bamboozled myself! Thank you @lhoestq for catching that! I've updated the dummy csv's to include headers and also confirmed that they all have headers, so I am not throwing away any information with that `next()` call. When I run the test locally for the dummy data it passes, so hopefully it is good to go :D
zaidalyafeai (CONTRIBUTOR) commented on 2020-12-21 04:56:00:
Can't pass dummy data tests. For the instructions, it asks me to generate the following file `dummy_data/Youm7_XML_utf_8.rar/Youm7_utf_8.xml` which is strange, any ideas @lhoestq ?

cc: I tested the data locally and it works, maybe the dummy tests doesn't support `rar` ?
SBrandeis (CONTRIBUTOR) commented on 2020-12-21 08:03:46:
Hi @jassimran,
The `jigsaw_toxicity_pred` dataset has not been released yet, it will be available with version 2 of `datasets`, coming soon.
You can still access it by installing the master (unreleased) version of datasets directly :
`pip install git+https://github.com/huggingface/datasets.git@master`
Please let me know if this helps
lhoestq (MEMBER) commented on 2020-12-21 09:59:41:
Thanks for the changes !

Sorry if I wasn't clear about the suggestion of adding the `raw` dataset as well.
By `raw` I meant the dataset with its original features, i.e. not tokenized to follow the conll format for NER.
The `raw` dataset has data fields `doc_text`, `doc_id` and `entities`.
lhoestq (MEMBER) commented on 2020-12-21 10:25:31:
Sorry for the delay, I was busy with the dataset sprint and the incredible amount of contributions to the library ^^'

What you can try to do to find what's wrong is check at which frequency the arrow writer writes all the examples from its in-memory buffer on disk. This happens [here](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_writer.py#L257-L258) in the code.

The idea is that `write_on_file` writes the examples every `writer_batch_size` examples and clear the buffer `self. current_rows`. As soon as `writer_batch_size` is small enough you shouldn't have memory issues in theory.

Let me know if you have questions or if I can help.

Since the dataset sprint is over and I will also be done with all the PRs soon I will be able to go back at it and take a look.
lhoestq (MEMBER) commented on 2020-12-21 10:37:17:
Hi !
Thanks for the changes :)

The only error left is the dummy data. Since we changed for standard downloads instead of manual downloads its structure changed. Fortunately you can auto-generate the dummy data with this command:

```
datasets-cli dummy_data ./datasets/poleval2019_mt --auto_generate --match_text_files "*"
```

Can you regenerate the dummy data using this command please ?
jonatasgrosman (CONTRIBUTOR) commented on 2020-12-21 11:17:13:
Alright, @lhoestq, now I understand your suggestion, but the JSON files the script downloads aren't actually the "raw" HAREM dataset, the [real raw version](https://www.linguateca.pt/primeiroHAREM/harem/ColeccaoDouradaHAREM.zip) of it is on XML format that needs a lot of preprocessing. Those JSON are just pre-processed versions of the dataset.

I can make the config of the raw version of the pre-processed dataset, but I'll leave a comment on the dataset summary about those details.
pietrolesci (NONE) commented on 2020-12-21 13:20:14:
Hi @SBrandeis,

Thanks a lot for picking up my request. 

Maybe I can clarify my use-case with a bit of context. Say I have the IMDb dataset. I create an ES index on it. Now I can save and reload the dataset from disk normally. Once I reload the dataset, it is easy to retrieve the ES index on my machine. I was wondering: is there a way I can share the (now) indexed version of the IMDb dataset with my colleagues without requiring them to re-index it?

Thanks a lot in advance for your consideration.

Best,

Pietro
dorlavie (NONE) commented on 2020-12-21 15:30:09:
> Hi dorlavie,
> You can find one solution that i have mentioned above, that can help you.
> And there is one more solution also which is downloading files locally

mahesh1amour, thanks for the fast reply

Unfortunately, in my case I can not read with pandas. The dataset is too big (50GB). 
In addition, due to security concerns I am not allowed to save the data locally
rabeehkarimimahabadi (NONE) commented on 2020-12-21 15:35:49:
@lhoestq  let me please expalin in more details, maybe you could help me suggesting an alternative to solve the issue for now, I have multiple large datasets using huggingface library, then I need to define a distributed sampler on top of it, for this I need to shard the datasets and give each shard to each core, but before sharding I need to shuffle the dataset, if you are familiar with distributed sampler in pytorch, this needs to be done based on seed+epoch generator to make it consistent across the cores they do it through defining a torch generator, I was wondering if you could tell me how I can shuffle the data for now, I am unfortunately blocked by this and have a limited time left, and I greatly appreciate your help on this. thanks
lhoestq (MEMBER) commented on 2020-12-21 15:45:21:
Right now the shuffle method only accepts the `seed` (optional int) or `generator` (optional `np.random.Generator`) parameters.

Here is a suggestion to shuffle the data using your own shuffle method using `select`.
`select` can be used to re-order the dataset samples or simply pick a few ones if you want.
It's what is used under the hood when you call `dataset.shuffle`.

To use `select` you must have the list of re-ordered indices of your samples.

Let's say you have a `shuffle` methods that you want to use. Then you can first build your shuffled list of indices:
```python
shuffled_indices = shuffle(range(len(dataset)))
```

Then you can shuffle your dataset using the shuffled indices with 
```python
shuffled_dataset = dataset.select(shuffled_indices)
```

Hope that helps
philschmid (CONTRIBUTOR) commented on 2020-12-21 16:11:36:
@dorlavie could use `boto3` to download the data to your local machine and then load it with `dataset`

boto3 example [documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-example-download-file.html)
```python
import boto3

s3 = boto3.client('s3')
s3.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME')
```

datasets example [documentation](https://huggingface.co/docs/datasets/loading_datasets.html)

```python
from datasets import load_dataset
dataset = load_dataset('csv', data_files=['my_file_1.csv', 'my_file_2.csv', 'my_file_3.csv'])
```
dorlavie (NONE) commented on 2020-12-21 16:38:02:
Thanks @philschmid for the suggestion.
As I mentioned in the previous comment, due to security issues I can not save the data locally.
I need to read it from S3 and process it directly.

I guess that many other people try to train / fit those models on huge datasets (e.g entire Wiki), what is the best practice in those cases?
zaidalyafeai (CONTRIBUTOR) commented on 2020-12-21 16:54:40:
Sorry it was my mistake, I missed up the directories, it works now. Not sure why you got that error. I have the same version of `rarfile`. Between, there were some suggestions to change the dataset from `1bn_words_arabic` to `arabic_billion_words` like https://github.com/huggingface/datasets/tree/master/datasets/spanish_billion_words.
lhoestq (MEMBER) commented on 2020-12-21 17:13:10:
I'm ok with renaming the dataset `arabic_billion_words` if you want.
Note that you will need to rename class name `ArabicBillionWords` instead of `BillionWords`
(though `BillionWords` was not matching `1bn_words_arabic` anyway)

You will need to regenerate the dataset_infos.json file after this change.
OR alternatively just replace all mentions of `billion_words` with `arabic_billion_words` in dataset_infos.json <- this trick should save you some time :)
lhoestq (MEMBER) commented on 2020-12-21 17:36:27:
Hmmm I'm still not able to run it on my side because of the rar error (I'm running macos)
I just tried with rarfile 3.1 and it didn't work either.
I would like to be able to run it end-to-end on my side before merging if you don't mind. Let me investigate this issue a little bit
bhavitvyamalik (CONTRIBUTOR) commented on 2020-12-21 18:20:37:
Hi @SapirWeissbuch,
When you are saying it freezes, at that time it is unzipping the file from the zip file it downloaded. Since it's a very heavy file it'll take some time. It was taking ~11GB after unzipping when it started reading examples for me. Hope that helps!
![Screenshot 2020-12-21 at 23 40 52](https://user-images.githubusercontent.com/19718818/102808355-3b380c00-43e6-11eb-81ab-c31019ae6322.png)
bhavitvyamalik (CONTRIBUTOR) commented on 2020-12-21 19:25:46:
I'm using datasets version: 1.1.3. I think you should drop `cache_dir` and use only
`dataset = datasets.load_dataset("trivia_qa", "rc")`

Tried that on colab and it's working there too
![image](https://user-images.githubusercontent.com/19718818/102814269-4db74300-43f0-11eb-8f26-ecfcf4632002.png)
SapirWeissbuch (NONE) commented on 2020-12-21 22:52:13:
I tried running this again - More details of the problem:
Code:
```
datasets.load_dataset("trivia_qa", "rc", cache_dir="/path/to/cache")
```

The output:
```
Downloading and preparing dataset trivia_qa/rc (download: 2.48 GiB, generated: 14.92 GiB, post-processed: Unknown size, total: 17.40 GiB) to path/to/cache/trivia_qa/rc/1.1.0/e734e28133f4d9a353af322aa52b9f266f6f27cbf2f072690a1694e577546b0d...                                                                          
Downloading: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2.67G/2.67G [03:38<00:00, 12.2MB/s]

```
The process continues (no progress bar is visible).
I tried `du -sh .` in `path/to/cache`, and the size keeps increasing, reached 35G before I killed the process.

Google Colab with custom `cache_dir` has same issue.
https://colab.research.google.com/drive/1nn1Lw02GhfGFylzbS2j6yksGjPo7kkN-?usp=sharing#scrollTo=2G2O0AeNIXan
bhavitvyamalik (CONTRIBUTOR) commented on 2020-12-22 05:24:05:
1) You can clear the huggingface folder in your `.cache` directory to use default directory for datasets. Speed of extraction and loading of samples depends a lot on your machine's configurations too.

2) I tried on colab `dataset = datasets.load_dataset("trivia_qa", "rc", cache_dir = "./datasets")`. After memory usage reached around 42GB (starting from 32GB used already), the dataset was loaded in the memory. Even Your colab notebook shows 
![image](https://user-images.githubusercontent.com/19718818/102852229-c7c4e780-4443-11eb-91d6-bf21024358a3.png)
which means it's loaded now.
SBrandeis (CONTRIBUTOR) commented on 2020-12-22 07:50:56:
Thanks for the clarification.

I am not familiar with ElasticSearch, but if I understand well you're trying to migrate your data along with the ES index.
My advice would be to check out ES documentation, for instance, this might help you: https://www.elastic.co/guide/en/cloud/current/ec-migrate-data.html

Let me know if it helps
tdiggelm (CONTRIBUTOR) commented on 2020-12-22 09:52:38:
> Hi @tdiggelm, thanks for the contribution! This dataset is really awesome.
> I believe creating a new branch from master and opening a new PR with your changes is the simplest option since no review has been done yet. Feel free to ping us when it's done.

Thank you very much for your quick reply! Will do ASAP and ping you when done.
lhoestq (MEMBER) commented on 2020-12-22 16:59:09:
Hi @tanmoyio indeed you're right.
We should *at least* add very explicit mentions in the dataset card that the content of this dataset contains very offensive language. We should also put it in perspective with the tasks it tries to solve, the annotation process and the limitations.

We have to make sure that nothing is unclear/misleading nor could lead to bad usage of the dataset.

What do you think @tanmoyio ?
Also feel free to suggest modifications in the dataset cards if you feel like some sections require corrections or more details
lhoestq (MEMBER) commented on 2020-12-22 17:26:03:
Hi ! Maybe we can rename it `narrativeqa_manual` to make it explicit that this one requires manual download contrary to `narrativeqa` ?
It's important to have this one as well, in case the `narrativeqa` one suffers from download issues (checksums or dead links for example).

You can also copy the dataset card from `narrativeqa` and add the dummy data as well
tanmoyio (CONTRIBUTOR) commented on 2020-12-22 18:24:09:
> Hi @tanmoyio indeed you're right.
> We should _at least_ add very explicit mentions in the dataset card that the content of this dataset contains very offensive language. We should also put it in perspective with the tasks it tries to solve, the annotation process and the limitations.
> 
> We have to make sure that nothing is unclear/misleading nor could lead to bad usage of the dataset.
> 
> What do you think @tanmoyio ?
> Also feel free to suggest modifications in the dataset cards if you feel like some sections require corrections or more details

yeah I agree with you. It would be good if "Personal and Sensitive Information" and "Considerations for Using the Data" is being explained properly in the README.md. @stevhliu
Phil1108 (NONE) commented on 2020-12-22 19:01:37:
> 
> 
> Hi @Phil1108 !
> Have you had a chance to take a look at my suggestions ?
> Feel free to ping me if you have questions or if you're ready for a review
> 
> Thanks again for adding this dataset, this one is very useful !

Hello @lhoestq , 

okay so should we go on with only the 2 parts or wait until the hosting of all 200 GB is cleared at HF? 
If you want to merge it with the 2 parts only at first und upgrade later to all, sure I'll include the minor changes in the next few days and update it here
philschmid (CONTRIBUTOR) commented on 2020-12-23 09:53:08:
@dorlavie are you using sagemaker for training too? Then you could use S3 URI, for example `s3://my-bucket/my-training-data` and pass it within the `.fit()` function when you start the sagemaker training job. Sagemaker would then download the data from s3 into the training runtime and you could load it from disk

**sagemaker start training job**
```python
pytorch_estimator.fit({'train':'s3://my-bucket/my-training-data','eval':'s3://my-bucket/my-evaluation-data'})
```

**in the train.py script**
```python
from datasets import load_from_disk

train_dataset = load_from_disk(os.environ['SM_CHANNEL_TRAIN'])
```

I have created an example of how to use transformers and datasets with sagemaker. 
https://github.com/philschmid/huggingface-sagemaker-example/tree/main/03_huggingface_sagemaker_trainer_with_data_from_s3

The example contains a jupyter notebook `sagemaker-example.ipynb` and an `src/` folder. The sagemaker-example is a jupyter notebook that is used to create the training job on AWS Sagemaker. The `src/` folder contains the `train.py`, our training script, and `requirements.txt` for additional dependencies.
lhoestq (MEMBER) commented on 2020-12-23 09:59:47:
I think we can wait a bit to find a solution for the hosting and update the PR to include all the files when it's done.
In the meantime you can update the PR to take into account the suggestions regarding the dataset card, the parallel downloads and the use of eval.

A safer alternative to eval is `ast.literal_eval`, it should do the job
bhavitvyamalik (CONTRIBUTOR) commented on 2020-12-23 13:37:16:
I think we’ll need complete dataset_infos.json to create YAML tags. I ran the script again with 100 files after going through your comments and it was occupying ~16 GB space. So in total it should take ~960GB and I don’t have this much memory available with me. Also, I'll have to download the whole dataset for generating dummy data, right?
lhoestq (MEMBER) commented on 2020-12-23 13:46:38:
This library uses Apache Arrow under the hood to store datasets on disk.
The advantage of Apache Arrow is that it allows to memory map the dataset. This allows to load datasets bigger than memory and with almost no RAM usage. It also offers excellent I/O speed.

For example when you access one element or one batch
```python
from datasets import load_dataset

squad = load_dataset("squad", split="train")
first_element = squad[0]
one_batch = squad[:8]
```

then only this element/batch is loaded in memory, while the rest of the dataset is memory mapped.