[Dataset created] some critical small issues when I was creating a dataset
Opened by richarddwang (closed)

Hi, I successfully created a dataset and has made a pr #248.
But I have encountered several problems when I was creating it, and those should be easy to fix.

1. Not found dataset_info.json
should be fixed by #241 , eager to wait it be merged.

2. Forced to install `apach_beam`
If we should install it, then it might be better to include it in the pakcage dependency or specified in `CONTRIBUTING.md`
```
Traceback (most recent call last):
  File "nlp-cli", line 10, in <module>
    from nlp.commands.run_beam import RunBeamCommand
  File "/home/yisiang/nlp/src/nlp/commands/run_beam.py", line 6, in <module>
    import apache_beam as beam
ModuleNotFoundError: No module named 'apache_beam'
```

3.  `cached_dir` is `None`
```
File "/home/yisiang/nlp/src/nlp/datasets/bookscorpus/aea0bd5142d26df645a8fce23d6110bb95ecb81772bb2a1f29012e329191962c/bookscorpus.py", line 88, in _split_generators
    downloaded_path_or_paths = dl_manager.download_custom(_GDRIVE_FILE_ID, download_file_from_google_drive)
  File "/home/yisiang/nlp/src/nlp/utils/download_manager.py", line 128, in download_custom
    downloaded_path_or_paths = map_nested(url_to_downloaded_path, url_or_urls)
  File "/home/yisiang/nlp/src/nlp/utils/py_utils.py", line 172, in map_nested
    return function(data_struct)
  File "/home/yisiang/nlp/src/nlp/utils/download_manager.py", line 126, in url_to_downloaded_path
    return os.path.join(self._download_config.cache_dir, hash_url_to_filename(url))
  File "/home/yisiang/miniconda3/envs/nlppr/lib/python3.7/posixpath.py", line 80, in join
    a = os.fspath(a)
```
This is because this line
https://github.com/huggingface/nlp/blob/2e0a8639a79b1abc848cff5c669094d40bba0f63/src/nlp/commands/test.py#L30-L32
And I add `--cache_dir="...."` to `python nlp-cli test datasets/<your-dataset-folder> --save_infos --all_configs`  in the doc, finally I could pass this error.
But it seems to ignore my arg and use `/home/yisiang/.cache/huggingface/datasets/bookscorpus/plain_text/1.0.0` as cahe_dir

4. There is no `pytest`
So maybe in the doc we should specify a step to install pytest

5. Not enough capacity in my `/tmp`
When run test for dummy data, I don't know why it ask me for 5.6g to download something, 
```
def download_and_prepare
...
if not utils.has_sufficient_disk_space(self.info.size_in_bytes or 0, directory=self._cache_dir_root):
                raise IOError(
                    "Not enough disk space. Needed: {} (download: {}, generated: {})".format(
                        utils.size_str(self.info.size_in_bytes or 0),
                        utils.size_str(self.info.download_size or 0),
>                       utils.size_str(self.info.dataset_size or 0),
                    )
                )
E               OSError: Not enough disk space. Needed: 5.62 GiB (download: 1.10 GiB, generated: 4.52 GiB)
```
I add a `processed_temp_dir="some/dir"; raw_temp_dir="another/dir"` to 71, and the test passed
https://github.com/huggingface/nlp/blob/a67a6c422dece904b65d18af65f0e024e839dbe8/tests/test_dataset_common.py#L70-L72

I suggest we can create tmp dir under the `/home/user/tmp` but not `/tmp`, because take our lab server for example, everyone use `/tmp` thus it has not much capacity. Or at least we can improve error message, so the user know is what directory has no space and how many has it lefted. Or we could do both.

6. name of datasets
I was surprised by the dataset name `books_corpus`, and didn't know it is from `class BooksCorpus(nlp.GeneratorBasedBuilder)` . I change it to `Bookscorpus` afterwards. I think this point shold be also on the doc.

7. More thorough doc to how to create `dataset.py`
I believe there will be.

**Feel free to close this issue** if you think these are solved.

Comments: Thanks for noticing all these :) They should be easy to fix indeed

Alright I think I fixed all the problems you mentioned. Thanks again, that will be useful for many people.
There is still more work needed for point 7. but we plan to have some nice docs soon.