abdulelahsm (CONTRIBUTOR) commented on 2020-12-18 19:02:44:
@lhoestq Hey, I tried using the first batch instead, the data was downloaded but I got this error, not sure why it can't find the path?

for content, I ran ``` "./datasets/saudinewsnet/test.py"```

which is a local test I'm running for the dataset, it contains the following code

```
from datasets import load_dataset

data = load_dataset("./datasets/saudinewsnet", split= "train")

print(data)

print(data[1])
```

this is the error I got 

```
2020-12-18 21:45:39.403908: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-12-18 21:45:39.403953: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Using custom data configuration default
Downloading and preparing dataset saudi_news_net/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/mesfas/.cache/huggingface/datasets/saudi_news_net/default/0.0.0/62ece5ef0a991415352d4b1efac681d75b5b3404064fd4f6a1d659499dab18f4...
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.42M/3.42M [00:03<00:00, 1.03MB/s]
Traceback (most recent call last):
  File "/home/mesfas/opensource/datasets/src/datasets/builder.py", line 604, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/mesfas/opensource/datasets/src/datasets/builder.py", line 902, in _prepare_split
    for key, record in utils.tqdm(
  File "/home/mesfas/environments/ar_res_reviews/lib/python3.8/site-packages/tqdm/std.py", line 1133, in __iter__
    for obj in iterable:
  File "/home/mesfas/.cache/huggingface/modules/datasets_modules/datasets/saudinewsnet/62ece5ef0a991415352d4b1efac681d75b5b3404064fd4f6a1d659499dab18f4/saudinewsnet.py", line 108, in _generate_examples
    with open(filepath, encoding="utf-8").read() as f:
IsADirectoryError: [Errno 21] Is a directory: '/home/mesfas/.cache/huggingface/datasets/downloads/extracted/314fd983aa07d3dada9429911a805270c3285f48759d3584a1343c2d86260765'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./datasets/saudinewsnet/test.py", line 3, in <module>
    data = load_dataset("./datasets/saudinewsnet", split= "train")
  File "/home/mesfas/opensource/datasets/src/datasets/load.py", line 607, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/mesfas/opensource/datasets/src/datasets/builder.py", line 526, in download_and_prepare
    self._download_and_prepare(
  File "/home/mesfas/opensource/datasets/src/datasets/builder.py", line 606, in _download_and_prepare
    raise OSError(
OSError: Cannot find data file. 
Original error:
[Errno 21] Is a directory: '/home/mesfas/.cache/huggingface/datasets/downloads/extracted/314fd983aa07d3dada9429911a805270c3285f48759d3584a1343c2d86260765'
```


this is the split code 

```
    def _split_generators(self, dl_manager):
        """Returns SplitGenerators."""
        my_urls = _URL
        datadir = dl_manager.download_and_extract(my_urls)
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN,
                # These kwargs will be passed to _generate_examples
                gen_kwargs={
                    "filepath": datadir,
                    "split": "train"
                },
            ),
        ]
```
and this is how I'm generating the examples

```
    def _generate_examples(self, filepath, split):
        
        #logging.info("generating examples from = %s", filepath)
        with open(filepath, encoding="utf-8") as f:
            articles = json.load(f)
            for article in articles:
                title = article.get("title", "").strip()
                source = article.get("source", "").strip()
                date = article.get("date_extracted", "").strip()
                link = article.get("url", "").strip()
                author = article.get("author", "").strip()
                content = article.get("content", "").strip()

                yield id_, {
                    "title": title,
                    "source": source,
                    "date": date,
                    "link": link,
                    "author": author,
                    "content": content
                }
```