t20062443/AT1ds huggingface dataset:
chiyuzhang94 (NONE) commented on 2020-09-17 05:43:28:
I solved the above issue by downloading text.py manually and passing the path to the `load_dataset` function. 

Now, I have a new issue with the Read-only file system.

The error is: 
```
I0916 22:14:38.453380 140737353971520 filelock.py:274] Lock 140734268996072 acquired on /scratch/chiyuzh/roberta/text.py.lock
Found main folder for dataset /scratch/chiyuzh/roberta/text.py at /home/chiyuzh/.cache/huggingface/modules/datasets_modules/datasets/text
Creating specific version folder for dataset /scratch/chiyuzh/roberta/text.py at /home/chiyuzh/.cache/huggingface/modules/datasets_modules/datasets/text/512f465342e4f4cd07a8791428a629c043bb89d55ad7817cbf7fcc649178b014
I0916 22:14:38.530371 140737353971520 filelock.py:318] Lock 140734268996072 released on /scratch/chiyuzh/roberta/text.py.lock
Traceback (most recent call last):
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 1019, in <module>
    main()
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 962, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 177, in load_and_cache_examples
    return HG_Datasets(tokenizer, file_path, args)
  File "/scratch/chiyuzh/roberta/run_language_modeling_hg.py", line 117, in HG_Datasets
    dataset = load_dataset('/scratch/chiyuzh/roberta/text.py', data_files=files, cache_dir = args.data_cache_dir, split="train")
  File "/arc/project/chiyuzh/evn_py36/datasets/src/datasets/load.py", line 590, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/arc/project/chiyuzh/evn_py36/datasets/src/datasets/load.py", line 385, in prepare_module
    os.makedirs(hash_folder_path)
  File "/project/chiyuzh/evn_py36/lib/python3.6/os.py", line 220, in makedirs
    mkdir(name, mode)
OSError: [Errno 30] Read-only file system: '/home/chiyuzh/.cache/huggingface/modules/datasets_modules/datasets/text/512f465342e4f4cd07a8791428a629c043bb89d55ad7817cbf7fcc649178b014'

```

I installed datasets at /project/chiyuzh/evn_py36/datasets/src where is a writable directory.
I also tried change the environment variables to the writable directory:
`export HF_MODULES_PATH=/project/chiyuzh/evn_py36/datasets/cache_dir/`
`export HF_DATASETS_CACHE=/project/chiyuzh/evn_py36/datasets/cache_dir/`
 
In my scripts, I also changed to:
`dataset = load_dataset('/scratch/chiyuzh/roberta/text.py', data_files=files, cache_dir = args.data_cache_dir, split="train")`
`data_cache_dir = $TMPDIR/data/` that also a writable directory.
 
But it still try to make directory at /home/chiyuzh/.cache/huggingface/modules/.
Do you have any idea about this issue? @thomwolf
shizhediao (NONE) commented on 2020-09-17 06:00:57:
> Hey @chiyuzhang94, I was also having trouble in loading a large text file (11GB).
> But finally got it working. This is what I did after looking into the documentation.
> 
> 1. split the whole dataset file into smaller files
> 
> ```shell
> mkdir ./shards
> split -a 4 -l 256000 -d full_raw_corpus.txt ./shards/shard_
> ```
> 
> 1. Pass paths of small data files to `load_dataset`
> 
> ```python
> files = glob.glob('shards/*')
> from datasets import load_dataset
> dataset = load_dataset('text', data_files=files, split='train')
> ```
> 
> (On passing the whole dataset file (11GB) directly to `load_dataset` was resulting into RAM issue)
> 
> 1. Tokenization
> 
> ```python
> def encode(examples):
>   return tokenizer(examples['text'], truncation=True, padding='max_length')
> dataset = dataset.map(encode, batched=True)
> dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])
> ```
> 
> Now you can pass `dataset` to `Trainer` or `pytorch DataLoader`
> 
> ```python
> dataloader = torch.utils.data.DataLoader(dataset, batch_size=4)
> next(iter(dataloader))
> ```
> 
> Hope this helps

When I run 'dataset = dataset.map(encode, batched=True)',
I encountered a problem like this:

> Testing the mapped function outputs
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/dataset_dict.py", line 300, in map
    for k, dataset in self.items()
  File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/dataset_dict.py", line 300, in <dictcomp>
    for k, dataset in self.items()
  File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1224, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1195, in does_function_return_dict
    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "<stdin>", line 3, in encode
TypeError: __init__() takes 1 positional argument but 2 were given
chiyuzhang94 (NONE) commented on 2020-09-17 06:03:27:
> > Hey @chiyuzhang94, I was also having trouble in loading a large text file (11GB).
> > But finally got it working. This is what I did after looking into the documentation.
> > 
> > 1. split the whole dataset file into smaller files
> > 
> > ```shell
> > mkdir ./shards
> > split -a 4 -l 256000 -d full_raw_corpus.txt ./shards/shard_
> > ```
> > 
> > 
> > 
> > 1. Pass paths of small data files to `load_dataset`
> > 
> > ```python
> > files = glob.glob('shards/*')
> > from datasets import load_dataset
> > dataset = load_dataset('text', data_files=files, split='train')
> > ```
> > 
> > 
> > (On passing the whole dataset file (11GB) directly to `load_dataset` was resulting into RAM issue)
> > 
> > 1. Tokenization
> > 
> > ```python
> > def encode(examples):
> >   return tokenizer(examples['text'], truncation=True, padding='max_length')
> > dataset = dataset.map(encode, batched=True)
> > dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])
> > ```
> > 
> > 
> > Now you can pass `dataset` to `Trainer` or `pytorch DataLoader`
> > ```python
> > dataloader = torch.utils.data.DataLoader(dataset, batch_size=4)
> > next(iter(dataloader))
> > ```
> > 
> > 
> > Hope this helps
> 
> When I run 'dataset = dataset.map(encode, batched=True)',
> I encountered a problem like this:
> 
> > Testing the mapped function outputs
> > Traceback (most recent call last):
> > File "", line 1, in 
> > File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/dataset_dict.py", line 300, in map
> > for k, dataset in self.items()
> > File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/dataset_dict.py", line 300, in 
> > for k, dataset in self.items()
> > File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1224, in map
> > update_data = does_function_return_dict(test_inputs, test_indices)
> > File "/anaconda3/envs/torch-xla-1.6/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1195, in does_function_return_dict
> > function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
> > File "", line 3, in encode
> > TypeError: **init**() takes 1 positional argument but 2 were given

What is your encoder function?
chiyuzhang94 (NONE) commented on 2020-09-17 06:09:51:
> > ```python
> > def encode(examples):
> >   return tokenizer(examples['text'], truncation=True, padding='max_length')
> > ```
> 
> It is the same as suggested:
> 
> > def encode(examples):
> > return tokenizer(examples['text'], truncation=True, padding='max_length')

Do you use this function in a `class` object?  

init() takes 1 positional argument but 2 were given. I guess the additional argument is self?
shizhediao (NONE) commented on 2020-09-17 06:16:03:
> > > ```python
> > > def encode(examples):
> > >   return tokenizer(examples['text'], truncation=True, padding='max_length')
> > > ```
> > 
> > 
> > It is the same as suggested:
> > > def encode(examples):
> > > return tokenizer(examples['text'], truncation=True, padding='max_length')
> 
> Do you use this function in a `class` object?
> 
> init() takes 1 positional argument but 2 were given. I guess the additional argument is self?

Thanks for your reply.
Could you provide some simple example here?
Currently, I do not use this function in a class object. 
I think you are right and I was wondering how to construct this class.
I try to modify it based on transformers' LineByLineTextDataset. Am I correct?

> class LineByLineTextDataset(Dataset):
    """
    This will be superseded by a framework-agnostic approach
    soon.
    """

    def __init__(self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int):
        assert os.path.isfile(file_path), f"Input file path {file_path} not found"
        # Here, we do not cache the features, operating under the assumption
        # that we will soon use fast multithreaded tokenizers from the
        # `tokenizers` repo everywhere =)
        #logger.info("Creating features from dataset file at %s", file_path)
        #with open(file_path, encoding="utf-8") as f:
        #    lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]
        #batch_encoding = tokenizer(lines, add_special_tokens=True, truncation=True, max_length=block_size)

	import glob
	files = glob.glob('/home/mtzhang111/fairseq/cs_doc/shards/shard_003*')
	from datasets import load_dataset
	dataset = load_dataset('text', data_files=files)
        batch_encoding= dataset.map(encode, batched=True)
        self.examples = batch_encoding["input_ids"]
	

    def encode(examples):
        return tokenizer(examples['text'], truncation=True, padding='max_length')

    def __len__(self):
        return len(self.examples)

    def __getitem__(self, i) -> torch.Tensor:
        return torch.tensor(self.examples[i], dtype=torch.long)
chiyuzhang94 (NONE) commented on 2020-09-17 06:43:21:
> > > > ```python
> > > > def encode(examples):
> > > >   return tokenizer(examples['text'], truncation=True, padding='max_length')
> > > > ```
> > > 
> > > 
> > > It is the same as suggested:
> > > > def encode(examples):
> > > > return tokenizer(examples['text'], truncation=True, padding='max_length')
> > 
> > 
> > Do you use this function in a `class` object?
> > init() takes 1 positional argument but 2 were given. I guess the additional argument is self?
> 
> Thanks for your reply.
> Could you provide some simple example here?
> Currently, I do not use this function in a class object.
> I think you are right and I was wondering how to construct this class.
> I try to modify it based on transformers' LineByLineTextDataset. Am I correct?
> 
> > class LineByLineTextDataset(Dataset):
> > """
> > This will be superseded by a framework-agnostic approach
> > soon.
> > """
> 
> ```
> def __init__(self, tokenizer: PreTrainedTokenizer, file_path: str, block_size: int):
>     assert os.path.isfile(file_path), f"Input file path {file_path} not found"
>     # Here, we do not cache the features, operating under the assumption
>     # that we will soon use fast multithreaded tokenizers from the
>     # `tokenizers` repo everywhere =)
>     #logger.info("Creating features from dataset file at %s", file_path)
>     #with open(file_path, encoding="utf-8") as f:
>     #    lines = [line for line in f.read().splitlines() if (len(line) > 0 and not line.isspace())]
>     #batch_encoding = tokenizer(lines, add_special_tokens=True, truncation=True, max_length=block_size)
> 
> import glob
> files = glob.glob('/home/mtzhang111/fairseq/cs_doc/shards/shard_003*')
> from datasets import load_dataset
> dataset = load_dataset('text', data_files=files)
>     batch_encoding= dataset.map(encode, batched=True)
>     self.examples = batch_encoding["input_ids"]
> 
> 
> def encode(examples):
>     return tokenizer(examples['text'], truncation=True, padding='max_length')
> 
> def __len__(self):
>     return len(self.examples)
> 
> def __getitem__(self, i) -> torch.Tensor:
>     return torch.tensor(self.examples[i], dtype=torch.long)
> ```

I am also struggling with this adaptation. 
I am not sure whether I am right.

I think you don't need to construct `class LazyLineByLineTextDataset(Dataset)` at all. 
torch.utils.data.Dataset is a generator. 

Now, we use `dataset = dataset.map(encode, batched=True)` as a generator. So we just pass dataset to torch.utils.data.DataLoader.
lhoestq (MEMBER) commented on 2020-09-17 10:05:22:
Hi ! sorry for the late response

Could you try to rebase from master ? We changed the named of the library last week so you have to include this change in your code.

Can you give me more details about the error you get when running the cli command ?

Note that in case of a manual download you have to specify the directory where you downloaded the data with `--data_dir <path/to/the/directory>`
lhoestq (MEMBER) commented on 2020-09-17 10:46:30:
Thanks for reporting.


Which tokenizers are you using ? What platform are you on ? Can you tell me which version of datasets and pyarrow you're using ? @timothyjlaurent @richarddwang @HuangLianzhe 

Also if you're able to reproduce the issue on google colab that would be very helpful.

I tried to run your code @richarddwang with the bert tokenizer and I wasn't able to reproduce
lhoestq (MEMBER) commented on 2020-09-17 15:32:42:
Currently `csv` doesn't support the `features` attribute (unlike `json`).
What you can do for now is cast the features using the in-place transform `cast_`

```python
from datasets import load_dataset

dataset = load_dataset('csv', data_files=file_dict, delimiter=';', column_names=['text', 'label'])
dataset.cast_(emotion_features)
```
stas00 (CONTRIBUTOR) commented on 2020-09-17 15:40:03:
Awesome! Thank you!

Any ideas how to eliminate this remaining log line from tensorflow (I know it's not `datasets` related, but perhaps you know).
```
python -c "import transformers"
2020-09-17 08:38:34.718410: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
```
chiyuzhang94 (NONE) commented on 2020-09-17 17:27:23:
Hi @shizhediao ,

Thanks! It looks great!

But my problem still is the cache directory is a read-only file system. 
[As I mentioned](https://github.com/huggingface/datasets/issues/610#issuecomment-693912285), I tried to change the cache directory but it didn't work. 

Do you have any suggestions?
ksjae (NONE) commented on 2020-09-17 22:23:53:
From transformers/examples/language-modeling/run-language-modeling.py :
```
def get_dataset(
    args: DataTrainingArguments,
    tokenizer: PreTrainedTokenizer,
    evaluate: bool = False,
    cache_dir: Optional[str] = None,
):
    file_path = args.eval_data_file if evaluate else args.train_data_file
    if True:
        dataset = load_dataset("text", data_files=glob.glob(file_path), split='train', use_threads=True, 
        ignore_verifications=True, save_infos=True, block_size=104857600)
        dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                                truncation=True, max_length=args.block_size), batched=True)
        dataset.set_format(type='torch', columns=['input_ids'])
        return dataset
    if args.line_by_line:
        return LineByLineTextDataset(tokenizer=tokenizer, file_path=file_path, block_size=args.block_size)
    else:
        return TextDataset(
            tokenizer=tokenizer,
            file_path=file_path,
            block_size=args.block_size,
            overwrite_cache=args.overwrite_cache,
            cache_dir=cache_dir,
        )
```

No, I'm not using multiprocessing.
stefan-it (CONTRIBUTOR) commented on 2020-09-18 14:13:26:
Hi @joeddav thanks for adding this! (I did a long webarchive.org session to actually find that dataset a while ago).

One question: should we manually correct the labeling scheme to (at least) IOB1?

That means "LOC" will be converted to "I-LOC". IOB1 is not explict. mentioned in the paper, but it is used in the documentation:

https://polyglot.readthedocs.io/en/latest/NamedEntityRecognition.html
joeddav (CONTRIBUTOR) commented on 2020-09-18 15:04:57:
@stefan-it I went back and forth on this. My biggest problem with it is that once you are in IOB, there is the expectation that the beginning of new entities are marked with a `B-` (at least in the case of two back-to-back entities):
```
Today    O
Alice    I-PER
Bob      B-PER
and      O
I        O 
ate      O
lasagna  O
```
If we just prepend `I-` to everything, `Bob` would be incorrectly tagged `I-PER`, meaning `Bob Alice` is a single entity. The current format is bad but is at least clear that it does not contain that information.

But I could go either way if someone has a strong opinion.
joeddav (CONTRIBUTOR) commented on 2020-09-18 16:18:48:
Related: https://issues.apache.org/jira/browse/ARROW-9773

It's definitely a size thing. I took a smaller dataset with 87000 rows and did:
```
for i in range(10,1000,20):
    table = pa.concat_tables([dset._data]*i)
    table.take([0])
```
and it broke at around i=300.

Also when `_indices` is not None, this breaks indexing by slice. E.g. `dset.shuffle()[:1]` breaks.

Luckily so far I haven't seen `_indices.column(0).take` break, which means it doesn't break `select` or anything like that which is where the speed really matters, it's just `_getitem`. So I'm currently working around it by just doing the arrow v0 method in `_getitem`:
```
#if PYARROW_V0:
data_subset = pa.concat_tables(
    self._data.slice(indices_array[i].as_py(), 1) for i in range(len(indices_array))
)
#else:
    #data_subset = self._data.take(indices_array)
```
Skyy93 (NONE) commented on 2020-09-19 08:22:38:
```python
Traceback` (most recent call last):
  File "main.py", line 281, in <module>
    main()
  File "main.py", line 190, in main
    train_data, test_data = data_factory(
  File "main.py", line 129, in data_factory
    train_data = load_dataset('text', 
  File "/home/me/Downloads/datasets/src/datasets/load.py", line 608, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/me/Downloads/datasets/src/datasets/builder.py", line 468, in download_and_prepare
    self._download_and_prepare(
  File "/home/me/Downloads/datasets/src/datasets/builder.py", line 546, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/me/Downloads/datasets/src/datasets/builder.py", line 888, in _prepare_split
    for key, table in utils.tqdm(generator, unit=" tables", leave=False, disable=not_verbose):
  File "/home/me/.local/lib/python3.8/site-packages/tqdm/std.py", line 1130, in __iter__
    for obj in iterable:
  File "/home/me/.cache/huggingface/modules/datasets_modules/datasets/text/512f465342e4f4cd07a8791428a629c043bb89d55ad7817cbf7fcc649178b014/text.py", line 103, in _generate_tables
    pa_table = pac.read_csv(
  File "pyarrow/_csv.pyx", line 617, in pyarrow._csv.read_csv
  File "pyarrow/error.pxi", line 123, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: CSV parse error: Expected 1 columns, got 2
```

Unfortunately i am still getting this issue on Linux. I installed datasets from source and specified script_version to master.
lewtun (MEMBER) commented on 2020-09-19 09:06:46:
Hi @lhoestq we've tried out your suggestion but are now running into the following error:

```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-163-81ffd5ac18c9> in <module>
----> 1 dataset.cast_(emotion_features)

/usr/local/lib/python3.6/dist-packages/datasets/dataset_dict.py in cast_(self, features)
    125         self._check_values_type()
    126         for dataset in self.values():
--> 127             dataset.cast_(features=features)
    128 
    129     def remove_columns_(self, column_names: Union[str, List[str]]):

/usr/local/lib/python3.6/dist-packages/datasets/fingerprint.py in wrapper(*args, **kwargs)
    161             # Call actual function
    162 
--> 163             out = func(self, *args, **kwargs)
    164 
    165             # Update fingerprint of in-place transforms + update in-place history of transforms

/usr/local/lib/python3.6/dist-packages/datasets/arrow_dataset.py in cast_(self, features)
    602         self._info.features = features
    603         schema = pa.schema(features.type)
--> 604         self._data = self._data.cast(schema)
    605 
    606     @fingerprint(inplace=True)

/usr/local/lib/python3.6/dist-packages/pyarrow/table.pxi in pyarrow.lib.Table.cast()

ValueError: Target schema's field names are not matching the table's field names: ['text', 'label'], ['label', 'text']
```

Looking at the types in `emotion_features` we see that `label` and `text` appear to be swapped in the Arrow table:

```
emotion_features.type
StructType(struct<label: int64, text: string>)
```

Did we define the `emotion_features` incorrectly? We just followed the instructions from the [docs](https://huggingface.co/docs/datasets/features.html?highlight=features#dataset-features), but perhaps we misunderstood something 😬
lhoestq (MEMBER) commented on 2020-09-21 13:50:53:
Indeed we convert tensors to list to be able to write in arrow format. Because of this conversion we lose the dtype information. We should add the dtype detection when we do type inference. However it would require a bit of refactoring since currently the conversion happens before the type inference..

And then for your information, when reading from arrow format we have to cast from arrow to numpy (which is fast since pyarrow has a numpy integration), and then to torch.

However there's one thing that can help you: we make sure that the dtypes correspond to what is defined in `features`.
Therefore what you can do is provide `features` in `.map(preprocess, feature=...)` to specify the output types.

For example in your case:
```python
from datasets import Features, Value, Sequence

features = Features({
    "input_ids": Sequence(Value("int32")),
    "sembedding": Sequence(Value("float32"))
})
preprocessed_dataset = dataset.map(preprocess, features=features)

preprocessed_dataset.set_format("torch", columns=["input_ids", "sembedding"])
print(preprocessed_dataset[0]["sembedding"].dtype)
# "torch.float32"
```

Let me know if it helps
lhoestq (MEMBER) commented on 2020-09-22 08:18:29:
Right after `Caching processed dataset`, your function is applied to the dataset and there's a progress bar that shows how much time is left. How much time does it take for you ?

Also caching isn't supposed to slow down your processing. But if you still want to disable it you can do `.map(..., load_from_cache_file=False)`
BramVanroy (CONTRIBUTOR) commented on 2020-09-22 08:37:23:
If the arrow format is basically lists, why is the intermediate step to numpy necessary? I am a bit confused about that part.

Thanks for your suggestion. as I have currently implemented this, I cast to torch.Tensor in my collate_fn to save disk space (so I do not have to save padded tensors to max_len but can pad up to max batch len in collate_fn) at the cost of a bit slower processing. So for me this is not relevant anymore, but I am sure it is for others!
lhoestq (MEMBER) commented on 2020-09-22 08:58:22:
I'm glad you managed to figure something out :)

Casting from arrow to numpy can be 100x faster than casting from arrow to list.
This is because arrow has an integration with numpy that allows it to instantiate numpy arrays with zero-copy from arrow.
On the other hand to create python lists it is slow since it has to recreate the list object by iterating through each element in python.