go-inoue (NONE) commented on 2020-08-26 16:17:09:
1.  Here's the log from the first step.
```
Downloading and preparing dataset text/default-e84dd29acc4ad9ef (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/*****/.cache/huggingface/datasets/text/default-e84dd29acc4ad9ef/0.0.0/
447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6fe661fe4d070d380d...
Dataset text downloaded and prepared to /home/*****/.cache/huggingface/datasets/text/default-e84dd29acc4ad9ef/0.0.0/447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6fe661fe4d070d380d. Subsequent calls will reuse this data.
```
There's a file named `cache-7b1440ba7077af0f0d9035b5a55d01fc.arrow`, so it did create a cached file.
```
drwxr-xr-x 2 ***** ***** 4.0K Aug 26 15:59 .
drwxr-xr-x 3 ***** ***** 4.0K Aug 26 15:58 ..
-rw------- 1 ***** *****  99M Aug 26 15:59 cache-7b1440ba7077af0f0d9035b5a55d01fc.arrow
-rw-r--r-- 1 ***** *****  670 Aug 26 15:58 dataset_info.json
-rw-r--r-- 1 ***** *****    0 Aug 26 15:58 LICENSE
-rw-r--r-- 1 ***** *****  33M Aug 26 15:58 text-train.arrow
```
2. Ideally, `cache-7b1440ba7077af0f0d9035b5a55d01fc.arrow` should be used in `run_language_modeling.py` (modified version using `nlp`) with `xla_spawn.py`. But it looks like it's creating a new cached file.

```
08/26/2020 16:13:03 - INFO - filelock -   Lock 139635836351096 released on /home/*****/.cache/huggingface/datasets/3e34209a2741375a1db1ff03bf1abba1a9bd0e6016912d3ead0114b9d1ca2685.202fa4f84f552bff1f5400ae012663839c61efb3de068c6c8722d34ac0ea6192
.py.lock
08/26/2020 16:13:03 - WARNING - nlp.builder -   Using custom data configuration default
08/26/2020 16:13:03 - INFO - nlp.builder -   Overwrite dataset info from restored data version.
08/26/2020 16:13:03 - INFO - nlp.info -   Loading Dataset info from /home/*****/.cache/huggingface/datasets/text/default-e84dd29acc4ad9ef/0.0.0/447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6fe661fe4d070d380d
08/26/2020 16:13:03 - INFO - nlp.builder -   Reusing dataset text (/home/*****/.cache/huggingface/datasets/text/default-e84dd29acc4ad9ef/0.0.0/447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6fe661fe4d070d380d)
08/26/2020 16:13:03 - INFO - nlp.builder -   Constructing Dataset for split train, from /home/*****/.cache/huggingface/datasets/text/default-e84dd29acc4ad9ef/0.0.0/447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6fe661fe4d070d380d
08/26/2020 16:13:03 - INFO - nlp.utils.info_utils -   All the checksums matched successfully for post processing resources
08/26/2020 16:13:03 - INFO - nlp.builder -   Overwrite dataset info from restored data version.
08/26/2020 16:13:03 - INFO - nlp.info -   Loading Dataset info from /home/*****/.cache/huggingface/datasets/text/default-e84dd29acc4ad9ef/0.0.0/447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6fe661fe4d070d380d
08/26/2020 16:13:03 - INFO - nlp.builder -   Reusing dataset text (/home/*****/.cache/huggingface/datasets/text/default-e84dd29acc4ad9ef/0.0.0/447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6fe661fe4d070d380d)
08/26/2020 16:13:03 - INFO - nlp.builder -   Constructing Dataset for split train, from /home/*****/.cache/huggingface/datasets/text/default-e84dd29acc4ad9ef/0.0.0/447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6fe661fe4d070d380d
08/26/2020 16:13:03 - INFO - nlp.utils.info_utils -   All the checksums matched successfully for post processing resources
08/26/2020 16:13:05 - INFO - nlp.arrow_dataset -   Caching processed dataset at /home/*****/.cache/huggingface/datasets/text/default-e84dd29acc4ad9ef/0.0.0/447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6fe661fe4d070d380d/cache-0d77dfce704493dbe
63f071eed6a5431.arrow
^M  0%|          | 0/100 [00:00<?, ?it/s]08/26/2020 16:13:05 - INFO - nlp.arrow_dataset -   Caching processed dataset at /home/*****/.cache/huggingface/datasets/text/default-e84dd29acc4ad9ef/0.0.0/447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6
fe661fe4d070d380d/cache-0d77dfce704493dbe63f071eed6a5431.arrow
```

There are two cached files in the directory:

```
drwxr-xr-x 2 ***** ***** 4.0K Aug 26 16:14 .
drwxr-xr-x 3 ***** ***** 4.0K Aug 26 15:58 ..
-rw------- 1 ***** *****  99M Aug 26 16:14 cache-0d77dfce704493dbe63f071eed6a5431.arrow
-rw------- 1 ***** *****  99M Aug 26 15:59 cache-7b1440ba7077af0f0d9035b5a55d01fc.arrow
-rw-r--r-- 1 ***** *****  670 Aug 26 16:13 dataset_info.json
-rw-r--r-- 1 ***** *****    0 Aug 26 16:13 LICENSE
-rw-r--r-- 1 ***** *****  33M Aug 26 15:58 text-train.arrow
```

If I kill the process, and run it again, it will use the second cached file.

```
08/26/2020 16:19:52 - WARNING - nlp.builder -   Using custom data configuration default
08/26/2020 16:19:52 - INFO - nlp.builder -   Overwrite dataset info from restored data version.
08/26/2020 16:19:52 - INFO - nlp.info -   Loading Dataset info from /home/*****/.cache/huggingface/datasets/text/default-e84dd29acc4ad9ef/0.0.0/447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6fe661fe4d070d380d
08/26/2020 16:19:52 - INFO - nlp.builder -   Reusing dataset text (/home/*****/.cache/huggingface/datasets/text/default-e84dd29acc4ad9ef/0.0.0/447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6fe661fe4d070d380d)
08/26/2020 16:19:52 - INFO - nlp.builder -   Constructing Dataset for split train, from /home/*****/.cache/huggingface/datasets/text/default-e84dd29acc4ad9ef/0.0.0/447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6fe661fe4d070d380d
08/26/2020 16:19:52 - INFO - nlp.utils.info_utils -   All the checksums matched successfully for post processing resources
08/26/2020 16:19:53 - INFO - nlp.arrow_dataset -   Loading cached processed dataset at /home/*****/.cache/huggingface/datasets/text/default-e84dd29acc4ad9ef/0.0.0/447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6fe661fe4d070d380d/cache-0d77dfce70
4493dbe63f071eed6a5431.arrow
08/26/2020 16:19:53 - INFO - nlp.arrow_dataset -   Set __getitem__(key) output type to torch for ['input_ids'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
```