Title: Bugs : dataset.map() is frozen on ELI5
Author: ratthachat (NONE)
State: closed
Created: 2020-08-07T08:23:35Z

Hi Huggingface Team!

Thank you guys once again for this amazing repo.

I have tried to prepare ELI5 to train with T5, based on [this wonderful notebook of Suraj Patil](https://github.com/patil-suraj/exploring-T5/blob/master/T5_on_TPU.ipynb) 

However, when I run `dataset.map()` on ELI5 to prepare `input_text, target_text`, `dataset.map` is **frozen** in the first hundreds examples. On the contrary, this works totally fine on SQUAD (80,000 examples). Both `nlp` version 0.3.0 and 0.4.0 cause frozen process . Also try various `pyarrow` versions from 0.16.0 / 0.17.0 / 1.0.0 also have the same frozen process.

Reproducible code can be found on [this colab notebook ](https://colab.research.google.com/drive/14wttOTv3ky74B_c0kv5WrbgQjCF2fYQk?usp=sharing), where I also show that the same mapping function works fine on SQUAD, so the problem is likely due to ELI5 somehow.

----------------------------------------
**More Info :** instead of `map`, if I run `for` loop and apply function by myself, there's no error and can finish within 10 seconds. However, `nlp dataset` is immutable (I couldn't manually assign a new key-value to `dataset `object)

I also notice that SQUAD texts are quite clean while ELI5 texts contain many special characters, not sure if this is the cause ?

Comments:
This comes from an overflow in pyarrow's array.
It is stuck inside the loop that reduces the batch size to avoid the overflow.
I'll take a look

I created a PR to fix the issue.
It was due to an overflow check that handled badly an empty list.

You can try the changes by using 
```
!pip install git+https://github.com/huggingface/nlp.git@fix-bad-type-in-overflow-check
```

Also I noticed that the first 1000 examples have an empty list in the `title_urls` field. The feature type inference in `.map` will consider it `null` because of that, and it will crash when it encounter the next example with a `title_urls` that is not empty.

Therefore to fix that, what you can do for now is increase the writer batch size so that the feature inference will take into account at least one example with a non-empty `title_urls`:

```python
# default batch size is 1_000 and it's not enough for feature type inference because of empty lists
valid_dataset = valid_dataset.map(make_input_target, writer_batch_size=3_000) 
```

I was able to run the frozen cell with these changes.

@lhoestq Perfect and thank you very much!!
Close the issue.

@lhoestq mapping the function `make_input_target` was passed by your fixing.

However, there is another error in the final step of `valid_dataset.map(convert_to_features, batched=True)`

`ArrowInvalid: Could not convert Thepiratebay.vg with type str: converting to null type`
(The [same colab notebook above with new error message](https://colab.research.google.com/drive/14wttOTv3ky74B_c0kv5WrbgQjCF2fYQk?usp=sharing#scrollTo=5sRrJ3_C8rLt))

Do you have some ideas? (I am really sorry I could not debug it by myself since I never used `pyarrow` before) 
Note that `train_dataset.map(convert_to_features, batched=True)` can be run successfully even though train_dataset is 27x bigger than `valid_dataset` so I believe the problem lies in some field of `valid_dataset` again .

I got this issue too and fixed it by specifying `writer_batch_size=3_000` in `.map`.
This is because Arrow didn't expect `Thepiratebay.vg` in `title_urls `, as all previous examples have empty lists in `title_urls `

I am clear now . Thank so much again Quentin!

I'm getting a hanging `dataset.map()` when running a gradio app with `gradio` for auto-reloading instead of `python`

Maybe this is an issue with gradio, could you open an issue on their repo ? `Dataset.map` simply uses `multiprocess.Pool` for multiprocessing

If you interrupt the program mayeb the stack trace would give some information of where it was hanging in the code (maybe a lock somewhere ?)