Title: Crash when `num_proc` > dataset length for `map()` on a `datasets.Dataset`.
Author: mbforbes (NONE)
State: closed
Created: 2021-06-09T22:40:22Z
Labels: bug

## Describe the bug
Crash if when using `num_proc` > 1 (I used 16) for `map()` on a `datasets.Dataset`.

I believe I've had cases where `num_proc` > 1 works before, but now it seems either inconsistent, or depends on my data. I'm not sure whether the issue is on my end, because it's difficult for me to debug! Any tips greatly appreciated, I'm happy to provide more info if it would helps us diagnose.

## Steps to reproduce the bug
```python
# this function will be applied with map()
def tokenize_function(examples):
    return tokenizer(
        examples["text"],
        padding=PaddingStrategy.DO_NOT_PAD,
        truncation=True,
    )

# data_files is a Dict[str, str] mapping name -> path
datasets = load_dataset("text", data_files={...})  

# this is where the error happens if num_proc = 16,
# but is fine if num_proc = 1
tokenized_datasets = datasets.map(
    tokenize_function,
    batched=True,
    num_proc=num_workers,
)
```

## Expected results
The `map()` function succeeds with `num_proc` > 1.

## Actual results
![image](https://user-images.githubusercontent.com/1170062/121404271-a6cc5200-c910-11eb-8e27-5c893bd04042.png)
![image](https://user-images.githubusercontent.com/1170062/121404362-be0b3f80-c910-11eb-9117-658943029aef.png)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.6.2
- Platform: Linux-5.4.0-73-generic-x86_64-with-glibc2.31
- Python version: 3.9.5
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes, but I think N/A for this issue
- Using distributed or parallel set-up in script?: Multi-GPU on one machine, but I think also N/A for this issue

Comments:
Hi ! It looks like the issue comes from pyarrow. What version of pyarrow are you using ? How did you install it ?

Thank you for the quick reply! I have `pyarrow==4.0.0`, and I am installing with `pip`. It's not one of my explicit dependencies, so I assume it came along with something else.

Could you trying reinstalling pyarrow with pip ?
I'm not sure why it would check in your multicurtural-sc directory for source files.

Sure! I tried reinstalling to get latest. pip was mad because it looks like Datasets currently wants <4.0.0 (which is interesting, because apparently I ended up with 4.0.0 already?), but I gave it a shot anyway:

```bash
$ pip install --upgrade --force-reinstall pyarrow
Collecting pyarrow
  Downloading pyarrow-4.0.1-cp39-cp39-manylinux2014_x86_64.whl (21.9 MB)
     |████████████████████████████████| 21.9 MB 23.8 MB/s
Collecting numpy>=1.16.6
  Using cached numpy-1.20.3-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (15.4 MB)
Installing collected packages: numpy, pyarrow
  Attempting uninstall: numpy
    Found existing installation: numpy 1.20.3
    Uninstalling numpy-1.20.3:
      Successfully uninstalled numpy-1.20.3
  Attempting uninstall: pyarrow
    Found existing installation: pyarrow 3.0.0
    Uninstalling pyarrow-3.0.0:
      Successfully uninstalled pyarrow-3.0.0
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
datasets 1.8.0 requires pyarrow<4.0.0,>=1.0.0, but you have pyarrow 4.0.1 which is incompatible.
Successfully installed numpy-1.20.3 pyarrow-4.0.1
```

Trying it, the same issue:

![image](https://user-images.githubusercontent.com/1170062/121730226-3f470b80-caa4-11eb-85a5-684c44c816da.png)

I tried installing `"pyarrow<4.0.0"`, which gave me 3.0.0. Running, still, same issue.

I agree it's weird that pyarrow is checking the source code directory for its files. (There is no `pyarrow/` directory there.) To me, that makes it seem like an issue with how pyarrow is called.

Out of curiosity, I tried running this with fewer workers to see when the error arises:

- 1: ✅
- 2: ✅
- 4: ✅
- 8: ✅
- 10: ✅
- 11: ❌  🤔
- 12: ❌
- 16: ❌
- 32: ❌

checking my datasets:

```python
>>> datasets
DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 389290
    })
    validation.sc: Dataset({
        features: ['text'],
        num_rows: 10  # 🤔
    })
    validation.wvs: Dataset({
        features: ['text'],
        num_rows: 93928
    })
})
```

New hypothesis: crash if `num_proc` > length of a dataset? 😅

If so, this might be totally my fault, as the caller. Could be a docs fix, or maybe this library could do a check to limit `num_proc` for this case?

Good catch ! Not sure why it could raise such a weird issue from pyarrow though
We should definitely reduce num_proc to the length of the dataset if needed and log a warning.

This has been fixed in #2566, thanks @connor-mccarthy !
We'll make a new release soon that includes the fix ;)