Title: `_pickle.PicklingError: logger cannot be pickled` in multiprocessing `map`
Author: ZhaofengWu (NONE)
State: closed
Created: 2022-10-03T23:51:38Z
Labels: bug

## Describe the bug
When I `map` with multiple processes, this error occurs. The `.name` of the `logger` that fails to pickle in the final line is `datasets.fingerprint`.
```
  File "~/project/dataset.py", line 204, in <dictcomp>
    split: dataset.map(
  File ".../site-packages/datasets/arrow_dataset.py", line 2489, in map
    transformed_shards[index] = async_result.get()
  File ".../site-packages/multiprocess/pool.py", line 771, in get
    raise self._value
  File ".../site-packages/multiprocess/pool.py", line 537, in _handle_tasks
    put(task)
  File ".../site-packages/multiprocess/connection.py", line 214, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File ".../site-packages/multiprocess/reduction.py", line 54, in dumps
    cls(buf, protocol, *args, **kwds).dump(obj)
  File ".../site-packages/dill/_dill.py", line 620, in dump
    StockPickler.dump(self, obj)
  File ".../pickle.py", line 487, in dump
    self.save(obj)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../pickle.py", line 902, in save_tuple
    save(element)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../site-packages/dill/_dill.py", line 1963, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File ".../site-packages/dill/_dill.py", line 1140, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File ".../pickle.py", line 717, in save_reduce
    save(state)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../pickle.py", line 887, in save_tuple
    save(element)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File ".../pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File ".../pickle.py", line 998, in _batch_setitems
    save(v)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../site-packages/dill/_dill.py", line 1963, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File ".../site-packages/dill/_dill.py", line 1140, in _save_with_postproc
    pickler.save_reduce(*reduction, obj=obj)
  File ".../pickle.py", line 717, in save_reduce
    save(state)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../pickle.py", line 887, in save_tuple
    save(element)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../site-packages/dill/_dill.py", line 1251, in save_module_dict
    StockPickler.save_dict(pickler, obj)
  File ".../pickle.py", line 972, in save_dict
    self._batch_setitems(obj.items())
  File ".../pickle.py", line 998, in _batch_setitems
    save(v)
  File ".../pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File ".../site-packages/dill/_dill.py", line 1963, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File ".../site-packages/dill/_dill.py", line 1154, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File ".../pickle.py", line 998, in _batch_setitems
    save(v)
  File ".../pickle.py", line 578, in save
    rv = reduce(self.proto)
  File ".../logging/__init__.py", line 1774, in __reduce__
    raise pickle.PicklingError('logger cannot be pickled')
_pickle.PicklingError: logger cannot be pickled
```

## Steps to reproduce the bug
Sorry I failed to have a minimal reproducible example, but the offending line on my end is
```python
dataset.map(
    lambda examples: self.tokenize(examples),  # this doesn't matter, lambda e: [1] * len(...) also breaks. In fact I'm pretty sure it breaks before executing this lambda
    batched=True,
    num_proc=4,
)
```
This does work when `num_proc=1`, so it's likely a multiprocessing thing.

## Expected results
`map` succeeds

## Actual results
The error trace above.

## Environment info
- `datasets` version: 1.16.1 and 2.5.1 both failed
- Platform: Ubuntu 20.04.4 LTS
- Python version: 3.10.4
- PyArrow version: 9.0.0

Comments:
This is maybe related to python 3.10, do you think you could try on 3.8 ?

In the meantime we'll keep improving the support for 3.10. Let me add a dedicated CI

I did some binary search and seems like the root cause is either `multiprocess` or `dill`. python 3.10 is fine. Specifically:
- `multiprocess==0.70.12.2, dill==0.3.4`: works
- `multiprocess==0.70.12.2, dill==0.3.5.1`: doesn't work
- `multiprocess==0.70.13, dill==0.3.5.1`: doesn't work
- `multiprocess==0.70.13, dill==0.3.4`: can't test, `multiprocess==0.70.13` requires `dill>=0.3.5.1`

I will pin their versions on my end. I don't have enough knowledge of how python multiprocessing works to debug this, but ideally there could be a fix. It's also possible that I'm doing something wrong in my code, but again the `.name` of the logger that failed to pickle is `datasets.fingerprint`, which I'm not using directly.

Do you know which logger fails at being pickled ?

I'm not 100% sure how to figure it out -- the stack trace above doesn't clearly give me a place where I can print out who owns the logger, etc. I only found out its `.name` is `datasets.fingerprint` by printing right before
```
  File ".../logging/__init__.py", line 1774, in __reduce__
    raise pickle.PicklingError('logger cannot be pickled')
```
If you have any idea on how to find it out, please let me know.

Ok I see, not sure why it triggers this error though, in `logging.py` the code is

https://github.com/python/cpython/blob/c9da063e32725a66495e4047b8a5ed13e72d9e8e/Lib/logging/__init__.py#L1769-L1775

and on my side it works on 3.10 with dill 0.3.5.1 and multiprocess 0.70.13
```python
>>> datasets.fingerprint.logger.__reduce__()                                                                                                         
(<function logging.getLogger(name=None)>, ('datasets.fingerprint',))
```
Could you try to run this code ?

Are you in an environment where the loggers are instantiated differently ? Can you check the source code of `logging.Logger.__reduce__` in `".../logging/__init__.py", line 1774` ?

Closing due to inactivity.