Title: Enum used in map functions will raise a RecursionError with dill.
Author: jorgeecardona (NONE)
State: open
Created: 2021-07-14T09:16:08Z
Labels: bug

## Describe the bug

Enums used in functions pass to `map` will fail at pickling with a maximum recursion exception as described here: https://github.com/uqfoundation/dill/issues/250#issuecomment-852566284

In my particular case, I use an enum to define an argument with fixed options using the `TraininigArguments` dataclass as base class and the `HfArgumentParser`. In the same file I use a `ds.map` that tries to pickle the content of the module including the definition of the enum that runs into the dill bug described above.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from enum import Enum

class A(Enum):
    a = 'a'

def main():
    a = A.a
    
    def f(x):
        return {} if a == a.a else x
    
    ds = load_dataset('cnn_dailymail', '3.0.0')['test']
    ds = ds.map(f, num_proc=15)

if __name__ == "__main__":
    main()
```

## Expected results
The known problem with dill could be prevented as explained in the link above (workaround.) Since `HFArgumentParser` nicely uses the enum class for choices it makes sense to also deal with this bug under the hood.

## Actual results

```python
  File "/home/xxxx/miniconda3/lib/python3.8/site-packages/dill/_dill.py", line 1373, in save_type
    pickler.save_reduce(_create_type, (type(obj), obj.__name__,
  File "/home/xxxx/miniconda3/lib/python3.8/pickle.py", line 690, in save_reduce
    save(args)
  File "/home/xxxx/miniconda3/lib/python3.8/pickle.py", line 558, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/xxxx/miniconda3/lib/python3.8/pickle.py", line 899, in save_tuple
    save(element)
  File "/home/xxxx/miniconda3/lib/python3.8/pickle.py", line 534, in save
    self.framer.commit_frame()
  File "/home/xxxx/miniconda3/lib/python3.8/pickle.py", line 220, in commit_frame
    if f.tell() >= self._FRAME_SIZE_TARGET or force:
RecursionError: maximum recursion depth exceeded while calling a Python object
```

## Environment info
- `datasets` version: 1.8.0
- Platform: Linux-5.9.0-4-amd64-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyArrow version: 3.0.0

Comments:
I'm running into this as well. (Thank you so much for reporting @jorgeecardona — was staring at this massive stack trace and unsure what exactly was wrong!)

Hi ! Thanks for reporting :)

Until this is fixed on `dill`'s side, we could implement a custom saving in our Pickler indefined in utils.py_utils.py
There is already a suggestion in this message about how to do it:
https://github.com/uqfoundation/dill/issues/250#issuecomment-852566284

Let me know if such a workaround could help, and feel free to open a PR if you want to contribute !

I have the same bug.
the code is as follows:
![image](https://user-images.githubusercontent.com/84262181/139785849-620dd4ac-86ce-4212-8163-942bbca305aa.png)
the error is: 
![image](https://user-images.githubusercontent.com/84262181/139785899-88a9bd75-c60b-45a5-b819-830c7c096f3d.png)

Look for the solution for this bug.

Hi ! I think your RecursionError comes from a different issue @BitcoinNLPer , could you open a separate issue please ?

Also which dataset are you using ? I tried loading `CodedotAI/code_clippy` but I get a different error
```python
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/load.py", line 1615, in load_dataset
    **config_kwargs,
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/load.py", line 1446, in load_dataset_builder
    builder_cls = import_main_class(dataset_module.module_path)
  File "/Users/quentinlhoest/Desktop/hf/datasets/src/datasets/load.py", line 101, in import_main_class
    module = importlib.import_module(module_path)
  File "/Users/quentinlhoest/.virtualenvs/hf-datasets/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/quentinlhoest/.cache/huggingface/modules/datasets_modules/datasets/CodedotAI___code_clippy/d332f69d036e8c80f47bc9a96d676c3fa30cb50af7bb81e2d4d12e80b83efc4d/code_clippy.py", line 66, in <module>
    url_elements = results.find_all("a")
AttributeError: 'NoneType' object has no attribute 'find_all'
```