sunny2309/githug-issues huggingface dataset:
map and filter not working properly in multiprocessing with the new release 2.6.0

## Describe the bug
When mapping is used on a dataset with more than one process, there is a weird behavior when trying to use `filter` , it's like only the samples from one worker are retrieved, one needs to specify the same `num_proc` in filter for it to work properly. This doesn't happen with `datasets` version 2.5.2

In the code below the data is filtered differently when we increase `num_proc` used in `map` although the datsets before and after mapping have identical elements.

## Steps to reproduce the bug
```python
import datasets
from datasets import load_dataset

def preprocess(example):
    return example

ds = load_dataset("codeparrot/codeparrot-clean-valid", split="train").select([i for i in range(10)])
ds1 = ds.map(preprocess, num_proc=2)
ds2 = ds.map(preprocess)

# the datasets elements are the same
for i in range(len(ds1)):
    assert ds1[i]==ds2[i]

print(f'Target column before filtering {ds1["autogenerated"]}')
print(f'Target column before filtering {ds2["autogenerated"]}')
print(f"datasets version {datasets.__version__}")

ds_filtered_1 = ds1.filter(lambda x: not x["autogenerated"])
ds_filtered_2 = ds2.filter(lambda x: not x["autogenerated"])

# all elements in Target column are false so they should all be kept, but for ds2 only the first 5=num_samples/num_proc are kept
print(ds_filtered_1)
print(ds_filtered_2)
```
```
Target column before filtering [False, False, False, False, False, False, False, False, False, False]
Target column before filtering [False, False, False, False, False, False, False, False, False, False]

Dataset({
    features: ['repo_name', 'path', 'copies', 'size', 'content', 'license', 'hash', 'line_mean', 'line_max', 'alpha_frac', 'autogenerated'],
    num_rows: 5
})
Dataset({
    features: ['repo_name', 'path', 'copies', 'size', 'content', 'license', 'hash', 'line_mean', 'line_max', 'alpha_frac', 'autogenerated'],
    num_rows: 10
})
```
## Expected results
Increasing `num_proc` in mapping shouldn't alter filtering. With the previous version 2.5.2 this doesn't happen

## Actual results
Filtering doesn't work properly when we increase `num_proc` in mapping but not when calling `filter`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.6.0
- Platform: Linux-4.19.0-22-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.9.13
- PyArrow version: 8.0.0
- Pandas version: 1.4.2
Map caching not working for some class methods

## Describe the bug
The cache loading is not working as expected for some class methods with a model stored in an attribute.
The new fingerprint for `_map_single` is not the same at each run. The hasher generate a different hash for the class method.
This comes from `dumps` function in `datasets.utils.py_utils` which generates a different dump at each run.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from transformers import AutoConfig, AutoModel, AutoTokenizer

dataset = load_dataset("ethos", "binary")
BASE_MODELNAME = "sentence-transformers/all-MiniLM-L6-v2"

class Object:
    def __init__(self):
        config = AutoConfig.from_pretrained(BASE_MODELNAME)
        self.bert = AutoModel.from_config(config=config, add_pooling_layer=False)
        self.tok = AutoTokenizer.from_pretrained(BASE_MODELNAME)

    def tokenize(self, examples):

        tokenized_texts = self.tok(
            examples["text"],
            padding="max_length",
            truncation=True,
            max_length=256,
        )
        return tokenized_texts

instance = Object()
result = dict()
for phase in ["train"]:
    result[phase] = dataset[phase].map(instance.tokenize, batched=True, load_from_cache_file=True, num_proc=2)

```

## Expected results
Load cache instead of recompute result.

## Actual results
Result recomputed from scratch at each run.
The cache works fine when deleting `bert` attribute.

## Environment info
- `datasets` version: 2.5.3.dev0
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.13
- PyArrow version: 7.0.0
- Pandas version: 1.5.0
Specifying an exisiting folder in download_and_prepare deletes everything in it

## Describe the bug
The builder correctly creates the `output_dir` folder if it doesn't exist, but if the folder exists everything within it is deleted. Specifying `"."` as the `output_dir` deletes everything in your current dir but also leads to **another bug** whose traceback is the following:

```
                          Traceback (most recent call last)
Input In [11], in <cell line: 1>()
----> 1 rotten_tomatoes_builder.download_and_prepare(output_dir=".", max_shard_size="200MB", file_format="parquet")

File ~/BIGSCIENCE/env/lib/python3.9/site-packages/datasets/builder.py:818, in download_and_prepare(self, output_dir, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, storage_options, **download_and_prepare_kwargs)

File /usr/lib/python3.9/contextlib.py:124, in _GeneratorContextManager.__exit__(self, type, value, traceback)
    122 if type is None:
    123     try:
--> 124         next(self.gen)
    125     except StopIteration:
    126         return False

File ~/BIGSCIENCE/env/lib/python3.9/site-packages/datasets/builder.py:760, in incomplete_dir(dirname)

File /usr/lib/python3.9/shutil.py:722, in rmtree(path, ignore_errors, onerror)
    720         os.rmdir(path)
    721     except OSError:
--> 722         onerror(os.rmdir, path, sys.exc_info())
    723 else:
    724     try:
    725         # symlinks to directories are forbidden, see bug #1669

File /usr/lib/python3.9/shutil.py:720, in rmtree(path, ignore_errors, onerror)
    718 _rmtree_safe_fd(fd, path, onerror)
    719 try:
--> 720     os.rmdir(path)
    721 except OSError:
    722     onerror(os.rmdir, path, sys.exc_info())

OSError: [Errno 22] Invalid argument: '/home/christopher/BIGSCIENCE/.'
``` 

## Steps to reproduce the bug
```python
rotten_tomatoes_builder = load_dataset_builder("rotten_tomatoes")
rotten_tomatoes_builder.download_and_prepare(output_dir="./test_folder", max_shard_size="200MB", file_format="parquet")
```
If `test_folder` contains any files they will all be deleted
 
## Expected results
Either a warning that all files will be deleted, but preferably that they not be deleted at all.

## Actual results
N/A

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.2
- Platform: Linux-5.15.0-48-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3
Error in create a dataset from a Python generator

## Describe the bug
In HOW-TO-GUIDES > Load > [Python generator](https://huggingface.co/docs/datasets/v2.5.2/en/loading#python-generator), the code example defines the `my_gen` function, but when creating the dataset, an undefined `my_dict` is passed in.  

```Python
>>> from datasets import Dataset
>>> def my_gen():
...    for i in range(1, 4):
...        yield {"a": i}
>>> dataset = Dataset.from_generator(my_dict)
```
datasets doesn't support # in data paths

## Describe the bug
dataset files with `#` symbol their paths aren't read correctly.

## Steps to reproduce the bug
The data in folder `c#`of this [dataset](https://huggingface.co/datasets/loubnabnl/bigcode_csharp) can't be loaded. While the folder `c_sharp` with the same data is loaded properly
```python
ds = load_dataset('loubnabnl/bigcode_csharp', split="train", data_files=["data/c#/*"])
```
```
FileNotFoundError: Couldn't find file at https://huggingface.co/datasets/loubnabnl/bigcode_csharp/resolve/27a3166cff4bb18e11919cafa6f169c0f57483de/data/c#/data_0003.jsonl
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.5.2
- Platform: macOS-12.2.1-arm64-arm-64bit
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.4.3

cc @lhoestq
Classes label error when loading symbolic links using imagefolder

**Is your feature request related to a problem? Please describe.**
Like this: #4015
When there are **symbolic links** to pictures in the data folder, the parent folder name of the **real file** will be used as the class name instead of the parent folder of the symbolic link itself. Can you give an option to decide whether to enable symbolic link tracking?

This is inconsistent with the `torchvision.datasets.ImageFolder` behavior.

For example:
![image](https://user-images.githubusercontent.com/49552732/195008591-3cce644e-aabe-4f39-90b9-832861cadb3d.png)
![image](https://user-images.githubusercontent.com/49552732/195008841-0b0c2289-eb7f-411a-977b-37426f23a277.png)

It use `others` in green circle as class label but not `abnormal`, I wish `load_dataset` not use the real file parent as label.
**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context about the feature request here.
Fatal error with pyarrow/libarrow.so

## Describe the bug
When using datasets, at the very end of my jobs the program crashes (see trace below).
It doesn't seem to affect anything, as it appears to happen as the program is closing down. Just importing `datasets` is enough to cause the error.

## Steps to reproduce the bug
This is sufficient to reproduce the problem:

```bash
python -c "import datasets"
```

## Expected results
Program should run to completion without an error.

## Actual results
```bash
Fatal error condition occurred in /opt/vcpkg/buildtrees/aws-c-io/src/9e6648842a-364b708815.clean/source/event_loop.c:72: aws_thread_launch(&cleanup_thread, s_event_loop_destroy_async_thread_fn, el_group, &thread_options) == AWS_OP_SUCCESS
Exiting Application
################################################################################
Stack trace:
################################################################################
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x200af06) [0x150dff547f06]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x20028e5) [0x150dff53f8e5]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x1f27e09) [0x150dff464e09]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x200ba3d) [0x150dff548a3d]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x1f25948) [0x150dff462948]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x200ba3d) [0x150dff548a3d]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x1ee0b46) [0x150dff41db46]
/u/user/miniconda3/envs/env/lib/python3.10/site-packages/pyarrow/libarrow.so.900(+0x194546a) [0x150dfee8246a]
/lib64/libc.so.6(+0x39b0c) [0x150e15eadb0c]
/lib64/libc.so.6(on_exit+0) [0x150e15eadc40]
/u/user/miniconda3/envs/env/bin/python(+0x28db18) [0x560ae370eb18]
/u/user/miniconda3/envs/env/bin/python(+0x28db4b) [0x560ae370eb4b]
/u/user/miniconda3/envs/env/bin/python(+0x28db90) [0x560ae370eb90]
/u/user/miniconda3/envs/env/bin/python(_PyRun_SimpleFileObject+0x1e6) [0x560ae37123e6]
/u/user/miniconda3/envs/env/bin/python(_PyRun_AnyFileObject+0x44) [0x560ae37124c4]
/u/user/miniconda3/envs/env/bin/python(Py_RunMain+0x35d) [0x560ae37135bd]
/u/user/miniconda3/envs/env/bin/python(Py_BytesMain+0x39) [0x560ae37137d9]
/lib64/libc.so.6(__libc_start_main+0xf3) [0x150e15e97493]
/u/user/miniconda3/envs/env/bin/python(+0x2125d4) [0x560ae36935d4]
Aborted (core dumped)
```

## Environment info
- `datasets` version: 2.5.1
- Platform: Linux-4.18.0-348.23.1.el8_5.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.4
- PyArrow version: 9.0.0
- Pandas version: 1.4.3
Transfer some canonical datasets under an organization namespace

As discussed during our @huggingface/datasets meeting, we are planning to move some "canonical" dataset scripts under their corresponding organization namespace (if this does not exist).

On the contrary, if the dataset already exists under the organization namespace, we are deprecating the canonical one (and eventually delete it).

First, we should test it using a dummy dataset/organization.

TODO:
- [x] Test with a dummy dataset
  - [x] Create dummy canonical dataset: https://huggingface.co/datasets/dummy_canonical_dataset
  - [x] Create dummy organization: https://huggingface.co/dummy-canonical-org
  - [x] Transfer dummy canonical dataset to dummy organization
- [ ] Transfer datasets
  - [x] babi_qa => facebook
  - [x] cord19 => allenai
  - [x] emotion => dair-ai
  - [ ] gem => GEM
  - [x] hendrycks_test => cais/mmlu
  - [x] indonlu => indonlp
  - [ ] multilingual_librispeech => facebook
    - It already exists "facebook/multilingual_librispeech"
  - [ ] oscar => oscar-corpus
  - [x] peer_read => allenai
  - [x] qasper => allenai
  - [x] reddit => webis/tldr-17
  - [x] russian_super_glue => russiannlp
  - [x] rvl_cdip => aharley
  - [x] s2orc => allenai
  - [x] scicite => allenai
  - [x] scifact => allenai
  - [x] scitldr => allenai
  - [x] swiss_judgment_prediction => rcds
  - [x] the_pile => EleutherAI
  - [ ] wmt14, wmt15, wmt16, wmt17, wmt18, wmt19,... => wmt
- [ ] Deprecate (and eventually remove) datasets that cannot be transferred because they already exist
  - [x] banking77 => PolyAI
  - [x] common_voice => mozilla-foundation
  - [x] german_legal_entity_recognition => elenanereiss
- ... 

EDIT: the list above is continuously being updated
Multiprocessing with `Dataset.map` and  `PyTorch` results in deadlock 

## Describe the bug
There seems to be an issue with using multiprocessing with `datasets.Dataset.map` (i.e. setting `num_proc` to a value greater than one) combined with a function that uses `torch` under the hood.  The subprocesses that `datasets.Dataset.map` spawns [a this step](https://github.com/huggingface/datasets/blob/1b935dab9d2f171a8c6294269421fe967eb55e34/src/datasets/arrow_dataset.py#L2663) go into wait mode forever.  

## Steps to reproduce the bug
The below code goes into  deadlock when `NUMBER_OF_PROCESSES` is greater than one.
```python

NUMBER_OF_PROCESSES = 2

from transformers import AutoTokenizer, AutoModel
from datasets import load_dataset

dataset = load_dataset("glue", "mrpc", split="train")
tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
model = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
model.to("cpu")

def cls_pooling(model_output):
    return model_output.last_hidden_state[:, 0]

def generate_embeddings_batched(examples):
    sentences_batch = list(examples['sentence1'])
    encoded_input = tokenizer(
        sentences_batch, padding=True, truncation=True, return_tensors="pt"
    )
    encoded_input = {k: v.to("cpu") for k, v in encoded_input.items()}
    model_output = model(**encoded_input)
    embeddings = cls_pooling(model_output)
    examples['embeddings'] = embeddings.detach().cpu().numpy() # 64, 384
    return examples

embeddings_dataset = dataset.map(
    generate_embeddings_batched, 
    batched=True,
    batch_size=10,
    num_proc=NUMBER_OF_PROCESSES
)
```
While debugging it I've seen that it gets "stuck" when calling `torch.nn.Embedding.forward` but some testing shows that the same happens with other functions from `torch.nn`. 

## Environment info
- Platform: Linux-5.14.0-1052-oem-x86_64-with-glibc2.31
- Python version: 3.9.14
- PyArrow version: 9.0.0
- Pandas version: 1.5.0

Not sure if this is a HF problem, a PyTorch problem or something I'm doing wrong..

Thanks!
Mismatch between tutoriel and doc

## Describe the bug
In the "Process text data" tutorial, [`map` has `return_tensors` as kwarg](https://huggingface.co/docs/datasets/main/en/nlp_process#map). It does not seem to appear in the [function documentation](https://huggingface.co/docs/datasets/main/en/package_reference/main_classes#datasets.Dataset.map), nor to work.

## Steps to reproduce the bug
MWE:
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")

from datasets import load_dataset
dataset  = load_dataset("lhoestq/demo1", split="train")

dataset = dataset.map(lambda examples: tokenizer(examples["review"]), batched=True, return_tensors="pt")

```

## Expected results
return_tensors to be a valid kwarg :smiley: 

## Actual results
```python
>> TypeError: map() got an unexpected keyword argument 'return_tensors'
```

## Environment info
- `datasets` version: 2.3.2
- Platform: Linux-5.14.0-1052-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyArrow version: 8.0.0
- Pandas version: 1.4.3
Use HTML relative paths for tiles in the docs

This PR replaces the absolute paths in the landing page tiles with relative ones so that one can test navigation both locally in and in future PRs (see [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5084/en/index) for an example PR where the links don't work).

I encountered this while working on the `optimum` docs and figured I'd fix it elsewhere too :)

Internal Slack thread: https://huggingface.slack.com/archives/C02GLJ5S0E9/p1665129710176619
Review sync issues from GitHub to Hub

## Describe the bug
We have discovered that sometimes there were sync issues between GitHub and Hub datasets, after a merge commit to main branch.

For example:
- this merge commit: https://github.com/huggingface/datasets/commit/d74a9e8e4bfff1fed03a4cab99180a841d7caf4b
- was not properly synced with the Hub: https://github.com/huggingface/datasets/actions/runs/3002495269/jobs/4819769684
```
[main 9e641de]  Add Papers with Code ID to scifact dataset (#4941)
 Author: Albert Villanova del Moral <albertvillanova@users.noreply.huggingface.co>
 1 file changed, 42 insertions(+), 14 deletions(-)
push failed !
GitCommandError(['git', 'push'], 1, b'remote: ----------------------------------------------------------        \nremote: Sorry, your push was rejected during YAML metadata verification:        \nremote: - Error: "license" does not match any of the allowed types        \nremote: ----------------------------------------------------------        \nremote: Please find the documentation at:        \nremote: https://huggingface.co/docs/hub/models-cards#model-card-metadata        \nremote: ----------------------------------------------------------        \nTo [https://huggingface.co/datasets/scifact.git\n](https://huggingface.co/datasets/scifact.git/n) ! [remote rejected] main -> main (pre-receive hook declined)\nerror: failed to push some refs to \'[https://huggingface.co/datasets/scifact.git\](https://huggingface.co/datasets/scifact.git/)'', b'')
```
We are reviewing sync issues in previous commits to recover them and repushing to the Hub.

TODO: Review
- [x] #4941
  - scifact
- [x] #4931
  - scifact
- [x] #4753
  - wikipedia 
- [x] #4554
  - wmt17, wmt19, wmt_t2t
  - Fixed with "Release 2.4.0" commit: https://github.com/huggingface/datasets/commit/401d4c4f9b9594cb6527c599c0e7a72ce1a0ea49
    - https://huggingface.co/datasets/wmt17/commit/5c0afa83fbbd3508ff7627c07f1b27756d1379ea
    - https://huggingface.co/datasets/wmt19/commit/b8ad5bf1960208a376a0ab20bc8eac9638f7b400
    - https://huggingface.co/datasets/wmt_t2t/commit/b6d67191804dd0933476fede36754a436b48d1fc
- [x] #4607
- [x] #4416
  - lccc
  - Fixed with "Release 2.3.0" commit: https://huggingface.co/datasets/lccc/commit/8b1f8cf425b5653a0a4357a53205aac82ce038d1
- [x] #4367
load_datasets("json", ...) don't read local .json.gz properly

## Describe the bug
I have a local file `*.json.gz` and it can be read by `pandas.read_json(lines=True)`, but cannot be read by `load_datasets("json")` (resulting in 0 lines)

## Steps to reproduce the bug
```python
fpath = '/data/junwang/.cache/general/57b6f2314cbe0bc45dda5b78f0871df2/test.json.gz'
ds_panda = DatasetDict(
    test=Dataset.from_pandas(
        pd.read_json(fpath, lines=True)
    )
)
ds_direct = load_dataset(
    'json', data_files={
        'test': fpath
    }, features=Features(
        text_input=Value(dtype="string", id=None),
        text_output=Value(dtype="string", id=None)
    )
)
len(ds_panda['test']), len(ds_direct['test'])
```

## Expected results
Lines of `ds_panda['test']` and `ds_direct['test']` should match.

## Actual results
```
Using custom data configuration default-c0ef2598760968aa
Downloading and preparing dataset json/default to /data/junwang/.cache/huggingface/datasets/json/default-c0ef2598760968aa/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab...
Dataset json downloaded and prepared to /data/junwang/.cache/huggingface/datasets/json/default-c0ef2598760968aa/0.0.0/e6070c77f18f01a5ad4551a8b7edfba20b8438b7cad4d94e6ad9378022ce4aab. Subsequent calls will reuse this data.

(62087, 0)
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform: Ubuntu 18.04.4 LTS 
- Python version: 3.8.13
- PyArrow version: 9.0.0
HTTPError: 404 Client Error: Not Found for url

## Describe the bug

I was following chap 5 from huggingface course: https://huggingface.co/course/chapter5/6?fw=tf

However, I'm not able to download the datasets, with a 404 erros


<img width="1160" alt="iShot2022-10-06_15 54 50" src="https://user-images.githubusercontent.com/54015474/194406327-ae62c2f3-1da5-4686-8631-13d879a0edee.png">





## Steps to reproduce the bug
```python
from huggingface_hub import hf_hub_url

data_files = hf_hub_url(
    repo_id="lewtun/github-issues",
    filename="datasets-issues-with-hf-doc-builder.jsonl",
    repo_type="dataset",
)
from datasets import load_dataset

issues_dataset = load_dataset("json", data_files=data_files, split="train")
issues_dataset
```


## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.5.2
- Platform: macOS-10.16-x86_64-i386-64bit
- Python version: 3.9.12
- PyArrow version: 9.0.0
- Pandas version: 1.4.4
Filtering on an empty dataset returns a corrupted dataset.

## Describe the bug
When filtering a dataset twice, where the first result is an empty dataset, the second dataset seems corrupted.

## Steps to reproduce the bug
```python
  datasets = load_dataset("glue", "sst2")
  dataset_split = datasets['validation']
  ds_filter_1 = dataset_split.filter(lambda x: False)  # Some filtering condition that leads to an empty dataset
  assert ds_filter_1.num_rows == 0
  sentences = ds_filter_1['sentence']
  assert len(sentences) == 0
  ds_filter_2 = ds_filter_1.filter(lambda x: False)  # Some other filtering condition
  assert ds_filter_2.num_rows == 0
  assert 'sentence' in ds_filter_2.column_names
  sentences = ds_filter_2['sentence']
```

## Expected results
The last line should be returning an empty list, same as 4 lines above.

## Actual results
The last line currently raises `IndexError: index out of bounds`.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.5.2
- Platform: macOS-11.6.6-x86_64-i386-64bit
- Python version: 3.9.11
- PyArrow version: 7.0.0
- Pandas version: 1.4.1