sunny2309/githug-issues huggingface dataset:
Add support for continuous metrics (RMSE, MAE)

**Is your feature request related to a problem? Please describe.**

I am uploading our dataset and models for the "Constructing interval measures" method we've developed, which uses item response theory to convert multiple discrete labels into a continuous spectrum for hate speech. Once we have this outcome our NLP models conduct regression rather than classification, so binary metrics are not relevant. The only continuous metrics available at https://huggingface.co/metrics are pearson & spearman correlation, which don't ensure that the prediction is on the same scale as the outcome.

**Describe the solution you'd like**
I would like to be able to tag our models on the Hub with the following metrics:
- RMSE
- MAE

**Describe alternatives you've considered**

I don't know if there are any alternatives.

**Additional context**
Our preprint is available here: https://arxiv.org/abs/2009.10277 . We are making it available for use in Jigsaw's Toxic Severity Rating Kaggle competition: https://www.kaggle.com/c/jigsaw-toxic-severity-rating/overview . I have our first model uploaded to the Hub at https://huggingface.co/ucberkeley-dlab/hate-measure-roberta-large

Thanks,
Chris
audio column not saved correctly after resampling

## Describe the bug
After resampling the audio column, saving with save_to_disk doesn't seem to save with the correct type. 

## Steps to reproduce the bug
- load a subset of common voice dataset (48Khz)
- resample audio column to 16Khz
- save with save_to_disk()
- load with load_from_disk()

## Expected results
I expected that after saving the data, and then loading it back in, the audio column has the correct dataset.Audio type (i.e. same as before saving it)
{'accent': Value(dtype='string', id=None),
 'age': Value(dtype='string', id=None),
 'audio': Audio(sampling_rate=16000, mono=True, _storage_dtype='string', id=None),
 'client_id': Value(dtype='string', id=None),
 'down_votes': Value(dtype='int64', id=None),
 'gender': Value(dtype='string', id=None),
 'locale': Value(dtype='string', id=None),
 'path': Value(dtype='string', id=None),
 'segment': Value(dtype='string', id=None),
 'sentence': Value(dtype='string', id=None),
 'up_votes': Value(dtype='int64', id=None)}

## Actual results
Audio column does not have the right type
{'accent': Value(dtype='string', id=None),
 'age': Value(dtype='string', id=None),
 'audio': {'bytes': Value(dtype='binary', id=None),
  'path': Value(dtype='string', id=None)},
 'client_id': Value(dtype='string', id=None),
 'down_votes': Value(dtype='int64', id=None),
 'gender': Value(dtype='string', id=None),
 'locale': Value(dtype='string', id=None),
 'path': Value(dtype='string', id=None),
 'segment': Value(dtype='string', id=None),
 'sentence': Value(dtype='string', id=None),
 'up_votes': Value(dtype='int64', id=None)}

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.17.0
- Platform: linux
- Python version:
- PyArrow version:
Adding Turkic X-WMT evaluation set for machine translation

This dataset is a human-translated evaluation set for MT crowdsourced and provided by the [Turkic Interlingua ](turkic-interlingua.org) community. It contains eval sets for 8 Turkic languages covering 88 language directions. Languages being covered are:

Azerbaijani (az)
Bashkir (ba)
English (en)
Karakalpak (kaa)
Kazakh (kk)
Kirghiz (ky)
Russian (ru)
Turkish (tr)
Sakha (sah)
Uzbek (uz)

More info about the corpus is here: [https://github.com/turkic-interlingua/til-mt/tree/master/xwmt](https://github.com/turkic-interlingua/til-mt/tree/master/xwmt)

A paper describing the test set is here: [https://arxiv.org/abs/2109.04593](https://arxiv.org/abs/2109.04593)
Dataset Viewer not showing Previews for Private Datasets

## Dataset viewer issue for 'abidlabs/test-audio-13'

It seems that the dataset viewer does not show previews for `private` datasets, even for the user who's private dataset it is. See [1] for example. If I change the visibility to public, then it does show, but it would be useful to have the viewer even for private datasets.

![image](https://user-images.githubusercontent.com/1778297/150200515-93ff1545-11fd-4793-be64-6bed3cd895e2.png)

**Link:** 

[1] https://huggingface.co/datasets/abidlabs/test-audio-13

**Am I the one who added this dataset?** 

Yes
Add British Library books dataset

This pull request adds a dataset of text from digitised (primarily 19th Century) books from the British Library. This collection has previously been used for training language models, e.g. https://github.com/dbmdz/clef-hipe/blob/main/hlms.md. It would be nice to make this dataset more accessible for others to use through datasets. 

This is still a WIP but I wanted to get some initial feedback in particular; I wanted to check:
- I am handling the use of `iter_archive` correctly - I intend to ensure that `dl_manager.download` gets the complete list of URLs to download upfront, so the progress bar knows how much is left to download and then to pass through the `gen_kwargs` a list of downloaded zip archives wrapped in `iter_archive`. I am unsure if there is a more elegant approach for this?
- the number of configs: I have aimed to keep this limited - there are a lot of URLs covering the entire dataset, but I have tried to base the configs on what I believe the majority of people will want to they are not presented with too many options - I am happy to hear suggestions for changing this

If there are other glaring omissions or mistakes, I'd be happy to hear them. If this approach seems sensible in general, I will finish all the remaining TODOs, generate dummy_data, etc.
Readme info not being parsed to show on Dataset card page

## Describe the bug
The info contained in the README.md file is not being shown in the dataset main page. Basic info and table of contents are properly formatted in the README.

## Steps to reproduce the bug

# Sample code to reproduce the bug
The README file is this one: https://huggingface.co/datasets/softcatala/Tilde-MODEL-Catalan/blob/main/README.md

## Expected results
README info should appear in the Dataset card page.

## Actual results
Nothing is shown. However, labels are parsed and shown successfully.
ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode: /content

## Bug
The install of streaming dataset is giving following error.

## Steps to reproduce the bug
```python
! git clone https://github.com/huggingface/datasets.git
! cd datasets
! pip install -e ".[streaming]"
```

## Actual results
Cloning into 'datasets'...
remote: Enumerating objects: 50816, done.
remote: Counting objects: 100% (2356/2356), done.
remote: Compressing objects: 100% (1606/1606), done.
remote: Total 50816 (delta 834), reused 1741 (delta 525), pack-reused 48460
Receiving objects: 100% (50816/50816), 72.47 MiB | 27.68 MiB/s, done.
Resolving deltas: 100% (22541/22541), done.
Checking out files: 100% (6722/6722), done.
ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode: /content
fix multiple language downloading in mC4

If we try to access multiple languages of the [mC4 dataset](https://github.com/huggingface/datasets/tree/master/datasets/mc4), it will throw an error. For example, if we do 
```python
mc4_subset_two_langs = load_dataset("mc4", languages=["st", "su"])
```
we got
```
FileNotFoundError: Couldn't find file at https://huggingface.co/datasets/allenai/c4/resolve/1ddc917116b730e1859edef32896ec5c16be51d0/multilingual/c4-st+su.tfrecord-00000-of-00002.json.gz
```
Now it should work. Check it (from the root dir of a project): 
```python
mc4_subset_two_langs = load_dataset("./datasets/mc4/", languages=["st", "su"])
```
Datasets streaming + map doesn't work for `Audio`

## Describe the bug
When using audio datasets in streaming mode, applying a `map(...)` before iterating leads to an error as the key `array` does not exist anymore.

## Steps to reproduce the bug

```python
from datasets import load_dataset

ds = load_dataset("common_voice", "en", streaming=True, split="train")


def map_fn(batch):
    print("audio keys", batch["audio"].keys())
    batch["audio"] = batch["audio"]["array"][:100]
    return batch


ds = ds.map(map_fn) 

sample = next(iter(ds))
```

I think the audio is somehow decoded before `.map(...)` is actually called.

## Expected results

IMO, the above code snippet should work.

## Actual results

```bash
audio keys dict_keys(['path', 'bytes'])
Traceback (most recent call last):
  File "./run_audio.py", line 15, in <module>
    sample = next(iter(ds))
  File "/home/patrick/python_bin/datasets/iterable_dataset.py", line 341, in __iter__
    for key, example in self._iter():
  File "/home/patrick/python_bin/datasets/iterable_dataset.py", line 338, in _iter
    yield from ex_iterable
  File "/home/patrick/python_bin/datasets/iterable_dataset.py", line 192, in __iter__
    yield key, self.function(example)
  File "./run_audio.py", line 9, in map_fn
    batch["input"] = batch["audio"]["array"][:100]
KeyError: 'array'
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.17.1.dev0
- Platform: Linux-5.3.0-64-generic-x86_64-with-glibc2.17
- Python version: 3.8.12
- PyArrow version: 6.0.1
Add The Medical Segmentation Decathlon Dataset

## Adding a Dataset
- **Name:** *The Medical Segmentation Decathlon Dataset*
- **Description:** The underlying data set was designed to explore the axis of difficulties typically encountered when dealing with medical images, such as small data sets, unbalanced labels, multi-site data, and small objects. 
- **Paper:** [link to the dataset paper if available](https://arxiv.org/abs/2106.05735)
- **Data:** http://medicaldecathlon.com/
- **Motivation:** Hugging Face seeks to democratize ML for society. One of the growing niches within ML is the ML + Medicine community. Key data sets will help increase the supply of HF resources for starting an initial community.

(cc @osanseviero @abidlabs )

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
conll 2003 dataset source url is no longer valid

## Describe the bug
Loading `conll2003` dataset fails because it was removed (just yesterday 1/14/2022) from the location it is looking for.

## Steps to reproduce the bug
```python
from datasets import load_dataset


load_dataset("conll2003")
```

## Expected results
The dataset should load.

## Actual results
It is looking for the dataset at `https://github.com/davidsbatista/NER-datasets/raw/master/CONLL2003/train.txt` but it was removed from there yesterday (see [commit](https://github.com/davidsbatista/NER-datasets/commit/9d8f45cc7331569af8eb3422bbe1c97cbebd5690) that removed the file and related [issue](https://github.com/davidsbatista/NER-datasets/issues/8)). 

- We should replace this with an alternate valid location.
- this is being referenced in the huggingface course chapter 7 [colab notebook](https://colab.research.google.com/github/huggingface/notebooks/blob/master/course/chapter7/section2_pt.ipynb), which is also broken.

```python
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-4-27c956bec93c> in <module>()
      1 from datasets import load_dataset
      2 
----> 3 raw_datasets = load_dataset("conll2003")

11 frames
/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token, ignore_url_params)
    610             )
    611         elif response is not None and response.status_code == 404:
--> 612             raise FileNotFoundError(f"Couldn't find file at {url}")
    613         _raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
    614         if head_error is not None:

FileNotFoundError: Couldn't find file at https://github.com/davidsbatista/NER-datasets/raw/master/CONLL2003/train.txt
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version:
- Platform:
- Python version:
- PyArrow version:
Bug in wiki bio load


wiki_bio is failing to load because of a failing drive link . Can someone fix this ?


![7E90023B-A3B1-4930-BA25-45CCCB4E1710](https://user-images.githubusercontent.com/3104771/149617870-5a32a2da-2c78-483b-bff6-d7534215a423.png)


![653C1C76-C725-4A04-A0D8-084373BA612F](https://user-images.githubusercontent.com/3104771/149617875-ef0e30b0-b76e-48cf-b3eb-93ba8e6e5465.png)
a
label information get lost after parquet serialization

## Describe the bug
In *dataset_info.json* file, information about the label get lost after the dataset serialization.

## Steps to reproduce the bug
```python
from datasets import load_dataset

# normal save
dataset = load_dataset('glue', 'sst2', split='train')
dataset.save_to_disk("normal_save")


# save after parquet serialization
dataset.to_parquet("glue-sst2-train.parquet")
dataset = load_dataset("parquet", data_files='glue-sst2-train.parquet')
dataset.save_to_disk("save_after_parquet")
```

## Expected results
I expected to keep label information in *dataset_info.json* file even after parquet serialization

## Actual results
In the normal serialization i got
```json
"label": {
      "num_classes": 2,
      "names": [
        "negative",
        "positive"
      ],
      "names_file": null,
      "id": null,
      "_type": "ClassLabel"
    },
```
And after parquet serialization i got
```json
"label": {
      "dtype": "int64",
      "id": null,
      "_type": "Value"
    },
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.18.0
- Platform: ubuntu 20.04
- Python version: 3.8.10
- PyArrow version: 6.0.1
ConnectionError in IndicGLUE dataset

While I am trying to load IndicGLUE dataset (https://huggingface.co/datasets/indic_glue) it is giving me with the error:
```
ConnectionError: Couldn't reach https://storage.googleapis.com/ai4bharat-public-indic-nlp-corpora/evaluations/wikiann-ner.tar.gz (error 403)
Fix push to hub to allow individual split push

# Description of the issue

If one decides to push a split on a datasets repo, he uploads the dataset and overrides the config. However previous config splits end up being lost despite still having the dataset necessary.

The new flow is the following:
 - query the old config from the repo
 - update into a new config (add/overwrite new split for example)
 - push the new config

# Side fix

 - `repo_id` in HfFileSystem was wrongly typed.
 - I've added `indent=2` as it becomes much easier to read now.
Dataset.from_pandas preserves useless index

## Describe the bug
Let's say that you want to create a Dataset object from pandas dataframe. Most likely you will write something like this:

```
import pandas as pd
from datasets import Dataset


df = pd.read_csv('some_dataset.csv')
# Some DataFrame preprocessing code...
dataset = Dataset.from_pandas(df)
```
If your preprocessing code contain indexing operations like this:
```
df = df[df.col1 == some_value]
```
then your df.index can be changed from (default) ```RangeIndex(start=0, stop=16590, step=1)``` to something like this ```Int64Index([    0,     1,     2,     3,     4,     5,     6,     7,     8,
                9,
            ...
            83979, 83980, 83981, 83982, 83983, 83984, 83985, 83986, 83987,
            83988],
           dtype='int64', length=16590)```

In this case, PyArrow (by default) will preserve this non-standard index. In the result, your dataset object will have the extra field that you likely don't want to have: '__index_level_0__'.

You can easily fix this by just adding extra argument ```preserve_index=False``` to call of ```InMemoryTable.from_pandas``` in ```arrow_dataset.py```.

If you approve that this isn't desirable behavior, I can make a PR fixing that.

## Environment info
- `datasets` version: 1.16.1
- Platform: Linux-5.11.0-44-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 6.0.1
Cannot load ‘bookcorpusopen’

## Describe the bug
Cannot load 'bookcorpusopen'


## Steps to reproduce the bug
```python
dataset = load_dataset('bookcorpusopen')
```
or
```python
dataset = load_dataset('bookcorpusopen',script_version='master')
```

## Actual results
ConnectionError: Couldn't reach https://the-eye.eu/public/AI/pile_preliminary_components/books1.tar.gz

## Environment info
- `datasets` version: 1.9.0
- Platform: Linux version 3.10.0-1160.45.1.el7.x86_64
- Python version:  3.6.13
- PyArrow version: 6.0.1
DuplicatedKeysError when loading tweet_qa dataset

When loading the tweet_qa dataset with `load_dataset('tweet_qa')`, the following error occurs: 

`DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 2a167f9e016ba338e1813fed275a6a1e
Keys should be unique and deterministic in nature
`
Might be related to issues #2433 and #2333

- `datasets` version: 1.17.0
- Python version: 3.8.5
set_format("np") no longer works for Image data

## Describe the bug
`dataset.set_format("np")` no longer works for image data, previously you could load the MNIST like this:

```python
dataset = load_dataset("mnist")
dataset.set_format("np")
X_train = dataset["train"]["image"][..., None] # <== No longer a numpy array
```
but now it doesn't work, `set_format("np")` seems to have no effect and the dataset just returns a list/array of PIL images instead of numpy arrays as requested.
Bug in `openbookqa` dataset

## Describe the bug
Dataset entries contains a typo.

## Steps to reproduce the bug

```python
>>> from datasets import load_dataset
>>> obqa = load_dataset('openbookqa', 'main')
>>> obqa['train'][0]
```

## Expected results
```python
{'id': '7-980', 'question_stem': 'The sun is responsible for', 'choices': {'text': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting'], 'label': ['A', 'B', 'C', 'D']}, 'answerKey': 'D'}
```

## Actual results
```python
{'id': '7-980', 'question_stem': 'The sun is responsible for', 'choices': {'text': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting'], 'label': ['puppies learning new tricks', 'children growing up and getting old', 'flowers wilting in a vase', 'plants sprouting, blooming and wilting']}, 'answerKey': 'D'}
```

The bug is present in all configs and all splits.

## Environment info
- `datasets` version: 1.17.0
- Platform: Linux-5.4.0-1057-aws-x86_64-with-glibc2.27
- Python version: 3.9.7
- PyArrow version: 4.0.1
Specify the feature types of a dataset on the Hub without needing a dataset script

**Is your feature request related to a problem? Please describe.**
Currently if I upload a CSV with paths to audio files, the column type is string instead of Audio.

**Describe the solution you'd like**
I'd like to be able to specify the types of the column, so that when loading the dataset I directly get the features types I want.

The feature types could read from the `dataset_infos.json` for example.

**Describe alternatives you've considered**
Create a dataset script to specify the features, but that seems complicated for a simple thing.

cc @abidlabs
Datasets created with `push_to_hub` can't be accessed in offline mode

## Describe the bug
In offline mode, one can still access previously-cached datasets. This fails with datasets created with `push_to_hub`.

## Steps to reproduce the bug
in Python:
```
import datasets
mpwiki = datasets.load_dataset("teven/matched_passages_wikidata")
```
in bash:
```
export HF_DATASETS_OFFLINE=1
```
in Python:
```
import datasets
mpwiki = datasets.load_dataset("teven/matched_passages_wikidata")
```

## Expected results
`datasets` should find the previously-cached dataset.

## Actual results
ConnectionError: Couln't reach the Hugging Face Hub for dataset 'teven/matched_passages_wikidata': Offline mode is enabled

## Environment info
- `datasets` version: 1.16.2.dev0
- Platform: Linux-4.18.0-193.70.1.el8_2.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.10
- PyArrow version: 3.0.0
Allow loading community metrics from the hub, just like datasets

**Is your feature request related to a problem? Please describe.**
Currently, I can load a metric implemented by me by providing the local path to the file in `load_metric`.
However, there is no option to do it with the metric uploaded to the hub.
This means that if I want to allow other users to use it, they must download it first which makes the usage less smooth.

**Describe the solution you'd like**
Load metrics from the hub just like datasets are loaded.
In order to not break stuff, the convention can be to put the metric file in a "metrics" folder in the hub.
Task search function on hub not working correctly

When I want to look at all datasets of the category: `speech-processing`  *i.e.* https://huggingface.co/datasets?task_categories=task_categories:speech-processing&sort=downloads , then the following dataset doesn't show up for some reason:

- https://huggingface.co/datasets/speech_commands

even thought it's task tags seem correct: 
https://raw.githubusercontent.com/huggingface/datasets/master/datasets/speech_commands/README.md
wmt19 is broken (zh-en)

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("wmt19", 'zh-en')
```

## Expected results
The dataset should download.

## Actual results
`ConnectionError: Couldn't reach ftp://cwmt-wmt:cwmt-wmt@datasets.nju.edu.cn/parallel/casia2015.zip`

## Environment info
- `datasets` version: 1.15.1
- Platform: Linux
- Python version: 3.8
Add PubMed Central Open Access dataset

## Adding a Dataset
- **Name:** PubMed Central Open Access
- **Description:** The PMC Open Access Subset includes more than 3.4 million journal articles and preprints that are made available under license terms that allow reuse.
- **Paper:** *link to the dataset paper if available*
- **Data:** https://www.ncbi.nlm.nih.gov/pmc/tools/openftlist/
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
`wiki_dpr` details for Open Domain Question Answering tasks

Hey guys!

Thanks for creating the `wiki_dpr` dataset!

I am currently trying to use the dataset for context retrieval using DPR on NQ questions and need details about what each of the files and data instances mean, which version of the Wikipedia dump it uses, etc. Please respond at your earliest convenience regarding the same! Thanks a ton!

P.S.: (If one of @thomwolf @lewtun @lhoestq could respond, that would be even better since they have the first-hand details of the dataset. If anyone else has those, please reach out! Thanks!)