sunny2309/githug-issues huggingface dataset:
Custom formatting for lazy map + arrow data extraction refactor

Hi !

This PR refactors the way data are extracted from pyarrow tables to extend it to the use of custom formatting functions.

While the internal storage of the dataset is always the Apache Arrow format, by setting a specific format on a dataset, you can cast the output of `datasets.Dataset.__getitem__` in NumPy/pandas/PyTorch/TensorFlow, on-the-fly.

A specific format can be activated with `datasets.Dataset.set_format`. For example: `dataset.set_format(type='torch', columns=['label'])`.

### What's new:

You can now also define your own formatting function that is applied on-the-fly. To do so you can pass your formatting function in the `transform` parameter of `datasets.Dataset.set_format`, and keep `type` to `None`.
A formatting function is a callable that takes a batch (as a dict, formatted as python) as input and returns a batch.

Here is an example to tokenize and pad tokens on-the-fly when accessing the samples:

```python
from datasets import load_dataset
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
def encode(batch):
    return tokenizer(batch["sentence1"], padding="longest", truncation=True, max_length=512, return_tensors="pt")

dataset = load_dataset("glue", "mrpc", split="train")
dataset.set_format(transform=encode)
dataset.format
# {'type': 'custom', 'format_kwargs': {'transform': <function __main__.encode(batch)>}, 'columns': ['idx', 'label', 'sentence1', 'sentence2'], 'output_all_columns': False}
dataset[:2]
# {'input_ids': tensor([[  101,  2572,  3217, ... 102]]), 'token_type_ids': tensor([[0, 0, 0, ... 0]]), 'attention_mask': tensor([[1, 1, 1, ... 1]])}
```

Let me know what you think of this API !
We can still change it if we want to.

Especially @sgugger since this may be useful when using `datasets` to train models.

EDIT: this was changed to `dataset.set_transform(encode)`

-------------------

Note:

I had to refactor the way data are extracted and formatted from pyarrow tables and I made it more robust and flexible. In particular I modularized it to be able to unit-test it properly. This was very helpful since I detected some bugs in the previous implementation and was able to fix them.

Some bugs I found and fixed:
- certain slices/ranges were not supported because negative ids were passed to pyarrow
- formatting as numpy/torch/tensorflow a column would make it lose its precision information (for example a column as `Value("float32")`) would be returned as a tensor of float64 (default behavior for numpy)
- on windows  integers formatted as numpy/torch/tensorflow were not always int64 tensors by default but were int32 

The unit tests for those are now really extensive :)
ModuleNotFoundError: No module named 'apache_beam', when specific languages.

```py
import datasets
wiki = datasets.load_dataset('wikipedia', '20200501.ja', cache_dir='./datasets')
```
then `ModuleNotFoundError: No module named 'apache_beam'` happend.

The error doesn't appear when it's '20200501.en'.
I don't know Apache Beam, but according to #498 it isn't necessary when it's saved to local. is it correct?
How to use split dataset 

![Capture1](https://user-images.githubusercontent.com/78090287/106057436-cb6a1f00-6111-11eb-8c9c-3658065b1fdf.PNG)

Hey,
I want to split the lambada dataset into corpus, test, train and valid txt files (like penn treebank) but I am not able to achieve this. What I am doing is, executing the lambada.py file in my project but its not giving desired results. Any help will be appreciated!
Not enough disk space (Needed: Unknown size) when caching on a cluster

I'm running some experiments where I'm caching datasets on a cluster and accessing it through multiple compute nodes. However, I get an error when loading the cached dataset from the shared disk.

The exact error thrown:

```bash
>>> load_dataset(dataset, cache_dir="/path/to/cluster/shared/path")
OSError: Not enough disk space. Needed: Unknown size (download: Unknown size, generated: Unknown size, post-processed: Unknown size)
```


[`utils.has_sufficient_disk_space`](https://github.com/huggingface/datasets/blob/8a03ab7d123a76ee744304f21ce868c75f411214/src/datasets/utils/py_utils.py#L332) fails on each job because of how the cluster system is designed (`disk_usage(".").free` can't compute on the cluster's shared disk).


This is exactly where the error gets thrown:
https://github.com/huggingface/datasets/blob/master/src/datasets/builder.py#L502

```python
if not utils.has_sufficient_disk_space(self.info.size_in_bytes or 0, directory=self._cache_dir_root):
    raise IOError(
          "Not enough disk space. Needed: {} (download: {}, generated: {}, post-processed: {})".format(
          utils.size_str(self.info.size_in_bytes or 0),
          utils.size_str(self.info.download_size or 0),
          utils.size_str(self.info.dataset_size or 0),
          utils.size_str(self.info.post_processing_size or 0),
       )
    )

```

What would be a good way to circumvent this? my current fix is to manually comment out that part, but that is not ideal. 
Would it be possible to pass a flag to skip this check on disk space?
JSONDecodeError on JSON with multiple lines

Hello :),

I have been trying to load data using a JSON file. Based on the [docs](https://huggingface.co/docs/datasets/loading_datasets.html#json-files), the following format is supported:

```json
{"key1":11, "key2":12, "key3":13}
{"key1":21, "key2":22, "key3":23}
```
 But, when I try loading a dataset with the same format, I get a JSONDecodeError : `JSONDecodeError: Extra data: line 2 column 1 (char 7142)`. Now, this is expected when using `json` to load a JSON file. But I was wondering if there are any special arguments to pass when using `load_dataset` as the docs suggest that this format is supported.

When I convert the JSON file to a list of dictionaries format, I get AttributeError: `AttributeError: 'list' object has no attribute 'keys'`. So, I can't convert them to list of dictionaries either.

Please let me know :)

Thanks,
Gunjan
Dataset Examples Explorer

In the Older version of the Dataset, there are a useful Dataset Explorer that allow user to visualize the examples (training, test and validation) of a particular dataset, it is no longer there in current version.

Hope HuggingFace can re-enable the feature that at least allow viewing of  the first 20  examples of a particular dataset, or alternatively can extract 20 examples for each datasets and make those part of the Dataset Card Documentation.
AttributeError: module 'pyarrow' has no attribute 'PyExtensionType' during import 

I'm using Colab. And suddenly this morning, there is this error. Have a look below!

![screenshot-colab research google com-2021 01 26-08-15-36](https://user-images.githubusercontent.com/45964869/105799890-fdaf3b80-5fae-11eb-8f06-11b65cdccc30.png)
[Question & Bug Report] Can we preprocess a dataset on the fly?

I know we can use `Datasets.map` to preprocess a dataset, but I'm using it with very large corpus which generates huge cache file (several TB cache from a 400 GB text file). I have no disk large enough to save it.  Can we preprocess a dataset on the fly without generating cache?

BTW, I tried raising `writer_batch_size`. Seems that argument doesn't have any effect when it's larger than `batch_size`, because you are saving all the batch instantly after it's processed. Please check the following code:

https://github.com/huggingface/datasets/blob/0281f9d881f3a55c89aeaa642f1ba23444b64083/src/datasets/arrow_dataset.py#L1532
Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.2.1/datasets/csv/csv.py

Hi,
When I load_dataset from local csv files, below error happened, looks raw.githubusercontent.com was blocked by the chinese government. But why it need to download csv.py? should it include when pip install the dataset?

```
Traceback (most recent call last):
  File "/home/tom/pyenv/pystory/lib/python3.6/site-packages/datasets/load.py", line 267, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/home/tom/pyenv/pystory/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 343, in cached_path
    max_retries=download_config.max_retries,
  File "/home/tom/pyenv/pystory/lib/python3.6/site-packages/datasets/utils/file_utils.py", line 617, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.2.1/datasets/csv/csv.py

```
Add Librispeech ASR

This PR adds the librispeech asr dataset: https://www.tensorflow.org/datasets/catalog/librispeech

There are 2 configs: "clean" and "other" whereas there are two "train" datasets for "clean", hence the name "train.100" and "train.360".

As suggested by @lhoestq, due to the enormous size of the dataset in `.arrow` format, the speech files are not directly prepared to a float32-array, but instead just the path to the array file is stored.
Issues when run two programs compute the same metrics

I got the following error when running two different programs that both compute sacreblue metrics. It seems that both read/and/write to the same location (.cache/huggingface/metrics/sacrebleu/default/default_experiment-1-0.arrow) where it caches the batches:

```
File "train_matching_min.py", line 160, in <module>ch_9_label
    avg_loss = valid(epoch, args.batch, args.validation, args.with_label)
  File "train_matching_min.py", line 93, in valid
    bleu += eval.compute()
  File "/u/tlhoang/projects/seal/match/models/eval.py", line 23, in compute
    return self.metric.compute()['score']
  File "/dccstor/know/anaconda3/lib/python3.7/site-packages/datasets/metric.py", line 387, in compute
    self._finalize()
  File "/dccstor/know/anaconda3/lib/python3.7/site-packages/datasets/metric.py", line 355, in _finalize
    self.data = Dataset(**reader.read_files([{"filename": f} for f in file_paths]))
  File "/dccstor/know/anaconda3/lib/python3.7/site-packages/datasets/arrow_reader.py", line 231, in read_files
    pa_table = self._read_files(files)
  File "/dccstor/know/anaconda3/lib/python3.7/site-packages/datasets/arrow_reader.py", line 170, in _read_files
    pa_table: pa.Table = self._get_dataset_from_filename(f_dict)
  File "/dccstor/know/anaconda3/lib/python3.7/site-packages/datasets/arrow_reader.py", line 299, in _get_dataset_from_filename
    pa_table = f.read_all()
  File "pyarrow/ipc.pxi", line 481, in pyarrow.lib.RecordBatchReader.read_all
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Expected to read 1819307375 metadata bytes, but only read 454396
```
Error iterating over Dataset with DataLoader

I have a Dataset that I've mapped a tokenizer over:

```
encoded_dataset.set_format(type='torch',columns=['attention_mask','input_ids','token_type_ids'])
encoded_dataset[:1]
```
```
{'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]),
 'input_ids': tensor([[  101,   178,  1198,  1400,  1714, 22233, 21365,  4515,  8618,  1113,
            102]]),
 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])}
```

When I try to iterate as in the docs, I get errors:

```
dataloader = torch.utils.data.DataLoader(encoded_dataset, batch_sampler=32)
next(iter(dataloader))
```

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-45-05180ba8aa35> in <module>()
      1 dataloader = torch.utils.data.DataLoader(encoded_dataset, batch_sampler=32)
----> 2 next(iter(dataloader))

3 frames
/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __init__(self, loader)
    411         self._timeout = loader.timeout
    412         self._collate_fn = loader.collate_fn
--> 413         self._sampler_iter = iter(self._index_sampler)
    414         self._base_seed = torch.empty((), dtype=torch.int64).random_(generator=loader.generator).item()
    415         self._persistent_workers = loader.persistent_workers

TypeError: 'int' object is not iterable


```
Connection Issues

Today, I am getting connection issues while loading a dataset and the metric.
```
Traceback (most recent call last):
  File "src/train.py", line 180, in <module>
    train_dataset, dev_dataset, test_dataset = create_race_dataset()
  File "src/train.py", line 130, in create_race_dataset
    train_dataset = load_dataset("race", "all", split="train")
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/load.py", line 591, in load_dataset
    path, script_version=script_version, download_config=download_config, download_mode=download_mode, dataset=True
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/load.py", line 267, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 343, in cached_path
    max_retries=download_config.max_retries,
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 617, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.2.1/datasets/race/race.py
```

Or

```
Traceback (most recent call last):
  File "src/train.py", line 105, in <module>
    rouge = datasets.load_metric("rouge")
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/load.py", line 500, in load_metric
    dataset=False,
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/load.py", line 267, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 343, in cached_path
    max_retries=download_config.max_retries,
  File "/Users/saeed/Desktop/codes/repos/dreamscape-qa/env/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 617, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach https://raw.githubusercontent.com/huggingface/datasets/1.2.1/metrics/rouge/rouge.py
```
Unable to format dataset to CUDA Tensors

Hi,

I came across this [link](https://huggingface.co/docs/datasets/torch_tensorflow.html) where the docs show show to convert a dataset to a particular format. I see that there is an option to convert it to tensors, but I don't see any option to convert it to CUDA tensors.

I tried this, but Dataset doesn't support assignment:
```
  columns=['input_ids', 'token_type_ids', 'attention_mask', 'start_positions','end_positions']

        samples.set_format(type='torch', columns = columns)
        for column in columns:
            samples[column].to(torch.device(self.config.device))
```
There should be an option to do so, or if there is already a way to do this, please let me know.

Thanks,
Gunjan
wikipedia dataset incomplete

Hey guys,

I am using the https://github.com/huggingface/datasets/tree/master/datasets/wikipedia dataset.
Unfortunately, I found out that there is an incompleteness for the German dataset.
 For reasons unknown to me, the number of inhabitants has been removed from many pages:
Thorey-sur-Ouche has 128 inhabitants according to the webpage (https://de.wikipedia.org/wiki/Thorey-sur-Ouche).
The pickle file however shows: französische Gemeinde mit  Einwohnern (Stand).
 Is it possible to fix this?

Best regards 
Chris
FewRel

## Adding a Dataset
- **Name:** FewRel
- **Description:**  Large-Scale Supervised Few-Shot Relation Classification Dataset
- **Paper:** @inproceedings{han2018fewrel,
               title={FewRel:A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation},
               author={Han, Xu and Zhu, Hao and Yu, Pengfei and Wang, Ziyun and Yao, Yuan and Liu, Zhiyuan and Sun, Maosong},
               booktitle={EMNLP},
               year={2018}}
- **Data:** https://github.com/ProKil/FewRel
- **Motivation:** relationship extraction dataset that's been used by some state of the art systems that should be incorporated.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).
COMET metric citation

In my last pull request to add COMET metric, the citations where not following the usual "format". Because of that they where not correctly displayed on the website: 

<img width="814" alt="Screenshot 2021-01-20 at 09 48 44" src="https://user-images.githubusercontent.com/17256847/105158000-686efb80-5b05-11eb-8bb0-9c85fdac2938.png">

This pull request is only intended to fix that.
Added metadata and correct splits for swda.

Switchboard Dialog Act Corpus

I made some changes following  @bhavitvyamalik recommendation in #1678:

* Contains all metadata.
* Used official implementation from the [/swda](https://github.com/cgpotts/swda) repo.
* Add official train and test splits used in [Stolcke et al. (2000)](https://web.stanford.edu/~jurafsky/ws97) and validation split used in [Probabilistic-RNN-DA-Classifier](https://github.com/NathanDuran/Probabilistic-RNN-DA-Classifier).
Issue while Creating Custom Metric

Hi Team,

I am trying to create a custom metric for my training as follows, where f1 is my own metric:

```python
 def _info(self):
        # TODO: Specifies the datasets.MetricInfo object
        return datasets.MetricInfo(
            # This is the description that will appear on the metrics page.
            description=_DESCRIPTION,
            citation=_CITATION,
            inputs_description=_KWARGS_DESCRIPTION,
            # This defines the format of each prediction and reference
            features = datasets.Features({'predictions':datasets.Sequence(datasets.Value("int32")), "references": datasets.Sequence(datasets.Value("int32")),"offset_mapping":datasets.Sequence(datasets.Value("int32")),'text':datasets.Sequence(datasets.Value('string')),"ground":datasets.Sequence(datasets.Value("int32")),}),
            # Homepage of the metric for documentation
            homepage="http://metric.homepage",
            # Additional links to the codebase or references
            codebase_urls=["http://github.com/path/to/codebase/of/new_metric"],
            reference_urls=["http://path.to.reference.url/new_metric"]
        )

    def _compute(self,predictions,references,text,offset_mapping,spans):

        pred_spans = []

        for i,preds in enumerate(predictions):
            current_preds = []
            for j,token_preds in enumerate(preds):
                if (preds>0.5):
                    current_preds+=list(range(offset_mapping[i][j][0],offset_mapping[i][j][1]))
            pred_spans.append(current_spans)
        
        return {
            "Token Wise F1": f1_score(references,predictions,labels=[0,1]),
            "Offset Wise F1": np.mean([f1(preds,gold) for preds,fold in zip(pred_spans,ground)])
        }

```

I believe this is not correct. But that's not the issue I am facing right now. I get this error :
```python
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-144-ed7349b50821> in <module>()
----> 1 new_metric.compute(predictions=inputs["labels"],references=inputs["labels"], text=inputs["text"], offset_mapping=inputs["offset_mapping"],ground=inputs["ground"] )

2 frames
/usr/local/lib/python3.6/dist-packages/datasets/features.py in encode_batch(self, batch)
    802         encoded_batch = {}
    803         if set(batch) != set(self):
--> 804             print(batch)
    805             print(self)
    806             raise ValueError("Column mismatch between batch {} and features {}".format(set(batch), set(self)))

ValueError: Column mismatch between batch {'references', 'predictions'} and features {'ground', 'predictions', 'offset_mapping', 'text', 'references'}
```
On checking the features.py file, I see the call is made from add_batch() in metrics.py which only takes in predictions and references.

How do I make my custom metric work? Will it work with a trainer even if I am able to make this metric work?

Thanks,
Gunjan
Add GLUE Compat (compatible with transformers<3.5.0)

Link to our discussion on Slack (HF internal)
https://huggingface.slack.com/archives/C014N4749J9/p1609668119337400

The next step is to add a compatible option in the new `run_glue.py`

I duplicated `glue` and made the following changes:
1. Change the name to `glue_compat`.
2. Change the label assignments for MNLI and AX.
error when run fine_tuning on text_classification

dataset:sem_eval_2014_task_1
pretrained_model:bert-base-uncased

error description:
when i use these resoruce to train fine_tuning a text_classification on sem_eval_2014_task_1,there always be some problem(when i use other dataset ,there exist the error too). And i followed the colab code (url:https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/text_classification.ipynb#scrollTo=TlqNaB8jIrJW).


the error is like this :
`File "train.py", line 69, in <module>
    trainer.train()
  File "/home/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/transformers/trainer.py", line 784, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/home/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 435, in __next__
    data = self._next_data()
  File "/home/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 475, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/projects/anaconda3/envs/calibration/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
KeyError: 2`

this is my code :
```dataset_name = 'sem_eval_2014_task_1'
num_labels_size = 3
batch_size = 4
model_checkpoint = 'bert-base-uncased'
number_train_epoch = 5

def tokenize(batch):
return tokenizer(batch['premise'], batch['hypothesis'], truncation=True, )

def compute_metrics(pred):
labels = pred.label_ids
preds = pred.predictions.argmax(-1)
precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='micro')
acc = accuracy_score(labels, preds)
return {
'accuracy': acc,
'f1': f1,
'precision': precision,
'recall': recall
}

model = BertForSequenceClassification.from_pretrained(model_checkpoint, num_labels=num_labels_size)
tokenizer = BertTokenizerFast.from_pretrained(model_checkpoint, use_fast=True)

train_dataset = load_dataset(dataset_name, split='train')
test_dataset = load_dataset(dataset_name, split='test')

train_encoded_dataset = train_dataset.map(tokenize, batched=True)
test_encoded_dataset = test_dataset.map(tokenize, batched=True)

args = TrainingArguments(
output_dir='./results',
evaluation_strategy="epoch",
learning_rate=2e-5,
per_device_train_batch_size=batch_size,
per_device_eval_batch_size=batch_size,
num_train_epochs=number_train_epoch,
weight_decay=0.01,
do_predict=True,
)
trainer = Trainer(
model=model,
args=args,
compute_metrics=compute_metrics,
train_dataset=train_encoded_dataset,
eval_dataset=test_encoded_dataset,
tokenizer=tokenizer
)

trainer.train()
trainer.evaluate()