Title: Passing `cache_dir` to a metric does not work
Author: sgugger (CONTRIBUTOR)
State: closed
Created: 2020-10-12T17:55:14Z

When passing `cache_dir` to a custom metric, the folder is concatenated to itself at some point and this results in a FileNotFoundError:

## Reproducer

```python
import datasets
import torch
from datasets import Metric

class GatherMetric(Metric):
    def _info(self):
        return datasets.MetricInfo(
            description="description",
            citation="citation",
            inputs_description="kwargs",
            features=datasets.Features({
                'predictions': datasets.Value('int64'),
                'references': datasets.Value('int64'),
            }),
            codebase_urls=[],
            reference_urls=[],
            format='numpy'
        )

    def _compute(self, predictions, references):
        return {"predictions": predictions, "labels": references}

metric = GatherMetric(cache_dir="test-metric")
inputs = torch.randint(0, 2, (1024,))
targets = torch.randint(0, 2, (1024,))

batch_size = 8
for i in range(0, 1024, batch_size):
    metric.add_batch(predictions=inputs[i:i+batch_size], references=targets[i:i+batch_size])
result = metric.compute()
```

## Stack trace:

```
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
~/git/datasets/src/datasets/metric.py in _finalize(self)
    349                 reader = ArrowReader(path=self.data_dir, info=DatasetInfo(features=self.features))
--> 350                 self.data = Dataset(**reader.read_files([{"filename": f} for f in file_paths]))
    351             except FileNotFoundError:

~/git/datasets/src/datasets/arrow_reader.py in read_files(self, files, original_instructions)
    227         # Prepend path to filename
--> 228         pa_table = self._read_files(files)
    229         files = copy.deepcopy(files)

~/git/datasets/src/datasets/arrow_reader.py in _read_files(self, files)
    166         for f_dict in files:
--> 167             pa_table: pa.Table = self._get_dataset_from_filename(f_dict)
    168             pa_tables.append(pa_table)

~/git/datasets/src/datasets/arrow_reader.py in _get_dataset_from_filename(self, filename_skip_take)
    291         )
--> 292         mmap = pa.memory_map(filename)
    293         f = pa.ipc.open_stream(mmap)

~/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.memory_map()

~/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.MemoryMappedFile._open()

~/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/.pyenv/versions/3.7.9/envs/base/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

FileNotFoundError: [Errno 2] Failed to open local file 'test-metric/gather_metric/default/test-metric/gather_metric/default/default_experiment-1-0.arrow'. Detail: [errno 2] No such file or directory

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-17-e42d43cc981f> in <module>
      2 for i in range(0, 1024, batch_size):
      3     metric.add_batch(predictions=inputs[i:i+batch_size], references=targets[i:i+batch_size])
----> 4 result = metric.compute()

~/git/datasets/src/datasets/metric.py in compute(self, *args, **kwargs)
    380         if predictions is not None:
    381             self.add_batch(predictions=predictions, references=references)
--> 382         self._finalize()
    383 
    384         self.cache_file_name = None

~/git/datasets/src/datasets/metric.py in _finalize(self)
    351             except FileNotFoundError:
    352                 raise ValueError(
--> 353                     "Error in finalize: another metric instance is already using the local cache file. "
    354                     "Please specify an experiment_id to avoid colision between distributed metric instances."
    355                 )

ValueError: Error in finalize: another metric instance is already using the local cache file. Please specify an experiment_id to avoid colision between distributed metric instances.
```

The code works when we remove the `cache_dir=...` from the metric.

Comments: