Repository: huggingface/evaluate
Issue: Trying to use metric.compute but get OSError
Root cause: Resource Mishandling

I want to use metric.compute from load_metric('accuracy') to get training accuracy, but receive OSError. I am wondering what is the mechanism behind the metric calculation, why would it report an OSError?

```python
195     for epoch in range(num_train_epochs):
196         model.train()
197         for step, batch in enumerate(train_loader):
198             # print(batch['input_ids'].shape)
199             outputs = model(**batch)
200
201             loss = outputs.loss
202             loss /= gradient_accumulation_steps
203             accelerator.backward(loss)
204
205             predictions = outputs.logits.argmax(dim=-1)
206             metric.add_batch(
207                 predictions=accelerator.gather(predictions),
208                 references=accelerator.gather(batch['labels'])
209             )
210             progress_bar.set_postfix({'loss': loss.item(), 'train batch acc.': train_metrics})
211
212             if (step + 1) % 50 == 0 or step == len(train_loader) - 1:
213                 train_metrics = metric.compute()
```

the error message is as below:

```
Traceback (most recent call last):
  File "run_multi.py", line 273, in <module>
    main()
  File "/home/yshuang/.local/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/yshuang/.local/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/yshuang/.local/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/yshuang/.local/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "run_multi.py", line 213, in main
    train_metrics = metric.compute()
  File "/home/yshuang/.local/lib/python3.8/site-packages/datasets/metric.py", line 391, in compute
    self._finalize()
  File "/home/yshuang/.local/lib/python3.8/site-packages/datasets/metric.py", line 342, in _finalize
    self.writer.finalize()
  File "/home/yshuang/.local/lib/python3.8/site-packages/datasets/arrow_writer.py", line 370, in finalize
    self.stream.close()
  File "pyarrow/io.pxi", line 132, in pyarrow.lib.NativeFile.close
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
OSError: error closing file
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.6.1
- Platform: Linux NAME="Ubuntu" VERSION="20.04.1 LTS (Focal Fossa)"
- Python version: python3.8.5
- PyArrow version: 4.0.0


also, I test the function on some little data ,  get the same message:

```
Python 3.8.5 (default, Jan 27 2021, 15:41:15)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from datasets import load_metric
>>> metric = load_metric('accuracy')
>>> metric.add_batch(predictions=[1, 1, 1, 1], references=[1, 1, 0, 0])
2021-05-15 16:39:17.240991: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
>>> metric.compute()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/yshuang/.local/lib/python3.8/site-packages/datasets/metric.py", line 391, in compute
    self._finalize()
  File "/home/yshuang/.local/lib/python3.8/site-packages/datasets/metric.py", line 342, in _finalize
    self.writer.finalize()
  File "/home/yshuang/.local/lib/python3.8/site-packages/datasets/arrow_writer.py", line 370, in finalize
    self.stream.close()
  File "pyarrow/io.pxi", line 132, in pyarrow.lib.NativeFile.close
  File "pyarrow/error.pxi", line 112, in pyarrow.lib.check_status
OSError: error closing file
```

Hi @hyusterr,
If you look at the example provided in `metrics/accuracy.py`, it only does `metric.compute()` to calculate the accuracy. Here's an example:
```
from datasets import load_metric
metric = load_metric('accuracy')
output = metric.compute(predictions=[1, 1, 1, 1], references=[1, 1, 0, 0])
print(output['accuracy'])  # 0.5
```


I thought I can use Metric to collect predictions and references, this follows the step from huggingface's sample colab.
BTW, I fix the problem by setting other cache_dir in load_metric, but I'm still wondering about the mechanism.

I tried this code on a colab notebook and it worked fine (with gpu enabled):
```
from datasets import load_metric
metric = load_metric('accuracy')
output = metric.add_batch(predictions=[1, 1, 1, 1], references=[1, 1, 0, 0])
final_score = metric.compute()
print(final_score)  # 0.5
```
Also, in `load_metric`, I saw `cache_dir` is optional and it defaults to `~/.datasets/`

Hi ! By default it caches the predictions and references used to compute the metric in `~/.cache/huggingface/datasets/metrics` (not `~/.datasets/`). Let me update the documentation @bhavitvyamalik .

The cache is used to store all the predictions and references passed to `add_batch` for example in order to compute the metric later when `compute` is called.

I think the issue might come from the cache directory that is used by default. Can you check that you have the right permissions ? Otherwise feel free to set `cache_dir` to another location.

Closing this for now. Will re-open it should the issue still persist.

Hi..
I'm facing the same issue.

I'm trying to run beir/examples/retrieval/evaluation/dense/evaluate_sbert_multi_gpu.py. Doing do I end up with the below error.

Traceback (most recent call last):
File "evaluate_sbert_multi_gpu.py", line 62, in
results = retriever.retrieve(corpus, queries)
File "/data/user/beir/beir/retrieval/evaluation.py", line 23, in retrieve
return self.retriever.search(corpus, queries, self.top_k, self.score_function, **kwargs)
File "/data/user/beir/beir/retrieval/search/dense/exact_search_multi_gpu.py", line 150, in search
cos_scores_top_k_values, cos_scores_top_k_idx, chunk_ids = metric.compute()
File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/evaluate/module.py", line 433, in compute
self._finalize()
File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/evaluate/module.py", line 390, in _finalize
self.data = Dataset(**reader.read_files([{"filename": f} for f in file_paths]))
File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/datasets/arrow_reader.py", line 260, in read_files
pa_table = self._read_files(files, in_memory=in_memory)
File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/datasets/arrow_reader.py", line 195, in _read_files
pa_table: Table = self._get_table_from_filename(f_dict, in_memory=in_memory)
File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/datasets/arrow_reader.py", line 331, in _get_table_from_filename
table = ArrowReader.read_table(filename, in_memory=in_memory)
File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/datasets/arrow_reader.py", line 352, in read_table
return table_cls.from_file(filename)
File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/datasets/table.py", line 1065, in from_file
table = _memory_mapped_arrow_table_from_file(filename)
File "/home/user/miniconda3/envs/beir/lib/python3.7/site-packages/datasets/table.py", line 52, in _memory_mapped_arrow_table_from_file
pa_table = opened_stream.read_all()
File "pyarrow/ipc.pxi", line 750, in pyarrow.lib.RecordBatchReader.read_all
File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: Expected to be able to read 80088040 bytes for message body, got 80088032
---
Steps to reproduce:
conda create -n beir python=3.7 -y
conda activate beir
pip install beir
pip install evaluate
Change the variable dataset from "nfcorpus" to "quora"
Run the command: CUDA_VISIBLE_DEVICES=0,1,2,3 python beir/examples/retrieval/evaluation/dense/evaluate_sbert_multi_gpu.py

@lhoestq any idea what might be causing this?