Repository: huggingface/evaluate
Issue: rouge-score not found by evaluate.load()
Root cause: Environment Incompatibility

Hello,
I have a problem loading the rouge metric with evaluate.load(). Apparently, the rouge_scores library (https://github.com/google-research/google-research/tree/master/rouge) is not found by evaluate. I followed your tutorial from https://huggingface.co/course/chapter7/5?fw=pt#preprocessing-the-data (section 'Metrics for Text Summarization') and installed the rouge_score library. I got access to two different clusters. On the first one, everything works like a charm, however, the second cluster throws this exception below:

Traceback

Traceback (most recent call last):
  File "/home/hd/hd_hd/hd_rk435/MA_Thesis/fine-tune/mnli_text/generation_MNLI.py", line 14, in <module>
    rouge_score = evaluate.load("rouge") #/home/hd/hd_hd/hd_rk435/rouge")  # evaluate.load("rouge") #/pfs/data5/home/hd/hd_hd/hd_rk435/ma_venv/lib64/python3.8/site-packages/rouge_score/rouge_scorer.py") #"rouge")
  File "/pfs/data5/home/hd/hd_hd/hd_rk435/ma_venv/lib64/python3.8/site-packages/evaluate/loading.py", line 716, in load
    evaluation_module = evaluation_module_factory(
  File "/pfs/data5/home/hd/hd_hd/hd_rk435/ma_venv/lib64/python3.8/site-packages/evaluate/loading.py", line 621, in evaluation_module_factory
    return LocalEvaluationModuleFactory(
  File "/pfs/data5/home/hd/hd_hd/hd_rk435/ma_venv/lib64/python3.8/site-packages/evaluate/loading.py", line 424, in get_module
    local_imports = _download_additional_modules(
  File "/pfs/data5/home/hd/hd_hd/hd_rk435/ma_venv/lib64/python3.8/site-packages/evaluate/loading.py", line 268, in _download_additional_modules
    raise ImportError(
ImportError: To be able to use rouge, you need to install the following dependencies['rouge'] using 'pip install rouge' for instance'



Also for Python 3.9

Traceback (most recent call last):
  File "/home/hd/hd_hd/hd_rk435/MA_Thesis/fine-tune/mnli_text/generation_MNLI.py", line 14, in <module>
    rouge_score = evaluate.load("/home/hd/hd_hd/hd_rk435/rouge/rouge.py")  # evaluate.load("rouge") #/pfs/data5/home/hd/hd_hd/hd_rk435/ma_venv/lib64/python3.8/site-packages/rouge_score/rouge_scorer.py") #"rouge")
  File "/pfs/data5/home/hd/hd_hd/hd_rk435/test_venv/lib64/python3.9/site-packages/evaluate/loading.py", line 774, in load
    evaluation_module = evaluation_module_factory(
  File "/pfs/data5/home/hd/hd_hd/hd_rk435/test_venv/lib64/python3.9/site-packages/evaluate/loading.py", line 671, in evaluation_module_factory
    return LocalEvaluationModuleFactory(
  File "/pfs/data5/home/hd/hd_hd/hd_rk435/test_venv/lib64/python3.9/site-packages/evaluate/loading.py", line 494, in get_module
    local_imports = _download_additional_modules(
  File "/pfs/data5/home/hd/hd_hd/hd_rk435/test_venv/lib64/python3.9/site-packages/evaluate/loading.py", line 269, in _download_additional_modules
    raise ImportError(
ImportError: To be able to use rouge, you need to install the following dependencies['rouge', 'rouge', 'rouge'] using 'pip install rouge rouge rouge' for instance'


Code to reproduce:
```
import evaluate
rouge_score = evaluate.load("rouge")
```

Different ways of loading the metric, e.g through a path results in the same exception.

**Versions**
I've tried to get the script running in different versions, none of them worked.
- Python 3.8.6/3.9.2
- transformers 4.19.2
- rouge-score 1.0.2/0.0.4
- evaluate 0.3.0/0.1.2
- Torch 1.12.1

On the first cluster I'm using the following versions:
- Python 3.8.10
- transformers 4.19.2
- rouge-score 0.0.4
- evaluate 0.1.2
- torch 1.12.0

Rouge Score seems to be correctly installed:
```
(test_venv) [ ~]$ ls test_venv/lib64/python3.9/site-packages/rouge_score/
__init__.py  create_pyrouge_files.py  io_test.py  rouge_scorer.py       scoring.py       test_util.py  tokenize_test.py
__pycache__  io.py                    rouge.py    rouge_scorer_test.py  scoring_test.py  tokenize.py
```


Thanks for reading! 

Hi @PhMeier, can you check if you can load the rouge module in the environment without `evaluate`?

```python
from rouge_score import rouge_scorer, scoring
```

Just to check that it is indeed correctly installed?

Hi @lvwerra thank you for the fast reply!

The library seems to be correctly installed. I've run this code and received this output:
```
from rouge_score import rouge_scorer, scoring
scorer = rouge_scorer.RougeScorer(['rouge1', 'rougeL'], use_stemmer=True)
scores = scorer.score('The quick brown fox jumps over the lazy dog',
                      'The quick brown dog jumps on the log.')
print("Scores: {}".format(scores))
```
Output:
```
(test_venv) $ python3 test.py
Scores: {'rouge1': Score(precision=0.75, recall=0.6666666666666666, fmeasure=0.7058823529411765), 'rougeL': Score(precision=0.625, recall=0.5555555555555556, fmeasure=0.5882352941176471)}
```

If I try now to use the evaluate.load() method, I receive this exception:
```
from rouge_score import rouge_scorer, scoring
import evaluate
rouge_score = evaluate.load("rouge")
```
```
Traceback (most recent call last):
  File "/pfs/data5/home/hd/hd_hd/hd_rk435/test.py", line 3, in <module>
    rouge_score = evaluate.load("rouge")
  File "/pfs/data5/home/hd/hd_hd/hd_rk435/test_venv/lib64/python3.9/site-packages/evaluate/loading.py", line 778, in load
    evaluation_instance = evaluation_cls(
TypeError: 'NoneType' object is not callable
```




I've used version 0.3.0 of evaluate and rouge-score 1.2.1 in Python 3.9.2 with this code example:
```
from rouge_score import rouge_scorer, scoring
import evaluate
from transformers import AutoTokenizer, BartTokenizer

rouge_score = evaluate.load("/home/hd/hd_hd/hd_rk435/evaluate/metrics/rouge")
tokenizer = BartTokenizer.from_pretrained("facebook/bart-large")
generated_summary = "Nearly all the tips useful and. I consider myself an intermediate to advanced user of OneNote."
reference_summary = "Review: Nearly all the tips useful and. I consider myself an intermediate to advanced user of OneNote."

res = rouge_score.compute(predictions=[generated_summary], references=[reference_summary], use_stemmer=True) #, tokenizer=tokenizer)
print(res) 
```

```
{'rouge1': 0.9696969696969697, 'rouge2': 0.967741935483871, 'rougeL': 0.9696969696969697, 'rougeLsum': 0.9696969696969697}
```

Now it works! I'm pretty sure that I've tried this combination yesterday.

Anyways thanks again for the fast reply and this great library! 

I think loading `rouge_score` before might lead to a namespace clash with the evaluate importer in the last example. 

What's odd in the original message is that it says you should install `rouge` whereas the import is actually `rouge_score`. The error comes from here:
https://github.com/huggingface/evaluate/blob/960dc92ef6a86e3b11d7b7276b4960135a3129b9/src/evaluate/loading.py#L262-L272

When looking at the `library_import_name`, and `library_import_path` I get:

```python
absl # Here to have a nice missing dependency error message early on
datasets datasets
nltk # Here to have a nice missing dependency error message early on
numpy # Here to have a nice missing dependency error message early on
six # Here to have a nice missing dependency error message early on
rouge_score rouge_score
evaluate evaluate
```

So it is not clear to me where `rouge` comes from since this is grabbed by scanning the Python files for imports.

Anyway, since it is fixed now I won't spend more time investigating :) 



I may want to add some information to my previous post. Adressing the rouge metric via evaluate.load("rouge") still yields the 'type Error' or the 'ImportError' above, even with  0.3.0 of evaluate and rouge-score 1.2.1 in Python 3.9.2.

I've cloned the repository and the load method receives the path to the rouge directory in the 'metrics' folder. 
``
rouge_score = evaluate.load("/home/hd/hd_hd/hd_rk435/evaluate/metrics/rouge")
``


After your comment about the import, I  looked into loading.py. The problem is in the function 'get_imports' in line  143.  I added some print statements:
```
        match = re.match(r"^import\s+(\.?)([^\s\.]+)[^#\r\n]*(?:#\s+From:\s+)?([^\r\n]*)", line, flags=re.MULTILINE)
        print("First match: ", match)
        if match is None:
            match = re.match(
                r"^from\s+(\.?)([^\s\.]+)(?:[^\s]*)\s+import\s+[^#\r\n]*(?:#\s+From:\s+)?([^\r\n]*)",
                line,
                flags=re.MULTILINE,
            )
            print("Match second try:", match)
```

The output shows that rouge_scorer is correctly found. However due to the match to group 2 the ImportError is caused since 'rouge' is matched and not 'rouge_score'.

```
Match second try: <re.Match object; span=(0, 20), match='from absl import app'>
Match group 2:  absl
First match:  None
Match second try: <re.Match object; span=(0, 22), match='from absl import flags'>
Match group 2:  absl
First match:  None
Match second try: <re.Match object; span=(0, 20), match='from rouge import io'>
Match group 2:  rouge
First match:  None
Match second try: <re.Match object; span=(0, 30), match='from rouge import rouge_scorer'>
Match group 2:  rouge
First match:  None
Match second try: <re.Match object; span=(0, 25), match='from rouge import scoring'>
Match group 2:  rouge
```








That's very odd, e.g. the line `from rouge import io` does not exist in the [file](https://github.com/huggingface/evaluate/blob/main/metrics/rouge/rouge.py), I wonder which file this is importing.

**Update:** did you clone [this](https://github.com/google-research/google-research/blob/master/rouge/rouge.py) repository and are pointing to that? You should be pointing to the above file/repository instead and install google's rouge with `pip install rouge_score`, as `evaluate`'s rouge score is a wrapper around it.

I only cloned [evaluate](https://github.com/huggingface/evaluate) and pointed then to metrics/rouge as stated in the documentation. 

`rouge_score = evaluate.load("/home/hd/hd_hd/hd_rk435/evaluate/metrics/rouge")`

In this way, no error occurs. Only calling evaluate.load("rouge") does not work.  [rouge-score](https://github.com/google-research/google-research/tree/master/rouge) was installed via pip. 


`from rouge import io` exists in for example in [`google-research/rouge/rouge.py`](https://github.com/google-research/google-research/blob/master/rouge/rouge.py)



This is very weird and I can't reproduce it on my end - can you reproduce this e.g. on Colab? A guess is that somehow the installed `rouge-score` package is in the path when `evalaute.load` is called and thus `load` tries to load `google-research/rouge/rouge.py` instead of the evaluate script.

Hi @lvwerra 
I've found the solution! I had an old cloned repository named 'rouge', which caused the confusion. Should have checked that earlier! Thanks again for your time and help.
Script:
```
from rouge_score import rouge_scorer, scoring
import evaluate
from transformers import AutoTokenizer, BartTokenizer

#"""
rouge_score = evaluate.load("rouge") #"/home/hd/hd_hd/hd_rk435/evaluate/metrics/rouge")
tokenizer = BartTokenizer.from_pretrained("facebook/bart-large")
generated_summary = "Nearly all the tips useful and. I consider myself an intermediate to advanced user of OneNote.\n"
reference_summary = "Review: Nearly all the tips useful and. I consider myself an intermediate to advanced user of OneNote.\n"

generated_summary = ["Nearly all the tips useful and. I consider myself an intermediate to advanced user of OneNote.\n", "Nearly all the tips useful and. I consider myself an intermediate to advanced user of OneNote.\n"]
reference_summary = ["Review: Nearly all the tips useful and. I consider myself an intermediate to advanced user of OneNote.\n", "Review: Nearly all the tips useful and. I consider myself an intermediate to advanced user of OneNote.\n"]

res = rouge_score.compute(predictions=generated_summary, references=reference_summary, use_aggregator=True) #, use_stemmer=True) #, tokenizer=tokenizer)
print(res)
```
Output:
```
(test_venv) [hd_rk435@uc2n996 ~]$ python3 ~/test.py
{'rouge1': 0.9696969696969697, 'rouge2': 0.967741935483871, 'rougeL': 0.9696969696969697, 'rougeLsum': 0.9696969696969697}
```