Title: Compare different Rouge implementations  
Author: ibeltagy (NONE)
State: closed
Created: 2020-09-11T15:49:32Z

I used RougeL implementation provided in `datasets` [here](https://github.com/huggingface/datasets/blob/master/metrics/rouge/rouge.py) and it gives numbers that match those reported in the pegasus paper but very different from those reported in other papers, [this](https://arxiv.org/pdf/1909.03186.pdf) for example.
Can you make sure the google-research implementation you are using matches the official perl implementation? 
There are a couple of python wrappers around the perl implementation, [this](https://pypi.org/project/pyrouge/) has been commonly used, and [this](https://github.com/pltrdy/files2rouge) is used in fairseq). 
There's also a python reimplementation [here](https://github.com/pltrdy/rouge) but its RougeL numbers are way off.

Comments:
Updates - the differences between the following three
(1) https://github.com/bheinzerling/pyrouge (previously popular. The one I trust the most)
(2) https://github.com/google-research/google-research/tree/master/rouge
(3) https://github.com/pltrdy/files2rouge (used in fairseq)
can be explained by two things, stemming and handling multiple sentences.

Stemming: 
(1), (2): default is no stemming. (3): default is with stemming ==> No stemming is the correct default as you did [here](https://github.com/huggingface/datasets/blob/master/metrics/rouge/rouge.py#L84)

Multiple sentences:
(1) `rougeL` splits text using `\n`
(2) `rougeL` ignores `\n`. 
(2) `rougeLsum` splits text using `\n`
(3) `rougeL` splits text using `.`

For (2), `rougeL` and `rougeLsum` are identical if the sequence doesn't contain `\n`. With `\n`, it is `rougeLsum` that matches (1) not `rougeL`. 

Overall, and as far as I understand, for your implementation here https://github.com/huggingface/datasets/blob/master/metrics/rouge/rouge.py#L65 to match the default, you only need to change `rougeL` [here](https://github.com/huggingface/datasets/blob/master/metrics/rouge/rouge.py#L86) to `rougeLsum` to correctly compute metrics for text with newlines.

Tagging @sshleifer who might be interested.

Thanks for the clarification !
We're adding Rouge Lsum in #701 

This is a real issue, sorry for missing the mention @ibeltagy

We implemented a more involved [solution](https://github.com/huggingface/transformers/blob/99cb924bfb6c4092bed9232bea3c242e27c6911f/examples/seq2seq/utils.py#L481) that enforces that sentences are split with `\n` so that rougeLsum scores match papers even if models don't generate newlines. 

Unfortunately, the best/laziest way I found to do this introduced an `nltk` dependency (For sentence splitting, all sentences don't end in `.`!!!), but this might be avoidable with some effort.

#### Sidebar: Wouldn't Deterministic Be Better?

`rouge_scorer.scoring.BootstrapAggregator` is well named but is not deterministic which I would like to change for my mental health, unless there is some really good reason to sample 500 observations before computing f-scores.

I have a fix on a branch, but I wanted to get some context before introducting a 4th way to compute rouge. Scores are generally within .03 Rouge2 of boostrap after multiplying by 100, e.g 22.05 vs 22.08 Rouge2.



> This is a real issue, sorry for missing the mention @ibeltagy
> 
> We implemented a more involved [solution](https://github.com/huggingface/transformers/blob/99cb924bfb6c4092bed9232bea3c242e27c6911f/examples/seq2seq/utils.py#L481) that enforces that sentences are split with `\n` so that rougeLsum scores match papers even if models don't generate newlines.
> 
> Unfortunately, the best/laziest way I found to do this introduced an `nltk` dependency (For sentence splitting, all sentences don't end in `.`!!!), but this might be avoidable with some effort.

Thanks for the details, I didn't know about that. Maybe we should consider adding this processing step or at least mention it somewhere in the library or the documentation

> #### Sidebar: Wouldn't Deterministic Be Better?
> `rouge_scorer.scoring.BootstrapAggregator` is well named but is not deterministic which I would like to change for my mental health, unless there is some really good reason to sample 500 observations before computing f-scores.
> 
> I have a fix on a branch, but I wanted to get some context before introducting a 4th way to compute rouge. Scores are generally within .03 Rouge2 of boostrap after multiplying by 100, e.g 22.05 vs 22.08 Rouge2.

I think the default `n_samples` of the aggregator is 1000. We could increase it or at least allow users to change it if they want more precise results.

Hi, thanks for the solution. 

I am not sure if this is a bug, but on line [510](https://github.com/huggingface/transformers/blob/99cb924bfb6c4092bed9232bea3c242e27c6911f/examples/seq2seq/utils.py#L510), are pred, tgt supposed to be swapped?

This looks like a bug in an old version of the examples in `transformers`

Hi, so I took this example from the HF implementation. What I can see is that  the precision of `Hello there` being summarized to `general kenobi` is 1. I don't understand how this calculation is correct.
Is the comparison just counting the words?
and if Yes, then how does this translates to summarization evaluation?
```
    >>> rouge = datasets.load_metric('rouge')
    >>> predictions = ["hello there", "general kenobi"]
    >>> references = ["hello there", "general kenobi"]
    >>> results = rouge.compute(predictions=predictions, references=references)
    >>> print(list(results.keys()))
    ['rouge1', 'rouge2', 'rougeL', 'rougeLsum']
    >>> print(results["rouge1"])
    AggregateScore(low=Score(precision=1.0, recall=1.0, fmeasure=1.0), mid=Score(precision=1.0, recall=1.0, fmeasure=1.0), high=Score(precision=1.0, recall=1.0, fmeasure=1.0))
    >>> print(results["rouge1"].mid.fmeasure)
    1.0
""", stored examples: 0)
```