Repository: huggingface/evaluate
Issue: rouge score giving non-deterministic results
Root cause: Algorithmic Error

I'm not sure if this is a bug or an expected behaviour, but I couldn't find any reference to this in the docs so I thought I'd open an issue.

I have noticed two things when calculating rouge scores on large chunks of long text documents

1. The scores are non-deterministic. If I re-start python and re-initialise the metric with `rouge_score = evaluate.load('rouge')` and then re-calculate the scores on the same predictions and references, I get different results (this does not happen if I don't re-initialize the metric)
2. The F1 score can be lower than both recall and precision. This is also the case for the mid-confidence level, which according to my understanding should be the average of all the F1 Rouge scores from all samples, rather than the boundaries of the standard deviation, so if F1 is calculated through `2 * precision * recall / (precision + recall)` then this shouldn't happen.

Are these two behaviours expected, and what is their reason?

As an example I copy below the scores I get from two different runs of exactly the same data, which show both the mentioned issues. 

1. Run 1

`{'rouge1': AggregateScore(low=Score(precision=0.3243700246113191, recall=0.32660699724381115, fmeasure=0.30412723729901875), mid=Score(precision=0.3375758153944711, recall=0.337393136278435, fmeasure=0.31151127030782255), high=Score(precision=0.3519158309481779, recall=0.3480565897561773, fmeasure=0.3194669480416914)), 'rouge2': AggregateScore(low=Score(precision=0.060371242698786964, recall=0.06139179363469908, fmeasure=0.056540214226806565), mid=Score(precision=0.06546256177158391, recall=0.06721841919354898, fmeasure=0.06127572467088304), high=Score(precision=0.07033209792591454, recall=0.07296091893833931, fmeasure=0.0656492161952344)), 'rougeL': AggregateScore(low=Score(precision=0.17994593956723304, recall=0.18408363827906665, fmeasure=0.1696635239862768), mid=Score(precision=0.18600734687280238, recall=0.1919674152470312, fmeasure=0.17405747167764546), high=Score(precision=0.19297648125708244, recall=0.1995519136798136, fmeasure=0.17806290369663147)), 'rougeLsum': AggregateScore(low=Score(precision=0.17977866130129155, recall=0.18417132629535324, fmeasure=0.17006844632251267), mid=Score(precision=0.18634291620489318, recall=0.19205676348354728, fmeasure=0.17412756897924064), high=Score(precision=0.1931381967971536, recall=0.19970515726239785, fmeasure=0.17823326745758938))}`. 

2. Run 2

`{'rouge1': AggregateScore(low=Score(precision=0.3249003718921296, recall=0.32643064107228026, fmeasure=0.30365387189059817), mid=Score(precision=0.3379200760591662, recall=0.3371345811393641, fmeasure=0.31145401622860025), high=Score(precision=0.3524766069909411, recall=0.34871487525786565, fmeasure=0.31924995849086085)), 'rouge2': AggregateScore(low=Score(precision=0.060176442640101885, recall=0.06109357204894354, fmeasure=0.056270475502950286), mid=Score(precision=0.06535272797598764, recall=0.06699344801939497, fmeasure=0.06105165110094797), high=Score(precision=0.07044258853079938, recall=0.07302818004000375, fmeasure=0.06560736801827798)), 'rougeL': AggregateScore(low=Score(precision=0.17931899861389008, recall=0.18436717091947302, fmeasure=0.16964017973067347), mid=Score(precision=0.18640832648184572, recall=0.19182174956286982, fmeasure=0.17401466805303323), high=Score(precision=0.19322295586591504, recall=0.20000972351673146, fmeasure=0.17859023960672574)), 'rougeLsum': AggregateScore(low=Score(precision=0.18007836594460425, recall=0.18426881056178082, fmeasure=0.16989966159414976), mid=Score(precision=0.18648124112813447, recall=0.19196327848629846, fmeasure=0.17388889183980177), high=Score(precision=0.19298871927634093, recall=0.2001710196023202, fmeasure=0.17866572339252532))}`

I believe question 1) is related to this post: https://github.com/huggingface/datasets/issues/361#issuecomment-655988938 and if you turn off the bootstrap aggregator (`use_aggregator=False`) it should be deterministic. 

I believe also for 2) the bootstrapping might be the issue. Look at the following toy example:


|P|R|F1|
|:-|:-|:-|
|0.1|0.01|0.02|
|0.1|0.01|0.02|
|0.01|0.1|0.02|
|0.01|0.1|0.02|

Assuming you only sample one from the list of and then take the median (in reality you sample several and take the mean of each sample batch) you will get the following medians (which are the mid scores):

|P|R|F1|
|:-|:-|:-|
|0.055|0.055|0.02|

I think this especially prone to happen when P and R are often asymmetric (one gets larger and the other smaller, like it is often the case for precision and recall). You can have a look at the underlying computation here: https://github.com/google-research/google-research/blob/f3033edb77b3f4549858482c91db6131a7a7af11/rouge/scoring.py#L133-L159

Hope this helps!

Thanks @lvwerra  for your prompt reply.
Regarding point 2), thanks a lot for the example, I believe this is to be expected then, it's not an issue with the code, it's just the way the formula is set up.

Regarding point 1), I have taken a look at the code you linked. My understanding is that the `_bootstrap_resample` takes some random samples and then calculates mean and confidence interval based on that rather than the full set of scores. Therefore the differing results on each run are due to the different subset of selected scores, rather than numerical errors due to the order in which scores have been added, or other operations performed. Is this correct? If so, do you happen to know why bootstrapping is necessary? I'm assuming it's just to make the function faster, but it would be good if you could confirm

Bootstrapping is just a way to estimate confidence intervals (low/mid/high). It is not necessary to run it at all and you can turn it off with `use_aggregator=False` in which case you get raw values directly. This should also be a bit faster.

Thanks again