Repository: Unbabel/COMET
Issue: Different score on Multi-GPU
Root cause: Resource Mishandling

## 🐛 Bug

I get very different scores when using 2 GPUs and bigger batch size.

### To Reproduce 

```
comet-score --gpus 1 -s data/ende/sample_415_de-en_new.en -t data/ende/sample_415_de-en_new.hyp-B.de -r data/ende/sample_415_de-en_new.de | grep -v Segment
```

```
Global seed set to 12
wmt20-comet-da is already in cache.
Some weights of the model checkpoint at xlm-roberta-large were not used when initializing XLMRobertaModel: ['lm_head.layer_norm.bias', 'lm_head.dense.weight', 'roberta.pooler.dense.weight', 'lm_head.layer_norm.weight', 'roberta.pooler.dense.bias', 'lm_head.decoder.weight', 'lm_head.bias', 'lm_head.dense.bias']
- This IS expected if you are initializing XLMRobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing XLMRobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Encoder model frozen.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
Predicting: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 52/52 [00:05<00:00,  9.16it/s]
data/ende/sample_415_de-en_new.hyp-B.de score: 0.6983
```


```
comet-score --gpus 2 -s data/ende/sample_415_de-en_new.en -t data/ende/sample_415_de-en_new.hyp-B.de -r data/ende/sample_415_de-en_new.de | grep -v Segment
```
```
Global seed set to 12
wmt20-comet-da is already in cache.
Some weights of the model checkpoint at xlm-roberta-large were not used when initializing XLMRobertaModel: ['lm_head.dense.weight', 'lm_head.layer_norm.weight', 'lm_head.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.weight', 'lm_head.layer_norm.bias', 'roberta.pooler.dense.bias', 'lm_head.dense.bias']
- This IS expected if you are initializing XLMRobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing XLMRobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Encoder model frozen.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
Global seed set to 12
wmt20-comet-da is already in cache.
Global seed set to 12
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
Some weights of the model checkpoint at xlm-roberta-large were not used when initializing XLMRobertaModel: ['lm_head.dense.bias', 'lm_head.dense.weight', 'roberta.pooler.dense.bias', 'lm_head.decoder.weight', 'roberta.pooler.dense.weight', 'lm_head.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias']
- This IS expected if you are initializing XLMRobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing XLMRobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Encoder model frozen.
Global seed set to 12
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2
Added key: store_based_barrier_key:1 to store for rank: 1
Added key: store_based_barrier_key:1 to store for rank: 0
Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
Predicting: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 26/26 [00:04<00:00,  6.24it/s]
Predicting: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 26/26 [00:04<00:00,  5.96it/s]
data/ende/sample_415_de-en_new.hyp-B.de score: 0.6977
```

```
comet-score --gpus 2 --batch_size 64 -s data/ende/sample_415_de-en_new.en -t data/ende/sample_415_de-en_new.hyp-B.de -r data/ende/sample_415_de-en_new.de | grep -v Segment
```
```
Global seed set to 12
wmt20-comet-da is already in cache.
Some weights of the model checkpoint at xlm-roberta-large were not used when initializing XLMRobertaModel: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.weight', 'lm_head.decoder.weight', 'lm_head.bias', 'lm_head.dense.bias', 'lm_head.layer_norm.bias']
- This IS expected if you are initializing XLMRobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing XLMRobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Encoder model frozen.
GPU available: True, used: True
TPU available: False, using: 0 TPU cores
Global seed set to 12
wmt20-comet-da is already in cache.
Global seed set to 12
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/2
Some weights of the model checkpoint at xlm-roberta-large were not used when initializing XLMRobertaModel: ['roberta.pooler.dense.bias', 'lm_head.layer_norm.weight', 'roberta.pooler.dense.weight', 'lm_head.decoder.weight', 'lm_head.bias', 'lm_head.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.bias']
- This IS expected if you are initializing XLMRobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing XLMRobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Encoder model frozen.
Global seed set to 12
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/2
Added key: store_based_barrier_key:1 to store for rank: 1
Added key: store_based_barrier_key:1 to store for rank: 0
Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
LOCAL_RANK: 1 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]
Predicting: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:03<00:00,  1.01it/s]
Predicting: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:04<00:00,  1.06s/it]
data/ende/sample_415_de-en_new.hyp-B.de score: 0.7801
```

### Expected behaviour
Same score for different batch sizes.

### Screenshots
If applicable, add screenshots to help explain your problem.

### Environment
OS: Ubuntu 18.04
Packaging: pip 
Version 1.0.1

### Additional context

This is coming from an in-domain test set that unfortunately I cannot share for reproduction. I can share individual segment scores if needed, omitted for simplicity.

The system from the example is Bing. I tested my finetuned system and also has similar differences in scores.

Could it be related to #60 

I'll take a look at it today... it could be related to issue #60 yes!

Hi @ZJaume!

I have been testing this (with the current master branch) and everything seems fine to me...

Here are my tests:
```bash
comet-score -s source.txt -t system-1.txt -r reference.txt --gpus 1 --quiet
```
System score: 0.6524

```bash
comet-score -s source.txt -t system-1.txt -r reference.txt --gpus 1 --batch_size 64 --quiet
```
System score: 0.6524

```bash
comet-score -s source.txt -t system-1.txt -r reference.txt --gpus 1 --batch_size 128 --quiet
```
System score: 0.6524

```bash
comet-score -s source.txt -t system-1.txt -r reference.txt --gpus 2 --quiet
```
System score: 0.6521

```bash
comet-score -s source.txt -t system-1.txt -r reference.txt --gpus 2 --quiet --batch_size 128
```
System score: 0.6521

```
comet-score -s source.txt -t system-1.txt -r reference.txt --quiet --gpus 0
```
System score: 0.6524


Those very small differences from 0.6524 to 0.6521 are expected due to batch normalization. But anything above those millesimal differences is not expected.

I am leaving here the data I used so you can test it in your environment.
[system-1.txt](https://github.com/Unbabel/COMET/files/8350139/system-1.txt)
[source.txt](https://github.com/Unbabel/COMET/files/8350140/source.txt)
[reference.txt](https://github.com/Unbabel/COMET/files/8350143/reference.txt)

If the problem persists, I'll need more information about your setup.