Repository: Unbabel/COMET
Issue: Multi-GPU support?
Root cause: Unimplemented Feature Gap

## 🚀 Feature
Multi-GPU support would be nice. 

### Motivation

Scoring larger test sets takes ages on a single GPU :)

Finally got some time to add support to this. 

The comet-score command is currently receiving the `--gpus` flag along with the `--batch_size`. Basically, with a batch size of 16 and 2 GPUs, the model will run batches of 8 in each GPU. With this, the user can set up a big batch_size and split predictions along GPUs to speed up inference for large datasets.

This is only available on the master branch. We still have a few more things to work on before releasing it to pip.

Thanks!



Awesome!

Hi, 
I think this needs to be re-opened or I can create a new issue, but multi-gpu scoring seems to be a lot slower than even running on a single GPU :) 

GPU utilization is basically not happening. 

Also getting this: 

```
home/marcinjd/.local/lib/python3.8/site-packages/torch/nn/modules/container.py:519: UserWarning: nn.ParameterList is being used with DataParallel but t>
  warnings.warn("nn.ParameterList is being used with DataParallel but this is not "
/home/marcinjd/.local/lib/python3.8/site-packages/torch/nn/modules/container.py:464: UserWarning: Setting attributes on ParameterList is not supported.
  warnings.warn("Setting attributes on ParameterList is not supported.")
```

Using torch==1.9.0+cu111 from https://download.pytorch.org/whl/torch_stable.html
This version is required for A100 support. 

I'll Investigate this but it's probably PyTorch lightning DP that is extremely inefficient: [Data Parallel](https://pytorch-lightning.readthedocs.io/en/stable/advanced/multi_gpu.html#data-parallel)

I tried DDP also but the problem with DDP is that the sampler is split into two and you get two system scores instead of one. I'll see if I can gather those outputs and merge everything in the end... Ill keep this open for now 

I can confirm, that multiple GPUs are detected but only one is utilised during calculations:

```GPU available: True, used: True                                                                                                                                                                                TPU available: False, using: 0 TPU cores                                                                                                                                                                       LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0,1,2,3]```

--gpus flag is int, so not sure how to use it to utilise more than one? I tried --gpus 2 (to use two I suppose) and got:

```/.local/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py:69: UserWarning: You requested multiple GPUs but did not specify a backend, e.g. `Trainer(accelerator="dp"|"ddp"|"ddp2")`. Setting `accelerator="ddp_spawn"` for you.```

I added some changes that I believe solve the above problems (at least it worked in our cluster using slurm).

Using the following data can you test the 2 commands below?

[Online-B.en.txt](https://github.com/Unbabel/COMET/files/7571654/Online-B.en.txt)
[Online-G.en.txt](https://github.com/Unbabel/COMET/files/7571655/Online-G.en.txt)
[ref.en.txt](https://github.com/Unbabel/COMET/files/7571656/ref.en.txt)
[src.ru.txt](https://github.com/Unbabel/COMET/files/7571657/src.ru.txt)

`comet-score -s src.ru.txt -t Online-B.en.txt -r ref.en.txt --gpus 2` ? 
`comet-score -s src.ru.txt -t Online-B.en.txt Online-G.en.txt -r ref.en.txt --gpus 2` ? 

PS: You must update the COMET version to unbabel-comet==1.0.1

Also, we added caching for sentence embeddings which speed up inference a lot (depends on the number of repeated source and reference segments batches) when scoring multiple systems on a single GPU.