zhiminy/EvalEng huggingface dataset:
Repository: Unbabel/COMET
Issue: [QUESTION] default model is not update?
Root cause: Configuration Error

## ❓ Questions and Help

### Before asking: 
1. Search for similar [issues](https://github.com/Unbabel/COMET/issues).   
3. Search the [docs](https://unbabel.github.io/COMET/html/index.html).    

<!-- If you still can't find what you need: -->

#### What is your question?

I am trying to evaluate my translation outputs with `wmt22-comet-da`, the new default model. I tried to re-install the package using `pip install -I unbabel-comet` and install the package locally. I also tried to manually specicify the model with the flag `--model Unbabel/wmt22-comet-da`, But I always receive the following error:
```
comet-score: error: Unbabel/wmt22-comet-da is not a valid checkpoint path or model choice. Choose from dict_keys(['emnlp20-comet-rank', 'wmt20-comet-da', 'wmt20-comet-qe-da', 'wmt20-comet-qe-da-v2', 'wmt21-comet-da', 'wmt21-comet-mqm', 'wmt21-cometinho-mqm', 'wmt21-cometinho-da', 'wmt21-comet-qe-mqm', 'wmt21-comet-qe-da', 'eamt22-cometinho-da', 'eamt22-prune-comet-da'])
```

#### Code

<!-- Please paste a code snippet if your question requires it! -->   

#### What have you tried?

#### What's your environment?

 - OS: [e.g. iOS, Linux, Win]
 - Packaging [e.g. pip, conda]
 - Version [e.g. 0.5.2.1]


Hi @minghao-wu  

I just tested the code on master and everything seems to work. You sure everything was well installed locally?

We have not updated Pip yet. Ill do it this week. But if you install it from source it should work well.



Hi @ricardorei ,

Thank you for your quick reply. Maybe there is some underlying problem on my server. No worries, I will wait for the pip update. Thanks.
Repository: Unbabel/COMET
Issue: Version 2.0
Root cause: Configuration Error

Hi

In the current PyPi package there a dependency on torch < 2, but this does not appear to be present on master. Do you either have a timeline on when latest might be on PyPi? If not, do you know if running comet v1.13 against torch 2.0 can expect bad things to happen?

Many thanks!

Hi @david-macleod 

torch 2.0 is still in beta, it' not even available on pypi. The latest version in pypi is 1.13.1

To be fair I have never tested 2.0 but if there is 100% backwards compatibility it will work well with COMET. The release of the new version to pypi will happen very soon. Frankly it has been delayed by several weeks already

Thanks, now that torch 2.0 is out we are still not able to bump due to unbabel-comet (even when using `unbabel-comet==2.0`). Could you point me to where the `Collecting torch<2.0.0,>=1.6.0` limitation is coming from?
Repository: Unbabel/COMET
Issue: Specifying GPU ID for inference
Root cause: Unimplemented Feature Gap

## 🚀 Feature

Adding parameters to specify GPU ids when calling `model.predict()`

### Motivation

Currently, the COMET API only takes the number of GPU cores to work on.
I wish to run COMET on multi-gpu environment, where each core is working on its own task.
So it would be nice to specify visible GPU ids for the model.
`predict()` API will probably look like:
```
model.predict(data, batch_size=BATCH_SIZE, gpus=2, gpu_ids=[0, 2])
```

### Alternatives



### Additional context

Since model prediction is made by ptl.Trainer, I presume that the modification will not be so hard if utilizing the ptl API.


Yep the modification should not be hard. 

For now I suggest you launch your script using CUDA_VISIBLE_DEVICES=[0,2]

I'll leave this open for future reference

This feature is a must in some use cases, like multi-GPU training with online feedback of COMET score.
It can be quite annoying to handle device ids in such cases...
I am looking forward for a update! Thanks for a fast & nice reply :)

@jinulee-v could you test the current master? I added the `devices` argument to predict. I tested it in the following way:

```python
# Backwards compatibility
model_output = model.predict(data, batch_size=8, gpus=1)

# Specify which device for single GPU
model_output = model.predict(data, batch_size=8, gpus=1, devices=[7])

# Specify devices for DDP
model_output = model.predict(data, batch_size=8, gpus=2, devices=[5, 7])
```

Feel free to suggest edits. 

The `gpus` argument could actually be replaced with `devices` only but I don't want to introduce breaking changes in the current predict interface.

PS: if this works you can close the issue and I'll release it in the next pip release. For now I'll wait a few weeks to see if more people report bugs/suggest features.
Repository: Unbabel/COMET
Issue: Models not accessible
Root cause: Documentation Deficiency

## 🐛 Bug

I get the following error when running COMET from the commandline:

```
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/api/models/wmt20-comet-qe-da/revision/main
```
Same behavior with other models as well. This was working perfectly till a couple of days back.


### To Reproduce 

```
model='wmt22-comet-qe-da'
ngpus=2
srcfname="src"
reffname="ref"
hypfname="hyp"
jsonfname=$hypfname.json


comet-score -s $srcfname \
       -t $hypfname \
       --to_json $jsonfname \
       --model $model \
       --gpus $ngpus \
       --quiet 
```

### Environment
OS: [ Linux ]
Packaging [ pip ]
Version [2.0.0]




because models are now hosted on hugging face you must specify the organisation also.

This should fix it:
```
model='Unbabel/wmt20-comet-qe-da'
ngpus=2
srcfname="src"
reffname="ref"
hypfname="hyp"
jsonfname=$hypfname.json


comet-score -s $srcfname \
       -t $hypfname \
       --to_json $jsonfname \
       --model $model \
       --gpus $ngpus \
       --quiet 
```

The above command will download, cache and score your data with [Unbabel/wmt20-comet-qe-da](https://huggingface.co/Unbabel/wmt20-comet-qe-da)

Thanks, that works.
Repository: Unbabel/COMET
Issue: Inefficient _layer_norm implementation in layerwise_attention.py
Root cause: Algorithmic Error

## 🐛 Bug

Iterations along `batch_size` axis are slow.

Code:

https://github.com/Unbabel/COMET/blob/26d5779574f5cbb88dc5fd7ef52052777bfcb1a1/comet/modules/layerwise_attention.py#L96-L122

This also prevents `wmt20-comet-qe-da` to be exported to onnx format.

### To Reproduce 

N/A

### Expected behaviour

Use `.view` to create a proper shape for broadcasting.

Preferred code:
```py
        def _layer_norm(tensor, broadcast_mask, mask):
            tensor_masked = tensor * broadcast_mask
            batch_size, _, input_dim = tensors[0].size()

            # mean for each sentence
            num_elements_not_masked = mask.sum(1) * input_dim
            mean = tensor_masked.view(batch_size, -1).sum(1)
            mean = (mean / num_elements_not_masked).view(batch_size, 1, 1)

            variance = (((tensor_masked - mean) * broadcast_mask) ** 2).view(batch_size, -1).sum(1) / num_elements_not_masked
            normalized_tensor = (tensor - mean) / torch.sqrt(variance + 1e-12).view(batch_size, 1, 1)
            return normalized_tensor
```

### Screenshots

N/A

### Environment

N/A

### Additional context

N/A


This is a good suggestion. I tested it and everything worked well its currently in master.
Repository: Unbabel/COMET
Issue: [QUESTION]__init__.py generates a wrong path for hparams.yaml in Windows
Root cause: Algorithmic Error

#### What is your question?
Hello Ricardo, 

i just installed COMET 2.0.0 and ran a basic test with the text examples after the full installation. And i got the Exception: 

hparams.yaml file is missing from C:Users\username\.cache\huggingface\hub\models--Unbabel--wmt22-comet-da\snapshots\371e9839ca4e213dde891b066cf3080f75ec7e72! 

I noticed that the path of the parent folder is incorrect, a "\\" is missing in "C:Users".
How can I fix it?

Thank you!

#### What's your environment?

 - OS: Win11
 - Packaging: Pip
 - Version: COMET 2.0.0


It happens here:

https://github.com/Unbabel/COMET/blob/77eb17921d597ffcea2ab6eea15a9c36ce78bd3e/comet/models/__init__.py#L56-L59

```
Python 3.10.10 (tags/v3.10.10:aad5f6a, Feb  7 2023, 17:20:36) [MSC v.1929 64 bit (AMD64)] on win32
>>> import os
>>> os.path.join("C:", "Users")
'C:Users'
```

For temporary usage, try to move the snapshot into the folder where your project locates in and use relative path to load it.

---

To developers: I'd suggest using `pathlib` for path manipulating.

```py
from pathlib import Path

# ...
checkpoint_path = Path(checkpoint_path)

if not checkpoint_path.is_file():
  raise Exception(f"Invalid checkpoint path: {checkpoint_path}")

parent_folder = checkpoint_path.parents[1] # .parent.parent
hparams_file = parent_folder / "hparams.yaml"
# ...
```

Thank you! @K024 

Thanks @K024!



@YZhou0413 I just pushed the changes suggested above. You can test the current master

@ricardorei It's working, Thanks!

@YZhou0413 I'll release this on the next pypi version! 

Since we just released version 2.0 and I expect more people reporting issues, I'll just wait a few weeks.
Repository: Unbabel/COMET
Issue: `Unbabel/wmt22-comet-da` model not working as part of Huggingface evaluate
Root cause: External Dependency Breakage

## 🐛 Bug

I tried using [Huggingface evaluate](https://huggingface.co/docs/evaluate/index) but it is not working with the newest `Unbabel/wmt22-comet-da` as part of unbabel-comet `2.0.0` release. I'm not sure if this is an issue with Huggingface `evaluate` lib or `COMET`.

### To Reproduce 

Please check this [collab notebook](https://colab.research.google.com/drive/1W1BjJkHJ-_QsNhUfRDLqotgOAlh4-1RT?usp=sharing).


### Screenshots
The issue:

![image](https://user-images.githubusercontent.com/3949616/228016566-7e402ecd-e082-4df7-955c-9917ddde3643.png)
.

### Environment
OS: Linux
Version 2.0.0


It seems like Hugging face evaluate has to be updated to support recent comet versions. This is a bug on `evaluate` and not on COMET.

I'll try to find some time to investigate  whats wrong with the current implementation of COMET in `evaluate`.


I created a pull request to `evaluate` to solve this: https://github.com/huggingface/evaluate/pull/443

Once it is merged, it should work.

Basically it has to do with an API change. You can see the difference [here](https://github.com/huggingface/evaluate/pull/443/commits/70ca582c9029bf05591ddf50e6bf187d221a7451#diff-04c571bf81d04b98363eda223cf1c8e5e995478d3eba2c3a9142a9a19ed7c441L144)
Repository: Unbabel/COMET
Issue: Can't reproduce Cometinho model scores
Root cause: Documentation Deficiency

## 🐛 Bug

Cometinho [paper](https://aclanthology.org/2022.eamt-1.9/) states that pruned model achieves 0.274 Kendall tau score on news subsection of WMT21 EN-RU MQM dataset. Their distilled model achieves 0.263 Kendall tau score. However, simple reproduction script using wmt21-cometinho-mqm models fails to show similar results on the same data.

### To Reproduce 

1. Prerequirements - Download model and dataset
```bash
# wmt2021-mqm link mentioned in https://github.com/Unbabel/COMET/tree/master/data
wget https://unbabel-experimental-data-sets.s3.eu-west-1.amazonaws.com/comet/data/2021-mqm.tar.gz
tar xzf 2021-mqm.tar.gz

# cometinho-mqm model from https://github.com/Unbabel/COMET/blob/master/MODELS.md
wget https://unbabel-experimental-models.s3.amazonaws.com/comet/wmt21/wmt21-cometinho-mqm.tar.gz
tar xzf wmt21-cometinho-mqm.tar.gz
```
2. Run the code
```python
import pandas as pd
import torch
torch.set_float32_matmul_precision("medium")
from scipy.stats import kendalltau
from comet import load_from_checkpoint

model = load_from_checkpoint("./wmt21-cometinho-mqm/checkpoints/model.ckpt")

dataset = pd.read_csv("./2021-mqm.csv")

eval_set = dataset.query("lp == 'en-ru' & domain == 'news'")

data = eval_set[['src', 'mt', 'ref']].to_dict(orient='records')
scores = eval_set['score']

cometinho_scores = model.predict(data, batch_size=256, gpus=1)

tau, p = kendalltau(cometinho_scores['scores'], scores)
print(f"Kendall tau: {tau:.6f}")
``` 

### Expected behaviour
Provided script should print `Kendall tau: 0.263` or a close number.

### Environment
OS: Ubuntu Linux 20.04 Kernel 5.17.5 inside docker [container](https://hub.docker.com/r/rexhaif/basic-ml)
Hardware: Nvidia RTX 3090
Packaging: pip
Version: 2.0.0

### Additional Comments

1. When testing on zh-en news subset reproduced scores are at least close to paper scores: 0.3549 reproduced vs 0.321 in paper
2. `torch.set_float32_matmul_precision("medium")` impacts only GPU utilization & processing speed, scores are the same

Hi @Rexhaif I think your problem is on the models you are using. Have you tried these ones?
- `eamt22-cometinho-da`
- `eamt22-prune-comet-da`

The confusion is that the "first" versions of cometinho were actually trained for [WMT 21](https://aclanthology.org/2021.wmt-1.111.pdf). They are not distilled versions of larger COMET models but rather smaller encoders trained with the same data. Then for EAMT Conference we experimented with distillation of larger models (which allowed us to train the smaller encoder in much more data) those are the results presented in [COMETINHO paper](https://aclanthology.org/2022.eamt-1.9/).

Sorry about the confusing names. I hope with this you are able to reproduce the results

Hi, thanks for the clarification and the model names!

Are there any distilled/pruned COMETINHO models available that were trained on the MQM scores rather than the DA scores?

Unfortunately no.
Repository: Unbabel/COMET
Issue: Comet QE asks for the reference
Root cause: Validation Gap

Using the comet score with QE model (wmt-large-qe-estimator-1719) through the command line asks for a reference, even when it shouldn't be used in the calculation (if I am not mistaken)

You are right. Using the command line references are mandatory. Yet COMET-QE does not use them.. you can pass whatever you want. 

Using python you dont even need the references:

```python
from comet.models import download_model
model = download_model("wmt-large-qe-estimator-1719")
data = [{
        "src": "Dem Feuer konnte Einhalt geboten werden",
        "mt": "The fire could be stopped",
}, {
        "src": "Schulen und Kindergärten wurden eröffnet.",
        "mt": "Schools and kindergartens were open"
}]
model.predict(data, cuda=True, show_progress=True)
```
> ([{'src': 'Dem Feuer konnte Einhalt geboten werden', 'mt': 'The fire could be stopped', 'predicted_score': 0.008310817182064056}, {'src': 'Schulen und Kindergärten wurden eröffnet.', 'mt': 'Schools and kindergartens were open', 'predicted_score': 0.7012745141983032}], [0.008310817182064056, 0.7012745141983032])
Repository: Unbabel/COMET
Issue: Support pandas 2
Root cause: Architectural Constraint

## 🚀 Feature

Comet 1.1.3 supported pandas 2

https://github.com/Unbabel/COMET/blob/0d58039da7861bebd920c890d857e93ec3f8ea12/pyproject.toml#L41

333 but unfortunately major releases are not allowed anymore in comet 2.0.0

https://github.com/Unbabel/COMET/blob/db918c6149c771509adcb427e1cf1c6ca94fd926/pyproject.toml#L41

As far as I can see, the only pandas functionality that is needed is reading simple CSV files:

https://github.com/search?q=repo%3AUnbabel%2FCOMET+pd.&type=code

So I think that allowing for pandas 2 does not break anything.
Repository: Unbabel/COMET
Issue: [QUESTION] How to finetune `wmt22-comet-da` and have results scaled to 0-1 range
Root cause: Documentation Deficiency

I am trying to fine-tune the `wmt22-comet-da` model on new DA data, using its checkpoint and the configs available in the `configs/` folder here. I know that this model scales the result scores between 0 and 1, but after fine-tuning, the new fine-tuned model generates negative scores as well, similarly to the older COMET models.

My main question is, where exactly is the configuration I should change for the model to scale the results like `wmt22-comet-da` does? I haven't been able to find that in the code. My configs are exactly the same as those in [`configs/`](https://github.com/Unbabel/COMET/tree/master/configs) except for filepaths and the checkpoint to load.


Yep, Julia you are right. You need first to rescale the data to be between 0 and 1.

The rescaling I used was actually borrowed from BLEURT. The rescaling is the following:
1) Find a reasonable "min value". This is done by finding all annotations with more than 1 annotator where all annotators agreed that the score was 0. Then your "min_value" is the average z-score for those segments.
2) Find a reasonable "max value". This is done by finding all annotations with more than 1 annotator where all annotators agreed that the translation is perfect (100 score). Then your "max_value" is the average z-score for those segments.
3) apply a min_max_scaler to your data and truncate every score above 1 and bellow 0.

This rescaling won't impact your model correlations and your model will output scores in that range.
Repository: Unbabel/COMET
Issue: UnifiedMetric test failing: test_multitask_with_references
Root cause: Validation Gap

## 🐛 Bug

Was working on some test cases and ran the test suite and found that one of the tests fail. Specifically test_unified_metric.test_multitask_with_references

https://github.com/Unbabel/COMET/blob/db918c6149c771509adcb427e1cf1c6ca94fd926/tests/integration/models/test_unified_metric.py#L159

```
Traceback (most recent call last):
  File "/home/local/vanroy/COMET/tests/integration/models/test_unified_metric.py", line 261, in test_multitask_with_references
    self.assertListEqual(word_level_example, subword_scores_example)
AssertionError: Lists differ: [0, 0[31 chars] 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0] != [0, 0[31 chars] 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0]

First differing element 18:
0
1

  [0,
-  0,
   0,
   0,
   0,
   0,
   0,
   0,
   0,
   0,
   0,
   0,
   0,
   0,
   0,
   0,
   0,
   0,
   0,
   1,
   1,
   1,
   1,
   1,
   1,
-  0,
+  1,
+  1,
   0,
   0,
   0,
   0,
   0,
   0,
   0]


```

### To Reproduce 

I simply ran the unittests in a fresh environment. Maybe the difference is caused by different torch/python versions?

### Environment
- Platform: Linux-5.14.0-162.6.1.el9_1.0.1.x86_64-x86_64-with-glibc2.34
- Python version: 3.10.10
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- pytorch-lightning==1.9.5



All tests seem to be passing on the current version BUT the word-level implementation is not optimal. We have been working on a new metric that uses word-level tagging and hopefully soon we will merge better code to deal with word-level
Repository: Unbabel/COMET
Issue: Segmentation error when tring to reproduce wmt22 results
Root cause: Resource Mishandling

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

### To Reproduce 
```
pip install unbabel-comet
comet-score -s ../mt-metrics-eval-v2/wmt22/sources/en-zh.txt -t ../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt -r ../mt-metrics-eval-v2/wmt22/references/en-zh.refA.txt > log.comet
Global seed set to 1
Fetching 5 files: 100%|█████████████████████████| 5/5 [00:00<00:00, 88487.43it/s]
Lightning automatically upgraded your loaded checkpoint from v1.8.3.post1 to v1.9.5. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint --file ../../../../../root/.cache/huggingface/hub/models--Unbabel--wmt22-comet-da/snapshots/371e9839ca4e213dde891b066cf3080f75ec7e72/checkpoints/model.ckpt`
Encoder model frozen.
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Predicting DataLoader 0: 100%|████████████████████| 128/128 [01:00<00:00,  2.10it/s]
[1]    13312 segmentation fault  comet-score -s ../mt-metrics-eval-v2/wmt22/sources/en-zh.txt -t  -r  > 
```
And the result in log.comet is like:
```
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 0   score: 0.8275                                                                                                                                                                    
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 1   score: 0.8833
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 2   score: 0.7753
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 3   score: 0.9103
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 4   score: 0.8103
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 5   score: 0.9792
...
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 2033    score: 0.9494
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 2034    score: 0.9332
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 2035    score: 0.9397
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  Segment 2036    score: 0.9048
../mt-metrics-eval-v2/wmt22/system-outputs/en-zh/HuaweiTSC.txt  score: 0.8622 
```
### Expected behaviour
Output the system score for candidates.

### Screenshots
If applicable, add screenshots to help explain your problem.

### Environment
OS: [e.g. iOS, Linux, Win]
Packaging [e.g. pip, conda] pip/py39
Version [e.g. 0.5.2.1] 2.0.1

### Additional context

<!-- Add any other context about the problem here. -->


And the results for wm22 en-zh HuaweiTSC.txt do not match that in the wmt22 released `COMET-22-refA.sys.score`.
The score in the file is 0.47647532625047895.
Repository: Unbabel/COMET
Issue: How to reproduce Unbabel/wmt22-comet-da model
Root cause: Documentation Deficiency

I want to train the model Unbabel/wmt22-comet-da model from scratch.
It will be very helpful for me if someone provides the list of datasets, pre-processing steps, and hyperparameters.

hyper-parameters are [these ones](https://github.com/Unbabel/COMET/blob/master/configs/models/regression_model.yaml) and regarding the data we used all DA data [here](https://github.com/Unbabel/COMET/tree/master/data) including the MLQE-PE data.

For making the model output a score between 0 and 1 you just have to rescale the data after concatenating everything. The rescaling is the following:

1) Find a reasonable "min value". This is done by finding all annotations with more than 1 annotator where all annotators agreed that the score was 0. Then your "min_value" is the average z-score for those segments.
2) Find a reasonable "max value". This is done by finding all annotations with more than 1 annotator where all annotators agreed that the translation is perfect (100 score). Then your "max_value" is the average z-score for those segments.
3) Apply a min_max_scaler to your data and truncate every score above 1 and bellow 0.

This rescaling won't impact your model correlations and your model will output scores in that range. This is the same rescaling used in BLEURT also.

Btw the seed used is 91 if I am not mistaken... `comet-train --cfg YOUR_CONFIGS.yaml --seed_everything 91`