zhiminy/EvalEng huggingface dataset:
Repository: Unbabel/COMET
Issue: Evaluate lines with newline characters
Root cause: Validation Gap

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->
Currently, the library does not allow sentences that contain newline characters (i.e. '\n') but rather will split them into subsentences to compute scores. This is due to how the input sentences are read (e.g. see here for the [scorer code](https://github.com/Unbabel/COMET/blob/979ef4787ba2ceb196db1a2e887bdc02b3465b5c/comet/cli/score.py#L167-L176)). A better way to achieve this would be to simply read the files as binary and then apply decoding to the individual lines. Happy to contribute with a small PR if you feel like this might be useful to other users.

@ricardorei 

### To Reproduce 

Simply executing COMET on the input files, either via scoring or compare.

### Expected behaviour

If I have a file that consists of 1000 lines (i.e. `wc -l output_it/src.txt`), I would expect exactly 1000 sentence-level scores.

### Environment
OS: Ubuntu 20.04.5 LTS (Focal Fossa)
Python 3.8.16 via Conda
Repository: Unbabel/COMET
Issue: [QUESTION] Can Different COMET Metrics Give Opposing Results for Same MT System
GitHub labels: question

Hello,

Our system is being validated against both "wmt-large-da-estimator-1719" and "wmt-large-hter-estimator" estimators with the same translations dataset, of course (70k+ translations).
  
The two estimators give completely opposite results.
The "da" estimator is placing our MT system in "...the bottom 25%" while the "HTER" estimator returns a "top 25%" score.

I know this is not a technical issue, but can you please provide some additional information on how we might be able to interpret those types of results?

Thank you very much


This issue label is exactly for this type of questions! I am happy to help

What are the scores exactly?

Sometimes when comparing two systems with similar quality these two models (wmt-large-da-estimator-1719 and wmt-large-hter-estimator) can differ regarding "which model is better". Yet, when scoring a single MT the scores should point into the same direction...



You are testing the model with 70k translations? can you compute a Pearson correlation between wmt-large-da-estimator-1719 and wmt-large-hter-estimator scores? 

> This issue label is exactly for this type of questions! I am happy to help
> 
> What are the scores exactly?
> 
> Sometimes when comparing two systems with similar quality these two models (wmt-large-da-estimator-1719 and wmt-large-hter-estimator) can differ regarding "which model is better". Yet, when scoring a single MT the scores should point into the same direction...

Please find below the results:

  | wmt-large-da-estimator-1719 | wmt-large-hter-estimator | emnlp-base-da-ranker
-- | -- | -- | --
Score | -0.21418807 | 0.212977027 | 0.145221945
Translations Count (same MT) | 70544 | 70544 | 70544

Thanks for the support!

> You are testing the model with 70k translations? can you compute a Pearson correlation between wmt-large-da-estimator-1719 and wmt-large-hter-estimator scores?

Unfortunately we don't have within our team experience with this type of computation but I will ask our engineers to have a look.

Ok, the scores make sense! HTER and DA's have different scales. HTER is a measure that you want to minimize. It reflects the effort required to "correct" the translation output in order to be semantically equivalent to the reference (higher HTER reflects more effort).

DA is a continuous scale of "how good is a translation" (a high DA score means that the translation is good).

Both models are telling you that your MT is not good. For a SOTA MT system, you should expect your HTER score to be close to 0 while the DA score should be between 0.6  and 1 

Its all here: [https://unbabel.github.io/COMET/html/models.html](https://unbabel.github.io/COMET/html/models.html) 

If you want to read more about HTER: [Snover et al., 2006](https://www.cs.umd.edu/~snover/pub/amta06/ter_amta.pdf)

and DA's: [Graham et al., 2013](https://www.aclweb.org/anthology/W13-2305.pdf) 

Thank you very much Ricardo! Makes sense now.
Repository: Unbabel/COMET
Issue: [QUESTION] Does COMET support Scoring multiple refs like scarebleu?
GitHub labels: question

## ❓ Questions and Help

Does COMET support Scoring multiple refs?
Or do I need to calculate each ref separately and take the average as the final COMET score?

Looking forward to your reply. : )




Hi @dsj96,

As you mentioned, in order to utilize COMET with multiple references, all you have to do is average the scores obtained from the various references. We have previously investigated the influence of multiple references, and our findings indicate that when it comes to neural metrics like COMET, using multiple references is not as crucial as it is for lexical metrics such as BLEU. In fact, having a single high-quality reference appears to yield better results.

For more details, please refer to [section 10.4 of this paper](https://aclanthology.org/2020.wmt-1.101.pdf).

Thank you very much for your reply!
Repository: Unbabel/COMET
Issue: [QUESTION] How are the older models supposed to be used?
Root cause: Documentation Deficiency

## ❓ Questions and Help

If I remember correctly, it used to be possible to use specific, older models, like so:

```python
from comet import download_model, load_from_checkpoint

if __name__ == "__main__":
    model_path = download_model("eamt22-cometinho-da")
    model = load_from_checkpoint(model_path)
    data = [
        {
            "src": "Dem Feuer konnte Einhalt geboten werden",
            "mt": "The fire could be stopped",
            "ref": "They were able to control the fire."
        },
        {
            "src": "Schulen und Kindergärten wurden eröffnet.",
            "mt": "Schools and kindergartens were open",
            "ref": "Schools and kindergartens opened"
        }
    ]
    model_output = model.predict(data, batch_size=8, gpus=0)
    print(model_output)
```

It seems that in v2.0, you have made the switch to the hugging face hub. I think that's a good thing but it is unclear how I can still use the older models now. If I try the code above, an error is thrown stating that that repo cannot be found on the HF hub.

Maybe the older checkpoints can also be uploaded to the hub? Is there an alternative, auto-download, way to use the older checkpoints? [This file](https://github.com/Unbabel/COMET/blob/master/MODELS.md) can be updated accordingly to tell users how to load/use which model.
Repository: Unbabel/COMET
Issue: v1.x and v2.x have different scores for wmt20-comet-qe-da model
Root cause: External Dependency Breakage

## 🐛 Bug
Scores do not match between v1.x and v2.x for wmt20-comet-qe-da model. 

### To Reproduce 
Assuming you have (mini)conda in your system ...
```bash
# create conda envs
conda create --yes -n cometv1 python=3.10
conda create --yes -n cometv2 python=3.10

# install unbabel-comet
conda run -n cometv1 --live-stream pip install unbabel-comet==1.1.3
conda run -n cometv2 --live-stream pip install unbabel-comet==2.0.1

# get some test data
sacrebleu -t wmt21/systems -l en-zh --echo src > src.txt
sacrebleu -t wmt21/systems -l en-zh --echo Online-A > mt.txt

# test 
conda run -n cometv1 --live-stream comet-score --model wmt20-comet-qe-da -s src.txt -t mt.txt | tail -1
conda run -n cometv2 --live-stream comet-score --model Unbabel/wmt20-comet-qe-da -s src.txt -t mt.txt | tail -1
```
v1 gives 0.2254
v2 gives 0.2046


### Additional Info: 
v2 gives negative scores, v1 has no negatives. 
```bash
conda run -n cometv2 --live-stream comet-score --model Unbabel/wmt20-comet-qe-da -s src.txt -t mt.txt | awk '$NF < 0'
```

### Expected behaviour
v2.x should have same scores for v1.x. 

### Environment
OS:  Linux
Packaging: pip, conda
Version: v1.1.3, v2.0.1

Hi @thammegowda, yes the [wmt20-comet-qe-da](https://huggingface.co/Unbabel/wmt20-comet-qe-da) from v2.0 is equivalent to the wmt20-comet-qe-da-v2 from v1.0.

This is because we found an error in the training of the initial wmt20-comet-qe-da. That model was trained with a sigmoid output while regressing on a score that was not normalizes between 0 and 1. This did not affect correlations but the scores were weird and many times close to 0. We fixed that by retraining the model and releasing a replacement.

Cheers,
Ricardo



Hi @ricardorei Thanks for the clarification.
Repository: Unbabel/COMET
Issue: [QUESTION]When I train my COMET model, I have the following problem when I am almost successful, it seems to be stuck
Root cause: Environment Incompatibility

## ❓ Questions and Help
When I train my COMET model, I have the following problem when I am almost successful, it seems to be stuck
Here is my training command：
CUDA_VISIBLE_DEVICES=0 comet-train --cfg /home/xusongcheng/COMET-master/configs/models/referenceless_model.yaml
This is the last part on Xshell after my command and it's stuck here!
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]

  | Name                | Type               | Params
-----------------------------------------------------------
0 | encoder             | XLMREncoder        | 558 M
1 | layerwise_attention | LayerwiseAttention | 26
2 | train_metrics       | RegressionMetrics  | 0
3 | val_metrics         | ModuleList         | 0
4 | estimator           | FeedForward        | 10.5 M
-----------------------------------------------------------
10.5 M    Trainable params
558 M     Non-trainable params
569 M     Total params
1,138.661 Total estimated model params size (MB)
Sanity Checking: 0it [00:00, ?it/s]
<img width="560" alt="c8eb7dca18e1f9bca663377397c4aec" src="https://github.com/Unbabel/COMET/assets/140067672/9bf29908-7957-4c67-a63a-a27ab5ca10cd">
I added “--num_workers 0” to the command is invalid
How should I solve it?
best wishes
Winsome

Hi @Winsome-A I have never seen this error maybe you can provide extra information? what pytorch-lightning version are you using?

Oh yeah .Thanks for your email. I have upgraded my cuda&cudnn version and eventually , the problem went away.
Repository: Unbabel/COMET
Issue: Unbabel/wmt22-cometkiwi-da doesn't work from Huggingface
Root cause: Environment Incompatibility

## 🐛 Bug

Unbabel/wmt22-cometkiwi-da doesn't work from Huggingface

### To Reproduce 

```python
from comet import download_model, load_from_checkpoint

model_path = download_model("Unbabel/wmt22-cometkiwi-da")
model = load_from_checkpoint(model_path)
data = [
    {
        "src": "The output signal provides constant sync so the display never glitches.",
        "mt": "Das Ausgangssignal bietet eine konstante Synchronisation, so dass die Anzeige nie stört."
    },
    {
        "src": "Kroužek ilustrace je určen všem milovníkům umění ve věku od 10 do 15 let.",
        "mt": "Кільце ілюстрації призначене для всіх любителів мистецтва у віці від 10 до 15 років."
    },
    {
        "src": "Mandela then became South Africa's first black president after his African National Congress party won the 1994 election.",
        "mt": "その後、1994年の選挙でアフリカ国民会議派が勝利し、南アフリカ初の黒人大統領となった。"
    }
]
model_output = model.predict(data, batch_size=8, gpus=1)
print (model_output)

```

### Screenshots
![image](https://github.com/Unbabel/COMET/assets/38466901/3fd7378d-eedd-4624-8192-ac59a36190ed)

### Environment
OS: Linux
Packaging: pip
unbabel-comet: 2.0.1



It seems to be authorization problem. manually download would do.
```bash
git lfs install
git clone https://huggingface.co/Unbabel/wmt22-cometkiwi-da
```

```python
from comet import download_model, load_from_checkpoint

# model_path = download_model("wmt22-cometkiwi-da")
model = load_from_checkpoint("local_path/wmt22-cometkiwi-da/checkpoints/model.ckpt")
data = [
    {
        "src": "The output signal provides constant sync so the display never glitches.",
        "mt": "Das Ausgangssignal bietet eine konstante Synchronisation, so dass die Anzeige nie stört."
    },
    {
        "src": "Kroužek ilustrace je určen všem milovníkům umění ve věku od 10 do 15 let.",
        "mt": "Кільце ілюстрації призначене для всіх любителів мистецтва у віці від 10 до 15 років."
    },
    {
        "src": "Mandela then became South Africa's first black president after his African National Congress party won the 1994 election.",
        "mt": "その後、1994年の選挙でアフリカ国民会議派が勝利し、南アフリカ初の黒人大統領となった。"
    }
]
model_output = model.predict(data, batch_size=8, gpus=1)
print (model_output)

```
Repository: Unbabel/COMET
Issue: [QUESTION] Why does training speed go down?
GitHub labels: question

I noticed that `comet-train` (after encoder finetuning) has speed of ~12it/s at e.g. 30% which drops to ~7it/s at 60% and to ~6it/s at 90% of the epoch. 

1. Is that something particular to only me or did anyone else observe this as well?
2. If yes, is this expected behaviour?



I'm using NVIDIA A10G GPUs and the following software versions:

  -  Python - 3.10.9
  -  COMET - upstream
  -  torch - 2.0.1
  -  pytorch-lightning - 1.9.5
  -  transformers - 4.29.0
  -  numpy - 1.24.3


Hi zouharvi,

I noticed this behavior as well. I think it has something to do with "Encoder model fine-tuning". After this the speed gradually decreases for me from 13.98it/s to 5.85it/s at the end of the epoch.

Could someone comment if this is an excepted behavior?

Indeed, without encoder fine-tuning (`nr_frozen_epochs=1`), this does not happen. Shot in the dark: I wonder if there is some memory leak associated with that which leaves some grad-able objects on the GPU?

hmmm and what happens on the second epoch? I actually never noticed this...

In the second and the next epochs it converges to ~5it/s for me (A10G with batch size 6).

Hi, I trained two reference-free QE models on in-domain data with 300k segments. One with `nr_frozen_epochs=0.3` (as proposed in the config in this repo), and the other with `nr_frozen_epochs=1`.  The rest of the parameters stayed the same.
The True Positive Rate of the prediction is lower by about 10% when using `nr_frozen_epochs=1`. So the model where the encoder-fine tuning takes place later, leads to worse performance.
The training was indeed faster until the first epoch, after this the "Encoder model fine-tuning" took place (as intended).

Was anyone able to trace this? I checked that there is no accidental length batching during training (there is none).

Finetuning the encoder is indeed slower but the gradual slowdown happens irrespective of whether the encoder is finetuned or not.

Among others I checked `pin_memory=True` and `fuse=True` but it does not affect the speed
Repository: Unbabel/COMET
Issue: Using comet-mbr for Multi-Model Translation Ranking: Questions About Input Format and GPU Disabling
Root cause: Validation Gap

I'm attempting to utilize the following command: `comet-mbr -s [SOURCE].txt -t [MT_SAMPLES].txt --num_sample [X] -o [OUTPUT_FILE].txt` for the purpose of ranking multi-model translations. However, I have a couple of questions:

Could you provide clarification on the expected format of the [MT_SAMPLES].txt file? How should the distinct translations be structured within it?
    I tried disabling the GPU using the --gpus=0 option, but it appears to be ineffective. Is there a different approach I should take to achieve this?

 - OS:  iOS
 - Packaging  pip



Hey! There is a bit more information about [data format here](https://unbabel.github.io/COMET/html/running.html#mbr-command).

You are right, the MBR command is [hardcoded to use a single GPU](https://github.com/Unbabel/COMET/blob/master/comet/cli/mbr.py#L269).

I assigned a bug label to this issue until its solved.


Thanks, @ricardorei! I followed the same approach a few days back. I locally commented out this line because the flag wasn't being considered in the code. Everything worked perfectly after that.


Another suggestion if possible it to add the  data format to the readme.
Repository: Unbabel/COMET
Issue: Is `wmt22-comet-da` the same as "COMET-22" & trained on MQM data?
GitHub labels: question

[Rei et al. 2022](https://aclanthology.org/2022.wmt-1.52/) proposed "COMET-22" as a new model which is "an ensemble between a COMET estimator model trained with DA **and a newly proposed multitask model trained to predict sentence-level scores along with OK/BAD word-level tags derived from MQM error annotations.**"  The COMET documentation, however, lists `wmt22-comet-da` as the current default model, and says it's been trained only on DA data from WMT 2017-2020, matching with what is listed in its [`hparams.yaml`](https://huggingface.co/Unbabel/wmt22-comet-da/blob/main/hparams.yaml) file.

So I just wanted to clarify, regarding `wmt22-comet-da`, the version that is available on HuggingFace Hub: is it a regular COMET-DA model trained only on WMT17-20 DA data, or was it fine-tuned on MQM scores as well?

Yep this is a bit confusing. The name choice was not the best... The `wmt22-comet-da` model is the DA model we used in that ensemble. The multitask model is actually not available atm.

We are working on a new and improved metric that will closely follow that multitask model described there and I hope to release that soon. 

Giving a bit more context, that metric seemed to work well for very high quality MT and specially for `zh-en`, `en-de` and `en-ru` but was blind to really bad quality MT and correlations were not good outside those languages. 

Thank you for clarifying!
Repository: Unbabel/COMET
Issue: Quantization
Root cause: Unimplemented Feature Gap

## 🚀 Feature
HF transformers implements 8 bit and 4 bit quantization. It would be nice if that feature can be leveraged for the xlm-r-xxl machine translation eval model. 

### Motivation

The large xlm-r-xxl model is too big for most commodity gpus. To increase access to top performance translation eval, please implement a quantize version. 

### Alternatives

I have seen a few libraries which quantize bert models outside the HF ecosystem. 

### Additional context

I tried to load the big model in 8 bit with HF, without autodevice, I could load the model, which then used 14gb vram but I don't know how to use it. 

Loading on 8bit and using flashattention would be great enhancements. There is a good example of [RoBERTa with flash-attention](https://huggingface.co/kiddothe2b/flash-roberta-base).



This also connects to @BramVanroy suggestion to use better transformer (#117 )
Repository: Unbabel/COMET
Issue: whar aspects of language gets considered while evaluating the tranlsation by COMET. as far as I am aware of MQM metric sare covered in it. Interested to know other terminolofy/features of language gets in COMET.
GitHub labels: question

## ❓ Questions and Help

### Before asking: 
1. Search for similar [issues](https://github.com/Unbabel/COMET/issues).   
3. Search the [docs](https://unbabel.github.io/COMET/html/index.html).    

<!-- If you still can't find what you need: -->

#### What is your question?

#### Code

<!-- Please paste a code snippet if your question requires it! -->   

#### What have you tried?

#### What's your environment?

 - OS: [e.g. iOS, Linux, Win]
 - Packaging [e.g. pip, conda]
 - Version [e.g. 0.5.2.1]


duplicate of #175
Repository: Unbabel/COMET
Issue: what aspects of language get considered while evaluating the translation by COMET. as far as I am aware of MQM metrics are covered in it. Interested in knowing other terminology/features of language gets in COMET. I am looking for metric along with COMET that can assess the translated content for cosmetic domain data. please suggest.
GitHub labels: question


I am not sure I understand the question very well. Can you give a specific example of what you want to evaluate?

COMET models are trained with different signals. Most models are trained with DA which basically measure adequacy of the output. Other models are trained with MQM. Models trained with MQM should ideally model all dimensions covered by the MQM typology.
Repository: Unbabel/COMET
Issue: [QUESTION] Reliability for en-th and th-en pair
GitHub labels: question

Hello,

I am interested in the reliability of Unbabel/wmt-22 model for en-th and th-en language pair (th - Thia). I went through several papers in the description, but it seems that Thai language was never selected specifically in any competition for direct evaluation. Though, it's written that this language is supported by the models, my question is the following:

What quality (kendall tau correlation, for instance) we might expect within these language pairs? Can we even state anything without explicit experiments?

I was thinking about referring to the similar language pairs, but it seems like there were no representatives from Tai-Kadai family or adjacent languages such as Burmese or Khmer. Correct me if I am wrong and thank you for your help.

Hi @zzzzzzzzzzzzz, Unfortunately without testing we can't say much about the performance of COMET on Thia. Just yesterday we added a paper where we tested COMET on african languages which was never done before. the `wmt22-comet-da` model without any fine-tuning on those languages, was not bad. I expect that for Thia, because its supported by the underlying encoder, we would get similar results... 

You can [read the paper here ](https://arxiv.org/pdf/2311.09828.pdf)



Hello @ricardorei, thank you for your response!

Hope that the direct evaluation will appear in the future. 

For now, I believe that at least COMET is valuable for sorting different approaches, since I believe the correlation will still be higher than for BLEU-like metrics.
Repository: Unbabel/COMET
Issue: AttributeError: 'dict' object has no attribute 'scores'
Root cause: Interface Contract Mismatch

#### What is your question?

Hello! I have an issue running COMET on Google Colab. I received the following error:

```bash
Predicting DataLoader 0: 100%|██████████| 79/79 [03:37<00:00,  2.75s/it]
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-6-82b44970d657>](https://localhost:8080/#) in <cell line: 14>()
     12 model = load_from_checkpoint(model_path)
     13 
---> 14 seg_scores, sys_score = model.predict(data, batch_size=128, gpus=1).values()
     15 comet = round(sys_score*100, 2)
     16 print("COMET:", comet)

1 frames
[/usr/local/lib/python3.10/dist-packages/comet/models/base.py](https://localhost:8080/#) in <listcomp>(.0)
    650             exit()
    651 
--> 652         scores = torch.cat([pred.scores for pred in predictions], dim=0).tolist()
    653         if "metadata" in predictions[0]:
    654             metadata = flatten_metadata([pred.metadata for pred in predictions])

AttributeError: 'dict' object has no attribute 'scores' 
```

#### Code example
```python
from comet import download_model, load_from_checkpoint

model = load_from_checkpoint(download_model("Unbabel/wmt22-comet-da"))
data = [
    {
        "src": "Ich esse gerne Pizza.",
        "mt": "I like to eat happy pizza.",
        "ref": "I like to eat pizza."
    },
]
model.predict(data, batch_size=1, gpus=0)[0]
```

#### What's your environment?

 - OS: Linux (Google Colab)
 - Python: 3.10.12
 - Packaging: pip 23.3.1
 - Version:
   - unbabel-comet 2.2.0
   - torch 2.1.1
   - torchmetrics 0.10.3
   - pytorch-lightning 2.1.2
   - transformers 4.35.2

#### What have you tried?

I have also tried it locally on macOS using a virtual environment on Python 3.9.0. I have got the same error message.


I have a same issue with you since yesterday

This can be fixed through this edit #185 

Changed:
```
scores = torch.cat([pred.scores for pred in predictions], dim=0).tolist()
```
to:
```
scores = torch.cat([pred["scores"] for pred in predictions], dim=0).tolist()
```