zhiminy/EvalEng huggingface dataset:
Repository: Unbabel/COMET
Issue: Different versions of COMET code give different scores with the same model and date.
Root cause: Algorithmic Error

## 🐛 Bug

Using COMET 2.2.1 (and Python 3.9) and the wmt22-comet-da model I get a score of 0.7982, but using COMET 1.1.2 (and Python 3.7) I get a score 0.8618. This is on exactly the same source, target and reference file.

I appreciate that 1.1.2 is an old version, and should not be used, but many people will have old versions installed, and be unaware that they should not be used with new models. The consequence of this bug is that research papers should provide both the model of COMET used, as well as the version of the software.

### To Reproduce 

Install COMET 2.2.1 on Python 3.9, score test files. I used an en->mt translation of NTREX with this model https://huggingface.co/HPLT/mt-mt-en-v1.0-hplt_opus. I have attached the src, hypo and ref files.

Install COMET 1.1.2 on Python 3.7, score the same files. 

Compare scores. 

### Expected behaviour

With the same model and data, COMET should give the same scores. 

### Screenshots
If applicable, add screenshots to help explain your problem.

### Environment
OS: Ubuntu 20.04.6 LTS
Versions: 2.2.1 and 1.1.2
wmt22-comet-da



[hypo.txt](https://github.com/Unbabel/COMET/files/14422847/hypo.txt)
[ref.txt](https://github.com/Unbabel/COMET/files/14422850/ref.txt)
[src.txt](https://github.com/Unbabel/COMET/files/14422851/src.txt)
Repository: Unbabel/COMET
Issue: [QUESTION] large file scoring
Root cause: Algorithmic Error


When scoring a large file (say > 100K records) why does it start with a high throughput , for instance say 50 it/sec, and quickly after a few 10K records it drops significantly (more than half)

Thanks


could it be the same as: https://github.com/Unbabel/COMET/issues/158

Training is typically influenced by various factors, but for inference, batch sorting is employed to minimize padding. Consequently, the longest batches end up being processed in the end resulting in a higher number of tokens per batch compared to the beginning.


you can check the difference by setting length_batching to False
Repository: Unbabel/COMET
Issue: [QUESTION] Splitting big models over multiple GPUs
Root cause: Unimplemented Feature Gap

When specifying the number of GPUs during inference, is it only for parallelism or is the model loaded piece-wise over multiple GPUs, if it's bigger than individual GPUs? For example I'd like to use XCOMET-XXL and our cluster has many 12GB GPUs.

At first I thought that the model parts will be loaded onto all GPUs, e.g.:
```
comet-score -s data/xcomet_ennl.src -t data/xcomet_ennl_T1.tgt --gpus 5 --model "Unbabel/XCOMET-XL"
```

However I'm getting GPU OOM on the first GPU:
```
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 26.00 MiB. GPU 0 has a total capacity of 10.75 GiB of which 11.62 MiB is free. ...
```

1. Is it correct that in the above setting the model is being loaded in full 5 times on all 5 GPUs?
2. Is there a way to split the model over multiple GPUs?

Thank you!

- `unbabel-comet 2.2.1`
- `pytorch-lightning 2.2.0.post0`
- `torch 2.2.1`

same question here

Last time I check this was not very easy to do with pytorch-lightning. 

We actually used a custom made implementation with FSDP to train these larger models (without using pytorch-lightning). I have to double check if the new versions support FSDP better than the currently used pytorch lightning version (2.2.0.post0).

But short answer: model parallelism is not something we are supporting in the current codebase.

idea here. Ctranslate2 just integrated tensor parallelism. It also support XMLRoberta, so just wondering if we could adapt a bit the converter so that we could run the model within CT2 which is very fast.
How different is it from XML Roberta at inference ?

Does it support XLM-R XL? the architecture also differs from XLM-R 

It seems like they improved documentation a lot actually: https://lightning.ai/docs/pytorch/stable/advanced/model_parallel/fsdp.html



> Does it support XLM-R XL? the architecture also differs from XLM-R

we can adapt if we have the detailed description somewhere.
cc @minhthuc2502
Repository: Unbabel/COMET
Issue: [INPUT] Text Length of Input (source, reference, and hypothesis)
GitHub labels: question

For translation quality estimation of COMET, I think there is no limitation of the text length. However, from my personal perspective, I do not think the estimation will be accurate if the text is too long. 

So, what text length (of source, reference, and hypothesis) do you recommend?


Hi @foreveronehundred! The code does not break when running very very large segments BUT the models truncate the input if it goes above 512 tokens. For models like `wmt22-cometkiwi-da` the input will be shared for both source and translation which means that the total number of tokens from source and translation should not be longer than 512 tokens....

Still, 512 tokens is a long input. Its more than enough to input several sentences together and evaluate entire paragraphs. Maybe not enough for an entire 2 page document tho.

A quick way to test it is to tokenize both inputs and get their length:
```python
from transformers import XLMRobertaTokenizer
source = ["Hello, how are you?", "This is a test sentence."]
translations = ["Bonjour, comment ça va?", "Ceci est une phrase de test."]
# This is the same for most COMET models
tokenizer = XLMRobertaTokenizer.from_pretrained("xlm-roberta-base") 
# Tokenize and count tokens for each pair
for src, trans in zip(source, translations):
     # Tokenize sentences
    src_tokens = tokenizer.encode(src, add_special_tokens=False)
    trans_tokens = tokenizer.encode(trans, add_special_tokens=False)

    # Jointly encode and count tokens
    joint_tokens = tokenizer.encode(src, trans, add_special_tokens=True, truncation=True)

    # Output token counts
    print(f"Source: {src}\nTranslation: {trans}")
    print(f"Source tokens: {len(src_tokens)}")
    print(f"Translation tokens: {len(trans_tokens)}")
    print(f"Jointly encoded tokens: {len(joint_tokens)}")
    print("="*30)
```




Thanks for reply. I think the length is enough for general cases.
By the way, I want to know the token length of the training data. Could you give some statistics (Mean, STD, etc.)?
Repository: Unbabel/COMET
Issue: Change the global variable logger to comet_logger
Root cause: Architectural Constraint


The point lies here: https://github.com/Unbabel/COMET/blob/master/comet/__init__.py#L22

When we import anything like "from comet import load_from_checkpoint" it will create a global variable "logger" and there are consequences if in the main code calling this import we also have a variable "logger".

Since the likelihood is more "calling comet from another toolkit" rather than the contrary, I would like to rename the logger variable to comet_logger which has to happen only in a few instances.

Do you agree?

I agree. I don't see any reason why not.
Repository: Unbabel/COMET
Issue: Training script for XCOMET
GitHub labels: question

Kindly provide the training scripts as well as data preparation scripts for the XCOMET metric.

Hi @babangain

The training script for XCOMET won't be available for now. We are trying to get approval to release them... the codebose for now only supports inference for XCOMET and parts of the training script are missing.
Repository: Unbabel/COMET
Issue: Safetensors Support
Root cause: Unimplemented Feature Gap

## 🚀 Feature
<!-- A clear and concise description of the feature proposal -->
It would ease the model usage if the weights of large (>1B) models were available in ```.safetensors```.
### Motivation
As for now, it takes ~1 hour to download ```unbabel/*-xxl``` models on A100. It can be resolved with other formats for model weights.
The safetensors format is faster than the Pytorch ones, as stated here: https://huggingface.co/docs/safetensors/en/speed.

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->

### Additional context
It seems that ```safetensors``` support would require:
- converting the weights for published models (i.e. ```save_model``` from ```safetensors.torch```)
- support within the classes, for example, new function for loading safetensors in ```class CometModel```
- another function similar to ```load_from_checkpoint```
<!-- Add any other context or screenshots about the feature request here. -->


My understanding is that safetensors would not fix the download speed (model size) problem. The link you sent talks about loading the models from the disk.
Repository: Unbabel/COMET
Issue: [QUESTION] why num_layers = num_hidden_layers + 1
GitHub labels: question

What is the reason for this ?

here: https://github.com/Unbabel/COMET/blob/master/comet/encoders/bert.py#L67

thanks.

Hi @vince62s yes because the config does not account for the embeddings, yet the representations are returned when you set `output_hidden_states=True`.
Repository: Unbabel/COMET
Issue: Training data and scripts used for wmt22-cometkiwi-da
Root cause: Documentation Deficiency

Hi Team,

Can you share the training data and training scripts used for wmt22-cometkiwi-da. We want it reference for training with our own sample reference data.

Hi @rohitk-cognizant,

To train `wmt22-cometkiwi-da` you just have to run:

```
comet-train --cfg configs/models/{your_model_config}.yaml
```

Your configs should be something like this:

```yaml
unified_metric:
  class_path: comet.models.UnifiedMetric
  init_args:
    nr_frozen_epochs: 0.3
    keep_embeddings_frozen: True
    optimizer: AdamW
    encoder_learning_rate: 1.0e-06
    learning_rate: 1.5e-05
    layerwise_decay: 0.95
    encoder_model: XLM-RoBERTa
    pretrained_model: microsoft/infoxlm-large
    sent_layer: mix
    layer_transformation: sparsemax
    word_layer: 24
    loss: mse
    dropout: 0.1
    batch_size: 16
    train_data: 
      - TRAIN_DATA.csv
    validation_data: 
      - VALIDATION_DATA.csv
    hidden_sizes:
      - 3072
      - 1024
    activations: Tanh
    input_segments:
      - mt
      - src
    word_level_training: False
    
trainer: ../trainer.yaml
early_stopping: ../early_stopping.yaml
model_checkpoint: ../model_checkpoint.yaml
```

Hi @ricardorei ,

Thanks for the update. Can I use the same training parameters mentioned in master branch trainer.yaml file?

Hmm maybe you should change them a bit. For example to train on a single GPU (which is usually faster) and with precision 16 use this:
```yaml
  accelerator: gpu
  devices: 1
  # strategy: ddp # Comment this line for distributed training
  precision: 16
```
You might also want to consider reducing the `accumulate_grad_batches` to 2 instead of 8
```yaml
  accumulate_grad_batches: 2
```

What is the format that the data should look like?
Repository: Unbabel/COMET
Issue: Add missing library stubs or py.typed marker
Root cause: Unimplemented Feature Gap

## 🚀 Feature
Add missing py.typed marker

### Motivation

This proposal aims to make the project adhere to the [PEP 561 standard for stub packages](https://mypy.readthedocs.io/en/stable/installed_packages.html#installed-packages), thereby enabling type checking. Given that the project is largely typed, it could also benefit other packages that depend on comet and utilize mypy.

### Alternatives

Another option is to configure mypy to disregard missing import errors in projects that rely on comet. For example by adding a # type: ignore at the end of the line containing the import.

### Additional context

I'm using comet in my personal project, which has a CI step for type checking.
Repository: Unbabel/COMET
Issue: Model download error
Root cause: External Dependency Breakage

## ❓ Questions and Help

I tried to download the model using:
model = download_model("wmt-large-da-estimator-1719"

But I get the following error:

'''
AttributeError                            Traceback (most recent call last)
<ipython-input-3-46133d14a8d8> in <module>
     
----> 4 model = download_model("wmt-large-da-estimator-1719")

7 frames
/proj/tools/anaconda3/lib/python3.8/site-packages/torch/nn/modules/module.py in __setattr__(self, name, value)
    817                     buffers[name] = value
    818                 else:
--> 819                     object.__setattr__(self, name, value)
    820 
    821     def __delattr__(self, name):

AttributeError: can't set attribute
'''


 - OS: [Ubuntu 18.04]
 - Packaging [pip]
 - Version [20.2.4]


which _torch_ version and _PyTorch-lightning_ version are you using?

what happens if you install version 0.0.6?

`pip install unbabel-comet==0.0.6`

pytorch version: 1.6.0
PyTorch-lightning version: 1.3.0

I managed to run it by commenting lines 133 to 136 in comet/models/model_base.py and replacing them with this:
```
for key in hparams.keys():
            self.hparams[key]=hparams[key]
```

I think this is related to a breaking change between Pytorch Lightning 1.2.* and 1.3.0. The requirement should be `pytorch-lightning = "<1.3"` and not `pytorch-lightning = "<=1.3"`.

If you still want, you can `pip install pytorch-lightning==1.2.8` to solve that issue

I see. Thank you for that!
Repository: Unbabel/COMET
Issue: I see Unbabel comet is downloading models--xlm-roberta-large folder every time, is there any way to load it from local, if yes please share the hack.[QUESTION]
Root cause: Documentation Deficiency

## ❓ Questions and Help

### Before asking: 
1. Search for similar [issues](https://github.com/Unbabel/COMET/issues).   
3. Search the [docs](https://unbabel.github.io/COMET/html/index.html).    

<!-- If you still can't find what you need: -->

#### What is your question?

#### Code

<!-- Please paste a code snippet if your question requires it! -->   

#### What have you tried?

#### What's your environment?

 - OS: [e.g. iOS, Linux, Win]
 - Packaging [e.g. pip, conda]
 - Version [e.g. 0.5.2.1]


I see Unbabel comet is downloading models--xlm-roberta-large folder every time, is there any way to load it from local, if yes please share the hack, because I am not allowed to hit hugging face in production environment.

if there's a way, it will be helpful

rewrite the reload_hparams.yaml:
pretrained_model:  (your model path)

and

in the code,you should write like:
model = load_from_checkpoint(
    CHECKPOINT_PATH,
    reload_hparams=True,
    strict=False  
)
Repository: Unbabel/COMET
Issue: [QUESTION] predict multiple times with a model
Root cause: Resource Mishandling

## ❓ Questions and Help

### Before asking: 
1. Search for similar [issues](https://github.com/Unbabel/COMET/issues).   
3. Search the [docs](https://unbabel.github.io/COMET/html/index.html).    

<!-- If you still can't find what you need: -->

#### What is your question?
When I tried to predict some data multiple times, there's problem.
First dataset prediction is complete.
While it try second dataset, this process is stucked and only GPU0 is occupied very long time.
So, I have to kill the process and start from second dataset again.
Below is final part of log of trying second dataset

GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
Killed

What should I do for fixing this problem?

#### Code

model_path = download_model("Unbabel/wmt22-cometkiwi-da")
model = load_from_checkpoint(model_path)
model.predict(data1, batch_size=256, gpus=8)["scores"]
...
model.predict(data2, batch_size=256, gpus=8)["scores"]

#### What have you tried?

#### What's your environment?
python 3.11.9
pyenv
Linux 22.04
torch==2.3.1
unbabel-comet==2.2.2

 - OS: [e.g. iOS, Linux, Win]
 - Packaging [e.g. pip, conda]
 - Version [e.g. 0.5.2.1]


I've encountered the same issue. Have you resolved it?
Repository: Unbabel/COMET
Issue: [QUESTION] how to enable multi-gpu when calling the predict method
Root cause: Documentation Deficiency

The predict method is called by setting gpus=2, but only one gpu is used. 
`model.predict(samples=samples,batch_size=2, gpus=2, devices=2, accelerator="gpu", strategy="ddp")`

you need to specify clear device number like "devices=[1,2]"：
```model.predict(samples=samples,batch_size=2,_ gpus=2, devices=[1,2], accelerator="gpu", strategy="ddp")```
or don‘t pass the  'devices'  like：
```model.predict(samples=samples,batch_size=2,_ gpus=2, accelerator="gpu", strategy="ddp")```
Repository: Unbabel/COMET
Issue: Cannot use `load_from_checkpoint` in an offline environment
Root cause: Unimplemented Feature Gap

### Motivation

Due to the policy of my company, I cannot access to an online server (e.g., huggingface).
So I tried to use `load_from_checkpoint` function after I manually downloaded `wmt22-comet-da` checkpoint.
However, it makes an SSL error "SSLError: HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded".

This error is raised because of missing `local_files_only` param to each model's `from_pretrained` method.

## 🚀 Feature

I think COMET needs to support `local_files_only` parameter to `load_from_checkpoint` function not to raise http connection error.

Below is my current solution to the problem.

- Add `local_files_only` param to `load_from_checkpoint`.
- Pass it to str2model value classes' `load_from_checkpoint`(https://github.com/Unbabel/COMET/blob/master/comet/models/__init__.py#L88)
    - it automatically passes the param to model instantiation through `LightningModule`'s `load_from_checkpoint` using `kwargs`.
- Add `local_files_only` param to `CometModel.__init__()` (https://github.com/Unbabel/COMET/blob/master/comet/models/base.py#L94)
- Pass it to str2encoder value classes `from_pretrained` (https://github.com/Unbabel/COMET/blob/master/comet/models/base.py#L119)
- Add `local_files_only` param to each encoder classes' `__init__` such as

```python
self.model = XLMRobertaModel(
    XLMRobertaConfig.from_pretrained(pretrained_model, local_files_only=local_files_only),
    add_pooling_layer=False,
)
```

Can I make a PR?
Please let me know if there is better way.


yep, this seems good to me!

I made a PR for this issue, please check it :)

Additionally, I think it is also needed to add `cache_dir` param too. (If you thinks it's okay, I will make another PR for this)

Hi @zzaebok thanks for the PR! 

I am on vocations and going to ACL next week so it might take me some time but I will look at it ASAP. 

> Hi @zzaebok thanks for the PR!
> 
> I am on vocations and going to ACL next week so it might take me some time but I will look at it ASAP.

any updates?

i need this update as well :)

do i simply place the downloaded (eg. xlm-roberta-large) and place it on the parent folder?

> do i simply place the downloaded (eg. xlm-roberta-large) and place it on the parent folder?

I modified the local comet configs `hparam.yaml`, where there is a path for the infoxlm ckpt path (`pretrained_model`). Then switch on the reload_hparams during comet loading:
          `  comet_scorer = comet.load_from_checkpoint(args.comet_ckpt, reload_hparams=True)
`

@vergilus appreciate your feedback 👍   working now :)
Repository: Unbabel/COMET
Issue: [QUESTION] MQM dataset on huggingface
GitHub labels: question

For the dataset shared here: https://huggingface.co/datasets/RicardoRei/wmt-mqm-human-evaluation

the data summary says the score is: MQM score. Sample row below:

> 1776	en-de	He said: "I know of several other guys over the internet who feel the same way," but added that they are "too cowardly to act on their anger."	Er sagte: „Ich weiß ganz genau, dass es noch mehr Typen im Internet gibt, die das Gleiche denken wie ich“, so Minassian, wenngleich er hinzufügte, dass diese „wohl zu feige wären, um ihrem Zorn freien Lauf zu lassen“.	Er sagte: „Ich kenne mehrere andere Typen über das Internet, die genauso empfinden“, fügte aber hinzu, dass sie „zu feige sind, um ihre Wut auszuleben“.	**-0.333333**	Human-A.0	3	news	2020

- Is the score here in *bold* a z-score already normalized per annotator?  
- If so, does it make sense to combine this MQM dataset with the DA dataset to train a COMET-like model from scratch?
Repository: Unbabel/COMET
Issue: [QUESTION] developing on mac with torch > 2.0.0
Root cause: Configuration Error

## ❓ Questions and Help

I installed Comet from source successfully on Linux with `pip install .`

On MacOS (for local development), I got the following errror:

```
ERROR: Could not find a version that satisfies the requirement torch<2.0.0,>=1.6.0 (from unbabel-comet[dev]) (from versions: 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0)
ERROR: No matching distribution found for torch<2.0.0,>=1.6.0
```

Could I install `torch>2.0.0` and use it with Comet?

I am using torch 2.1.1 on my MacBook with an M2 chip without any problems

probably my bad ... I haven't used the latest master
Repository: Unbabel/COMET
Issue: [QUESTION] finetuning COMET base models
GitHub labels: question

Hi,

I am trying to do some ablation studies on my custom pSQM dataset. I am constrained for compute and memory and cannot train the large (L/XL) or explainable models yet.

I would like to do the following:

1. train a **DA model** from scratch using my own data: I use the config as defined in the `regression_model.yaml` config file.
2. finetune `wmt22-comet-da` using the `regression_model.yaml` config file
3. train a **QE model** from scratch using my own data: I use the config as defined in the `referenceless_model.yaml` config file.
4. finetune `wmt22-cometkiwi-da` using the `unified_metric.yaml` config file with `input_segments:
      - mt
      - src`

My question is whether 3 and 4 are analogous to 1 and 2? 
Or should I be training and finetuning the QE models as per the `unified_metric.yaml` config?

Would appreciate any pointers regarding this.
Repository: Unbabel/COMET
Issue: docCOMET available ? 
Root cause: Documentation Deficiency

https://github.com/Unbabel/COMET/blob/332dfb0dd16d30eb68a0a61e16d3da2b19a0b06f/README.md?plain=1#L13

Is docCOMET really available?

the original repo hasn't been updated for a while (https://github.com/amazon-science/doc-mt-metrics/tree/main)


regards

I have the same confusion. From https://github.com/Unbabel/COMET/blob/master/comet/cli/score.py we can see a parameter "--enable-context" has been added to the script, but it seems the package installed from pip has not been updated, since running `comet-score --enable-context` will raise the following error:
```
comet-score: error: Unrecognized arguments: --enable-context
```
Repository: Unbabel/COMET
Issue: Comet installation fails
Root cause: External Dependency Breakage

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

### To Reproduce 

My Python environment is 3.12.4 and sentencepiece dependency require python < 3.12 and there is a new sentencepiece 0.2.0 released when can we expect new release of commet library

### Expected behaviour
Installation should proceed normally

### Error Trace

```INFO: pip is looking at multiple versions of unbabel-comet to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install -r requirements.txt (line 1) and sentencepiece==0.2.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested sentencepiece==0.2.0
    unbabel-comet 2.2.2 depends on sentencepiece<0.2.0 and >=0.1.96

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict```

### Environment
iOS: 15.0 [e.g. iOS, Linux, Win]
Packaging pip
pip Version: 24.0
Python Version: 3.12.4

### Additional context

<!-- Add any other context about the problem here. -->


Any update on this ?

I am bumping the sentencepiece version and publishing a new version. Thanks for raising this!