zhiminy/EvalEng huggingface dataset:
Repository: Unbabel/COMET
Issue: Make models artifacts on HuggingFace adhere to conventional structure
Root cause: Unimplemented Feature Gap

## 🚀 Feature
Models artifacts on HuggingFace should adhere to a more conventional structure including serialized model binary and json config files

### Motivation

This would allow to directly the deploy the models as HuggingFace predictors and serve them in an endpoint. This allows to start experimentation/testing faster.

### Alternatives

Creating a new HF space including the requested artifacts, which would not be practical

### Additional context

The end goal is to deploy comet within Amazon Sagemaker
Repository: Unbabel/COMET
Issue: Error while running explainable metrics
Root cause: External Dependency Breakage

Hi @kepler 
https://github.com/Unbabel/COMET/blob/770c2900c93ad6ed30bd43227531325e76d9b019/explainable-metrics/explain_unite.py#L277

I got an error in this line while running comet explainable:
`python explainable-metrics/explain_comet.py -m COMET/wmt22-comet-da/checkpoints/model.ckpt -t ende-2021-concat.csv --batch_size 8`

**raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'XLMREncoder' object has no attribute 'add_span_tokens'**

**requirements.txt**
```
sentencepiece== 0.1.96
transformers ==4.17
pytorch-lightning ==2.0.0
jsonargparse = "3.13.1"
torch==2.4.0"
```
Repository: Unbabel/COMET
Issue: Runtime error on Apple Silicon
Root cause: Environment Incompatibility

## 🐛 Bug
After installation and set-up on Apple Silicon, running the following command gives error `RuntimeError: _share_filename_: only available on CPU`
```
comet-score -s src.txt -t hyp.txt --model Unbabel/wmt22-cometkiwi-da
```

Proposed fix here: #236 

### Environment
OS: macOS
Packaging: conda
Version: lastest master branch

### Additional context

<!-- Add any other context about the problem here. -->


Thanks for the PR I am merging it and bumping the version
Repository: Unbabel/COMET
Issue: [QUESTION] About Z-score
GitHub labels: question

Hi, I find that COMET is trained on the z-score of DA. However, I am not sure about the implementation. 

**Is it rescaled on the translation direction level or something else?**

By essence the z-score is rescaled by annotator to make sure there are no big discrepancies. Therefore it should make scores consistent across languages but it is not so meaningfull to compare scores between languages.
However since I used the same method as COMET in my estimator for EuroLLM here: https://medium.com/p/7dccfe167814
and as wmt24 provides the same English source for most language pair, you can see that scores are not so much different across traditional pairs. The question is more about: do we really trust the DA scores year after year ....
Repository: Unbabel/COMET
Issue: [QUESTION] XCOMETXXL score decreases as the length of the text increases
Root cause: Algorithmic Error

## ❓ Questions and Help

<!-- If you still can't find what you need: -->

#### What is your question?
Hello, I conducted an experiment here: the ref and target were set to exactly the same text, and the Comet score and XCOMETXXL score were run separately. Here, a phenomenon was found that as the length of the target text became longer and longer, the Comet score was relatively stable at around 0.9, but the XCOMETXXL score dropped sharply, even reaching below 0.1.
This phenomenon is puzzling.

#### Code
self.model = load_from_checkpoint(self.comet_config[model_name]['model_path'])
model_output = self.model.predict(data, batch_size=self.batch_size, gpus=1)

<!-- Please paste a code snippet if your question requires it! -->   

#### What have you tried?

#### What's your environment?

 - OS: Linux
 - Packaging pip
 - Version unbabel-comet==2.2.2  python==3.9
Repository: Unbabel/COMET
Issue: Refless example doesn't work with 1.0.0rc4
Root cause: Validation Gap

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

### To Reproduce 

Following the README:

```
> echo -e "Dem Feuer konnte Einhalt geboten werden\nSchulen und Kindergärten wurden eröffnet." >> src.de
> echo -e "The fire could be stopped\nSchools and kindergartens were open" >> hyp.en
> comet-score -s src.de -t hyp.en --model wmt20-comet-qe-da
```

results in the following output:
```
Global seed set to 12
usage: comet-score [-h] [-s SOURCES] [-t TRANSLATIONS] [-r REFERENCES] [--batch_size BATCH_SIZE] [--gpus GPUS]
                   [--to_json TO_JSON]
                   [--model {emnlp20-comet-rank,wmt20-comet-da,wmt20-comet-qe-da,wmt21-cometinho-da}]
                   [--mc_dropout MC_DROPOUT] [--seed_everything SEED_EVERYTHING]
comet-score: error: wmt20-comet-qe-da requires -r/--references.
```

Looking at the code:
https://github.com/Unbabel/COMET/blob/61caa5a37cd4db35525cd2122d8c55423929455d/comet/cli/score.py#L79-L80
it seems that the model has to have `refless` in the name, which none of the available models have:
```
comet-score: error: argument --model: invalid choice: [...] (choose from 'emnlp20-comet-rank', 'wmt20-comet-da', 'wmt20-comet-qe-da', 'wmt21-cometinho-da')
```

### Expected behaviour
I'd hope to get a score for my hypotheses given the sources.

### Screenshots
n/a

### Environment
OS: MacOS
Packaging: pip
Version: unbabel-comet==1.0.0rc4

### Additional context

<!-- Add any other context about the problem here. -->


You are right Ill fix it. I changed the name and that test skipped. As a workaround, you can pass the source sentences "as references". They are ignored by the model

Thanks @ricardorei! Would it make sense to also adjust `comet-compare`? Currently it throws the same error w/o references and the `-qe-da-` model:
```
usage: comet-compare [-h] [-s SOURCES] [-x SYSTEM_X] [-y SYSTEM_Y] [-r REFERENCES] [--batch_size BATCH_SIZE]
                     [--gpus GPUS] [--num_splits NUM_SPLITS] [--sample_ratio SAMPLE_RATIO] [--to_json TO_JSON]
                     [--model {emnlp20-comet-rank,wmt20-comet-da,wmt20-comet-qe-da,wmt21-cometinho-da}]
                     [--seed_everything SEED_EVERYTHING]
comet-compare: error: wmt20-comet-qe-da requires -r/--references.
```

yes, the same error has to be fixed on comet-compare
Repository: Unbabel/COMET
Issue: [QUESTION]How to set language pairs for Reference-free Model
Root cause: Documentation Deficiency

## ❓ Questions and Help

I used Reference Free Model [Unbabel/wmt23-cometkiwi-da-xxl](https://huggingface.co/Unbabel/wmt23-cometkiwi-da-xxl)
My source langauge is English, target language is Chinese.
However there is some cases that the source(also target) text contains both Chinese and English. 
I want the model to test fixed language pair English->Chinsese.
Is there any method ? I check both document and comet-score --help, there is no such function explaination.
Repository: Unbabel/COMET
Issue: [QUESTION] About COMET Train and Test Data
Root cause: Documentation Deficiency

Hi, I have some questions about the dataset provided here: https://github.com/Unbabel/COMET/tree/master/data

1) If I understand correctly, the training data for each year's WMT is accumulated from all previous WMTs. Does this mean that here (https://github.com/Unbabel/COMET/tree/master/data), DA data for 2021 is a subset of 2022? 

2) The training data here (https://github.com/Unbabel/COMET/blob/master/configs/models/referenceless_model.yaml) is set to `data/1720-da.csv`. Does this mean that just merge all DA data here (https://github.com/Unbabel/COMET/tree/master/data) from 2017 to 2020? Are there any duplication issues?

3) Where can I find the test data if I want to formally test my metric on the WMT21 DA Task?
Repository: Unbabel/COMET
Issue: KeyError: "Model 'Unbabel/XCOMET-XXL' not supported by COMET."
Root cause: Environment Incompatibility

## 🐛 Bug

```
from comet import download_model, load_from_checkpoint

model_path = download_model("Unbabel/XCOMET-XXL")
model = load_from_checkpoint(model_path)
data = [
    {
        "src": "Boris Johnson teeters on edge of favour with Tory MPs", 
        "mt": "Boris Johnson ist bei Tory-Abgeordneten völlig in der Gunst", 
        "ref": "Boris Johnsons Beliebtheit bei Tory-MPs steht auf der Kippe"
    }
]
model_output = model.predict(data, batch_size=8, gpus=1)
# Segment-level scores
print (model_output.scores)

# System-level score
print (model_output.system_score)

# Score explanation (error spans)
print (model_output.metadata.error_spans)
```

<!-- KeyError: "Model 'Unbabel/XCOMET-XXL' not supported by COMET."-->

KeyError: "Model 'Unbabel/XCOMET-XXL' not supported by COMET."

### Environment
OS: Linux
Packaging: pip
Version: 2.2.4


Solved! 
It got resolved upon vacating space in /home folder and downloading the model again.

> Solved! It got resolved upon vacating space in /home folder and downloading the model again.

How did you do that? (I have the same issue on Windows 11.)
Repository: Unbabel/COMET
Issue: Comet reference models reference text parameter
Root cause: Documentation Deficiency

## ❓ Questions and Help

#### What is your question?
I have seen in the sample and documentation that the reference text is translation of the source text. Is that always the expectation?

#### Code

I am using this command 

comet-score -s src.de -t hyp1.en -r ref.en

#### What's your environment?

 - MAC
Repository: Unbabel/COMET
Issue: Support for EuroLLM encoder
Root cause: Unimplemented Feature Gap

## 🚀 Feature

Support for more recent encoders such as the one use for EuroLLM.

### Motivation

First, thanks a lot for the great work on this framework! We have been using it to train our own UnifiedMetric. However, with the limitation of 24GB of vRAM, our options are somehow limited. We get the best results with XLM-RoBERTa as encoder and pretrained models like InfoXLM or RoBERTa. Unfortunately, these models are not very recent and do not cover all the languages we use. So, I wonder if it would be possible to add support for other encoders, such as the one used in EuroLLM.

Many thanks in any case!



EuroLLM is a causal LLM so that would require some continued training as a bidirectional encoder.
EuroBERT should be plug-n-play but unfortunately the "Euro" in EuroBERT is not very "european languages".

I see, thanks! I initially mentioned EuroBERT but then realized it only covers 15 languages, which is indeed far from the 24 official EU languages supported by EuroLLM. This remains the main limitation we're facing, so it would still be great to have more encoders supporting more recent pretrained models.

I actually have experimented EuroBERT and I do have the code for it. I can make the PR.

Yep, EuroBERT does not support many languages. Btw a casual decoder can also work within COMET and I have done something similar in the past. It complicates some features but for regression is totally fine to use the EOS token instead of the BOS token (which is the default for bidirectional models)
Repository: Unbabel/COMET
Issue: jsonargparse==3.13.1 required for comet is incompatible with Python>=3.12
Root cause: External Dependency Breakage

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

### To Reproduce 

Hello !

I attempt to use the `--enable-context` feature, so I tried to install the recent version of comet via 

`pip install unbabel-comet`

in a conda environment with Python==3.12.9.

Then I discovered that the specific version (3.13.1) of jsonargparse, that is required for comet, is not compatible with Python 3.12, so that it results in the following error:

```error: ArgumentParser._parse_known_args() missing 1 required positional argument: 'intermixed'```

Installing comet with Python3.11 can avoid this problem.


### Environment
OS: Linux
Packaging : conda environment
Version:  Python==3.12.9,  jsonargparse==3.13.1, unbabel-comet 2.2.6, unbabel-comet 2.2.4



I use comet-score on python3.12 also have this error, demote to python3.10 can run normally :)

And why limit to `jsonargparse==3.13.1`? it seems incompatible with pathlib's `Path`
Repository: Unbabel/COMET
Issue: Protobuf 6 support
Root cause: External Dependency Breakage

## 🚀 Feature
Bump `protobuf` dependency to 6

### Motivation
Currently, the `protobuf` dependency is set to version 4. This was deprecated in Q1 2025. 

### Alternatives

We could also use `protobuf` version 5, as at least it has not reached End of Life yet.

### Additional context

When trying to use the library with `opentelemetry`, I end up with version conflict on protobuf. Is it possible to upgrade to the latest version of `protobuf`?


I'm also having issues with conflicts due to the protobuf pin. 

I'm wondering if the dependency is even needed, it doesn't seem like the project uses it, and according to `poetry show protobuf` no other package requires it. Also, the tests pass without it.

I can prepare a small PR if it's OK to remove the dependency pin.
Repository: Unbabel/COMET
Issue: [QUESTION]
Root cause: Configuration Error

#### What is your question?

As per [pypi](https://pypi.org/project/unbabel-comet/#history) c2.2.7 is not released yet whereas it's released as per github
Pip index versions also suggest 2.2.6 to be the latest.




@aminfarajian asked to tag @ricardorei for help.
Repository: Unbabel/COMET
Issue: Cannot disable the progress bar
Root cause: Documentation Deficiency

## 🐛 Bug

Hi,

I'm using version 1.0.0rc6 and scoring within Python, `seg_scores, sys_score = model.predict(data, gpus=1)`.

However, I cannot disable the progress bar using `show_progress=False`. It would be nice to have this option.

Thanks!


@antonio-farinhas can you test it now?
`seg_scores, sys_score = model.predict(data, gpus=1, progress_bar=False)`
Repository: Unbabel/COMET
Issue: Negative score Unbabel/XCOMET-XL
Root cause: Validation Gap

## 🐛 Bug

Hi!

I am using Unbabel/XCOMET-XL for reference-based MT evaluation and observed a negative score. Is this expected?

I am aware from [Issue #38](https://github.com/Unbabel/COMET/issues/38) that older COMET models were trained on unbounded z-scores. However, the [XCOMET paper](https://aclanthology.org/2024.tacl-1.54) explicitly states: "we employ min-max scaling on our DA corpus to set its range of scores to [0, 1]".

If the model architecture does not include a final sigmoid layer or hard clipping, I assume it is possible for limit cases to output values slightly below 0 or above 1. Is this the case? If so, is clipping a good method to force the values to be in the range [0, 1]?

### To Reproduce 

```python
from comet import download_model, load_from_checkpoint

model_name = "Unbabel/XCOMET-XL"
model_path = download_model(model_name)
model = load_from_checkpoint(model_path)

src = ["There were protests worldwide, several criminal prosecutions, and the leaders of the governments of Iceland and Pakistan both resigned."]
mt = ["Було проведено протести по всьому світу, кілька кримінальних переслідувань, а лідери урядів Ісландії та Пакистану both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both both"]
ref = ["У світі відбулися протести, кілька кримінальних переслідувань, а обидва лідери урядів Ісландії та Пакистану пішли у відставку."]

# inference
data = [{"src": s, "mt": t, "ref": r} for s, t, r in zip(src, mt, ref)]
scores = model.predict(data, batch_size=1, gpus=0, accelerator="cpu")

print(scores) # "scores": -0.0025453418493270874

assert scores["scores"][0] < 0.0, scores
```

### Expected behaviour
I expected to obtain a value for the score in the range [0, 1].

### Environment
OS: Linux (Ubuntu 24.04.3 LTS)
Packaging: comet installed through pip inside a conda environment
Version: comet 2.2.7 (pypi), python 3.11.9 (conda)
Repository: Unbabel/COMET
Issue: How can I use QE model with HTER? (wmt20-comet-qe-hter)
GitHub labels: question

I'm using wmt20-comet-qe-da for MT quality estimation. I wanted to use HTER based QE model for better interpretability. Is that model supported yet? If not can you guide me a bit to understand DA system scores? eg. I have source and MT. I get a DA score - what's the threshold score to be considered good or bad?


Hi @ShivaliGoel for HTER we have reference-based metrics only. The HTER models are not as "good" (the correlations with other types of human judgments like MQM or DA is lower) as the DA models and I advise you to use the DA models because of that.

DA is a very simple human metric. You basically ask an annotator to look at a source segment and a translation hypothesis and to assign a score between 0-100 that reflects the adequacy of the translation.

Since they are often collected on a large scale, they need to be normalized between annotators and converted to z-scores. That's why they are sometimes negative and a bit harder to "understand". Usually, a z-score above 0 means that the translation is better than average, and below zero is the opposite. If you want a more deep explanation of DA's and how they correlate with other human assessments, please just asks!

The most famous tool to perform this type of annotation is [Appraise](https://github.com/cfedermann/Appraise).



@ricardorei May I ask if DA scores are bounded to be lower than 1.0? I barely see any numbers that are larger than 1.0 in the figure below.
![image](https://user-images.githubusercontent.com/16165404/138646336-a96064b4-3054-4a52-a8d7-d2e5f054f90f.png)


Hey @chenming-wu! The `wmt20-comet-da` (previously called `wmt-da-estimator-1719`) model is not bounded and some scores can go over 1. In the plot above you can see that the smoothed line goes over 1.0. It is very rare but that can happen...

The corresponding QE metric (which was developed afterward) is bounded with a sigmoid function


Thanks for clarifying!
Repository: Unbabel/COMET
Issue: Read from STDIN
Root cause: Unimplemented Feature Gap

## 🚀 Feature
<!-- A clear and concise description of the feature proposal -->

It would be really nice if COMET could read input from STDIN, e.g.,

```bash
# three fields triggers comet-ref
$ paste source.txt hyps.txt ref.txt | comet [args]

# two fields -> comet-src
$ paste source.txt hyps.txt | comet [args]
```

### Motivation

This is consistent with standard UNIX usage. It is also slightly less cumbersome, and allows comet to be used in settings without writing files to disk.


I'll investigate this. It seems like a good feature. 

Do you have support for this in `sacrebleu`? 

If so I can start by looking into `sacrebleu` implementation

Sacrebleu supports STDIN for the system output, but not for the ref (and doesn’t use the source). So the COMET-style use would be

```
$ cat system.txt | comet -s source.txt -r ref.txt
```

which is different from what I proposed.

But it gives me another idea, which would be to add support for sacrebleu-style builtin test sets, e.g.,

```
# one option
$ cat system.txt | comet -t wmt20 -l de-en [other args]

# another option
$ cat system.txt | comet --sacrebleu-testset wmt20/de-en
$ cat system.txt | comet --sacrebleu-testset mtedx/valid/pt-es
```

You could accomplish this by just using sacrebleu as a library. It’s pretty easy:

```
from sacrebleu.utils import get_source, get_references, get_files

# trigger sacrebleu test set
# make these optional: nargs=“?” for argparse
if args.source is None and args.references is None:
    if args.sacrebleu_dataset is None:
        # throw error

    # some test sets are hierarchical, e.g., “mtedx/valid”
    test_set, langpair = args.sacrebleu_dataset.rsplit(“/“, maxsplit=1)
    source = get_source(test_set, langpair)
    ref = get_referencees(test_set, langpair)

     # alternative
    source, ref, _ = get_files(test_set, langpair)
```

(Note that, via @ozancaglayan, sacrebleu also supports tab-delimited system output on STDIN, and will then do significance testing among them, e.g.,

```
$ paste system1.txt system2.txt | sacrebleu -t wmt20 -l de-en
```

Just something else to consider in terms of CLI).

Thanks, Matt! These are nice features indeed! Do you want to submit a PR 😁 ?

If not I can still try to allocate some time to do them before the new release

Hmm, good question. What’s your timeline for the release?



> On Oct 19, 2021, at 15:30, Ricardo Rei ***@***.***> wrote:
> 
> ﻿
> Thanks, Matt! These are nice features indeed! Do you want to submit a PR 😁 ?
> 
> If not I can still try to allocate some time to do them before the new release
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.


We were planning the release for the end of November beginning of December
Repository: Unbabel/COMET
Issue: wrong output of comet-compare
Root cause: Algorithmic Error

## 🐛 Bug

comet-compare output wrong info in the json file:
-  the same src,ref, x and y text are reported
- the scores are probably correct

I think that the problems are due to this line
https://github.com/Unbabel/COMET/blob/6009f675e80f7070b576b20b7438295d39889409/comet/cli/compare.py#L142
and the 3 following lines

It seems that the entry 0 is always output, while it should output the entry "i".




### To Reproduce 
comet-compare -s SRC.txt -r REF.txt -x SysX.txt -y SysY.txt --to_json JSON.txt

SRC.txt, REF.txt, SysX.txt and SysY.txt   must have more than one lines.


### Screenshots
If applicable, add screenshots to help explain your problem.

### Environment
OS: ubuntu
Packaging: pip3
Version: unbabel-comet==1.0.0rc8




You are right. 

The JSON outputs always have the same text, even tho the scores are correct. 

The resulting computations seem to be correct.

Would you like to do a PR?

Actually, I would avoid to do a PR.
I would like to install COMET as a package though pip3, without going through the source code.
When do you plan to fix the bug in the Python package?

I'll try to do it today. 

If you like, I can pass my bug-fixed version

Nicola

Nicola Bertoldi
VP Research & Development
***@***.***

-
Le informazioni contenute nella presente comunicazione sono di natura privata e come tali sono da considerarsi riservate ed indirizzate esclusivamente ai destinatari indicati e per le finalità strettamente legate al relativo contenuto. Se avete ricevuto questo messaggio per errore, vi preghiamo di eliminarlo e di inviare una comunicazione all’indirizzo e-mail del mittente.
-
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you received this in error, please contact the sender and delete the material.

> On 21 Oct 2021, at 18:19, Ricardo Rei ***@***.***> wrote:
> 
> 
> I'll try to do it today.
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub, or unsubscribe.
> Triage notifications on the go with GitHub Mobile for iOS or Android. 
> 



I think it's not necessary. These are small fixes

Thanks anyway!