zhiminy/EvalEng huggingface dataset:
Repository: Unbabel/COMET
Issue: small score difference for identical outputs
Root cause: Algorithmic Error

## 🐛 Bug

Using comet-compare
I noticed that the exact same outputs of two different systems receive different score, although almost negligible.
But, although negligibly different, these scores are non consider a tie; and hence  there is an impact on the number of wins/losses reported.

        "ties (%)": 0.0,
        "x_wins (%)": 1.0,
        "y_wins (%)": 0.0

    {
        "src": "Nedávno prohrál s Raonicem v Brisbane Open.",
        "system_x": {
            "mt": "He recently lost to Raonic at the Brisbane Open.",
            "score": 0.8726277947425842
        },
        "system_y": {
            "mt": "He recently lost to Raonic at the Brisbane Open.",
            "score": 0.872564971446991
        },
        "ref": "He recently lost against Raonic in the Brisbane Open."
    },
### To Reproduce 
comet-compare -s SRC -r REF -x SysX -y SysY --to_json JJJ

Actually, I am using directly the function "compare_command()" included in "cli/compare.py"

### Expected behaviour
Either an identical score for identical outputs, or a bit more flexible counts of wins/losses/ties

### Screenshots
If applicable, add screenshots to help explain your problem.

### Environment
OS: ubuntu
Packaging: pip3
Version: unbabel-comet==1.0.0rc8



Hey, Nicola thanks for reporting this... The way bootstrap resampling works will not be affected by very small segment level differences because for the `wins, losses, ties` count we will average across several segments and we will do this several times across different splits.

If two systems have the exact same translations I can ensure you that you will have `x_win = 0.5` and `y_win = 0.5` or at least something really close to that which is not statistically significant.

Btw the root of this problem comes from the layerwise normalization we do which can be affected by the `batch_size`. In practice, it's not desirable but negligible as you said.

why do not apply a smooth decision for the ties count?
Something like

```
        if subsample_x_scr > subsample_y_scr + epsilon:
            win_count[0] += 1
        elif subsample_y_scr > subsample_x_scr + epsilon:
            win_count[1] += 1
        else:
            win_count[2] += 1
```
with a reasonably small value for ``epsilon``

that's a reasonable idea for the system-level score. I'll fix it along with the other issues you reported

@nicolabertoldi thanks for the issues! I believe everything is working properly now. Please tell me if not
Repository: Unbabel/COMET
Issue: Problems with refless model
Root cause: Validation Gap

## 🐛 Bug

there are several issues with refless models:
- the default refless model "wmt20-comet-qe-da" si not included in the _REFLESS_MODELS; https://github.com/Unbabel/COMET/blob/6009f675e80f7070b576b20b7438295d39889409/comet/cli/score.py#L43
- even if added, the tool fails here https://github.com/Unbabel/COMET/blob/6009f675e80f7070b576b20b7438295d39889409/comet/cli/compare.py#L118

- the tool outputs the ref even if the model is reflesse; hence, it raises an error; https://github.com/Unbabel/COMET/blob/6009f675e80f7070b576b20b7438295d39889409/comet/cli/compare.py#L145


Actually, I solved all the bugs in my local code.


### To Reproduce 

Before reporting a bug, make sure that the bug can be reproduced with a minimal example and add your relevant changes, to see if the issue persists. 

If the test is failing, please add your test cases to the issue (as a draft PR, or simply paste the code to the issue description here).

### Environment
OS: ubuntu
Packaging: pip3
Version: unbabel-comet==1.0.0rc8
Repository: Unbabel/COMET
Issue: Calling xlm-roberta-large via HuggingFace
Root cause: Unimplemented Feature Gap

## 🚀 Feature
I would like to use this library for COMET scores but I want to host xlm-roberta-large elsewhere like HuggingFace. How we can enable in this library? Would modifying xlmr.py be enough to call the XLM model that is deployed to a remote GPU server?

### Motivation
I would like to place the computationally intensive encoder of the model on a GPU server (for faster batch-inference) that is shared by multiple COMET scorers but also maybe with other applications that may also be benefiting from xlm-roberta-large.

I am not sure I understand this. But we added Embedding Caching to COMET.

This means that after you instantiate a COMET model and that model will reuse computations for repeated batches (during inference only). Yet I think you mean something else... I think you want the "heavy" weight to be performed in a different machine and then we just reuse those embeddings? That seems over-complicated... What is your use case for this?
Repository: Unbabel/COMET
Issue: [QUESTION] About HTER models in download list.
Root cause: Unimplemented Feature Gap

## ❓ Questions and Help

### Before asking: 
1. Search for similar [issues](https://github.com/Unbabel/COMET/issues).   
3. Search the [docs](https://unbabel.github.io/COMET/html/index.html).    

<!-- If you still can't find what you need: -->

#### What is your question?
Hi, I found that the HTER models are off the download list of the current codes. 
https://github.com/Unbabel/COMET/blob/master/comet/models/__init__.py
I wonder whether they are still supported in the current version. 

I used version 1.0.0rc9, and it report this.
"Exception: wmt-large-hter-estimator is not in the `availale_metrics` or is a valid checkpoint folder."
Is that normal or should I use the previous version?
Thanks.

#### Code

#### What have you tried?

#### What's your environment?

 - OS: Linux
 - Packaging pip
 - Version 1.0.0rc9


Hey @ElliottYan 
I was planning to remove those as they were outperformed by the DA models but I can add them back!

For now, you will have to resort to the previous version... Those checkpoints still depend on fairseq but I have a script to convert the models into the new codebase (with transformers)

I see. Thanks for your reply!

As I need a bounded score, I will try the previous version for now, but I would really appreciate it if you add them back to the current one.

In addition, I see you explain about HTER models are not as good as DA ones. Can I ask how bad are we talking about? Is there any result reported that I missed? 

Thanks

we are not talking about a huge performance decrease but its usually a few points behind. You can look for example at the WMT20 Metrics task findins where we submitted several COMET versions. I also know that the authors of  the [To Ship or Not to Ship](https://arxiv.org/abs/2107.10821) compared the DA model with the HTER model. The HTER model was even outperformed by the reference-less DA model (`wmt20-comet-qe-da`)

Btw the `wmt20-comet-qe-da` is also bounded with a sigmoid. If you only care about having a bounded score this one might be useful. 

I'll add the HTER models meanwhile.

Oh, great! The "wmt20-comet-qe-da" model also performs very well in the paper, and I will try that. 
Thanks!
Repository: Unbabel/COMET
Issue: Error when running inference on newly trained metric
Root cause: Validation Gap

While inference (comet-score) using the provided metrics works well, if I try to train a new metric with comet-train and the use it to predict quality scores I get the following error:

`comet-score: error: argument --model: invalid choice: 'lightning_logs/version_19/checkpoints/epoch=1-step=129339.ckpt' (choose from 'emnlp20-comet-rank', 'wmt20-comet-da', 'wmt20-comet-qe-da', 'wmt21-cometinho-da')`

The error disappears once I comment out line 64: `choices=available_metrics.keys(),` in comet/cli/score.py

```
parser.add_argument(
        "--model",
        type=Union[str, Path_fr],
        required=False,
        default="wmt20-comet-da",
        choices=available_metrics.keys(),
        help="COMET model to be used.",
    )
```



Done!

You can now do: comet-score -s src.de -t hyp1.en hyp2.en -r ref.en --model YOUR/CHECKPOINT
Repository: Unbabel/COMET
Issue: Add WMT test sets via sacrebleu
Root cause: Unimplemented Feature Gap

[Extracting from #30]

It would be nice to would be to add support for sacrebleu-style builtin test sets, e.g.,

```
# one option
$ cat system.txt | comet -t wmt20 -l de-en [other args]

# another option
$ cat system.txt | comet --sacrebleu-testset wmt20/de-en
$ cat system.txt | comet --sacrebleu-testset mtedx/valid/pt-es
```

You could accomplish this by just using sacrebleu as a library. It’s pretty easy:

```
from sacrebleu.utils import get_source, get_references, get_files

# trigger sacrebleu test set
# make these optional: nargs=“?” for argparse
if args.source is None and args.references is None:
    if args.sacrebleu_dataset is None:
        # throw error

    # some test sets are hierarchical, e.g., “mtedx/valid”
    test_set, langpair = args.sacrebleu_dataset.rsplit(“/“, maxsplit=1)
    source = get_source(test_set, langpair)
    ref = get_referencees(test_set, langpair)

     # alternative
    source, ref, _ = get_files(test_set, langpair)
```

_Originally posted by @mjpost in https://github.com/Unbabel/COMET/issues/30#issuecomment-946825063_
Repository: Unbabel/COMET
Issue: Requirements conflicts
Root cause: Architectural Constraint

Hi,
I am using your tool in my python pipelines, but I had a problem, that your requirements are too strict and I get conflict with many tools. Could you please reinvestigate if you must specify all packages on exact version?
Additionally, are you planning to shift into transformer3? 
Thank you,
TK

Hey! Besides pytorch-transformers do you have any requirement that is giving you conflicts? 

We plan to move to pytorch-transformers soon. 

I have a conflict with several libraries, however, I created a temporary solution by having a second environment. But basically, transformers 3 was the main conflict which cannot be solved by downgrading, others conflicts were only in different versions (for example you don't use the latest pandas 1.1.4). 

If the modules are not strictly reliable on given libraries, maybe could you be more relaxed with requirements? For example:
pandas==1.* instead of pandas==1.0.* 
I know there could be conflicts introduced this way, but maybe not all modules have to be set on the exact version
Repository: Unbabel/COMET
Issue: TypeError: 'NoneType' object is not subscriptable when calling comet-score
Root cause: Validation Gap

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->


### To Reproduce 

Using comet 1.0.1 with Python 3.9, I get the following (using testset in /data of mt-telescope):

$ comet-score -s newstest2020-ruen.src.ru.txt -t newstest2020-ruen.OnlineA.txt -r newstest2020-ruen.ref.en.txt 
Global seed set to 12
wmt20-comet-da is already in cache.
Traceback (most recent call last):
  File ".../telescope-venv/bin/comet-score", line 8, in <module>
    sys.exit(score_command())
  File ".../telescope-venv/lib/python3.9/site-packages/comet/cli/score.py", line 180, in score_command
    model = load_from_checkpoint(model_path)
  File ".../telescope-venv/lib/python3.9/site-packages/comet/models/__init__.py", line 57, in load_from_checkpoint
    model_class = str2model[hparams["class_identifier"]]
TypeError: 'NoneType' object is not subscriptable

### Expected behaviour

Output comet score.

### Environment
OS: Linus
Packaging Pip
Version 1.0.1

If that makes a difference, this was run without gpu.

which version of PyTorch-lightning got installed?

It's 1.3.5

That's the correct version. I just tested it locally using python 3.9 and everything went well... I am not able to reproduce the error. Can you send me your exact virtual env?
Repository: Unbabel/COMET
Issue: [QUESTION] Training my own metric using Comet regression model
GitHub labels: question

Hi Ricardo, I read all your code implementations. From my understanding, when I use comet-train to train my own evaluation metric, I will not load in any pretrained Comet weights except the initialization of XLM-Roberta weights. I just want to double check if this is correct. Currently, I set "resume_from_checkpoint" in the trainer.yaml in the config to be "null".

Thank you in advance! This is a great work.

Yes, typically when you use `comet-train` you are just loading XLM-R weights (or any other encoder) and finetuning on top.

The only exception is if you use the model flag `load_weights_from_checkpoint`. This flag can be used to load an existing COMET checkpoint.


    

For our models from WMT21 shared task we actually did that.. we train a checkpoint on DA's and then we fine-tune them in MQM z-scores using that same flag: [Are References Really Needed?
Unbabel-IST 2021 Submission for the Metrics Shared Task](https://statmt.org/wmt21/pdf/2021.wmt-1.111.pdf)

Btw the advantage of `load_weights_from_checkpoint` over `resume_from_checkpoint` is that the architecture doesn't need to be exactly the same! we only load the matching (key, values) between the new model and the checkpoint. While in `resume_from_checkpoint` the model architecture needs to be exactly the same. Does this make sense?

Let's imagine you want to reuse the weights learned with a Ranking Model... with `load_weights_from_checkpoint` you will be able to do that and then fine-tune a Regression Model.


Thank you so much for your detailed explanations!
Repository: Unbabel/COMET
Issue: Why is COMET_DA default model?
GitHub labels: question

Hi Unbabel,

I wanted to ask, why is the wmt20-comet-da default model when using COMET? I am a bit worried that people using it off the shelf won't understand the underlying difference and will mistakingly report COMET scores on the QE metric.

Why not set the reference-based COMET as the default, since it seems to be outperforming comet-da (also I fear that comet-da will have more biases and potential problems than reference-based). 

Thank you for the answer,
Tom

Hi Tom, the default model is the reference-based model (`wmt20-comet-da` same as `wmt-large-da-estimator-1719` from previous versions). The reference-free models are "tagged" with the `qe` on the name, e.g: `wmt20-comet-qe-da`.

The naming `comet-da` only has to do with the fact that the model was trained to predict DA z-scores. 

This is the exact same model you used on your paper [Ship not to ship](https://arxiv.org/abs/2107.10821):

What you used as COMET in your tables is the same as `wmt20-comet-da` and the COMET-src is the `wmt20-comet-qe-da`.

Hi Ricardo,

Oh, I really apologize, I should not be working nights. Thank you for your clarification.

Have a lovely end of the year!
Repository: Unbabel/COMET
Issue: [QUESTION] Train my own metrics without source sentences
Root cause: Unimplemented Feature Gap

Hi Ricardo, thank you so much for your previous answers.
I have a follow-up question regarding to train Comet myself without using any source sentences. 

So the input to the system will be translated sentences, reference sentences and the rating of the translations.
Is there a way to do this in the current code base? Thank you in advance.

No, the current models accept either `source` and `translation` (Quality estimation setting) or `source`, `translation` and `reference` (MT metrics setting). To support your use case you will have to change the codebase.

Do you want to create a metric that receives `translation`, `reference`, and a `score` right? I think the easiest way is to create something similar to the [referenceless.py](https://github.com/Unbabel/COMET/blob/master/comet/models/regression/referenceless.py) metric but replacing the `source` with the `reference` as text inputs. Regarding the `score` I would incorporate that input in the feed-forward on top, after the encoding of the text inputs.



tell me if you need any help.

Thank you very much! I will go ahead and modify referenceless.py.
Btw, do you think it is possible for me to change "monitor: val_pearson" in earlystopping.yaml to some customized metrics like kendall correlation? 

the WMT kendall-tau like metric based on daRR or just plain Kendall tau?

Modifying it for monitoring "plain" Kendall tau is easy although you will have to implement it. The WMT Kendall-tau based on daRR is going to give you more work ...

Our implementation uses the [regression metrics from torchmetrics](https://torchmetrics.readthedocs.io/en/latest/references/modules.html#regression-metrics). They dont have kendall-tau and thats why we only look at Pearson and Spearman... but you can extend their library and [build your custom metric](https://torchmetrics.readthedocs.io/en/latest/pages/implement.html) like kendall

I did exactly that for the WMT Kendall when building the translation ranking model: [ wmt_kendall.py](https://github.com/Unbabel/COMET/blob/master/comet/models/ranking/wmt_kendall.py)

Cool! Thank you for your quick response. I am trying to build WMT kendall-tau. In this case, can I directly use your script wmt_kendall.py?

you can yes, but you will have to pass the score for the better and worse MT. This means that for regression metrics, during validation, you will have to load a different corpus with 4 texts (source, better MT, worse MT, reference).

Oh I see. Thank you so much! I assumed for ranking model that I can just call your wmt kendall metric with the same corpus.

if you are using the ranking model yes. If you are using regression models you have to adapt the validation part

Great! Thank you for all your answers again.
Repository: Unbabel/COMET
Issue: Continue to fine-tune from wmt20-comet-da checkpoint
Root cause: Unimplemented Feature Gap

Hi Ricardo, is it possible for me to train from your checkpoint wmt20-comet-da? Currently, I saw that "wmt20-comet-da" contains only the model.

It is possible but as you said, I don't have the optimizer and trainer state stored. But you can always load the model weights and initialize the rest from scratch.

I don't think this is a major problem since we did not use any fancy scheduler etc..
Repository: Unbabel/COMET
Issue: GPU support information into README
Root cause: Documentation Deficiency

COMET takes 30-40 minutes to evaluate 400 sentences testset on CPU. Therefore GPU is necessary, but it took me some time before I found out there is a flag "cuda". Can you add an example with cuda parametr into README?

Thanks for the feedback. I'll add that. Actually, we will work on documentation this week. 

We will add an explanation to all CLI flags and a few results from WMT20

If you have anything else, just say and we will try to add it!
Repository: Unbabel/COMET
Issue: Is there a theoretical range of values for the COMET regressor? 
GitHub labels: question

Is there a theoretical range of values for the COMET regressor? 

Since the final estimator layer is an FFN https://github.com/Unbabel/COMET/blob/master/comet/models/regression/regression_metric.py#L95

Which goes to https://github.com/Unbabel/COMET/blob/85b0c8ff24fb63273283807cd892d3ef44b89a3a/comet/modules/feedforward.py#L58 

Is the theoretical range `(-inf, inf)`, https://pytorch.org/docs/1.9.1/generated/torch.nn.Linear.html?

Are there some table of practical ranges that the comet owners/contributors/users have found for varying languages and length?




Hi @alvations 

Yes, the scores are unbounded for most COMET models (except `wmt20-comet-da-qe` that has a sigmoid in the end). Nonetheless, since they are trained to regress on a [z-scores](https://www.statology.org/z-score-normalization/) they usually range somewhere between -2.5 to 1.2. You can observe that in [this plot](https://unbabel.github.io/COMET/html/models.html#faq):

![wmt20_scores_distribution](https://user-images.githubusercontent.com/17256847/149377413-46798cec-6a03-4588-b8ea-8bbcf0a8ae37.png)

That this plot from WMT20 en-de data which is a high resource and high quality language pair. You can expect this distribution to have a smaller mean for low resource languages.

The max value and min value should be more or less the same because the model does not "risk" much outside those values.


Thanks for the explanation! It'll make sense if we somehow distill the model and put a softmax at the final layer so that it spits out `[0, 1.0]`.
Repository: Unbabel/COMET
Issue: [QUESTION] What is the data format for QE ranker model?
Root cause: Documentation Deficiency

I want to train my own metric which is a ranker model and referenceless. According to https://github.com/Unbabel/COMET/blob/0.1.0/docs/source/training.md the data format is a csv file with src, mt, ref and score. Because the ranker model needs to have a pos hypothesis and a neg hypothesis to train and as I understand it doesn't need score to train so is the data format the same for training ranker model? 


Hi @taquynhnga2001 yes you need another format for the ranker. You need the daRR data.

You can find that data in this issue: #36

WMT 17-> 19:
This includes relative ranks and DA scores.

```bash
wget https://unbabel-experimental-data-sets.s3-eu-west-1.amazonaws.com/comet/da/wmt-metrics.zip
```

and 2020 DA Relative-Ranks:
```bash
wget https://unbabel-experimental-data-sets.s3.eu-west-1.amazonaws.com/wmt/2020-daRR.csv.tar.gz
```
Repository: Unbabel/COMET
Issue: Benchmark tests?
Root cause: Environment Incompatibility

I've installed COMET on two different machines, (Python 3.8.10, Ubuntu, running on GPU, and 3.9.10, MacOS, running on CPU) and am getting vastly different results on each machine for the same model and the same texts. Clearly, they can't both be correct.

I'm wondering whether there are any benchmark results that I can compare to, for example the simple examples in the installation instructions, so I can try to figure out what's going on and validate the installation. Also, I'm wondering whether the fact that I'm running COMET on Chinese characters might have something to do with the different results. Are there any benchmark results for EN<>CN?

Thanks.

can you share a bit more details?

For example: 
- exact command you are calling
- samples (I don't need the entire test set but just a few samples that are causing different outputs)
- the output that is being produced? 

It is possible to have small differences on the same samples (things like 0.3456 vs 0.3462) but more than that, something is wrong...

Thanks for the quick response. The command I'm calling follows the examples in the README. E.g.
comet-score -s SRCA.txt -t HTA1_MT_EN-US_-_ZH-CN.txt --model wmt20-comet-qe-da
SRCA.txt is a plain text file in English. HTA1_MT_EN-US_-_ZH-CN.txt is a translation in Chinese.
For the first segment (sentence) I get the scores: 0.2642 and 0.4263, respectively so fairly different and not a rounding issue.
This segment/sentence in SRCA is "From the factory floor to air-to-air combat, artificial intelligence will soon replace humans not only in jobs that involve basic, repetitive tasks but advanced analytical and decision-making skills."
The target text is "不久之后，人工智能将在许多工作中取代人类，从工厂车间到空对空作战，不仅基本和重复性的工作将由人工智能进行，需要高级分析和决策技能的工作亦是如此。" (I don't speak/read Chinese, so I don't have a clue what it says.)

The output in my computer was:
mt.txt	Segment 0	score: 0.4283
mt.txt	score: 0.4283

On a completely different machine, using a GPU, I get the same output:
mt.txt  Segment 0       score: 0.4283
mt.txt  score: 0.4283



I don't get where the 0.2642 score comes from... I can't replicate it with your example

Thanks, yes, that's what I'm getting on my Mac, too. I get the other score (0.2642) on my Linux machine, where I had some error messages while trying to install comet because of package incompatibilities. I had to wipe those packages and reinstall. This leads me to think that there's some residual incompatibility that wasn't resolved which leads to random result numbers. Which is why I was asking for a benchmark to figure out which results to trust. Since I'm getting 0.4283 (or 0.4263 with a newline), that's it. Thanks.