zhiminy/EvalEng huggingface dataset:
Repository: UKGovernmentBEIS/inspect_ai
Issue: GAIA web browser tool returns <empty> when affected by packet loss
Root cause: Validation Gap

When infra issues are causing packets from the browser container to the Internet to by dropped, we should probably return an error rather than an ambiguous <empty> message.

(Note this issue is a poorly refined stub and may need to be re-directed to another component, but filing it here until we know more).

@alex-treebeard, can you say a little bit more about the setup that exhibited this behavior? If you have any logs that may have captured the issue would be valuable.

I believe that the recent refactor of the web browser likely resolved these issues. Please open another issue if you see this sort of behaviour again.
Repository: Unbabel/COMET
Issue: Segmentation fault error
Root cause: External Dependency Breakage

Hello! 

## 🐛 Bug

Whenever I am try to run your demo, when using : 

`from comet.models import download_model`

I get:

  **Segmentation fault (core dumped)**

### To Reproduce 

I'm using Amazon EC2 instances with ubuntu 18.04 and I've also tried with ubuntu 20.04 and got the same errors.
I log into the instance and do:

`sudo su `
`apt-get update `
`apt-get install python3-pip`
`pip3 install unbabel-comet`

If I'm on 18.04 I have to pip upgrade because otherwise sentence piece breaks.

I've tried building from source installing from pip nothing seems to be working it installs and I can even do:

`import comet`

But whenever : 

`from comet.models import download_model`

segmentation fault, also if I use the cli command:

`comet download -d apequest --saving_path data/`

again the same error! 

### Environment
OS: Linux
Packaging pip
Version : pip 20.0.2

I've tried multiple instances, virtual environments but nothing seems to be effective! With ubuntu 20.04 I used python 3.8, I've seen that the recommendation is python 3.6, On ubuntu 18.04 I've used the default python3 which is 3.6.9 and still had the same issues! 



Here is the output of my pip freeze

```

absl-py==0.11.0
cachetools==4.1.1
certifi==2020.11.8
cffi==1.14.3
chardet==3.0.4
click==7.1.2
Cython==0.29.15
fairseq==0.9.0
fastBPE==0.1.0
filelock==3.0.12
fsspec==0.8.4
future==0.18.2
google-auth==1.23.0
google-auth-oauthlib==0.4.2
grpcio==1.33.2
idna==2.10
joblib==0.17.0
Markdown==3.3.3
numpy==1.19.4
oauthlib==3.1.0
pandas==1.0.5
portalocker==2.0.0
protobuf==3.14.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
python-dateutil==2.8.1
pytorch-lightning==1.0.7
pytorch-nlp==0.5.0
pytz==2020.4
PyYAML==5.3.1
regex==2020.11.13
requests==2.25.0
requests-oauthlib==1.3.0
rsa==4.6
sacrebleu==1.4.14
sacremoses==0.0.43
scikit-learn==0.23.1
scipy==1.5.4
sentencepiece==0.1.94
six==1.15.0
sphinx-markdown-tables==0.0.15
tensorboard==2.2.0
tensorboard-plugin-wit==1.7.0
threadpoolctl==2.1.0
tokenizers==0.7.0
torch==1.4.0
tqdm==4.52.0
transformers==2.10.0
-e git+https://github.com/Unbabel/COMET@c9ac4c9cbdb8484aa5ee286c9cbe13002c16a193#egg=unbabel_comet
urllib3==1.26.2
Werkzeug==1.0.1
wget==3.2
```

If you could help me figure this thing out it would be wonderful! 


Thank you for your time and willingness to share your tool! I'm eager to try it :-) 




Can you please try: `pip install sentencepiece==0.1.91` ?


I managed to replicate the error you were getting and after installing sentencepiece 0.1.91 it worked


https://github.com/huggingface/transformers/issues/4857 

That seem to have solved my issue :-) 
Obrigado! 

> 

hi, have downgraded the sentencepiece version to 0.19.1, but still report this error.
Repository: Unbabel/COMET
Issue: Segment order incorrect when running on >1 GPU
Root cause: Resource Mishandling

## 🐛 Bug

Segment order is incorrect when using more than one GPU. 

### To Reproduce 

`comet-score -s src.de -t junk.en -r ref.en --model wmt21-comet-mqm --gpus 0`
junk.en Segment 0 score: 0.0036
junk.en Segment 1 score: -0.0050
junk.en Segment 2 score: 0.0029
junk.en Segment 3 score: 0.0426
junk.en Segment 4 score: 0.0481
junk.en Segment 5 score: 0.0511
junk.en Segment 6 score: 0.0501
junk.en Segment 7 score: 0.0325
junk.en Segment 8 score: 0.0434
junk.en Segment 9 score: 0.0652
junk.en Segment 10 score: 0.0538
junk.en Segment 11 score: 0.0465
junk.en Segment 12 score: 0.0490
junk.en Segment 13 score: 0.0564
junk.en Segment 14 score: 0.0448
junk.en Segment 15 score: 0.0564
junk.en Segment 16 score: 0.0549
junk.en Segment 17 score: 0.0461
junk.en Segment 18 score: 0.0577
junk.en Segment 19 score: 0.0573
junk.en score: 0.0429

`comet-score -s src.de -t junk.en -r ref.en --model wmt21-comet-mqm --gpus 4`
junk.en Segment 0 score: 0.0036
junk.en Segment 1 score: 0.0481
junk.en Segment 2 score: 0.0434
junk.en Segment 3 score: 0.0490
junk.en Segment 4 score: 0.0549
junk.en Segment 5 score: -0.0049
junk.en Segment 6 score: 0.0511
junk.en Segment 7 score: 0.0650
junk.en Segment 8 score: 0.0563
junk.en Segment 9 score: 0.0460
junk.en Segment 10 score: 0.0029
junk.en Segment 11 score: 0.0501
junk.en Segment 12 score: 0.0538
junk.en Segment 13 score: 0.0449
junk.en Segment 14 score: 0.0577
junk.en Segment 15 score: 0.0426
junk.en Segment 16 score: 0.0325
junk.en Segment 17 score: 0.0465
junk.en Segment 18 score: 0.0564
junk.en Segment 19 score: 0.0573
junk.en score: 0.0429

In this case, segment 0 -> segment 0, segment 1-> segment 5, segment 2 -> segment 10, etc

See also: https://github.com/amazon-science/doc-mt-metrics/issues/8


Thanks @thompsonb

I have to refactor the code to use a CustomWriter from [Pytorch Lightning](https://pytorch-lightning.readthedocs.io/en/stable/api/pytorch_lightning.callbacks.BasePredictionWriter.html#pytorch_lightning.callbacks.BasePredictionWriter).

This was something that was not supported a few pytorch lightning versions ago.

Ill try to work on it after EMNLP

I made some significant changes on inference with multi-gpu. The segment order issue is solved and its also faster to run inference with multi-gpu now. 

This will be merged on the next release.
Repository: Unbabel/COMET
Issue: [QUESTION] Batch-normalization in pooling layer
GitHub labels: question

## ❓ Question

Hi,
Is the batch-normalization component from the pooling layer from here:
https://github.com/Unbabel/COMET/blob/a96a688e450f7b61f9d884240ff0749223d79113/comet/modules/layerwise_attention.py#L139
mentioned in any of the COMET papers? I don't seem to see it in the main COMET citation. 

Can you elaborate a bit why this was useful? It seems unusual to see batch-normalization in NLP applications. 

Also, it seems this could be fixed to normalize over a single sentence during inference (use dimensions to only average over time dim) to achieve deterministic behavior consistent with batch-size 1. Would there be any downsides to that?

Thanks,
Marcin

Hi @emjotde! 

This is a good observation. In the paper we only say that we use the _Layerwise scalar mix_ to combine multiple layers. The implementation we used actually comes from this [model](https://github.com/Hyperparticle/udify). 

The idea is that the batch normalisation helps generalisation but in experiments for this year shared task we had more time to play with different hyper parameters and we ended up deactivating it. New models from this year do not use it anymore.

Also, your suggestion is good. I'll test it out

Thanks for the answer! Feel free to close.
Repository: Unbabel/COMET
Issue: t-test is two-tailed instead of one-tailed
Root cause: Algorithmic Error

## 🐛 Bug

The code to perform paired t-test is two-tailed instead of one-tailed. The alternative hypothesis that users typically care about is that the baseline mean score is _less_ than sys1's mean score, but that is not reflected in the test.

### To Reproduce 

See [here](https://github.com/Unbabel/COMET/blob/master/comet/cli/compare.py#L465) :-) 

### Expected behaviour

The test should _probably_ use `alternative="less"` or otherwise be configurable.

### Screenshots
If applicable, add screenshots to help explain your problem.

### Environment
OS: [e.g. iOS, Linux, Win] N/A
Packaging [e.g. pip, conda] N/A
Version [e.g. 0.5.2.1] N/A

### Additional context

<!-- Add any other context about the problem here. -->


I am adding a flag for t_test alternative and setting it by default to "less".


This will be available on the next release



Perhaps for the sake of not breaking the API, setting the default to `"two-sided"` might be a better option until the next major release? I'd hate to be the cause of people's papers comparing apples and oranges. I think with the ability to configure it, this should cover the issue well.

@erip Thats a good point thanks!

The next release will be 2.0 and we are also going to replace the default models with new ones. 

We will make it clear that scores (with default settings) won't be directly comparable to the previous version (1.1.3).

There will be backward compatibility but default options will probably change for all 3 commands: `comet-score`, `comet-mbr` and `comet-compare`

I agree with you that people typically want to know if baseline mean score is less than sys1's mean score. I think its a good call to change the t_test to `less`. What you think?

Atm this is only updated in the [`fix-multigpu` branch](https://github.com/Unbabel/COMET/tree/fix-multigpu) and not merged into master.

It seems very reasonable to me. I'm hoping there's not some nuance that I've overlooked here. I can look at sacrebleu to see what their alternative hypothesis is in their tests (for sake of consistency more than correctness).

Perfect! Thanks!

Unless I'm misreading their code, it [seems](https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/significance.py#L101) like they're testing using a [two-sided](https://github.com/mjpost/sacrebleu/blob/078c440168c6adc89ba75fe6d63f0d922d42bcfe/sacrebleu/significance.py#L264) alternative hypothesis due to the absolute value.

@erip I looked a bit more into this and indeed two-sided t_test is more usual and results made more sense in my tests. Nonetheless I am keeping the option to change that in the command line. I am going to merge v2.0 into master.

The release was delayed but at least master will contain the new changes
Repository: Unbabel/COMET
Issue: Evaluate excessively long sequence pair[QUESTION]
Root cause: Documentation Deficiency

## ❓ Questions and Help

### Before asking: 
1. Search for similar [issues](https://github.com/Unbabel/COMET/issues).   
3. Search the [docs](https://unbabel.github.io/COMET/html/index.html).    

<!-- If you still can't find what you need: -->
I briefly read though the source code and didn't get the answer.

#### What is your question?
I attempted to apply COMET to some excessively long document pairs (more than 3000 words for both source and target sequence). I didn't get any error report when doing so. How does COMET process the excessively long sequence?

#### Code

<!-- Please paste a code snippet if your question requires it! -->   

#### What have you tried?

#### What's your environment?

 - OS: Linux
 - Packaging: pip
 - Version: 1.1.3


Hi @minghao-wu, COMET truncates the sentence for very long sentences. Hugging face tokenizer already does that for you if you activate truncate=True. You can check the [code here](https://github.com/Unbabel/COMET/blob/master/comet/encoders/base.py#L235)

Also, since we are using a _triplet encoder_ architecture this means that both source, MT and reference have an equal capacity of 512 tokens (1536 tokens in total).

In my experience with typical MT testsets this is enough. What is your use case exactly?

Hi @ricardorei ,

Thank you for your answer. 

Yes, 512 is typically long enough for a single sentence. However, I am working on DocNMT and trying to use COMET as a document-level measure by concatenating all the source sentences, references and hypotheses within the same document. For example, one of the common benchmark dataset in DocNMT is [IWSLT2017 En-De](https://github.com/sameenmaruf/selective-attn/tree/master/data/IWSLT2017). The document length in the test set is typically longer than 1000 words. Do you have any suggestions for this situation?

You can take a look into [this work](https://github.com/amazon-science/doc-mt-metrics)

They extended COMET and other metrics to take into account context.

You still need to score each individual segment within the document but, for each segment, the score will be computed using the previous sentences and more context sensitive.

I want to integrate that into the next release of COMET but for now I think you should take a look at their work.

You can look at their paper here: [Embarrassingly Easy Document-Level MT Metrics: How to Convert Any Pretrained Metric Into a Document-Level Metric](https://statmt.org/wmt22/pdf/2022.wmt-1.6.pdf)

Thank you for your suggestion. I will have a look.

@ricardorei are you planning to add document level context in COMET sometime soon already?

same question
Repository: Unbabel/COMET
Issue: [QUESTION] Is WMT data prior to WMT17 unseen?
GitHub labels: question

Hi again,
just to confirm, is it safe to assume that WMT test sets (or metrics shared task data) prior to WMT17, i.e. WMT08 to WMT16 are unseen to the COMET models?

Asking for evaluation purposes.
Thanks,
Marcin


Yes, WMT08 to WMT16 was never seen by any COMET models. 

WMT 21 and WMT 22 are also great datasets that have never been used to train any COMET model. 



Thanks!
Repository: Unbabel/COMET
Issue: [QUESTION] Load the model from path
Root cause: Documentation Deficiency

Hello!

I downloaded `wmt20-comet-da` to Google Drive. I am trying to use it in Google Colab, which is perfectly connected to Google Drive. However, every time I run the code, the model is downloaded again. I am using the lines below. Is there a way to force COMET to use the model from the path rather than re-downloading it? Maybe there are other downloads? Thanks!

```
model_path = "/content/drive/MyDrive/models/wmt20-comet-da/checkpoints/model.ckpt"
model = load_from_checkpoint(model_path)
```


hmmm thats strange because the load_from_checkpoint function does not perform any download. Can you provide more information? for example a snipped of the script?

Thanks, Ricardo!

Here is the whole script that I use:
```
# Calculate COMET
df = pd.DataFrame({"src":source_sentences, "mt":translations, "ref":references})
data = df.to_dict('records')
model_path = "/content/drive/MyDrive/models/wmt20-comet-da/checkpoints/model.ckpt"
model = load_from_checkpoint(model_path)
seg_scores, sys_score = model.predict(data, batch_size=128, gpus=1)
print("COMET:", round(sys_score*100, 2))
```

Here is a screenshot of what I see.
<img width="1149" src="https://user-images.githubusercontent.com/48152713/210431295-3ed56547-7a3c-4b4c-bb47-39149ff12539.png">

I think it does not download `wmt20-comet-da`, but it downloads that 2.24GB every time, and I am not sure what it is. Thanks!



From my experience what the `load_from_checkpoint` downloads extra is the `models--xlm-roberta-large`. You can download it yourself and save alongside the `wmt20-comet-da` model and set the environment variable `TRANSFORMERS_CACHE` to point to where the `models--xlm-roberta-large` is. This will avoid the download. 
I'm using comet version `1.1.3`.

Yep exactly, The downloads you are observing are the hugging face XLM-R weights. You can also cache those weights to avoid download them multiple times.


Thanks, @avaz and @ricardorei! This worked.

First, download XLM-RoBERTa.
```python
from transformers import AutoModelForMaskedLM 
model = AutoModelForMaskedLM.from_pretrained("xlm-roberta-large", cache_dir="/content/drive/MyDrive/models/")
```

Then, set the cache folder for Transformers:
```python
import os
os.environ['TRANSFORMERS_CACHE'] = "/content/drive/MyDrive/models/"
```
OR
```
export TRANSFORMERS_CACHE="/content/drive/MyDrive/models/"
```
Finally, install COMET
```
pip3 install unbabel-comet -q
```
Repository: Unbabel/COMET
Issue: Tag v1.1.3
GitHub labels: documentation

enhancement

I just noticed that while 1.1.3 is the latest version on [PyPi](https://pypi.org/project/unbabel-comet/#history), it is not tagged on GitHub [releases](https://github.com/Unbabel/COMET/releases)/[tags](https://github.com/Unbabel/COMET/tags). Just for transparency, it might be useful to tag that release here as well.


You are right. The changes for v1.1.3 are just minor bumps but we should add it to tags.

Thanks for the suggestion

Done
Repository: Unbabel/COMET
Issue: [QUESTION]Interpretation and comparison of COMET scores across several languages
GitHub labels: question

## ❓ Questions and Help

### Before asking: 
1. Search for similar [issues](https://github.com/Unbabel/COMET/issues).   
3. Search the [docs](https://unbabel.github.io/COMET/html/index.html).    

<!-- If you still can't find what you need: -->

#### What is your question?
1. Is there a way to interpret the COMET score other than using it as a ranking system?
2. For example, we have a French dataset of 500 sentences from TED Talks and a Spanish dataset consisting of 500 sentences from Parliament sessions.  We are comparing the COMET scores for French and Spanish, the scores are denoted as comet_fr and comet_es. If comet_fr > comet_es, does that mean that the machine translation quality of French is better than the machine translation quality of Spanish? Is the COMET score comparable across languages? Or is this comparison invalid because the source data is not the same? 


#### Code

<!-- Please paste a code snippet if your question requires it! -->   

#### What have you tried?

#### What's your environment?

 - OS: [e.g. iOS, Linux, Win]
 - Packaging [e.g. pip, conda]
 - Version [e.g. 0.5.2.1]


Hi @clairehua1,

You should avoid comparing scores between languages and even between domains. This is not just for COMET but for any MT Metric.

For example BLEU, even tho is lexical, highly depends on the underlying tokenizer thus the results vary a lot between different languages.  

PS: even human annotation has a lot of variability between languages and domains. If we want reliable and comparable results we need to make sure the test conditions are the _same_ (same data, same annotators)

Cheers,
Ricardo


Thanks for the answer Ricardo! Is there a way to interpret the COMET score other than using it as a ranking system?

@clairehua1 for a specific setting (language pair and domain) you could plot the distribution of scores and analyse it by looking at quantiles. The scores usually follow a normal distribution.

To give a bit more context most models are trained to predict a z-normalized _direct assessment_ (a z-score). Z-scores have a mean at 0 and follow a normal distribution which means that ideally a score of 0 should represent an average translation.

In practise the distribution of scores (for the default models `wmt20-comet-da`) is slightly skewed towards positive scores which means that an average translation is usually assigned a score of 0.5.  I have an explanation [here](https://unbabel.github.io/COMET/html/faqs.html#is-there-a-theoretical-range-of-values-for-the-comet-regressor)


In the plots above you can see how different is the scores between English-German and English-Hausa. But you can see that the "peak" for German is a bit higher than Hausa. 

Nonetheless this is expected due to the fact that German translations tend to have better quality than Hausa ones.

![Screenshot 2023-02-19 at 18 39 20](https://user-images.githubusercontent.com/17256847/219968255-d4efcff8-9b04-4fff-8784-6bf70736257f.png)
Repository: Unbabel/COMET
Issue: Avoid downloading XLM-R checkpoint from huggingface
Root cause: Unimplemented Feature Gap

## 🚀 Feature

Allow users to download only the COMET checkpoint without the XLM-R checkpoint from huggingface hub.

### Motivation

Currently, all encoders in COMET repo are loaded by `XXModel.from_pretrained`, which also downloads model weights from huggingface. For end users without the demand to train a custom model, the weights are not needed. Code reference:

https://github.com/Unbabel/COMET/blob/9a84de1e7efc9966822ad786f86c3b5514cf824d/comet/encoders/xlmr.py#L35-L41

The code can be refactored with `XXConfig.from_pretrained` like:

```py
    def __init__(self, pretrained_model: str, load_pretrained_weights: bool=False) -> None:
        super(Encoder, self).__init__()
        self.tokenizer = XLMRobertaTokenizer.from_pretrained(pretrained_model)
        if load_pretrained_weights:
            self.model = XLMRobertaModel.from_pretrained(
                pretrained_model, add_pooling_layer=False
            )
        else:
            self.model = XLMRobertaModel(
                XLMRobertaConfig.from_pretrained(pretrained_model),
                add_pooling_layer=False
            )
        self.model.encoder.output_hidden_states = True
```

This avoids downloading unnecessary weights and only keeps the important config file.

### Alternatives

A better approach for ease of use is to export the whole model to ONNX format, but requires much more effort.

### Additional context

No


@K024 I refactored the code to avoid doing it. Thanks for the issue.
Repository: Unbabel/COMET
Issue: Use BetterTransformer for fast inference
Root cause: Unimplemented Feature Gap

## 🚀 Feature

PyTorch 1.12 came with a speed bump in Transformer encoder inference speeds through [BetterTransformer](https://pytorch.org/blog/a-better-transformer-for-fast-transformer-encoder-inference/). Because of Unbabel's efforts to improve usability and efficiency (~COMETINHO), such improvements might be a useful addition to the library.

### Implementation

Implementation is relatively straightforward ([blog](https://medium.com/pytorch/bettertransformer-out-of-the-box-performance-for-huggingface-transformers-3fbe27d50ab2), [docs](https://huggingface.co/docs/optimum/main/en/bettertransformer/tutorials/convert)) to implement. This requires `accelerate` and `optimum`, which can be optional dependencies (`pip install unbabel-comet[bettertransformer]`). If both of these are installed (with sufficient versions) alongside PyTorch 1.13 (min. requirement to work with Optimum API), then we can set a global constant `BETTER_TRANSFORMER_AVAILABLE = True`. 

 As an example, this line:

https://github.com/Unbabel/COMET/blob/master/comet/encoders/bert.py#L38

should be followed by:

```python
if self.use_bettertransformer and BETTER_TRANSFORMER_AVAILABLE:
    self.model = BetterTransformer.transform(self.model, keep_original_model=False)
```

And that's about it.

Of course, which class to implement `use_bettertransformer` is something that has to be decided but apart from that I think this is a feasible addition that can lead to significant speed improvements. 

I can work on this if needed but I need guidance on which class to implement `self.use_bettertransformer` in and whether the logic (if-statement) above should be implemented on a per-model basis or if we can generalize it somehow. 

yep this seems like a good idea. 

I'll try to find some time to test it! Thanks!