zhiminy/EvalEng huggingface dataset:
Repository: Unbabel/COMET
Issue: [QUESTION] AttributeError: An error occurred during model prediction: 'dict' object has no attribute 'scores'
Root cause: Interface Contract Mismatch

## ❓ Questions and Help

<!-- If you still can't find what you need: -->

#### What is your question?
An error occurred during model prediction: 'dict' object has no attribute 'scores'
<img width="1067" alt="Screenshot 2023-11-19 at 7 47 08 PM" src="https://github.com/Unbabel/COMET/assets/42874576/2f4c4540-7c0f-4c6e-87bd-252e41233476">
<img width="648" alt="Screenshot 2023-11-19 at 7 47 14 PM" src="https://github.com/Unbabel/COMET/assets/42874576/6d70e827-fd89-4d29-8cb6-1ae3ad6307f1">



#### Code
```
echo -e "10 到 15 分钟可以送到吗\nPode ser entregue dentro de 10 a 15 minutos?" >> src.txt
echo -e "Can I receive my food in 10 to 15 minutes?\nCan it be delivered in 10 to 15 minutes?" >> hyp1.txt
echo -e "Can it be delivered within 10 to 15 minutes?\nCan you send it for 10 to 15 minutes?" >> hyp2.txt
echo -e "Can it be delivered between 10 to 15 minutes?\nCan it be delivered between 10 to 15 minutes?" >> ref.txt
comet-score -s src.txt -t hyp1.txt -r ref.txt
```

#### What have you tried?
I tried unbabel-comet== 2.0.1, which previously worked well with my env.
Also, tried unbabel-comet==2.1.0 and 2.2.0, both encountered same Attribute Error

#### What's your environment?

 - OS: Linux (AWS SageMaker)
 - Packaging 
pip = 23.3.1
 - Version 
unbabel-comet = 2.0.1/2.1.0/2.2.0
pytorch = 1.13
python = 3.9.0


Same bug here

You can simply replace the code `pred.scores` in the original python file in site-package to `pred["scores"]` for quick fix, that works for me.

For me, installation from source worked (as opposed to the PyPi package).

> You can simply replace the code `pred.scores` in the original python file in site-package to `pred["scores"]` for quick fix, that works for me.

Thanks! Editing site package works for me.
Hopefully we can get an fix with PyPi and huggingface soon

Hi @Evstrife This was fixed by this merged pull request #185 

For now, you can reinstall COMET using this command:
```
pip3 install git+https://github.com/Unbabel/COMET.git
```
Repository: Unbabel/COMET
Issue: AttributeError: 'dict' object has no attribute 'scores' when using the model 'Unbabel/wmt22-cometkiwi-da'
Root cause: Interface Contract Mismatch

## 🐛 Bug

When I try to use the mode **'Unbabel/wmt22-cometkiwi-da'** it pops the following error

Traceback (most recent call last):
  File "/COMETKIWI/main.py", line 19, in <module>
    model_output = model.predict(data, batch_size=8, gpus=1)
  File "Anaconda3/envs/comet/lib/python3.9/site-packages/comet/models/base.py", line 652, in predict
    scores = torch.cat([pred.scores for pred in predictions], dim=0).tolist()
  File "Anaconda3/envs/comet/lib/python3.9/site-packages/comet/models/base.py", line 652, in <listcomp>
    scores = torch.cat([pred.scores for pred in predictions], dim=0).tolist() 

AttributeError: 'dict' object has no attribute 'scores' 

I can see the error is corrected in the repo, but when using the model directly(from huggingface) with the following command, it still shows the error. I think the model needs to be updated too.

**comet-score -s src.txt -t hyp1.txt --model Unbabel/wmt22-cometkiwi-da**


### Environment
OS: [ Linux]
Packaging [conda]




This issue was sorted when I modified the base.py inside the package installed in conda env

Is it corrected from huggingface?

The issue is within the COMET code, not on huggingface. The [PR to fix this](https://github.com/Unbabel/COMET/commit/45cb572516398b6994f112ed8ee7058dc5fb84ef) has been merged on main - but the package on PyPI has not been yet updated to include this fix. The only option rn is to download the repo and use the newest code.

Requesting here to please release a new package!

Can we please reopen this?
Until the PyPI package is not updated, this issue is not really "solved".
Repository: Unbabel/COMET
Issue: When loading huggingface Unbabel/wmt22-cometkiwi-da checkpoint, UnpicklingError
Root cause: External Dependency Breakage

UnpicklingError Traceback (most recent call last)

  1 import torch
----> 3 a = torch.load("wmt22-cometkiwi-da/checkpoints/model.ckpt", map_location=torch.device("cpu"))

File ~/anaconda3/envs/AMCT/lib/python3.9/site-packages/torch/serialization.py:1028, in load(f, map_location, pickle_module, weights_only, mmap, **pickle_load_args)
1026 except RuntimeError as e:
1027 raise pickle.UnpicklingError(UNSAFE_MESSAGE + str(e)) from None
-> 1028 return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)

File ~/anaconda3/envs/AMCT/lib/python3.9/site-packages/torch/serialization.py:1246, in _legacy_load(f, map_location, pickle_module, **pickle_load_args)
1240 if not hasattr(f, 'readinto') and (3, 8, 0) <= sys.version_info < (3, 8, 2):
1241 raise RuntimeError(
1242 "torch.load does not work with file-like objects that do not implement readinto on Python 3.8.0 and 3.8.1. "
1243 f"Received object of type "{type(f)}". Please update to Python 3.8.2 or newer to restore this "
1244 "functionality.")
-> 1246 magic_number = pickle_module.load(f, **pickle_load_args)
1247 if magic_number != MAGIC_NUMBER:
1248 raise RuntimeError("Invalid magic number; corrupt file?")

UnpicklingError: invalid load key, 'v'.
Repository: Unbabel/COMET
Issue: [QUESTION] Unable to use ReLU as activation function
Root cause: Validation Gap

I am currently training my custom metric using the UnifiedMetric and want to use [ReLU](https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html#relu)  instead of the default Tanh.
I thought simply replacing "Tanh" with "ReLU" in yaml file might work.
However, I am encountering the following error message: `Exception: ReLU is not a valid activation function!`
I want to know  if there is a way to use ReLU as the activation function.

Additionally, It seems that an error occurs at `activation.title()`.
```python
def build_activation(self, activation: str) -> nn.Module:
    if hasattr(nn, activation.title()):
        return getattr(nn, activation.title())()
    else:
        raise Exception(f"{activation} is not a valid activation function!")

```

Thankyou‼️



Hey @nkw011! The build_activation function seems to be lacking some flexibility. 

I think the best thing to do is to just modify it to be the following:

```python
def build_activation(self, activation: str) -> nn.Module:
    if hasattr(nn, activation):
        return getattr(nn, activation)()
    else:
        raise Exception(f"{activation} is not a valid activation function!")
```

and then everything should work.

the reason its not working atm is that when you pass "ReLU" it will be converted to the string "Relu" by the title function and thus it will not match any of the activation functions in the [torch.nn package](https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity)

@ricardorei Thank you for replying my question! I'm sorry but have one more question. The build activation function is located in comet/modules/feedforward.py. Can I get a way to modify the bulid activation function in pip package 'unbabel-comet' installed in local computer as you explain?

You should be able to do it. When you installed the package the code was stored somewhere in your computer. You just have to locate the source code and modify the function.

A simpler alternative is to just install the code from source:

Just clone the repo to your favourite location and install it:

```
git clone https://github.com/Unbabel/COMET
cd COMET
pip install poetry
poetry install
```
Then you can modify whatever you like.
Repository: Unbabel/COMET
Issue: [QUESTION] Keep getting scores of '0' no matter what input used 
Root cause: External Dependency Breakage

#### What is your question?
I keep getting scores of 0 no matter what input I give it
#### Code
`from comet import download_model, load_from_checkpoint

model_path = download_model("Unbabel/wmt22-comet-da")
model = load_from_checkpoint(model_path)

data = [
    {
        "src": "10 到 15 分钟可以送到吗",
        "mt": "Can I receive my food in 10 to 15 minutes?",
        "ref": "Can it be delivered between 10 to 15 minutes?"
    },
    {
        "src": "Pode ser entregue dentro de 10 a 15 minutos?",
        "mt": "Can you send it for 10 to 15 minutes?",
        "ref": "Can it be delivered between 10 to 15 minutes?"
    }
]


if __name__ == '__main__':
    model_output = model.predict(data, batch_size=8, gpus=1)
    print(model_output)
    print(model_output["scores"]) # sentence-level scores
    print(model_output["system_score"]) # system-level score
`

-output
Prediction([('scores', [0.0, 0.0]), ('system_score', 0.0)])
[0.0, 0.0]
0.0

#### What's your environment?

 - OS: Windows 10
 - Packaging Pip 23.3.1
 - Version Comet 2.2.0


Hey @Brecony76. I am not able to replicate this error. I just tried it and I get the following scores:

Prediction([('scores', [0.8417137265205383, 0.7745385766029358]), ('system_score', 0.8081261515617371)])



Hi @Brecony76 I'm observing the same issue.
* OS: Windows 10
* unbabel-comet  2.2.1
* pip 23.3.1
* Python 3.10.13
* torch 2.1.2+cu121
* Geforce 250MX (Driver Version: 537.79  CUDA Version: 12.2) (Yeah... it's my work laptop)

The behavior is particularly odd, because sometimes it does actually return a score, with no change in code or data... I'm not sure how to reproduce the 0.0 scores, nor the proper scores. Sometimes it just works, sometimes it doesn't. I will retest this tomorrow, to see if I can make any sense of it. For now I completed my task of evaluating some translations with Comet (thanks to the devs and researchers for making this so intuitive!)

I can confirm that this issue exists on Windows. It might be related to this CUDA warning: 

> [W CudaIPCTypes.cpp:16] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

But I am not sure and do not have time to dig into this deeper. It is a shame though, as this makes COMET unfortunately unreliable on Windows. 

I've done some digging but haven't found a solution, although I have pinpointed the place in the PL Trainer where something goes wrong. The model weights are turned to zero but I do not know why.

To put this into higher priority, feel free to comment on the issue that I raised over at PyTorch Lightning to indicate that you are also experiencing this problem. https://github.com/Lightning-AI/pytorch-lightning/issues/19537

I left a reply in https://github.com/Lightning-AI/pytorch-lightning/issues/19537#issuecomment-1974787881 with a suggestion. I hope it provides some useful insights.
Repository: Unbabel/COMET
Issue: HF Hub dependency conflicts with latest `tranformers` version.
Root cause: External Dependency Breakage

## 🐛 Bug

COMET has an explicit dependency for both `transformers = "^4.17"` and `huggingface-hub = "^0.16.0"` which will only allow versions up to 0.17.0, but the newest version of transformers (v4.36.1) has its _own_ `huggingface-hub` [dependency](https://github.com/huggingface/transformers/blob/ebfdb9ca62205279d5019ef1403877461b3b2da4/setup.py#L120) which is `>=0.19.3,<1.0`. This creates dependency conflicts for anyone trying to build a package on `COMET` and the latest `transformers` package. Can we do one of the following: 
1. Remove the `huggingface-hub` dependency from `COMET` and just keep the `transformers` dependency given that the former is implicit in the latter;
2. Use the same style of `huggingface-hub` dependency that `transformers` uses (namely `"huggingface-hub>=0.19.3,<1.0"`) to mitigate dependency conflicts; 
3. At very least, update the `huggingface-hub` dependency to be `^0.19.3`.

Hi @scottfleming! I like this suggestions but I rather go for 2. Because if for some reason HF decides to decouple the two packages, COMET will stop working (unable to download models).

I just released a new version that should solve this dependency issue. Ping me if the error still persists. 

PS: I went for option 2

Thanks so much, @ricardorei! I'll test this out and let you know if I run into any problems.
Repository: Unbabel/COMET
Issue: if tgt is same with src, the score is still high
Root cause: Algorithmic Error

#### Question
My mt system `mt_1` translates some sentences producing bad translations that are same with the source text, but it has a higher score. My another translation system `mt_2` translates the same source text and only partially translates it, yet it has a lower score. 

Is such a comet score credible? Do I need to keep the comet score intact or do I need to modify it manually (such as change the mt_1 comet to 0)? Thanks for any helpful answers.

Here are two examples:
```txt
src: 没有可能
mt_1: 没有可能
mt_2: impossible
ref: not possible
```
`mt_1` gets comet 92.40, `mt_2` gets comet 90.15.

```txt
src: 张迪鸣介绍说，这套传感器与智能手机相连，呼出气体后，健康报告就会显示在手机上。
mt_1: 张迪鸣介绍说，这套传感器与智能手机相连，呼出气体后，健康报告就会显示在手机上。
mt_2: 张 Dubai introduced the introduction, this set of sensors and smart phones are connected, after the call out the gas, health reports will be displayed on the phone.
ref: Zhang Diming introduced, this set of sensor is connected to the smart phone; after you breathe out, the health report will be shown on your phone.
```
`mt_1` gets comet 71.19, `mt_2` gets comet 69.13.

Then I tried to make the whole tgt be same with src and don't change ref on 1875 sents, that means a mt system don't translate any sentence, and gets a Avg comet 65.72. It seems too high.

#### Code
I used the comet model `wmt22-comet-da`, my srcipt is:
```bash
comet-score -s /home/data/tmp.zh -t /home/data/tmp.en -r /home/data/tmp.ref --gpus 1 --quiet --model /home/models/hf/wmt22-comet-da/checkpoints/model.ckpt > /home/data/tmp.en.comet
```

#### Environment
 - OS: CentOS Linux release 7.9.2009 (Core)
 - python 3.8.18
 - torch 2.0.1
 - transformers 4.30.0.dev0
 - tokenizers 0.13.3
 - datasets 2.14.6
 - accelerate 0.21.0
 - unbabel-comet 2.2.0


I think the COMET classifier operates on source/output/reference embeddings and there is no explicit penalty when a hypothesis is in an incorrect language. Maybe you could run language identification on the target side to force-set sentences in a wrong language to a low score.

Translating into the wrong language ('off-target translations') are a problem with LLM-based translation. I would always recommend running both a neural metric (like COMET) and a string-based metric (bleu or chrF) as well since the latter are more sensitive to off-target translations.
Repository: Unbabel/COMET
Issue: [QUESTION] Train UnifiedMetric/XCOMET with word level predictions. 
Root cause: Documentation Deficiency

#### What is your question?
I was trying to train a model both at sentence and word level, but was unable to find any examples of it. I believe XCOMET is able to do so, since it has a `word_level_training` parameter, but I could not find any examples on how to feed the word level data to the model. 

Would you have some examples available for this?

I still was unable to get word level predictions. Would it be possible to add an small example to reproduce?
Repository: Unbabel/COMET
Issue: Invalid link reference of reference-free model in readme
GitHub labels: documentation

enhancement

## 📚 Documentation

![image](https://github.com/Unbabel/COMET/assets/12471844/1f3e08a5-6fb1-4a14-9a9d-bc3c25217dda)

The **wmt22-cometkiwi-da** link should link to https://huggingface.co/Unbabel/wmt22-cometkiwi-da but in readme it is linked to invalid link of https://huggingface.co/Unbabel/wmt23-cometkiwi-da
Repository: Unbabel/COMET
Issue: Minimizing cpu RAM vs only use GPU RAM
Root cause: Architectural Constraint

## 🚀 Feature
Load model directly on GPU when available instead of 1) CPU 2) GPU

### Motivation
Trying to use comet-score with cometkiwi-xl on Colab

Currently, the load_checkpoint method forces to load on torch.device("cpu").
On Colab Free there is only 12GB of cpu RAM, hence XL does not fit.

Then I switched in __init__.py torch.device() to "cuda"
Now it loads the model on GPU fine

BUT just before starting to score, the cpu RAM suddenly jumps to > 12GB, not sure to understand why.

Any clue ?


usually, the way it should work is:
1) build model on meta device (empty weights) so it takes zero ram
2) load directly weights from checkpoint to GPU
I am trying to amend the code, but no luck so far.
Repository: Unbabel/COMET
Issue: [QUESTION] How to calculate corpus level COMET?
Root cause: Documentation Deficiency

I understand COMET returns a single score for each sentence it evaluates. I was wondering if there is any way to report a corpus level metric, and what would one be? Similar to how BLEU is reported.


Hey @AmitMY to get a corpus score you just have to average the segment level scores. 

We did that for the WMT20 shared task and it works pretty well: [Unbabel’s Participation in the WMT20 Metrics Shared Task](http://www.statmt.org/wmt20/pdf/2020.wmt-1.101.pdf)

Also, the CLI score command already returns you a corpus score.

Thanks!
I am using the python API, and only getting segment level scores, but averaging them is easy enough :) 

Not closing because I think its worth adding to the README?

Also, the mean sometimes comes out negative, and the more data I have the more the mean tends towards 0.

Yes, that can happen with models trained on DA's... but the fact that the means tends towards zero only depends on the distribution of the samples you are adding.

To better interpret the final score you can check our [FAQs](https://unbabel.github.io/COMET/html/models.html#faq)
Repository: Unbabel/COMET
Issue: [QUESTION] Does COMET support multiple references?
Root cause: Documentation Deficiency

Thanks for the tool! I'm wondering if COMET supports multiple references, or if we can just score each sentence with all the references and take the maximum value? Sorry if this has already been mentioned somewhere.

Hi! This is actually an open question. We did some experiments where we tried to experiment with multiple references and from our experiments, a simple average is what works best. We found that using a reliable reference is actually more important than having multiple references (if they are of lower quality)
Repository: Unbabel/COMET
Issue: what is the precision when load_from_checkpoint?
Root cause: Documentation Deficiency

## ❓ Questions and Help

### Before asking: 
1. Search for similar [issues](https://github.com/Unbabel/COMET/issues).   
3. Search the [docs](https://unbabel.github.io/COMET/html/index.html).    

<!-- If you still can't find what you need: -->

#### What is your question?
what is the precision of the model when `load_from_checkpoint`, can we specify the precision manually? Thank you.
#### Code

<!-- Please paste a code snippet if your question requires it! -->   

#### What have you tried?

#### What's your environment?

 - OS: [e.g. iOS, Linux, Win]
 - Packaging [e.g. pip, conda]
 - Version [e.g. 0.5.2.1]
Repository: Unbabel/COMET
Issue: Runtime error when loading wmt23-cometkiwi-da-xl
Root cause: External Dependency Breakage

## 🐛 Bug

I get the following error when trying to load the model from HuggingFace:

```bash
RuntimeError: Error(s) in loading state_dict for UnifiedMetric:
        Missing key(s) in state_dict: "encoder.model.embeddings.position_ids". 
```

### To Reproduce 

I am following the example on the HuggingFace page:

```python
from comet import download_model, load_from_checkpoint

model_path = download_model("Unbabel/wmt23-cometkiwi-da-xl")
model = load_from_checkpoint(model_path)
```

### Environment
OS: MacOS

Dependencies:
```
huggingface_hub = "0.12.1"
python = ">=3.9,<4.0.0"
unbabel-comet= "2.0.1"
torch = "2.0.1"
pytorch-lightning = "1.9.5"
```

Issue resolved by updating `unbabel-comet` package  to latest version.
Repository: Unbabel/COMET
Issue: Different scores from different COMET package versions 1.1.2 and 2.2.1
Root cause: External Dependency Breakage

## 🐛 Bug

When the same source, target, reference files are evaluated using the same wmt22-comet-da checkpoint, `unbabel-comet 2.2.1` under `python3.9` and `unbabel-comet 1.1.2` under `python3.7` gave me dramatically different numbers.

### To Reproduce 

In python3.7, `pip install --upgrade unbabel-comet` gives `1.1.2` as the latest version, while in python3.9 it gives `2.2.1`.

Scoring the same source, target, and reference files under the above two environments gave different scores. `unbabel-comet 1.1.2` results in a score of 0.86 while the `2.2.1` version gave 0.79. I used WMT22-COMET-DA downloaded from Hugging Face https://huggingface.co/Unbabel/wmt22-comet-da.

Attaching the files which gave 0.79 and 0.86 below, but I think any file combination can be used to reproduce this behaviour since it's associated with the COMET package version.
[target.en.txt](https://github.com/Unbabel/COMET/files/14419623/target.en.txt)
[source.mt.txt](https://github.com/Unbabel/COMET/files/14419624/source.mt.txt)
[hypothesis.en.txt](https://github.com/Unbabel/COMET/files/14419625/hypothesis.en.txt)

### Expected behaviour
I would expect different COMET package versions to give the same score if the same checkpoint and files are given.

### Environment
Managed python3.7 and python3.9 with conda.

### Additional context

If there is indeed some package mismatch between unbabel-comet 1.1.2 and 2.2.1, it might be difficult to go back and fix the problem. Users probably are unaware of this and will not update. Moreover, python3.7 only supports 1.1.2 as the latest even if users upgrade COMET in python3.7. Maybe this behaviour can be highlighted in README to encourage the user to use specific Python and unbabel-comet versions . On the other hand, this could imply that research papers should report COMET package version in addition to COMET version. Would it be possible to implement some kind of COMET signature just like that in `sacrebleu`?

<!-- Add any other context about the problem here. -->


This confirms what we learnt for BLEU, too: one should ALWAYS report version numbers (signatures), also for COMET!

Side note: in my [MATEO](https://mateo.ivdnt.org/), I added a custom signature for neural metrics like bertscore, bleurt and comet, too. For COMET it looks like this (inspired by sacrebleu):

```
comet: nrefs:1|bs:1000|seed:12345|c:Unbabel/wmt22-comet-da|version:2.0.1|mateo:1.1.3
```

where `c` stands for the checkpoint used and `version` is self-explanatory. Wasn't sure how far one had to go with this because difference in torch, cuda and transformers versions may or may not also lead to difference in results. Hell, even then the CUDA optimisation might lead to different results on different hardware.

Admittedly the README currently says it requires 3.8, so maybe I installed COMET in the stone age and `pip install —upgrade unbabel-comet` never warned me. Anyway I think the score mismatch should not be expected

Your signature is very thoughtful!