zhiminy/EvalEng huggingface dataset:
Repository: Unbabel/COMET
Issue: Shortened version of comet-score output
Root cause: Unimplemented Feature Gap

## 🚀 Feature

`Comet-score` should be able to print a shortened score, as an average of all segment scores, when passed a particular flag (maybe something like `--quiet`). To the best of my knowledge, this does not seem possible currently (though of course, I could be wrong as I am new to this package)

### Motivation

Currently, `Comet-score` prints a line-by-line score for each segment. This can be quite an overkill, especially if one is only interested in the score for the whole test set (which is currently calculated as an average for each segment score). Displaying only the average would be useful in these cases.

### Additional context

This is the current output when I run `comet-score` from the CLI: 

<img width="574" alt="Screenshot 2022-02-06 at 11 02 53 pm" src="https://user-images.githubusercontent.com/26062692/152705269-94c3daf3-0153-4dfb-905f-6003a72aa493.png">



I agree! A flag-like `--quiet` would be a nice feature!

Do you want to do a PR 😃 ? 



Sure @ricardorei ! Really sorry for the late response, I had been busy with a few tasks. But I have created a PR now, hopefully it is okay.
Repository: Unbabel/COMET
Issue: Warnings like "Some weights of the model checkpoint at xlm-roberta-large were not used when initializing XLMRobertaModel"
Root cause: Documentation Deficiency

## ❓ Questions and Help

#### What is your question?
When I run "comet-score -s test.en-zh.en -t decoder-out -r test.en-zh.zh", I got the following warnings. Is that normal? or am I missing something?

/root/.cache/torch/unbabel_comet/wmt20-comet-da//checkpoints/model.ckpt
**Some weights of the model checkpoint at xlm-roberta-large were not used when initializing XLMRobertaModel: ['lm_head.bias', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias', 'lm_head.dense.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight']
 This IS expected if you are initializing XLMRobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
This IS NOT expected if you are initializing XLMRobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Encoder model frozen.
/usr/local/python3/lib/python3.8/site-packages/torch/nn/modules/container.py:435: UserWarning: Setting attributes on ParameterList is not supported.**
  warnings.warn("Setting attributes on ParameterList is not supported.")
GPU available: True, used: True


#### What's your environment?

 - Linux
 - python 3.8
 - Version 


I'll see if I can fix the warning but you should not worry because after we load the XLMR from hugging face we reload the final weights [from our final checkpoints](https://github.com/Unbabel/COMET/blob/master/comet/models/__init__.py#L44).

To be fair this is a bit inefficient as we are loading XLMR twice, but what counts in the end are the weights from the checkpoint downloaded from the list of [available models](https://github.com/Unbabel/COMET/blob/master/comet/models/__init__.py#L31).

Thanks for the reply! 
I want to ask one more question. When evaluating on English->Chinese, should the source/hypothesis/reference file be tokenized or detokenized? 


The text should always be detokenized. We run our own tokenization (in most cases XLM-R tokenizer)

Btw as a sanity check I just tried:

```python
from transformers import XLMRobertaModel
model = XLMRobertaModel.from_pretrained("xlm-roberta-large")
```

And the warnings are exactly the same. This is the expected behaviour 

This is because XLM-R in hugging face is an `XLMRobertaForMaskedLM` model class. I don't care about the prediction head in COMET and I initialize instead a plain `XLMRobertaModel` (the base architecture without the LM head)

Great! That helps a lot!
Repository: Unbabel/COMET
Issue: Problems with MULTI-GPU scoring
Root cause: Resource Mishandling

## 🐛 Bug

Hi, there! I found that there were some problems with MULTI-GPU usage with COMET. When using 4 GPUs the segments seem to be getting unordered, making it impossible to retrieve the scores in the original order.

### To Reproduce 

I was running the following command:

comet-score -s source.txt -t target.txt –model wmt20-comet-qe-da --gpus 4

where ‘source.txt’ is a .txt file containing the source segments separated with the ‘\n’ character, and ‘target.txt’ a file containing the target segments to be scored, separated in the same way. Since I did not have a reference, I used the QE model.

I ran the scorer on a sample of the source.txt in a CPU, in 1 GPU and in 4 GPUs. The results for CPU and 1 GPU were exactly the same, but the results when using 4 GPUs showed that the scores were unordered. (See below)

(Experiment on 1 GPU)
tgt.en  Segment 0       score: 0.0000
tgt.en  Segment 1       score: 0.0000
tgt.en  Segment 2       score: 0.3730
tgt.en  Segment 3       score: 0.0002
tgt.en  Segment 4       score: 0.2696
tgt.en  Segment 5       score: 0.2466
tgt.en  Segment 6       score: 0.2901
tgt.en  Segment 7       score: 0.2161
tgt.en  Segment 8       score: 0.2324
tgt.en  Segment 9       score: 0.2536
tgt.en  score: 0.1882 

(Experiment on 1 GPU)
tgt.en  Segment 0       score: 0.0000
tgt.en  Segment 1       score: 0.0000
tgt.en  Segment 2       score: 0.3730
tgt.en  Segment 3       score: 0.0002
tgt.en  Segment 4       score: 0.2696
tgt.en  Segment 5       score: 0.2466
tgt.en  Segment 6       score: 0.2901
tgt.en  Segment 7       score: 0.2161
tgt.en  Segment 8       score: 0.2324
tgt.en  Segment 9       score: 0.2536
tgt.en  score: 0.1882

(Experiment on 4 GPUs)
tgt.en  Segment 0       score: 0.0000
tgt.en  Segment 1       score: 0.2695
tgt.en  Segment 2       score: 0.2323
tgt.en  Segment 3       score: 0.0000
tgt.en  Segment 4       score: 0.2469
tgt.en  Segment 5       score: 0.2536
tgt.en  Segment 6       score: 0.3732
tgt.en  Segment 7       score: 0.2899
tgt.en  Segment 8       score: 0.0002
tgt.en  Segment 9       score: 0.2165
tgt.en  score: 0.1882

### Environment

Experiments were carried out on an AWS EC2 g4dn.xlarge instance and using COMET's 1.0.1 version.

Thank you!

Just to add on this: System-level scores are correct, the problem is that we are currently losing the order of the segment-level scores.

This should not affect system comparisons but it will affect possible segment-by-segment analysis!

I refactored the multiGPU inference. 

This issue is the same as #101. 

Fix will be merged in next release.
Repository: Unbabel/COMET
Issue: Turning off multi-processing for DataLoader
Root cause: Algorithmic Error

## 🐛 Bug

Currently `CometModel.predict()` method takes `num_workers` param; however, it is currently impossible to turn-off multiprocessing due to this bug. The implementation works as expected for `num_workers > 0`, but it fails when `num_workers=0`. 

See [the pytorch DataLoader documentation](https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader) for further info. The default value for `num_worker` is `0` for `DataLoader`, however in the current `predict()` method value 0 is always overridden as max cpu.

The bug occurs due to [this line](https://github.com/Unbabel/COMET/blob/0a0bc3c4e28a6b26563944a72c8e21b115da9a2c/comet/models/base.py#L497)

```python
num_workers=num_workers or multiprocessing.cpu_count(),
```

Since in this expression, value `0` evaluates as False, it is always overridden by `multiprocessing.cpu_count()`, and thus there is currently no way making score prediction without multiprocessing.

### Environment
OS: Linux, Ubuntu 20
Packaging: pip
Version: 1.0.1
Repository: Unbabel/COMET
Issue: Multi-system Evaluation
Root cause: Unimplemented Feature Gap

## 🚀 Feature
It will be great if COMET makes a multi-system evaluation both with Paired Bootstrap Resampling and Paired Approximate Randomization for paired significance tests.

### Motivation

Paired Approximate Randomization is claimed to be more accurate than Paired Bootstrap Resampling when it comes to Type-I errors ([Riezler and Maxwell III, 2005](https://www.aclweb.org/anthology/W05-0908.pdf)).

### Alternatives

Besides Paired Bootstrap Resampling, please consider including Paired Approximate Randomization for paired significance tests.

### Additional context

When reporting Paired Bootstrap Resampling test results, could you please report the standard deviations along with the means?


@andmek do you know any python implementation from this method that I can take a look at?

SacreBLEU (the current version, v2.0.0) has such implementations for BLUE, chrF, and Translation error rate (TER) metrics. I am not sure whether they are implemented purely in python.

sacrebleu is python only I believe. I'll take a look! thanks!
Repository: Unbabel/COMET
Issue: No quiet flag
Root cause: Documentation Deficiency

## 🐛 Bug

<!-- A clear and concise description of what the bug is. -->

### To Reproduce 

Use `comet-score` with the `--quiet` flags described in the docs.

### Expected behaviour

Per the docs, the `--quiet` flag should suppress all output but the final score.

### Screenshots

N/A

### Environment
OS: Linux
Packaging: Pip
Version: 1.0.1



I'll check this soon and keep you updated. Thanks, @erip!

hi @erip! how are you?

I just looked into this and it seems to be working well. What do you mean by `suppress all output but the final score`?

The goal is just to suppress the segment-level scores. It still keeps the progress bar etc. 

I only went the final scores, none of the other output. I'm not sure what's meant by "working well" since I believe the flag doesn't exist in 1.0.1. 

The quiet flag was added to master recently. It is not released in pip yet. 

I mean that it suppresses the other segment-level scores.

I'll probably release the new version soon. Meanwhile, that feature is only available if you install the package from source.



Ah, understood! Thanks very much.
Repository: Unbabel/COMET
Issue: pip install conflict
Root cause: External Dependency Breakage

When I tried a clean install of current COMET version via pip, I got an error that there is a conflict in versions. I managed to resolve it by manually degrading PyYAML to 3.3.*

Could you check it, please?

Maybe it will be resolved by https://github.com/Unbabel/COMET/pull/6 ... I am sorry for possibly old bug report :)

I think you are right, it's not an old bug. What python version are you using?

I am using Python 3.8.5. And I used a clean virtual environment installing only comet (I did this last Friday).

and what about the operative system? 

I had also other similar bug reports from users (see #8 ) since I updated COMET. Yet, both in my computer (Mac OS) and in a Ubuntu VM worked well. I am trying to figure out the real issue here. 

We use Ubuntu 20.04.1 LTS
I can rerun it again. Basically, the problem is that "pytorch-lightning" wanted a different version of PyYAML than something else has installed. The only command I ran on a new virtual environment was "pip install unbabel-comet"
Btw. I made a typo, the PyYAML version that solved the problem was 5.3.*

Thanks for updating, I checked the installation on our machines and now it works perfectly :)

Thanks for the feedback!
I tried Poetry for the first time I hope it makes everything easier with dependencies now
Repository: Unbabel/COMET
Issue: [QUESTION] Are all of the encoders fine-tuned?
GitHub labels: question

Hello!

Are all of the encoders themselves fine-tuned? If so, I would like to use their embeddings for some experiments.

Sincerely,
Kamer

Hi @kayuksel! In almost all models the entire XLM-R model is fine-tuned except for the embedding layer. Also, the learning rate is higher on the top layers. This typically helps generalization because lower layers keep most of their original "representations".

Thank you very much!
Repository: Unbabel/COMET
Issue: [QUESTION] Importing a multilingual model from huggingface into COMET?
Root cause: Documentation Deficiency

I've been doing some basic experimentation with COMET, but I ran into a wall when I tried to use a different XLM-R model, I couldn't quite find any flags to indicate a custom XLM-R model loaded from a local file. 

I know you can train your own metrics with COMET, but could you also use a domain-specific model without compromising the quality of the output?

Hi @lfritzke-mitre I am not sure I understood it right? do you want to train on top of a custom XLM-R model?

If so you need to change the configs file used to train the model, more specifically you need to change the [`pretrained_model`](https://github.com/Unbabel/COMET/blob/master/configs/models/regression_metric.yaml#L11) and also set the [`encoder_model`](https://github.com/Unbabel/COMET/blob/master/configs/models/regression_metric.yaml#L10) to XLM-RoBERTa.

After training, you can pass it to the `comet-score` CLI or just load it using the `load_from_checkpoint` function.

Aha, this mostly answers my question!

To clarify, I want to use the default metrics provided by COMET (WMT20 etc.) in conjunction with a custom pretrained model. This would be a domain-specific XLM-R model, so would the default COMET metrics be appropriate, or would it be best to train a new one?

Just to make sure I have this right, I would keep the YAML file of the metric the same, but change `pretrained_model` to a local path to some XLM-R model? Would it be the folder or a specific file within the model?

Apologies for any confusions.

Hmmm, I am not sure this will work. I think the best option is to train a new one. 

All COMET models finetune the underlying XLM-R model. This means that the top layers (like the Feed Forward Estimator) rely on the representations coming from the underlying XLM-R which is also updated during training. If you change the encoder for something else, with different representations, this will affect the quality predictions in an unexpected way.

If you believe that your domain-specific model provides more quality representations than the original XLM-R I would recommend you to train a new COMET model possibly by keeping your XLM-R encoder frozen during training.

I hope this was helpful

This answers my question perfectly, thanks!
Repository: Unbabel/COMET
Issue: Model link broken (wmt21-comet-qe-da)
Root cause: External Dependency Breakage

## 🐛 Bug
Taken from #53 

When I pass --model wmt21-comet-qe-da

> Global seed set to 12
> Downloading refless-wmt21-comet-da.tar.gz
> refless-wmt21-comet-da.tar.gz: 0.00B [00:00, ?B/s]
> Traceback (most recent call last):
> File "comet/cli/score.py", line 327, in
> score_command()
> File "comet/cli/score.py", line 170, in score_command
> model_path = download_model(cfg.model, saving_directory=cfg.model_storage_path)
> File "/content/COMET/comet/download_utils.py", line 209, in download_model
> available_metrics[model], directory=saving_directory
> File "/content/COMET/comet/download_utils.py", line 165, in download_file_maybe_extract
> urllib.request.urlretrieve(url, filename=filepath, reporthook=_reporthook(t))
> File "/usr/lib/python3.7/urllib/request.py", line 247, in urlretrieve
> with contextlib.closing(urlopen(url, data)) as fp:
> File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
> return opener.open(url, data, timeout)
> File "/usr/lib/python3.7/urllib/request.py", line 531, in open
> response = meth(req, response)
> File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response
> 'http', request, response, code, msg, hdrs)
> File "/usr/lib/python3.7/urllib/request.py", line 569, in error
> return self._call_chain(*args)
> File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
> result = func(*args)
> File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
> urllib.error.HTTPError: HTTP Error 403: Forbidden
> 
> which means the model name (taken from doc) is not wrong according to your post, but not able to download.
> I'll use wmt20-comet-qe-da instead
> 

### Expected behavior

This command should work fine!
```
comet-score -s src.de -t hyp1.en --model wmt21-comet-qe-da
```

### Environment
OS: [e.g. iOS, Linux, Win]
unbabel-comet v1.1.0]
Repository: Unbabel/COMET
Issue: How to configure the precision of the p-value scores?
Root cause: Documentation Deficiency

## How to configure the precision of the p-value scores?

_comet-compare_ normally displays the p-value scores with four decimal digits (e.g., p-value: 0.6059). In my evaluation, some of the p-values have zero scores (p-value: 0.0000). I want to have p-values with more than four decimal digits. Is there an option to do that?


Hi @andmek! Not optionally no.. but that's something easy to change. You just have to change [this function ](https://github.com/Unbabel/COMET/blob/master/comet/cli/compare.py#L77) and replace all `print("{}:\t{:.4f}".format(k, v))` with `print("{}:\t{}".format(k, v))` for example.

Thanks, @ricardorei! I made the precision 80 decimal points! Still, I've got a result like this:

Bootstrap Resampling Results:
x-mean:	-0.26859185950079933480338922890950925648212432861328125000000000000000000000000000
y-mean:	0.28345951165515925751492432027589529752731323242187500000000000000000000000000000
ties (%):	0.00000000000000000000000000000000000000000000000000000000000000000000000000000000
x_wins (%):	0.00000000000000000000000000000000000000000000000000000000000000000000000000000000
y_wins (%):	1.00000000000000000000000000000000000000000000000000000000000000000000000000000000

Paired T-Test Results:
statistic:	-54.90133764783409020537874312140047550201416015625000000000000000000000000000000000
p_value:	0.00000000000000000000000000000000000000000000000000000000000000000000000000000000
Null hypothesis rejected according to t-test.

Is it alright?

By looking at the mean scores it looks right. System Y is much better than System X.

The null hypothesis is that both systems are equally good. This result is telling you that with 100% certain they are not equally good! System Y is much better.

Maybe that's the confusing part....but the null hypothesis is that both systems are equally good. If rejected, this means that one system is statistically better than the other.

Thanks, @ricardorei! It really gives a sense.
Repository: Unbabel/COMET
Issue: Force encodings to match across commands
Root cause: Validation Gap

## 🐛 Bug

Encoding of io operations is done inconsistently. Scoring is performed expecting `utf-8`, but MBR decoding is not (see lines [188](https://github.com/Unbabel/COMET/blob/98aeb826112c04baf703290c7f06edc277710a85/comet/cli/mbr.py#L188) and [191](https://github.com/Unbabel/COMET/blob/98aeb826112c04baf703290c7f06edc277710a85/comet/cli/mbr.py#L191)), and compare only sometimes (see lines [439](https://github.com/Unbabel/COMET/blob/98aeb826112c04baf703290c7f06edc277710a85/comet/cli/compare.py#L439) and [446](https://github.com/Unbabel/COMET/blob/98aeb826112c04baf703290c7f06edc277710a85/comet/cli/compare.py#L446)). Using these commands in succession can throw encoding errors.

### To Reproduce 
```bash
comet-mbr -s "./nmt_adapt/translations/${TRANSLATIONS_FILE}/source.txt" -t "./nmt_adapt/translations/${TRANSLATIONS_FILE}/mbr_samples.txt" --num_samples 12 -o "./nmt_adapt/translations/${TRANSLATIONS_FILE}/mbr_comet.txt"

comet-score -s "./nmt_adapt/translations/${TRANSLATIONS_FILE}/source.txt" -t "./nmt_adapt/translations/${TRANSLATIONS_FILE}/mbr_comet.txt" -r "./nmt_adapt/translations/${TRANSLATIONS_FILE}/references.txt" --quiet

```
Here all files except for "./nmt_adapt/translations/${TRANSLATIONS_FILE}/mbr_comet.txt" are explicitly in `utf-8`, and throw no issues when using `comet-score.`

Throws error:
```python
Traceback (most recent call last):
  File "/home/ivov/.conda/envs/nmt_eval/bin/comet-score", line 8, in <module>
    sys.exit(score_command())
  File "/home/ivov/.conda/envs/nmt_eval/lib/python3.9/site-packages/comet/cli/score.py", line 198, in score_command
    translations.append([line.strip() for line in fp.readlines()])
  File "/home/ivov/.conda/envs/nmt_eval/lib/python3.9/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 5824: invalid continuation byte

```

The system's default encoding is `latin-1`, but can't be certain.

### Expected behaviour
Scoring to continue as usual.

### Screenshots
If applicable, add screenshots to help explain your problem.

### Environment
OS: Linux-4.19.0-20-amd64-x86_64-with-glibc2.28
Packaging: pip
Version: 1.1.0
Repository: Unbabel/COMET
Issue: Referenceless QE metrics require reference input
Root cause: Validation Gap

## 🐛 Bug

When using comet from the `evaluate` package, it seems to require the reference also for QE metrics. There's an easy but dirty hotfix.

### To Reproduce 

```python3
comet_metric = evaluate.load("comet", config_name='wmt21-comet-qe-da')
# erorr!
comet_metric.compute(predictions=["Hallöchen"], sources=["Heyo"])
```

### More info

Passing a reference with an empty string appears to satisfy the library. I checked that the output scores are not dependent on the reference:

```python3
# {'mean_score': 0.08140071481466293, 'scores': [0.08140071481466293]}
comet_metric.compute(predictions=["Hallöchen"], sources=["Heyo"]. references=[""])
# {'mean_score': 0.08140071481466293, 'scores': [0.08140071481466293]}
comet_metric.compute(predictions=["Hallöchen"], sources=["Heyo"], references=["Howdy"])
```

Hi @zouharvi, This seems to be an issue on the `evaluate` package and not on `unbabel-comet`. I suggest you open and issue there...
Repository: Unbabel/COMET
Issue: comet-score [other_params] --to_json True doesn't output a json file
Root cause: Validation Gap

## 🐛 Bug

Passing `--to_json` as `True` or `true` doesn't output any JSON file. Whereas passing any other value creates a file with that name and it has the JSON output as expected.

### To Reproduce 

- Run `comet-score` (or comet-compare) with `--to_json` as `True`
- Observe that there is no json file created

If the test is failing, please add your test cases to the issue (as a draft PR, or simply paste the code to the issue description here).

### Expected behaviour
I think the expected behaviour should be the same for any string, but if it is a boolean and is `True` we can default the output to file `output.json`. If you think this is not the expected behaviour, then changing the help text as following will avoid the confusion.


I changed the flag accept only strings. By default its the empty string and thus it does not export to a json.