Repository: Unbabel/COMET
Issue: How to get data for training of own metric? [QUESTION]
Root cause: Documentation Deficiency

I want to train my own metric, but I need data for that. In the README of the data folder it says, I'm supposed to run ```comet download --help```, but I don't know, what program or script is meant by this command. Although I have ```unbabel-comet``` installed trough pip, my command-line still says ```Command 'comet' not found, but can be installed with: sudo apt install crimson``` (I guess the suggested package is not the right one?)

Unfortunately I also couldn't find any info on the needed data format otherwise I would just download some data somewhere else and put it there.

Hi @jbgruenwald, please check issue #36 and you will find everything you need 

Ok, thanks. Sorry, I didn't find that issue before

No problem! Please feel free to reopen the issue if you do not find something!

Hi, yes, I do have another question:

for your model wmt20-comet-da you used the following training data?
```
https://unbabel-experimental-data-sets.s3.eu-west-1.amazonaws.com/wmt/2020-da.csv.tar.gz
```

If I train a model on that and then evaluate translations with it, I get way worse results (correlation ~0.60) than if I evaluate with your pretrained model (~0.80).

That's why I'm wondering, whether I am using the right file. Btw which hyperparameters (nr of epochs, learning rate, etc) did you use? In case that info somehow helps finding a mistake, I used the following train config:
```
regression_metric:
  class_path: metrics.comet.models.RegressionMetric
  init_args:
    nr_frozen_epochs: 0.3
    keep_embeddings_frozen: True
    optimizer: AdamW
    encoder_learning_rate: 1.0e-05
    learning_rate: 3.1e-05
    layerwise_decay: 0.95
    encoder_model: XLM-RoBERTa
    pretrained_model: xlm-roberta-base
    pool: avg
    layer: mix
    dropout: 0.15
    batch_size: 4
    train_data: /home/COMET/data/2020-da.train.csv
    validation_data: /home/COMET/data/2020-da.test.csv
    hidden_sizes:
      - 768

trainer: trainer-5.yaml
early_stopping: early_stopping.yaml
model_checkpoint: model_checkpoint.yaml
```

And could you please release the files you used for your 21 models?

`wmt20-comet-da` is trained on the concatenation of 2017, 2018, and 2019 data.

- https://unbabel-experimental-data-sets.s3.eu-west-1.amazonaws.com/wmt/2017-da.csv.tar.gz
- https://unbabel-experimental-data-sets.s3.eu-west-1.amazonaws.com/wmt/2018-da.csv.tar.gz
- https://unbabel-experimental-data-sets.s3.eu-west-1.amazonaws.com/wmt/2019-da.csv.tar.gz


And the files to replicate the models are:

[wmt20-comet-da-hyper-parameters.zip](https://github.com/Unbabel/COMET/files/8502021/wmt20-comet-da-hyper-parameters.zip)



For `wmt21-comet-da` you just need to concatenate the data for 2020 (but use the `zh-en` and `en-de` data for validation only) .

For `wmt21-comet-mqm` you must train the `wmt21-comet-da` and then fine-tune the MQM data from 2020. The train test split can be found on the other issue (#36)




If you can't replicate the model just tell me

thanks a lot for your fast answer. I will try it later and tell you whether it's working