Repository: Unbabel/COMET
Issue: Reimplementing results in COMET EMNLP'20 paper
Root cause: Documentation Deficiency

Hi,

Recently I'm reimplementing the results in [your COMET EMNLP'20 paper](https://aclanthology.org/2020.emnlp-main.213.pdf). I also carefully referred to [documentation](https://unbabel.github.io/COMET/html/models.html) for more details. However, when reimplementing experiments over ```wmt-metrics``` data, I find something unexpected. Here are my steps for preparing:

1. I create a virtual environment with ```conda```:
```
conda create -n comet python=3.8
conda activate comet
pip install unbabel-comet
```

2. Download the ```wmt-metrics``` data via:
```
comet download -d wmt-metrics --saving_path data/wmt-metrics/
```

After these steps I continue the implementation with the released model:

1. I want to have a test using language pair ```en-de```, so I first split  ```test19-relative-ranking.csv```  file into multiple files: ```storing source```, ```reference```, ```positive_hypothesis```, and ```negative_hypothesis``` line by line. Here is the content of python source file ```script_language_filter.py```
```
from argparse import ArgumentParser
from csv import reader

parser = ArgumentParser()

parser.add_argument('--input_file', type=str, required=True)
parser.add_argument('--language', type=str, required=True)
parser.add_argument('--output_src', type=str, required=True)
parser.add_argument('--output_ref', type=str, required=True)
parser.add_argument('--output_pos', type=str, required=True)
parser.add_argument('--output_neg', type=str, required=True)

args = parser.parse_args()


def main():
    with open(args.input_file, mode='r', encoding='utf-8') as f1, \
            open(args.output_src, mode='w', encoding='utf-8') as f2, \
            open(args.output_ref, mode='w', encoding='utf-8') as f3, \
            open(args.output_pos, mode='w', encoding='utf-8') as f4, \
            open(args.output_neg, mode='w', encoding='utf-8') as f5:
        csv_reader = reader(f1)
        next(csv_reader) # escape the first line
        for _, row in enumerate(csv_reader):
            # csv_file title:
            # data, lp, src, ref, pos, neg, pos.model, neg.model, bestmodel
            # indexes of our interest:
            #       1 , 2  , 3  , 4  , 5
            if row[1] != args.language:
                continue

            f2.write(row[2].strip() + '\n')
            f3.write(row[3].strip() + '\n')
            f4.write(row[4].strip() + '\n')
            f5.write(row[5].strip() + '\n')
    return


if __name__ == '__main__':
    main()
```

Then, run this command:
```
python script_language_filter.py \
    --input_file test19-relative-ranks.csv \
    --language "de-en" \
    --output_src test19-relative-ranks.src \
    --output_ref test19-relative-ranks.ref \
    --output_pos test19-relative-ranks.pos \
    --output_neg test19-relative-ranks.neg
```

After this step, I get 4 more files ```test19-relative-ranks.{src,ref,pos,neg}```, each yielding 17,073 lines.

2. Scoring each sentence pair:
For positive_hypothesis-reference:
```
comet score -s test19-relative-ranks.src \
    -h test19-relative-ranks.pos \
    -r test19-relative-ranks.ref \
    --batch_size 16 \
    --to_json test19-relative-ranks.pos.json \
    --model emnlp-base-da-ranker
```
For negative_hypothesis-reference:
```
comet score -s test19-relative-ranks.src \
    -h test19-relative-ranks.neg \
    -r test19-relative-ranks.ref \
    --batch_size 16 \
    --to_json test19-relative-ranks.neg.json \
    --model emnlp-base-da-ranker
```
Then I get two files storing predicted scores ```test19-relative-ranks.{pos,neg}.json```.

3. I didn't find presented script for directly computing WMT DARR Kendall score. So I simply write a script:
```
from argparse import ArgumentParser
from json import load

parser = ArgumentParser()
parser.add_argument('--pos_json', type=str, required=True)
parser.add_argument('--neg_json', type=str, required=True)

args = parser.parse_args()


def main():
    with open(args.pos_json, mode='r', encoding='utf-8') as f1, \
            open(args.neg_json, mode='r', encoding='utf-8') as f2:
        pos_data = load(f1)
        neg_data = load(f2)

    concor = 0
    discor = 0

    for pos, neg in zip(pos_data, neg_data):
        if pos['predicted_score'] > neg['predicted_score']:
            concor += 1
        else:
            discor += 1

    print('%d items in total. Concor: %d, Discor: %d, WMTKendall: %f' % (concor + discor, concor, discor, (concor - discor) / (concor + discor)))
    return


if __name__ == '__main__':
    main()

```

And I run:
```
python script_compute_rr.py \
    --pos_json test19-relative-ranks.pos.json \
    --neg_json test19-relative-ranks.neg.json 
```

The results are:
```
17073 items in total. Concor: 11244, Discor: 5829, WMTKendall: 0.317167
```

So here I find my result is extremely higher than the reported result ```0.202``` in your paper (column de-en, row COMET-RANK in Table 2). I'm not sure which step I'm wrong. Besides, I also want to know whether the model tagged ```emnlp-base-da-ranker``` is exactly the well-trained model corresponding to the reported results in Table 2 of your paper.

Could you answer these questions for me? Many thanks!


 - OS: [Ubuntu 20.0]
 - Packaging [conda]
 - Version [0.1.0]


Hi @wanyu2018umac, 

The naming of the file test19-relative-ranks is a bit misleading. That file is just a random split that I did to validate data from 2019 without having to run the entire corpus from that year. To reproduce the results you have to use the entire en-de data from 2019.

The `emnlp-base-da-ranker` is trained on data from 2017 and 2018 only and the reported results use the entire 2019 data. 

We do have a script to reproduce the result [https://github.com/Unbabel/COMET/blob/master/wmt-shared-task/segment-level/segment_level_comet.py](https://github.com/Unbabel/COMET/blob/master/wmt-shared-task/segment-level/segment_level_comet.py)

You just need to download the data as you did and download the checkpoint locally also. Then it should be easy to run

Please send me an email if you have any problems reproducing the results (I am usually faster at replying to emails).

Ah I see it. It's a little misleading though. Thanks for your response!