Repository: Unbabel/COMET
Issue: mbr-decode triggers an issue with models other than default
Root cause: Validation Gap

## 🐛 Bug
works fine with default model.

Encoder model frozen.
Encoding sentences...: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 627/627 [00:15<00:00, 40.93it/s]
MBR Scores...:   0%|                                                                                                                                                          | 0/1002 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/usr/local/bin/comet-mbr", line 8, in <module>
    sys.exit(mbr_command())
  File "/usr/local/lib/python3.8/dist-packages/comet/cli/mbr.py", line 201, in mbr_command
    mbr_matrix = mbr_decoding(src_embeddings, mt_embeddings, model)
  File "/usr/local/lib/python3.8/dist-packages/comet/cli/mbr.py", line 127, in mbr_decoding
    scores = model.estimate(source, translation, pseudo_refs)[
  File "/usr/local/lib/python3.8/dist-packages/comet/models/regression/regression_metric.py", line 186, in estimate
    return {"score": self.estimator(embedded_sequences)}
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/comet/modules/feedforward.py", line 69, in forward
    return self.ff(in_features)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/container.py", line 141, in forward
    input = module(input)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1110, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/linear.py", line 103, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (5x6144 and 4096x2048)





@vince62s which model have you tried? I am running it fine with `wmt21-comet-da`

I think you tried to run MBR with a reference-free model. Reference-free models are not supposed to be used for MBR.

In MBR you will build a matrix where you will compare each hypothesis against each other. The hypothesis with the higher "consensus" is the one that is selected. With a reference-free model, each hypothesis will always be assigned the same score over and over again and the final results would be a simple ranking that does not take into consideration other hypotheses.

In any case, I'll try to add an exception to avoid users using a reference-free model for MBR.

Not 100% sure but I think I tried wmt21-comet-mqm too.

second thought on this.
In a real world if we want to use MBR decoding to output the best (according to comet) sentence out of an n-best list, you are not supposed to have a reference, hence it should choose the best out of N based on a QE model. Am I wrong ?
In other words the point would be to score each sentence on a N-best list, then pick the best one.

That's one way to do it yes. It is a combination of N-best Reranking with MBR. 

The current implementation does not do it. We currently score everything against everything. But one could easily get QE scores, and then pick the best hypothesis to use as references during MBR.

If you are interested in this topic you can take a look at our recent paper: [Quality-Aware Decoding](https://arxiv.org/pdf/2205.00978.pdf)

We explore reranking with QE metrics. MBR and the combination of both.

The best results are exactly what you are proposing: Nbest reranking + MBR (Table 2 and 3)