Repository: facebookresearch/fairseq

Issue #1903: CNN evaluation on translation task gives unexpected keys error 
Created: 2020-03-25T02:51:49Z
❓ Questions and Help
What is your question?
I followed the instructions to train the fconv_wmt_en_de model and then was trying to evaluate the checkpoint. But I get the following error (all layers with linearized_weight are not there in state_dict):
Traceback (most recent call last):
  File "/home/skhurana/fairseq/venv/bin/fairseq-generate", line 11, in <module>
    load_entry_point('fairseq', 'console_scripts', 'fairseq-generate')()
  File "/home/skhurana/fairseq/fairseq/fairseq_cli/generate.py", line 234, in cli_main
    main(args)
  File "/home/skhurana/fairseq/fairseq/fairseq_cli/generate.py", line 35, in main
    return _main(args, sys.stdout)
  File "/home/skhurana/fairseq/fairseq/fairseq_cli/generate.py", line 71, in _main
    task=task,
  File "/home/skhurana/fairseq/fairseq/fairseq/checkpoint_utils.py", line 186, in load_model_ensemble
    ensemble, args, _task = load_model_ensemble_and_task(filenames, arg_overrides, task)
  File "/home/skhurana/fairseq/fairseq/fairseq/checkpoint_utils.py", line 205, in load_model_ensemble_and_task
    model.load_state_dict(state["model"], strict=True, args=args)
  File "/home/skhurana/fairseq/fairseq/fairseq/models/fairseq_model.py", line 93, in load_state_dict
    return super().load_state_dict(new_state_dict, strict)
  File "/home/skhurana/fairseq/venv/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 830, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for FConvModel:
	Unexpected key(s) in state_dict: "decoder.convolutions.0._linearized_weight", "decoder.convolutions.1._linearized_weight", "decoder.convolutions.2._linearized_weight", "decoder.convolutions.3._linearized_weight", "decoder.convolutions.4._linearized_weight", "decoder.convolutions.5._linearized_weight", "decoder.convolutions.6._linearized_weight", "decoder.convolutions.7._linearized_weight", "decoder.convolutions.8._linearized_weight", "decoder.convolutions.9._linearized_weight", "decoder.convolutions.10._linearized_weight", "decoder.convolutions.11._linearized_weight", "decoder.convolutions.12._linearized_weight", "decoder.convolutions.13._linearized_weight", "decoder.convolutions.14._linearized_weight". 

Code
I train the model on a single V100 GPU using this command
fairseq-train \
    data-bin/conv-large \
    --arch fconv_wmt_en_de \
    --lr 0.25 --clip-norm 0.1 --dropout 0.2 --max-tokens 4096 \
    --criterion label_smoothed_cross_entropy --label-smoothing 0.1 \
    --lr-scheduler fixed --force-anneal 50 \
    --save-interval 1 --max-epoch 10 \
    --save-dir checkpoints/conv-large \
    --eval-bleu     --eval-bleu-args '{"beam": 5, "max_len_a": 1.2, "max_len_b": 10}'     --eval-bleu-detok space     --eval-bleu-remove-bpe     --eval-bleu-print-samples \
    --best-checkpoint-metric bleu --maximize-best-checkpoint-metric

and my evaluation command (which I run on CPU) is:
fairseq-generate data-bin/conv-large \
    --path checkpoints/conv-large/checkpoint_best.pt \
    --beam 5 --remove-bpe --cpu

What have you tried?
Tried using different dataset and but I get the same error. Also checked to see if it's an architecture-specific error - turns out transformer evaluation is okay, just the CNN throws this error.
What's your environment?

fairseq Version (e.g., 1.0 or master): master
PyTorch Version (e.g., 1.0): 1.4.0+cpu on inference, 1.4.0 while training
OS (e.g., Linux): Linux
How you installed fairseq (pip, source): source
Build command you used (if compiling from source): pip install --editable .
Python version: 3.6.8

Duplicate issue #1949: Bug while decoding a trained es-en translation model
Created: 2020-03-31T15:25:30Z
This is the error that I get on using the fairseq-generate command on a trained model
load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for FConvModel:
Unexpected key(s) in state_dict: "decoder.convolutions.0._linearized_weight", "decoder.convolutions.1._linearized_weight", "decoder.convolutions.2._linearized_weight".
Surprisingly, the translation and generate work for en-es pair and not reverse.

Closing comment (2020-03-31T15:32:13Z): Seems like a duplicate of #1903