Repository: facebookresearch/DomainBed
Issue: KeyError: 'env5_in_acc' | DomainNet experiment
Root cause: Validation Gap


While running "domainbed.scripts.collect_results", I am getting the "KeyError: 'env5_in_acc" error. Also, I notice the "DomainNet experiment" just print env0-env4 while it supposed to print env5 as well. I was wondering if you could help me to fix it. 



The error:
`Traceback (most recent call last):
  File "./anaconda3/envs/wavenet3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "./anaconda3/envs/wavenet3/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/./DomainBed/domainbed/scripts/collect_results.py", line 187, in <module>
    print_results_tables(records, selection_method, args.latex)
  File "/./DomainBed/domainbed/scripts/collect_results.py", line 91, in print_results_tables
    grouped_records = get_grouped_records(records).map(lambda group:
  File "/./DomainBed/domainbed/lib/query.py", line 111, in map
    return Q([fn(x) for x in self._list])
  File "/./DomainBed/domainbed/lib/query.py", line 111, in <listcomp>
    return Q([fn(x) for x in self._list])
  File "/./DomainBed/domainbed/scripts/collect_results.py", line 92, in <lambda>
    { **group, 'sweep_acc': selection_method.sweep_acc(group['records']) }
  File "/./DomainBed/domainbed/model_selection.py", line 34, in sweep_acc
    .map(lambda _, run_records: self.run_acc(run_records))
  File "/./DomainBed/domainbed/lib/query.py", line 109, in map
    return Q([fn(*x) for x in self._list])
  File "/./DomainBed/domainbed/lib/query.py", line 109, in <listcomp>
    return Q([fn(*x) for x in self._list])
  File "/./DomainBed/domainbed/model_selection.py", line 34, in <lambda>
    .map(lambda _, run_records: self.run_acc(run_records))
  File "/./DomainBed/domainbed/model_selection.py", line 89, in run_acc
    return test_records.map(self._step_acc).argmax('val_acc')
  File "/./DomainBed/domainbed/lib/query.py", line 111, in map
    return Q([fn(x) for x in self._list])
  File "/./DomainBed/domainbed/lib/query.py", line 111, in <listcomp>
    return Q([fn(x) for x in self._list])
  File "/./DomainBed/domainbed/model_selection.py", line 81, in _step_acc
    'test_acc': record[test_in_acc_key]
KeyError: 'env5_in_acc'`

DomainNet experiment output:
`env0_in_acc   env0_out_acc  env1_in_acc   env1_out_acc  env2_in_acc   env2_out_acc  env3_in_acc   env3_out_acc  env4_in_acc   env4_out_acc  epoch         loss          step          step_time    
0.0028049034  0.0023896104  0.0046749346  0.0036818138  0.0031480809  0.0030443507  0.0027463768  0.0024927536  0.0041326210  0.0050911376  0.0000000000  5.1279506683  0             0.6886174679 
`

That's interesting, attaching my command and output where I can see all environment statistics:

```bash
python -m domainbed.scripts.train --data_dir /checkpoint/dlp/datasets --dataset DomainNet --checkpoint_freq 1 --hparams '{"batch_size": 16}'
```

```bash
Args:
        algorithm: ERM
        checkpoint_freq: 1
        data_dir: /checkpoint/dlp/datasets
        dataset: DomainNet
        holdout_fraction: 0.2
        hparams: {"batch_size": 16}
        hparams_seed: 0
        output_dir: train_output
        seed: 0
        steps: None
        test_envs: [0]
        trial_seed: 0
HParams:
        batch_size: 16
        class_balanced: False
        groupdro_eta: 0.01
        irm_lambda: 100.0
        irm_penalty_anneal_iters: 500
        lr: 5e-05
        mixup_alpha: 0.2
        mldg_beta: 1.0
        mlp_depth: 3
        mlp_dropout: 0.0
        mlp_width: 256
        mmd_gamma: 1.0
        resnet_dropout: 0.0
        weight_decay: 0.0
env0_in_acc   env0_out_acc  env1_in_acc   env1_out_acc  env2_in_acc   env2_out_acc  env3_in_acc   env3_out_acc  env4_in_acc   env4_out_acc  env5_in_acc   env5_out_acc  epoch         loss          step          step_time    
0.0017660503  0.0021818182  0.0022042438  0.0018409069  0.0015913376  0.0015221753  0.0024927536  0.0024057971  0.0033030255  0.0026019833  0.0014646583  0.0013743219  0.0000000000  5.8700618744  0             1.6476581097
...
```

I wasn't able to reproduce this either. Could you post instructions to reproduce?

I meet this problem too，do you solve it?