zhiminy/EvalEng huggingface dataset:
Repository: facebookresearch/DomainBed
Issue: About error bars reported in paper
Root cause: Documentation Deficiency

Hi. Thanks for the nice work!
I am truly motivated by the work and trying to use it as baseline for my work.
While analyzing the code and the paper, I have a question about the standard deviation reported in the paper. 
According to the paper, std is calculated by running 3 experiments with entirely different random choices.
Can I follow the way you did it by changing hparams_seed, seed, trial_seed in your code?
Thank you!

@igul222 Can you shed some light on this one?

We try 20 different `hparams_seed` (which selects 20 different hyper-parameter configurations). We repeat each hyper-parameter configuration 3 `trial_seed` times (which re-runs the same choice of hyperparameters to take into account the stochasticity everywhere else, such as data splitting and shuffling). Mean +- stds reported in tables are over the 3 repetitions of the best hyper-parameter configuration.

Based on the code, the hparam in each trial is generated by `hash(hparams_seed, trial_seed)`. Each set of hparam is run for only one time. In addition, the result is collected by selecting the best hparm in each trial, while the best hparms among different trials might not be the same. I am afraid that current implementation is not as what you described.

You are right, I misremembered. Thanks.

In one sweep, 20 random hyper-parameter configurations (`hparams_seed`) are tried per algorithm. The best hyper-parameter configuration is chosen accordingly to the model selection strategy, and we take note of its test accuracy.

We run 3 of such sweeps (`trial_seed`). We report the mean +- err of the three noted test accuracies.
Repository: facebookresearch/DomainBed
Issue: Error in discriminator regularisation in `AbstractDANN`
Root cause: Algorithmic Error

While working my way through the `AbstractDANN` class, I think I came across a mistake in the calculation of the domain discriminator's loss.

https://github.com/facebookresearch/DomainBed/blob/2ed9edf781fe4b336c2fb6ffe7ca8a7c6f994422/domainbed/algorithms.py#L270-L274

If I understand the code correctly, here we want to regulariser the discriminator with the square of the gradients that go to the featurizer. The above code therefore aims to get each sample's predicted probabilities per domain (line 270), get the cross-entropy loss and calculate the gradient with respect to the featurizer output (line 271), square and average the gradient (line 273) and finally adds it to the loss (line 274). 

### Problem
In line 271, in order to get the probability for the true domain, we index the softmax like this `disc_softmax[:, disc_labels]` with (I think) the intention to get `disc_softmax[i, 0]` if i's domain is 0, `disc_softmax[i, 1]` if i's domain is 1, etc. Unfortunately, this is not what happens. Running this line e.g. on ColoredMNIST with batch size of 64 gives: 

```python
disc_softmax[:, disc_labels]
#> tensor([[0.5825, 0.5825, 0.5825,  ..., 0.4175, 0.4175, 0.4175],
#>        [0.5513, 0.5513, 0.5513,  ..., 0.4487, 0.4487, 0.4487],
#>        [0.5800, 0.5800, 0.5800,  ..., 0.4200, 0.4200, 0.4200],
#>        ...,
#>        [0.5869, 0.5869, 0.5869,  ..., 0.4131, 0.4131, 0.4131],
#>        [0.5305, 0.5305, 0.5305,  ..., 0.4695, 0.4695, 0.4695],
#>        [0.5332, 0.5332, 0.5332,  ..., 0.4668, 0.4668, 0.4668]],
#>       grad_fn=<IndexBackward0>)

disc_softmax[:, disc_labels].shape
#> torch.Size([128, 128])
``` 

That is, rather than indexing the last dimension, it first copies `disc_softmax[:, 0]` 64 times and then `disc_softmax[:, 1]` 64 times. It is actually a mistake I have made myself many many times, because it always intuitively feels like this should work. 


### Solution 

Instead, I think what we want is

```python
disc_softmax.gather(dim=-1, index=disc_labels[:, None]).shape
torch.Size([128, 1])
```
We probably also want to log before we sum to truly make it a cross-entropy but not sure if this makes much of a difference here. Overall, we could -- more transparently -- replace it with 

```python
F.cross_entropy(disc_out, disc_labels, reduction='sum')
```
(note that the sign is different to the original code but I think it doesn't matter because of the power 2 later). 


Let me know if I got anything wrong here or if there's something I've missed :)
Repository: facebookresearch/DomainBed
Issue: `Transfer` algorithm is missing function
Root cause: Validation Gap

@Gordon-Guojun-Zhang the implementation of your Transfer algorithm in this repo currently results in the following error:

```bash
python -m domainbed.scripts.train --data_dir PATH_TO_MNIST --algorithm Transfer --dataset ColoredMNIST
#> NameError: name 'loss_gap' is not defined
```

The error originiates here as `loss_gap` does not exist in this DomainBed repo

https://github.com/facebookresearch/DomainBed/blob/2ed9edf781fe4b336c2fb6ffe7ca8a7c6f994422/domainbed/algorithms.py#L1839

I think it is missing this function from your forked repo:

https://github.com/Gordon-Guojun-Zhang/Transferability-NeurIPS2021/blob/86fba39cbe291c65e494236c2a2a3ccba5bee1cc/domainbed/algorithms.py#L364-L374


Thanks for raising this issue. I have fixed it and submitted a PR. https://github.com/facebookresearch/DomainBed/pull/108
Repository: facebookresearch/DomainBed
Issue: Library versions
Root cause: Documentation Deficiency

Could you please specify the library versions (pytorch, torchvision and CUDA) for this scipts. 

In my machine:
```python
Python 3.8.3 (default, Jul  2 2020, 16:21:59) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
(ins)>>> import torch
(ins)>>> import torchvision
(ins)>>> torch.__version__
'1.6.0'
(ins)>>> torchvision.__version__
'0.7.0'
(ins)>>> torch.version.cuda
'10.1'
```

> In my machine:
> 
> ```python
> Python 3.8.3 (default, Jul  2 2020, 16:21:59) 
> [GCC 7.3.0] :: Anaconda, Inc. on linux
> Type "help", "copyright", "credits" or "license" for more information.
> (ins)>>> import torch
> (ins)>>> import torchvision
> (ins)>>> torch.__version__
> '1.6.0'
> (ins)>>> torchvision.__version__
> '0.7.0'
> (ins)>>> torch.version.cuda
> '10.1'
> ```

That really helps a lot!
Repository: facebookresearch/DomainBed
Issue: Enquiry about hyper-parameter tuning using sweep
Root cause: Documentation Deficiency

Hi authors, 

Thank you for providing the community with this fabulous test bed. I found this repo extremely helpful to my research about domain generalization. When reading the code in `sweep.py` and `train.py`, there is a line that I am not very sure about its rationale.

When we tune hyper parameters by calling `sweep.py`, we should keep the generated parameter the same across different trials. However, current seed of random hyper parameter generation in `train.py` is set to be `misc.seed_hash(args.hparams_seed, args.trial_seed)` (line79), which means that the set of hyper parameters will only be given one chance in the run to show its performance during sweep. I am wondering if a seed `misc.seed_hash(args.hparams_seed)` will be enough when we call `train.py` via `sweep.py`, as we should test the randomly generated set of hyper parameters in the same number of trials as we do for the default hyper parameters.

I will really appreciate it if you could enlighten me on my above question about the hyper-parameter tuning in `sweep.py`.

Thank you!

Thanks Nancy. Same question here! It feels that the same seed (hparams_seed excluding the trial_seed) should be used to set the hyperparameters across the corresponding trials. Otherwise, the resultant mean and error bar would not be for the same set of hyperparameters. 

@igul222 Can you shed some light on this one?

Thank you for the message. With further investigation on code in `collect_results.py` with @qiaoruiyt, we found that the top test performance among each trial is collected. Top results across trials are used to calculate mean and std. With current implementation, it seems that we randomly search the hyper-parameter space and use the result of several best hyper-parameters as the performance of an algorithm. 

My understanding about the need of repetition in experiment is to account for other sources of stochasticity (e.g. dataset partition, model initialization) rather than the stochasticity that hyper-parameters bring. Thus, I am wondering if the result of the same set of hyper-parameters across trials shoud be averaged instead, and the best result should be selected among different sets of hyper-parameters.

Please correct me if my understanding about the need of repetition is not your intention. 

Yes, that's how it is! The mean +- std reported in tables are indeed the averages for the chosen set of hyper-parameters. Closing.

For more explanation, see https://github.com/facebookresearch/DomainBed/issues/103#issuecomment-1330008560
Repository: facebookresearch/DomainBed
Issue: ColoredMNIST with IRM Implementation is different from the original IRM papers
Root cause: Documentation Deficiency

The ColoredMNIST with IRM implementation uses the following MLP network in original papers, which is very different from the Conv network implementation in DomainBed, and the results are also quite different (acc=66.9 on test_env[0.9] in original IRM paper; acc=10.1 on test_env[0.9] in this paper). Could you please let me know if this different implementation is reasonable for comparison?

```python
class MLP(nn.Module):
  def __init__(self):
    super(MLP, self).__init__()
    if flags.grayscale_model:
      lin1 = nn.Linear(14 * 14, flags.hidden_dim)
    else:
      lin1 = nn.Linear(2 * 14 * 14, flags.hidden_dim)
    lin2 = nn.Linear(flags.hidden_dim, flags.hidden_dim)
    lin3 = nn.Linear(flags.hidden_dim, 1)
    for lin in [lin1, lin2, lin3]:
      nn.init.xavier_uniform_(lin.weight)
      nn.init.zeros_(lin.bias)
    self._main = nn.Sequential(lin1, nn.ReLU(True), lin2, nn.ReLU(True), lin3)
  def forward(self, input):
    if flags.grayscale_model:
      out = input.view(input.shape[0], 2, 14 * 14).sum(dim=1)
    else:
      out = input.view(input.shape[0], 2 * 14 * 14)
    out = self._main(out)
    return out
```

The `binary_cross_entropy_with_logits` is used in the original IRM implementation; however, the general conv network is used in this repository. Please let me know if there is a reason for this. Under this implementation, it seems the IRM may not be suitable; thus, ColoredMNIST[0.9]'s benchmark score is 10.1 as opposed to 66.9 in the IRM paper.

Hi lccurious, I think using a ConvNet rather than an MLP for ColoredMNIST is reasonable since this dataset has image inputs. Another reason may be that since the same architecture should be used by all methods for fair benchmarking, it may not be appropriate if this architecture is itself tailored to a specific method (e.g., IRM). For your question on the discrepancy between test results, you can check Appendix B.1 of the DomainBed paper, which has indicated this discrepancy is largely due to different model selection methods rather than architectures.
Repository: facebookresearch/DomainBed
Issue: Issue in using InfiniteDataLoader with Lightning
Root cause: Unimplemented Feature Gap

I'm trying to use `InfiniteDataLoader` in `lib.fast_data_loader` with the Pytorch Lightning Module
```
train_loaders = [InfiniteDataLoader(
    dataset=env,
    weights=env_weights,
    batch_size=batch_size,
    num_workers=dataset.N_WORKERS)
    for i, (env, env_weights) in enumerate(in_splits)
    if i not in test_envs + val_envs]
    
trainer.fit(algorithm, train_loaders)
```

The training loop goes on forever in this case. If I add a `__len__` function similar to `FastDataLoader` in `InfiniteDataLoader` class, the problem is solved. However, I wanted to ask if this would cause issues with the expected behaviour of the `InfiniteDataLoader`.

I don't see any issues, but we do not support Lightning.
Repository: facebookresearch/DomainBed
Issue: Error "Object of type int64 is not JSON serializable"
Root cause: Validation Gap

When I run train.py with the MLDG algorithm, an error saying that "Object of type int64 is not JSON serializable" occurs. At line 
     File "/home/zx/DomainBed/domainbed/scripts/train.py", line 257, in <module>
         f.write(json.dumps(results, sort_keys=True) + "\n")

After debugging I find it comes from the "_hparam('n_meta_test', 2, lambda r:  r.choice([1, 2]))" line in the hparams_registry file. It is because that r.choice returns a numpy.int64 type data, which may lead to an error sometimes. I think we should change it to "_hparam('n_meta_test', 2, lambda r:  int(r.choice([1, 2])))" to transfer the selected number to the int type

Mind sending a pull request?

Hello!
Recently, this error occurred to me, too.
Are you fixing this error now? Just like ZouXinn said, it works!
Repository: facebookresearch/DomainBed
Issue: Question about the results

Hi, thanks for your great work!

I'm wondering that are those results released on your paper coming from the script [sweep.py](https://github.com/facebookresearch/DomainBed/blob/main/domainbed/scripts/sweep.py)?

Another question is that, for example, when we train a model for PACS using DomainBed with env0 as the test_env, should we use the highest result in env0_in_acc or just the last one or anything else?

Thanks.

Please use `collect_results.py` to produce tables. The results of the sweep are mentioned in other closed issues.
Repository: facebookresearch/DomainBed
Issue: What **query** means in the paper?

Hi, thanks for your brilliant work! The questions raised are thought-provoking, and the standards help the game be rigorous!

In Section 3.1, I'm wondering what query means in **oracle** model selection method? To me, one query seems like a random search to find the fit hyperparameters, but do you make queries in the other two model selection methods?

Many thanks to your work and have a good day :>


One query = one hparam configuration that we try. Oracle is only different in the way it chooses the best model (looking at validation set drawn from the test domain).

Hi David@lopezpaz , thx for your reply! Have a good day xD
Repository: facebookresearch/DomainBed
Issue: Bug in MixUp hyperparameter grid
Root cause: Algorithmic Error

Hi, I'm not entirely sure if this is a bug, but noticed it and figured I'd file it and let the maintainers decide.

It looks like there is a bug in the hyperparameter grid for the MixUp model, where the "range" of the random selection is between 10^-1 and 10^-1 (effectively fixing the value of that hyperparameter):

https://github.com/facebookresearch/DomainBed/blob/2ed9edf781fe4b336c2fb6ffe7ca8a7c6f994422/domainbed/hparams_registry.py#L67

Specifically, I am referring to `10**r.uniform(-1, -1)`.

Is this an issue, or am I just misunderstanding how the code works?

Seems like a bug indeed. The appendix in the DomainBed paper seems wrong as well. Looking at the original mixup paper, a  reasonable grid would be `10**r.uniform(-1, +1)`. Mind sending a PR?
Repository: facebookresearch/DomainBed
Issue: RotatedMNIST doesn't work with torchvision 0.8.2 ('torchvision.transforms' has no attribute 'InterpolationMode')
Root cause: External Dependency Breakage

Running `train.py` with `--dataset RotatedMNIST` causes an exception when using torchvision 0.8.2, which is the version currently specified in [domainbed/requirements.txt](https://github.com/facebookresearch/DomainBed/blob/3ab2b63e4e270686cd45c8ea7444c4418c0d77bc/domainbed/requirements.txt).

The problem is that the call to [`torchvision.transforms.functional.rotate`](https://pytorch.org/vision/0.8/transforms.html#torchvision.transforms.functional.rotate) was changed in commit 27e115e66 from https://github.com/facebookresearch/DomainBed/blob/4f4e6a7fec7ed81f93a5d218962480498503e0dd/domainbed/datasets.py#L170 to https://github.com/facebookresearch/DomainBed/blob/27e115e66bb84412db7017874bd8ee446d057610/domainbed/datasets.py#L167

But the `interpolation` argument for `rotate` was not added until torchvision 0.9.

It's also worth pointing out that the semantics here have actually changed as well since the interpolation mode was changed from BICUBIC to BILINEAR, which may have an impact on results. A change that impacts the results of the benchmark seems like something that should be avoided.


We didn't observe any changes at the time, but please feel free to reopen if you do. We should update `requirements.txt` anyway.
Repository: facebookresearch/DomainBed
Issue: MyLauncher for SLURM
Root cause: Documentation Deficiency

I'd like to launch a sweep on a SLURM based cluster. Can someone please advise what the MyLauncher function should look like?

Thanks!

I have something for that! #56 (Disclaimer: Use at own risk. It might not be appropriate for you cluster, but the general structure is there)

Take a look at `submitit`, too.
Repository: facebookresearch/DomainBed
Issue: PrototypicalDomainGeneralization code implementation
Root cause: Documentation Deficiency

I tried to reproduce prototypical domain generalization code.
Since there is no explanation in DomainBed readme file about PrototypicalDomainGeneralization, I downloaded code from this url (https://github.com/abhimanyudubey/PrototypicalDomainGeneralization/blob/master/domainbed/lib/fast_data_loader.py).
However, PrototypicalDomainGeneralization readme is as same as DomainBed, so I'm suffering to use the code.
Is there any protocol to train and test PrototypicalDomainGeneralization?


You would have to ask the authors of PrototypicalDomainGeneralization! Sorry.
Repository: facebookresearch/DomainBed
Issue: DomainBed leave-one-out results table is not showing up
Root cause: Validation Gap

Generated by:
```
CUDA_VISIBLE_DEVICES=2 python -m domainbed.scripts.sweep launch\
       --data_dir=/pub2/data \
       --output_dir=/pub2/podg \
       --command_launcher local \
       --single_test_envs \
       --algorithms ERM \
       --datasets PACS \
       --n_hparams 1\
       --n_trials 1
```

Collected with:
```
CUDA_VISIBLE_DEVICES=2 python -m domainbed.scripts.collect_results\
	--input_dir=/pub2/podg
```

Results:
```
Total records: 72

-------- Dataset: PACS, model selection method: training-domain validation set
Algorithm             A                     C                     P                     S                     Avg
ERM                   83.3 +/- 0.0          78.7 +/- 0.0          96.7 +/- 0.0          66.8 +/- 0.0          81.4

-------- Averages, model selection method: training-domain validation set
Algorithm                  PACS                       Avg
ERM                        81.4 +/- 0.0               81.4

-------- Dataset: PACS, model selection method: leave-one-domain-out cross-validation
Algorithm             A                     C                     P                     S                     Avg
ERM                   X                     X                     X                     X                     X

-------- Averages, model selection method: leave-one-domain-out cross-validation
Algorithm                  PACS                       Avg
ERM                        X                          X

-------- Dataset: PACS, model selection method: test-domain validation set (oracle)
Algorithm             A                     C                     P                     S                     Avg
ERM                   82.7 +/- 0.0          81.2 +/- 0.0          94.3 +/- 0.0          78.7 +/- 0.0          84.3

-------- Averages, model selection method: test-domain validation set (oracle)
Algorithm                  PACS                       Avg
ERM                        84.3 +/- 0.0               84.3
```

leave-one-out domain requires two test sets therefore I cant use `--single_test_env`
Repository: facebookresearch/DomainBed
Issue: Where are unlabelled data used in DANN?
Root cause: Unimplemented Feature Gap

First thank you for this really nice collection of implementations for domain adaptation 😄 and with a really clean code!

I was wondering if it is normal if the `unlabeled` parameter is not used in the `update` function of the `DANN` algorithm. More specifically the `update` code is [here](https://github.com/facebookresearch/DomainBed/blob/main/domainbed/algorithms.py#L247) and if I understand it right the call to `update` is in the `training.py` script [here](https://github.com/facebookresearch/DomainBed/blob/main/domainbed/scripts/train.py#L216) where it looks like the `uda_device` data are the unlabelled data passed to the `update` function as the second parameter so corresponding to the `unlabeled`.

So, my current understanding of the code is that the "target domain" is not being used (when it should). If my understanding is good I can try fixing this and submit a PR to the repository.

would love to have an example code of an algorithm that uses `unlabled`. From what I see, `unlabled` isn't used in any algorithm listed

Hi, as noticed, the code base has scaffolding for domain adaptation but it is not yet enabled for any algorithm. If anybody would like to enable this, feel free to submit a PR. Closing for now.
Repository: facebookresearch/DomainBed
Issue: DANN seems to not converge at all
Root cause: Configuration Error

Could anyone provide some details about how to train DANN?
I run the following code:
`
python3 -m domainbed.scripts.train\
       --data_dir=./domainbed/data/\
       --algorithm DANN\
       --dataset PACS\
       --test_env 2
`
And get the following result:
![image](https://user-images.githubusercontent.com/87475993/232716231-f79cbff3-ac7a-4423-8ba1-9d7073b732ea.png)
It seems that disc_loss and gen_loss don't converge at all !!

Hi @Crestina2001 , things indeed seem to diverge in this run. Your command looks correct, but perhaps different hyper-parameters are needed? See https://github.com/facebookresearch/DomainBed/issues/32 to identify the top hparams for each model, found in the original sweep for the paper.  

Closing due to inactivity, feel free to reopen if needed.
Repository: facebookresearch/DomainBed
Issue: Question about the train & val splits
Root cause: Algorithmic Error

Hi, in the code for splitting the dataset into train and val sets, it seems that the samples in a domain are randomly splitted without consideration for their categories, which may cause a problem: chances are that the samples in the train or val sets may not contain all categories (e.g., the val sets may only contain 100 categories, while there are 345 categories in total for the domain). I wonder whether this will impact the effectiveness for hyperparameter searching and result for leave-one-domain-out evaluation?
```python
def split_dataset(dataset, n, seed=0):
    """
    Return a pair of datasets corresponding to a random split of the given
    dataset, with n datapoints in the first dataset and the rest in the last,
    using the given random seed
    """
    assert(n <= len(dataset))
    keys = list(range(len(dataset)))
    np.random.RandomState(seed).shuffle(keys)
    keys_1 = keys[:n]
    keys_2 = keys[n:]
    return _SplitDataset(dataset, keys_1), _SplitDataset(dataset, keys_2)
```


Hi @X-funbean , 

You're right that the code does not consider categories in random splitting, but I'm not sure how much it will affect accuracy. If a class is so infrequent that it might not be represented in the validation split, it also likey will not change overall accuracy much. 

An interesting follow up question is if the class distribution for the source data is similar to the class distribution in the target data. If not, then perhaps a resampling of the validation data based on class might be useful. 

Closing for now, feel free to reopen if you have more questions.