zhiminy/EvalEng huggingface dataset:
Repository: facebookresearch/DomainBed
Issue: about domain adaptation
Root cause: Validation Gap

Hi, when I run "python train.py --data_dir=../../dataset/ --algorithm ERM --dataset RotatedMNIST --task domain_adaptation --uda_holdout_fraction 0", I got the error below. I did't modify the code, is there someting wrong? What' the meaning of "uda_holdout_fraction"?  Thanks!
`Traceback (most recent call last):
File "/home/weiyuhua/Code/DomainBed/domainbed/scripts/train.py", line 211, in <module>
for x,_ in next(uda_minibatches_iterator)]
StopIteration`

Fixed in master. The argument `--uda_holdout_fraction` is the percentage of examples from the test domain that DomainBed will provide (unlabeled) to the algorithm during training, provided that `--task=domain_adaptation`. Note that the current implementation of algorithms do not use the unlabeled data at all, you would have to include that functionality.
Repository: facebookresearch/DomainBed
Issue: Severe decrease in MMD performance on DomainNet dataset

It seems the CORAL and MMD algorithms are very similar, and based on the "current result" table, in most datasets, they achieve close results.  However,  there is a big difference in their accuracy on the DomainNet dataset (41.5-23.4). I was wondering, is there any specific reason for this, or might it be a bug?

No known reason. Please re-open if you investigate further.
Repository: facebookresearch/DomainBed
Issue: IGA implementation
Root cause: Validation Gap

Thank you for the great work! Your code is highly functional and I hope it will become the go-to open-source software for domain generalization with fair comparisons.

My question is related to the current implementation of the IGA algorithm, in https://github.com/facebookresearch/DomainBed/blob/54c2f8c614a96067dee2d961f0b34575753c9df0/domainbed/algorithms.py#L986

```
for i, (x, y) in enumerate(minibatches):
    [....]
    grads.append( autograd.grad(env_loss, self.network.parameters(), retain_graph=True) )
    
mean_loss = total_loss / len(minibatches)
mean_grad = autograd.grad(mean_loss, self.network.parameters(), retain_graph=True)
[....]
        penalty_value += (g - mean_g).pow(2).sum()
[....]
(mean_loss + self.hparams['penalty'] * penalty_value).backward()
```

I believe there is a small error: a "create_graph" is missing when calling autograd.grad. 

In details, with the current implementation the penalty_value is useless in the training. As proof, if we replace the last "backward" by
```
(self.hparams['penalty'] * penalty_value).backward()
```
we obtain the error:
```
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
```

Another proof is the lack of impact of the hyper-parameter `self.hparams['penalty']`

The fix is simple: just add create_graph in autograd.grad, to explicitly say to do further operations on gradients, to have a backpropable graph for those gradients

```
for i, (x, y) in enumerate(minibatches):
    [....]
    grads.append( autograd.grad(env_loss, self.network.parameters(), retain_graph=True, create_graph=True) )
    
mean_loss = total_loss / len(minibatches)
mean_grad = autograd.grad(mean_loss, self.network.parameters(), retain_graph=True, create_graph=True)
```

Overall, I am still (currently) unable to reproduce the results from the paper, but I believe this is a step forward.

@jc-audet, as the contributor of the IGA implementation, could you please take a look?

Hello @alexrame,

Wow! Good catch! I was also having issue reproducing the results from the paper and the lack of open code from the paper made it hard to check my implementation. When I was coding it, I somehow managed to get ~60% test accuracy on CMNIST with the right timing of penalty activation and number of steps of training so I figured it was alright.

We should add this fix asap. I will create a PR with `create_graph` along with penalty annealing I have in my personal fork for IGA.

Sorry for the wait

Hello!
Thank you for your time and answers. That would definitely be nice if you could push your fork.
I am eager to see how you achieve 60%. It seems like, without the "create_graph", IGA should behave like ERM ...

Left a quick review, almost ready to merge @jc-audet .

Merged #51
Repository: facebookresearch/DomainBed
Issue: ERM results from a randomly initialized network.
Root cause: Unimplemented Feature Gap

The paper provides results for ERM using ResNet50 models initialized with ImageNet weights. Please let me know if there are results available for models trained from scratch as well.

It'll be interesting to see how much of an effect ImageNet weights have on the final results. Using an ImageNet initialization is in a way equivalent to indirectly using the entire ImageNet dataset for training. While this will not invalidate the claims in the paper as most DG methods use ImageNet weights as a starting point, training a model from scratch will give a fairer idea of the contribution of the ERM algorithm itself for DG.

We do not provide such results (as we believe in any realistic setting pre-trained networks would be used), but shall you run them and find something interesting, please reopen this issue.
Repository: facebookresearch/DomainBed
Issue: Training output and settings explaination

Hi!
Thank you for your repository. I'd like to have some clarification about the training output; setting for example test_envs to 0 I get this output:
![image](https://user-images.githubusercontent.com/74110683/119803055-6536b280-bedf-11eb-905f-5cb2a9446991.png)
What does the columns represent?
Moreover, what's the difference between different test_envs configurations?
Thank you for your help!


On which dataset are you evaluating? In the case of Office-Home for example, the environments `env0` to `env3` would respond to the domains `A`, `C`, `P`, and `R` respectively.

From section 5 in the paper:

> We split the data from each domain into 80% and 20% splits. We use the larger splits for training and final evaluation, and the smaller splits to select hyperparameters (for an illustration, see Appendix B.3)

`env_x_in` is the larger split, `env_x_out` is the smaller split.

This is the illustration from Appendix B.3:

![Screenshot 2021-05-27 at 12 05 24](https://user-images.githubusercontent.com/7551272/119807846-d5dfce00-bee3-11eb-99e1-803b9d30ad8b.png)


Ok thanks for your reply! So, the results on the testing domains are only given at the end of the whole procedure? (I'm using PACS)
Moreover, what about the test_envs parameter?


If you pass `--test_envs 0` to the training script, then that means you train on the remaining envs `1`, `2`, and `3` and use `0` only for testing. If you were to pass `--test_envs 0 1` to the training script, you would only train on `2` and `3` and use both `0` and `1` only for testing.

At each training checkpoint (in your case after every step, for PACS the default is every 300 steps) you evaluate the model on all splits of the data. This is necessary later for model selection.

For the model selection method Leave-one-domain-out validation, you need two test domains: you evaluate on one domain to find the best model, but then you report the accuracy on the other test domain.

The relation between dataset split, test environments and model selection is explained in Appendix B.3.

Thank you @daysm
Repository: facebookresearch/DomainBed
Issue: MLP with depth=1

Hi,
Thanks for this great library.
We encountered a minor issue: the MLP class does not support the case of depth=1. The minimum length for the MLP is 2 right now. It could be a nice addition to create linear layers.



Hi! Linear classifiers can be used by setting `hparams['nonlinear_classifier'] = False`, as implemented [here](https://github.com/facebookresearch/DomainBed/blob/master/domainbed/networks.py#L207).
Repository: facebookresearch/DomainBed
Issue: ColorMNIST domain error
Root cause: Validation Gap

In paper, the colormnist domain is {0.1, 0.3, 0.9}
![image](https://user-images.githubusercontent.com/39073236/124071134-7ee49000-da71-11eb-9524-b3f76386910b.png)


But in code, , the colormnist domain is {0.1, 0.2, 0.9}
![image](https://user-images.githubusercontent.com/39073236/124071203-97ed4100-da71-11eb-88c3-f17522cf22f5.png)


The original IRM paper considers {0.1, 0.2, 0.9}.
Repository: facebookresearch/DomainBed
Issue: cdpl parameters in SelfReg are not registered in the optimizer
Root cause: Validation Gap

In SelfReg algorithm, cdpl is defined in L 1031-1040. However these parameters are not appended to parameters in self.network or self.optimizer. This leads to no update in self.cdpl whatsoever. 
I went through the paper's main code base independent of domainbed at https://github.com/dnap512/SelfReg/blob/334af5fc2430d4183210956bc4d86d86816feeee/codes/model/resnet18_selfreg.py#L174
and observed that self.proj which is self.cdpl is directly registered to the model and hence optimizer will optimize these parameters too. 
Thank you

Hi, @kowshikthopalli !

First of all, thank you for your interest in our research.

It is true that the experiment was conducted with the current DomainBed code.
We also experimented with the CDPL initialized (untrained) in our code, and found interesting results.
The results are as below. (Data: PACS, Model: ResNet18)
![image](https://user-images.githubusercontent.com/44395361/127083636-e8451257-3b58-456f-994b-535add901586.png)

It is found that the performance difference between (with trained CDPL) and (with untrained CDPL) is not significant.
Maybe it's because SelfReg is not big enough to train CDPL.
We are currently discussing these results.

Thank you.

@kowshikthopalli Thank you for opening the issue.

@dnap512 Thank you for the clarification. Should we close?

@lopezpaz Yes, Thank you.

@dnap512 @lopezpaz  thank you for your efforts in replying to me in a timely manner. 
@dnap512  please update us if you can when you finish analyzing the results. 
A minor comment- why is cdpl called class-dependent ? There are no multiple cdpl s, nor does SelfReg impose any conditions on the class labels to apply a cdpl layer. It is thus just a projection layer right? 


@kowshikthopalli We will update our analysis to you soon.
You know, the CDPL is only trained by SelfReg, not classification loss. And SelfReg is MSEloss between positive pairs (the same class samples). Thus, we designed the CDPL to slightly perturb distribution of features, while preserving class-specific information.
But, on current condition, yes, it's a projection layer. 

@kowshikthopalli 

We experimented by increasing the learning rate of the CDPL, and confirmed the highest performance when the learning rate was 15 times larger than the existing learning rate. 
As mentioned before, it is probably that SelfReg is not large enough to train the CDPL.
The results are as below.
![이슈용](https://user-images.githubusercontent.com/44395361/128830083-c2357031-52f1-44ec-8481-0977a77739ad.PNG)

If you have any questions about our work, please use [this repository](https://github.com/dnap512/SelfReg).

Thank you for your interest!
Repository: facebookresearch/DomainBed
Issue: Rotated-MNIST results
Root cause: Documentation Deficiency

Would it be possible to provide the results for Rotated-MNIST for commit 7df6f06 with 2 digits precision?
Indeed, most methods are between 97.8 and 98.0 with the oracle model selection - complexifying the comparison.
Thank you in advance.

Full sweep logs available at: https://drive.google.com/file/d/16VFQWTble6-nB5AdXBtQpQFwjEC7CChM/view?usp=sharing
Repository: facebookresearch/DomainBed
Issue: Unable to get the code running
Root cause: Documentation Deficiency

Getting the following error when even the simple python3 -m domainbed.scripts.train is run:

FileNotFoundError: Found no valid file for the classes raw. Supported extensions are: .jpg, .jpeg, .png, .ppm, .bmp, .pgm, .tif, .tiff, .webp

Trying out for PACS dataset, and found that all image types are indeed proper (i.e. png)

We need more context, i.e., exact command, path of dataset...

Thanks for your response. While trying out your repo for the first time, I ended up running multiple commands in arbitrary order. In doing so, I recall running python -m unittest discover as well. I cannot confidently point out what exactly caused the issue. Very likely some hidden files got created inside the dataset directory, which when tried to be picked up by the dataloader might have given this error.

As a remedy I resorted to deleting my entire dataset directories, downloaded them again, ran the commands for launching the sweep.py , and then things started working well.

I just have few more questions:
1. Is it mandatory to use python3 for running python3 -m domainbed.scripts.download and python3 -m domainbed.scripts.train respectively ? When I tried python -m domainbed.scripts.download, it was not working

2. Also, is it necessary that the datasets need to be downloaded into --data_dir=./domainbed/data , or it's fine in downloading to them to any path in my local system ?

3. Perhaps it is there somewhere in the repo, but as I did not explicitly find this in the readme, it was bit confusing to me about the nature of the environment where the code needs to be run upon. I found the following to be helpful (hence, thinking of sharing here, so that if someone else finds a similar issue, it would be helpful to them):

conda create -p /<path_to_your_envs>/dgbed python=3.8
conda activate dgbed 
pip install gdown
pip install wilds
pip install parameterized

P.S. As my main issue of getting the code running is resolved, you may close this issue as well (but please do answer my above 3 questions. Apologies if something is obvious/ trivial. )

1. It is mandatory to use `python3`. Note that your script `python` can be already pointing to `python3`. To know this, execute `python` without arguments and take a look at the banner.
2. Any path is fine, as long as you specify it correctly using `--data_dir`.
3. Any `python3` environment with the necessary dependencies should work.
Repository: facebookresearch/DomainBed
Issue: Setting the Seed.
Root cause: Documentation Deficiency

Hi,
I am unable to clearly understand the seed setup.
If I understand correctly, you have set the various seeds once in the beginning in train.py and then multiple runs are executed using the same process. 
Is this correct or my interpretation is wrong.
Can you please elaborate on the exact protocol used.

There are two main seeds:

- The **hyper-parameter seed** decides the hyper-parameter values (learning rate, weight decay) for the algorithm.
- The **trial seed** decides everything else (initial weights, data partitions).

When sweeping, for each hyper-parameter seed we run a few trial seeds. This allows us to perform model selection across hyper-parameter seeds taking into account standard deviation (arising from the trial seeds, given each hyper-parameter seed).

> There are two main seeds:
> 
> * The **hyper-parameter seed** decides the hyper-parameter values (learning rate, weight decay) for the algorithm.
> * The **trial seed** decides everything else (initial weights, data partitions).
> 
> When sweeping, for each hyper-parameter seed we run a few trial seeds. This allows us to perform model selection across hyper-parameter seeds taking into account standard deviation (arising from the trial seeds, given each hyper-parameter seed).

Hi, I do not think so according to the code.
https://github.com/facebookresearch/DomainBed/blob/93ebb6bccdb46d40471ce0713a165aaf7b5e21b6/domainbed/scripts/train.py#L75-L79
From line 78-79, the hparams are controlled by both **hyper-parameter seed** and **trial seed**, which means that for each hyper-parameter seed we **do not** run a few trial seeds.

Looking forward to your reply soon. Thanks!
Repository: facebookresearch/DomainBed
Issue: Wrong batch size being used
Root cause: Documentation Deficiency

If I run the ERM on PACS via the following command:

`python -m domainbed.scripts.train --data_dir=datasets/ --algorithm ERM --dataset PACS --hparams='{"resnet18": "True"}'`

and print the `all_x` and `all_y` in the `ERM` algorithm like:

```
 def update(self, minibatches):
        all_x = torch.cat([x for x,y in minibatches])
        all_y = torch.cat([y for x,y in minibatches])
        print(all_x.shape)
        print(all_y.shape)
```
I get this output for the parameters and the sizes: 

```
Args:
        algorithm: ERM
        checkpoint_freq: None
        data_dir: ../../projects/DomainBed/datasets/
        dataset: PACS
        holdout_fraction: 0.2
        hparams: {"resnet18": "True"}
        hparams_seed: 0
        output_dir: train_output
        seed: 0
        skip_model_save: False
        steps: None
        test_envs: [0]
        trial_seed: 0
HParams:
        batch_size: 32
        class_balanced: False
        data_augmentation: True
        groupdro_eta: 0.01
        irm_lambda: 100.0
        irm_penalty_anneal_iters: 500
        lr: 5e-05
        mixup_alpha: 0.2
        mldg_beta: 1.0
        mlp_depth: 3
        mlp_dropout: 0.0
        mlp_width: 256
        mmd_gamma: 1.0
        mtl_ema: 0.99
        resnet18: True
        resnet_dropout: 0.0
        sag_w_adv: 0.1
        weight_decay: 0.0
torch.Size([96, 3, 224, 224])
torch.Size([96])
```

Since `all_x` and `all_y` represent batches of shape `batch size x C x H x W` shouldn't `batch_size = 32` instead of the batch size suggested by the shape `batch_size = 96` since it is stated with `32` as hyperparameter? 

Maybe I missed something since this is exactly multiplied by factor 3.  


`args.batch_size` is the batch size per training environment (in your example, a batch of 32 examples for each of the 3 training environments amounts to a total of 96 examples in `all_x`).
Repository: facebookresearch/DomainBed
Issue: Requirements up to date ?
Root cause: External Dependency Breakage

I just cloned the repo and tried to run the train command. However I got the following error:
`AttributeError: module 'torchvision.transforms' has no attribute 'InterpolationMode'`

Is the requirements.txt file up to date ?
Here is my setting: 
```
Environment:
        Python: 3.8.5
        PyTorch: 1.7.1
        Torchvision: 0.8.2
        CUDA: 11.0
        CUDNN: 8004
        NumPy: 1.20.3
        PIL: 8.3.2
Args:
        algorithm: ERM
        checkpoint_freq: None
        data_dir: ./domainbed/data/MNIST/
        dataset: RotatedMNIST
        holdout_fraction: 0.2
        hparams: None
        hparams_seed: 0
        output_dir: train_output
        save_model_every_checkpoint: False
        seed: 0
        skip_model_save: False
        steps: None
        task: domain_generalization
        test_envs: [0]
        trial_seed: 0
        uda_holdout_fraction: 0
HParams:
        batch_size: 64
        class_balanced: False
        data_augmentation: True
        lr: 0.001
        nonlinear_classifier: False
        resnet18: False
        resnet_dropout: 0.0
        weight_decay: 0.0
```

Hi @jmamath, I was the one who created the `requirements.txt` file through [this PR](https://github.com/facebookresearch/DomainBed/pull/71/commits).

I put the requirements file after training only coloredMNIST; I just saw that R-MNIST has this command.

Feel free to correct the requirements file (I believe the torch version and torchvision version are tightly coupled, and they are also coupled with the CUDA version).

Closing in favor of a PR (@ashok-arjun ?)
Repository: facebookresearch/DomainBed
Issue: Extending Domainbed code for a new algorithm

Thank you so much for providing such good and efficient responses to all issues. 
In the supp material of the paper, it is mentioned that:
**Algorithms are classes that implement two methods: .update(minibatches) and
.predict(x). The update method receives a list of minibatches, one minibatch per training
domain, and each minibatch containing one input and one output tensor**
what is one input and output tensor ? and also if you can provide a bit more explanation on how to integrate a new model by inheriting the ERM, it would be helpful. 


Check for instance the mixup implementation at:

https://github.com/facebookresearch/DomainBed/blob/main/domainbed/algorithms.py#L406-L435

Or any other method from that file.
Repository: facebookresearch/DomainBed
Issue: cannot access PACS dataset
Root cause: Configuration Error

I cannot download the PACS dataset and could you please have a check the provided download link?
https://drive.google.com/uc?id=0B6x7gtvErXgfbF9CSk53UkRxVzg

I can access it, try again after deleting your gdown cache, namely `rm -rf ~/.cache/gdown/`

The previous 'PACS' google drive link has expired due to google's security update (could be accessed by people who can access it before). 
The new link is updated as follows,
https://drive.google.com/drive/folders/0B6x7gtvErXgfUU1WcGY5SzdwZVk?resourcekey=0-2fvpQY_QSyJf2uIECzqPuQ&usp=sharing

Best,
Da

Oh, I see; updated the link. Thanks @zwvews and @dali-dl.
Repository: facebookresearch/DomainBed
Issue: Strange results on OfficeHome datasets
Root cause: Configuration Error

I downloaded the code and ran it according to REARME, and got the results as follows. I am strange that 1) leave-one-domain-out results were not shown, 2) oracle results are worse than that in "training-domain validation set" setting. I expect to ask that what is wrong with my training.
![image](https://user-images.githubusercontent.com/37394054/136639469-453260de-5f90-4e12-95b8-028fc2d5f1a5.png)


Did all jobs finish? Any content in stderr? What exact command did you use?

I guess that the reason that leave-one-domain-out cross-validation results are lacking is that only the experiments that leaving one domain as the test domain and setting the others as training domains are conducted. Otherwise,  stderr is missing since n_hparams=0, n_trials=0. 

Indeed, n_hparams=0 and n_trials=0 is insufficient to compute error bars or to find good hyper-parameters.
Repository: facebookresearch/DomainBed
Issue: Restrict the model selection criterion to just "train_domain_validation"
Root cause: Documentation Deficiency

Hi,

It has been observed that there are a significantly large number of jobs that are being launched while running the sweep. Wanted to know how can we restrict the model selection criterion to just "train_domain_validation" (for eg)?

Can you please give the precise command if possible ?




The sweep is a collection of jobs to be run. The model selection criteria is an strategy to pick models from a sweep. These are two independent processes. You can hack `sweep.py` to limit the amount of jobs to the ones you care about.
Repository: facebookresearch/DomainBed
Issue: office-home dataset download
Root cause: External Dependency Breakage

` download_office_home(data_dir)` function is not working.

The google drive link for downloading the office-home dataset seems to be broken.
https://drive.google.com/uc?id=0B81rNlvomiwed0V1YUxQdC1uOTg

Could you please help with this?

Thank you

Fixed.
Repository: facebookresearch/DomainBed
Issue: Would it be possible to run a new sweep?
Root cause: Unimplemented Feature Gap

The [last sweep](https://github.com/facebookresearch/DomainBed/blob/main/domainbed/results/2020_10_06_7df6f06/results.png) is from 14 months ago and since then 10 new algorithms have been added. These newly added algorithms generally present only partial or no results at all on DomainBed. To the best of my knowledge, [Fishr](https://arxiv.org/pdf/2109.02934.pdf) (Table 4) includes the most up-to-date sweep, but it still lacks 6 of the currently available algorithms.

I would be curious to get an updated overview of how the current set of algorithms compare to one another, but I personally do not have access to the compute to run such a sweep. Would it by any chance be possible for you to schedule a new sweep sometime soon?

We are indeed planning it, but given the amount of compute, we are first considering some refactoring of the code. We will update the repository as soon as we have new results available.