zhiminy/EvalEng huggingface dataset:
Repository: facebookresearch/DomainBed
Issue: CMNIST Accuracy
Root cause: Documentation Deficiency

Hello, 

I have tried reproducing the results for CMNIST that are attached to the image in the readme:

<img width="977" height="377" alt="Image" src="https://github.com/user-attachments/assets/6e279d11-c321-4dee-b75c-cd3cb2094bc7" />

but I am unable to do so. For ERM and many other methods, I get far below the average 51% reported (18% to 20% avg. across multiple trials).

Curious, I checked the IRM paper where CMNIST was introduced and it seems they get similar results as I:

<img width="1054" height="300" alt="Image" src="https://github.com/user-attachments/assets/0250fb3a-666d-4a7e-a788-66520a43280a" />

How exactly were the results for ERM generated?

It looks like they're taking the average of the +90% +80% and the -90% to come up with the figure.

<img width="775" height="286" alt="Image" src="https://github.com/user-attachments/assets/47595875-7251-4b0d-9743-f6f19f582321" />
Repository: facebookresearch/DomainBed
Issue: Saving and loading trained models 

Hi I was wondering how to save amodel and then use it later. Turned out that each training instance saves 'model.pkl' in to the directory and have 'algorithm.module' only. Is there a way to save the model and use it later as an standalone module. I am moving from Keras to Pytorch so kind of new to the functionalities. Appreciate an answer for this. 

```python
import torch
from domainbed import algorithms

def load_model(fname):
    dump = torch.load(fname)
    algorithm_class = algorithms.get_algorithm_class(dump["args"]["algorithm"])
    algorithm = algorithm_class(
        dump["model_input_shape"],
        dump["model_num_classes"],
        dump["model_num_domains"],
        dump["model_hparams"])
    algorithm.load_state_dict(dump["model_dict"])
    return algorithm

model = load_model("model.pkl")
```
Repository: facebookresearch/DomainBed
Issue: Get best performing hyperparameters / Re-run hyperparameters with different trial seed
Root cause: Unimplemented Feature Gap

As it is currently implemented, launching a sweep with `n_trials` > 1 allows for reporting std. deviation of the best performing hyperparameter setting based on the different validation techniques. If I correctly understood the implementation, this computes  `n_trials`  different in- and out-splits and runs them for each of the sampled hyperparameter settings. 

Users with limited resources who are implementing a new algorithm might not want to do that. A more convenient and faster way would be an option where users are able to tune hyperparameters and then run ONLY the best performing setting for each of the validation techniques on additional `n_trials` seeds to report std. deviations of choosing different in- and out-splits.

Also: What is the current way of obtaining the best performing hyperparameter setting for each of the validation techniques? 

I'm hesitant to support this case as a built-in for two reasons:
- the meaning of reported errors won't be consistent across papers (all else equal, more standardization is good)
- I believe that ignoring variability in the hparam selection process can lead to misleadingly good results, e.g. if you run the sweep multiple times and only report the best number. In theory you could get arbitrarily good numbers (and low stdev) this way, but they wouldn't be reproducible.

There's no easy way to see hparams corresponding to the best run, but a starting point for writing this functionality yourself would be [here](https://github.com/facebookresearch/DomainBed/blob/3fe9d7bb4bc14777a42b3a9be8dd887e709ec414/domainbed/model_selection.py#L34). You want to change this to something like `map(lambda _, run_records: (self.run_acc(run_records), run_records[0]["hparams_seed"]) )` and then propagate that hparams_seed all the way to the final results table.

Yeah I can see the reasons, thanks for giving a starting point! 

Just wanted to flag for you that this is implemented now in https://github.com/facebookresearch/DomainBed/blob/master/domainbed/scripts/list_top_hparams.py . It's still experimental so there might be bugs / frequent breaking changes.

Ah cool, I actually had a quick implementation running for myself as well. Yours seems a little cleaner though!
Repository: facebookresearch/DomainBed
Issue: Getting access to domain knowledge
Root cause: Documentation Deficiency

Currently, samples from the training domains are just passed as `minibatches` to the algorithms in `algorithms.py`. If I'm correct, this leaves the implemented algorithms no option for splitting them into the different domains and revokes access to domain knowledge. However,  this is a common strategy for domain generalization algorithms.

Am I missing this option or how would one go about implementing this? Can one split the minibatches based on `num_domains` to reconstruct the domains or are they shuffled?

@SirRob1997 Hi, did you find the answer to your question posted above?


It's been a while since I've worked with this but if I remember correctly you can do something similar to what is done in the `AbstractDANN` class in [this line](https://github.com/facebookresearch/DomainBed/blob/main/domainbed/algorithms.py#L250).
Repository: facebookresearch/DomainBed
Issue: An issue related to model selection

Thanks for your work and code framework. 

When I run your code with the default hyperparameters, I find the performance of oracle selection is weaker than that of the training domain selection. 

Is there anything wrong?

 
![1381603161814_ pic_hd](https://user-images.githubusercontent.com/26360112/96533817-2396b700-12c1-11eb-8141-7f8a33a090f6.jpg)


Could you let us know exactly what you ran (commit hash and command strings) to produce this result?

I seem to know what's wrong. For the oracle selection, the code only saves the model after finishing training. Just running one model may perform poorly (This is exactly what I did). Therefore, for oracle, you run 20 models with different hyperparameters, and select the best hyperparameter.

Why not also use the validation set to save the optimal model during the training?



In short, querying the test set too many times makes the results uninformative (e.g. with infinite queries you could just brute-force overfit). We wanted the number of test-set queries to be relatively small and decoupled from things like the checkpoint frequency. We go into more detail in the paper.

Then why do you post the best performance instead of the final performance? Essentially, I think the two are the same, although I agree with the situation of brute-force overfit.

I'm not sure what you mean. We collect final-timestep accuracies of all models and report the max over these.

I mean in the IID Accuracy Selection

Oh, in that case we don't use the test set at all to select checkpoints; we use a validation set split off from the training data. This means it's fair game to query the validation set as much as we want.

Okay. Thanks for your quick answer.
Repository: facebookresearch/DomainBed
Issue: Implement DFAS
Root cause: Unimplemented Feature Gap

[This ICLR 2021 submission](https://openreview.net/pdf/27ab253487ff63c70073277ac34d59cee897ec6f.pdf) shows great results on PACS and OfficeHome. It would be cool to try to replicate these numbers.
Repository: facebookresearch/DomainBed
Issue: About MLDG's update()

This was written in the MLDG's update function.
"TODO: update() has at least one bug, possibly more. Disabling this whole algorithm until it gets figured out."
Maybe this bug is about second-order derivatives?
If not, please explain what kind of problem this bug is about.

We fixed a couple of bugs in MLDG in the past, and we forgot to remove this comment. Thank you.
Repository: facebookresearch/DomainBed
Issue: Extension to Unsupervised Domain Adaptation

Thank you for your research and work on DomainBed.

In Appendix E.1 you mention the possibility of extending the method `.update(minibatches, unlabeled)` to accept a minibatch of unlabeled examples.

Could you explain how you think this extension would be implemented best? Would `unlabeled` specify whether `minibatches` is unlabeled? I'm working on extending the `.update` method of `AbstractDANN` for UDA.

What I had in mind is redefining all update methods to `.update(minibatches, unlabeled=None)`. When `unlabeled=None`, DomainBed operates as usual (domain generalization). Otherwise, `unlabeled` is expected to contain a minibatch of unlabeled examples from the test domain, and it is up to each of the algorithms to leverage it to their advantage (domain adaptation).

The second change would be to add a flag `--domain_adaptation` to `train.py`, which would split the test loader into the unlabeled examples used for training and the actual test set used for evaluation (and oracle model selection).

Done in b953488d4dcfcc76427f07958b133b87d24a48e5.
Repository: facebookresearch/DomainBed
Issue: Interpreting results after training algorithms

I  tried to train the algorithm ERM on PACS dataset. The final logs at the end of training are as follows:

env0_in_acc   env0_out_acc  env1_in_acc   env1_out_acc  env2_in_acc   env2_out_acc  env3_in_acc   env3_out_acc  epoch         loss          step          step_time

0.8377059182  0.8239608802  0.9845415778  0.9423076923  0.9947604790  0.9670658683  0.9812340967  0.9299363057  3900.0000000  0.0155880866  3900          0.5439552005

However, I do not understand what do they represent. If out_acc represents the out of domain accuracy then the numbers do not make sense, since its greater than 90 percent for 3 domains, while in PACS only the test domain Photo obtains more than 90 percent out of domain accuracy

How do I obtain the out of domain accuracy on individual test domains ( Photo, Art Painting, Cartoon, Sketch )?

`envX_in_acc` and `envX_out_acc` are classification accuracies for the `in` and `out` splits for environment `X`, respectively. The `out` split is often smaller, containing `args.holdout_fraction` of the examples for that environment. When environment `X` is a training domain, the `in` split is used for training, and the `out` split is used for model selection. When environment `X` belongs to the testing domains, the `in` split is used for computing the final test accuracy, and the `out` split is used for model selection (only oracle).

Since we sort domains alphabetically, in your example the numbers represent:
```
env0_in_acc (domain A, in split)
env0_out_acc (domain A, out split)
env1_in_acc (domain C, in split)
env1_out_acc (domain C, out split)
env2_in_acc (domain P, in split)
env2_out_acc (domain P, out split)
env3_in_acc (domain S, in split)
env3_out_acc (domain S, out split)
```
Without knowing your `args.test_envs` I cannot say which one is your *test domain*, but it should be easy to find out for you.
Repository: facebookresearch/DomainBed
Issue: 2 test environments?

Hi, really cool repository :)

In the sweep, the all_combinations consider both 1 and 2 test envs. However, Appendix B of your paper seems only to consider the 1 test env (indicated by each column). Could you elaborate on the missing part of my understanding?

Best regards,

Found out that it is for leave-one-domain-out cross-validation

@lopezpaz Could you please confirm this?
Thanks
Repository: facebookresearch/DomainBed
Issue: Could you share the selected hyperparameters to reproduce the results?
Root cause: Documentation Deficiency

Hi, thanks to your great work.

I'm trying to reproduce the results reported in the paper. For the ERM, the default hyperparameter setting yields similar results as reported in the PACS, but it shows a little worse in the VLCS than reported. I suspect the difference come from the hyperparamters, so I wonder the searched values. Could you share the selected hyperparameters, at least for the ERM?

See: https://github.com/facebookresearch/DomainBed/blob/master/domainbed/scripts/list_top_hparams.py

@lopezpaz Thanks to fast answer. I know the script, but there is no log file in this repo and I have no resource to search hparams myself. Do you have no plan to share the search results, for people who lack resource like me?

@khanrc Here you go all the logs from the last public sweep: https://drive.google.com/file/d/16VFQWTble6-nB5AdXBtQpQFwjEC7CChM/view?usp=sharing

Thanks a lot! :)
Repository: facebookresearch/DomainBed
Issue: Implement checkpointing
Root cause: Unimplemented Feature Gap

The way it is currently implemented, we save the model parameters [after the last training step](https://github.com/facebookresearch/DomainBed/blob/master/domainbed/scripts/train.py#L196) is finished. However, this doesn't conveniently allow users to access the best performing model parameters for different validation techniques. In fact, I think only the parameters for oracle validation can be accessed this way, as early stopping is disabled there and with the [printing of the best-performing hyperparameters](https://github.com/facebookresearch/DomainBed/blob/master/domainbed/scripts/list_top_hparams.py), we can obtain the respective tuning step. 

To support the other validation techniques, it would probably be helpful to have a flag where we persist model parameters at each validation checkpoint. Once we run [`collect_results.py`](https://github.com/facebookresearch/DomainBed/blob/master/domainbed/scripts/collect_results.py), we might also have an option to prune these model checkpoints and only keep the ones that correspond to the best-performing one for a specific validation technique and test environment. This would help to save disk space (we keep only <img src="https://render.githubusercontent.com/render/math?math=\mathrm{num_domains} \times \mathrm{num_validation_techniques}"> model checkpoints) and make it even more convenient to browse through them.

Maybe there is even an option of implementing this that would be better.

I've added a flag to train.py; see https://github.com/facebookresearch/DomainBed/commit/32cbebbb719aec259b4470522d9c8efea27b1624 . I'm hesitant to add support for this to sweep.py because the changes are complicated and even a single sweep would end up generating quite a lot of weight files (also, it's best if we can keep collect_results.py idempotent). As an alternative, I suggest running a sweep and re-running specific hparam combinations whose weights you want to inspect in more detail. Closing for now; feel free to reopen if you see any problems.
Repository: facebookresearch/DomainBed
Issue: Multi-GPU Launcher
Root cause: Unimplemented Feature Gap

It's a very excellent repo!  Thanks for building it.

But I am wondering if there's a way to add a multi-GPU launcher to "command_launchers.py" for more convenient use.

For E.g. If I have an 8-GPUs machine, and I want to launch 8 jobs every time for the 8 GPU.

Is there a way to do it? or some ideas that I would then contribute some efforts to build one.

https://github.com/facebookresearch/DomainBed/blob/cecd7693929ddf8ce23fc9909dff546c2291b20a/domainbed/scripts/sweep.py#L77

I modified these lines in "sweep.py", from line 80 to line 99. 

I am not sure if this would be appropriate. At least, they work well on my system.

If so, would I be able to create a PR?
```
        all_gpu_counts = torch.cuda.device_count()
        commands_per_gpu = {}
        for i in range(all_gpu_counts):
            commands_per_gpu[i] = []

        idx = 0
        while idx < len(commands):
            for i in range(all_gpu_counts):
                if idx < len(commands):
                    commands_per_gpu[i].append("CUDA_VISIBLE_DEVICES={} {}".format(i, commands[idx]))
                    idx += 1

        multi_gpus_processes = []
        for i in range(all_gpu_counts):
            p = Process(target=launcher_fn, args=(commands_per_gpu[i],))
            p.start()
            multi_gpus_processes.append(p)

        for p in multi_gpus_processes:
            p.join()

Hi, thanks for the interest in contributing! I think this is a good idea. A cleaner way to implement this would be as a command launcher in command_launchers.py. Just write a function that runs each command in a list, and add the function to the registry at the bottom of the file. You can use your command launcher by running sweep.py with the --command_launcher flag.

I've gone ahead and taken a first shot at an implementation at https://github.com/facebookresearch/DomainBed/commit/56196cc5f8a8d53d3d7f547e385c443cbcb73dcd . Hope this helps!

Thank you @igul222. Please re-open if necessary, @Luodian.
Repository: facebookresearch/DomainBed
Issue: About test-domain validation set Score...
Root cause: Documentation Deficiency

Thanks for making a great library.
However, while I was measuring the performance of the algorithm with sweep.py, I found that the test-domain validation set came out lower than the training-domain validation set.
It's not just my algorithm, it's common in the existing sweep data.
I think this is not normal, Any idea why this is happening?

The "test-domain validation" metric [is limited to one query per hyper-parameter (in fact, the last epoch)](https://github.com/facebookresearch/DomainBed/blob/master/domainbed/model_selection.py#L69) in order to avoid extremely optimistic results. Therefore, it could return worse-than-optimal results. If we allowed the "test-domain validation" metric to peek at all epochs for all hyper-parameters, it would almost surely report better results than "training-domain validation" in all cases.
Repository: facebookresearch/DomainBed
Issue: Different results in the README page and the paper

Hi,

I may have missed something, but it seems what's reported in the "Current results" section in the README page is different from Table 4 in the original paper. I wonder has it been updated, and how?

Thank you!

The last version of the paper is the one accepted in ICLR 2021 (https://openreview.net/forum?id=lQdXeXDoWtI), uses commit 7df6f06, and matches the results from the README.md file.

Thanks for responding so quickly! You might wanna update the link in the README file though.

sorry to open this again. There is quite a discrepancy in the results reported in arxiv version and the open review version atleast for VLCS and other datasets. Just to be clear, I will have to compare my results with openreview version itself correct?


Openreview version for now, it is the latest. Thanks.
Repository: facebookresearch/DomainBed
Issue: Predefined groups for training groupDRO
Root cause: Documentation Deficiency

Hello,
Thank you very much for this amazing comparison. I have slight confusion regarding the implementation of groupDRO. 
In the [groupDRO](https://arxiv.org/abs/1911.08731) paper authors carefully generate the training datasets to contain a mixture of groups that are defined by leveraging human knowledge (identify some spurious correlations).

However, in this repository and comparison in the paper I fail to see such a construction of predefined groups for groupDRO. As far as I understand the [implementation,](https://github.com/facebookresearch/DomainBed/blob/master/domainbed/algorithms.py) dataset is **randomly** sampled and each sample in the minibatch is considered to arrive from one particular training group.  I understand that predefining groups on a new dataset is difficult but is it justifiable to compare groupDRO with such a relaxation. I tried to look into the paper associated with this repo, however, I did not see any mention on the same.  

Thank you so much for your time and patience!

This repository deals with domain generalization problems. That is, algorithms receive minibatches = ((X1, Y1), ..., (XG, YG)), where (Xg, Yg) is a minibatch of labeled examples from domain (group) g, for all g = 1, ..., G.
Repository: facebookresearch/DomainBed
Issue: About reproduced RSC result in DomainBed

Hi, I want to know about RSC results(reproduced on DomainBed) experimented on datasets(PACS,OfficeHome,VLCS) in DomainBed. I can't find specific results of RSC(results for each domain) in your paper.
If you can, could you show me results like this form?
     A              C             P               S
88.1±0.1  77.9±1.3  97.8±0.0  79.1±0.9

Here are what I want to know!


PACS

   
1. Average accuracy of each domain (ResNet18 with RSC)
   
2. Average accuracy of each domain (ResNet50 with RSC)
   

OfficeHome

   
3. Average accuracy of each domain (ResNet18 with RSC)


4. Average accuracy of each domain (ResNet50 with RSC)   


VLCS

   
5. Average accuracy of each domain (ResNet18 with RSC)     


6 .Average accuracy of each domain (ResNet50 with RSC)    




  


For ResNet50 results, see Appendix C in https://openreview.net/pdf?id=lQdXeXDoWtI

We only have ablation studies (e.g., experiments with ResNet18) for ERM.
Repository: facebookresearch/DomainBed
Issue: About the IRM implementation

Hi,

Thanks very much for this amazing project! 

I found the [penalty term](https://github.com/facebookresearch/DomainBed/blob/7b0c1cecfdaa6d06a11b58317385b8950a236251/domainbed/algorithms.py#L237) in IRM is summed over batch samples while the [ERM term](https://github.com/facebookresearch/DomainBed/blob/7b0c1cecfdaa6d06a11b58317385b8950a236251/domainbed/algorithms.py#L254) is averaged. Since both terms are averaged in original [IRM code](https://github.com/facebookresearch/InvariantRiskMinimization/blob/fc185d0f828a98f57030ba3647efc7394d1be95a/code/experiment_synthetic/models.py#L68), may I know whether there is any particular reason behind this implementation?

Thank you for your time.
Repository: facebookresearch/DomainBed
Issue: Incorrect reference to CDANN paper in Readme

I suppose the reference to the CDANN paper in Readme is wrong. It points to the following paper (https://arxiv.org/abs/1807.08479), which actually uses MMD in their method, and not domain discriminators that are used in the implementation of CDANN in DomainBed ([code](https://github.com/facebookresearch/DomainBed/blob/master/domainbed/algorithms.py#L214)).

I think the correct reference to the approach CDANN should be the following paper, which uses domain discriminators.

Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks.ECCV, 2018d  ([Link](https://openaccess.thecvf.com/content_ECCV_2018/papers/Ya_Li_Deep_Domain_Generalization_ECCV_2018_paper.pdf)).
Repository: facebookresearch/DomainBed
Issue: Different Train/Val split for PACS
Root cause: Documentation Deficiency

Hi, I find that you do not use the default train/val split for PACS data-set, and for this data set we usually follow the original train/val split by the author, will this cause the problem?

As described on the paper, we avoid fixed train-test splits as these lead to overfitting. Random train-test splits also allow us to compute error bars over splits.
Repository: facebookresearch/DomainBed
Issue: About algorithm MMD

I'm wondering if you really implement the "Domain Generalization with Adversarial Feature Learning" the MMD algorithm?
Since it seems no adversarial autoencode in the code?

That's correct, for us MMD only uses MMD. The alternative using an adversary is DANN. We will clarify.

> That's correct, for us MMD only uses MMD. The alternative using an adversary is DANN. We will clarify.

Got it! 
SO can I understand that the feature extraction is not so important in this paper, so you keep the MMD part (as I found that in the code AbstractMMD part)