zhiminy/EvalEng huggingface dataset:
Repository: facebookresearch/DomainBed
Issue: Add Adaptiope Dataset
Root cause: Unimplemented Feature Gap

Hi, 

Thanks for the framework first of all. Very recently I came across the adaptiope dataset which seems like a new benchmark. Can you add this ? 

Link to paper: https://openaccess.thecvf.com/content/WACV2021/papers/Ringwald_Adaptiope_A_Modern_Benchmark_for_Unsupervised_Domain_Adaptation_WACV_2021_paper.pdf

Hi @iamharsha1999, this repository is not under active development. However if you'd like to add this dataset feel free to submit a PR, and we will take a look.
Repository: facebookresearch/DomainBed
Issue: Is there a way to adapt domainbed for single domain generalization?
GitHub labels: feature request

Is there a way to adapt domainbed for single domain generalization?

It seems like a little bit of change in the train.py script can start the training but the collect functions to display won't work

Hi @williamium3000 , this would be a great contribution! If you're interested, feel free to submit a pull request!
Repository: facebookresearch/DomainBed
Issue: cannot download the terra incognita dataset
Root cause: Configuration Error

Thanks for the great effort in domain generalization. Now, I cannot download the terra incognita dataset even using the newest source at https://[lilablobssc.blob.core.windows.net/caltechcameratraps/eccv_18_all_images_sm.tar.gz](https://lilablobssc.blob.core.windows.net/caltechcameratraps/eccv_18_all_images_sm.tar.gz).

I have encountered the same issue; have you resolved it yet?

Hi @bugwriterLee , looking into it. 

https://lila.science/datasets/caltech-camera-traps

It seems like it is here, under "CCT20 Benchmark subset". I will edit the download script when I get a moment. 

Thanks!

Fixed in 3cf2e20fb885f3ecf2e97b83268bb6c896568075. Feel free to re-open if there are further issues.
Repository: facebookresearch/DomainBed
Issue: please provide a standard format to train custom dataset on domainbed
Root cause: Documentation Deficiency


@welcome-to-my-webpage ; closing for now due to inactivity and lack of detailed issue description. Feel free to re-open with more detail.  

In my case, I have a dataset with 35 classes equally distributed over 4-5 domains, I would like to know the exact procedure to test the algorithms provided with my data and get the results in the table specified. 

In the paper, there was a lot of emphasis over the model selection, if I already have an efficient net model pre-trained on the ImageNet1k would the procedure differ?
Repository: facebookresearch/DomainBed
Issue: Jobs for leaveone-out model selection

Hi,
Thanks for the library. In cases where a job includes multiple test environments e.g. `test_envs: [0,1]`, which method selection uses the results of these jobs? From what I understood, only training and oracle use them. Is that correct?

Hi @AmirEstiri , currently none of the model selection methods are compatible with multiple test environments. You can see in https://github.com/facebookresearch/DomainBed/blob/main/domainbed/model_selection.py#L6, that all records are filtered to only have a single test env.
Repository: facebookresearch/DomainBed
Issue: An issue about the way to calculate the mean / error on one test env.
Root cause: Documentation Deficiency

Hello, I have a question about the calculation method of the mean / error on one test env.

According to the code,  the calculation steps are

1. the test acc of one trial is from the hparams with max val acc across 20 different hparam sets in this trial.
2. the mean / error of test acc is calucated across multiple trials.

The steps expose an issue that **the test acc of different trials are from trained model with different hparam values**,  I think the test acc of each trial  should be based on the same hparams.

And the calculation steps **SHOULD** be
1. for each hparam set, calculate the mean of val acc across multiple trials, also the mean / error of test acc accross multiple trials
2. The final mean / error of test acc is from the hparam set with the max mean of val acc 

Welcome to point out my mistakes and looking forward to your reply. Thanks.

Hi @bad-meets-joke , this is a design choice discussed in the paper (https://arxiv.org/pdf/2007.01434, Section 5, **Standard Error Bars**). The general idea is that the entire process of splitting the data, initializing the model, and doing the hyper-parameter search IS the learning algorithm, just that some parts are search-based (hyper-parameters) and some parts are optimization based (eg ERM). 

The reason this kind of reasoning is important is that it allows one to test the robustness of the whole system. For example, maybe one set of hyper-parameters is good for algorithm A, and another set of hyper-parameters is good for Algorithm B. Using your proposed methodology, we sample 20 sets of hyper-parameters. It's not unlikely that amongst those 20,  Algorithm A hits the jackpot hparams. Simply sampling the hyper-parameters once, and rerunning across multiple trials for each hparam set (as you propose) would unfairly show that Algorithm A is better.  

In summary, the error bars/mean show **how the system as whole** performs, not just the **optimization part**. 

Let me know if you have additional questions. 

Closing for now, feel free to reopen with more questions.
Repository: facebookresearch/DomainBed
Issue: A question about in/out accuracy

Hello. I have a question about in/out accuracy when measuing performance.

If there are n domains for a dataset, we divide each domain into a train set (in) and a test set (out). Then, train the model using the train data of the domains except the target domain.
After model selection, when measuring performance, why do we evaluate with the test_in_acc_key of the target domain, i.e., the train set?
Why not the whole dataset and why not the test set?

Thank you!

Hello, I think there is a bit of confusion here.

This is a good thread to a similar question explaining what in/out accuracy means
https://github.com/facebookresearch/DomainBed/issues/53#issuecomment-849509189

If I understood correctly, `env_x_out` is a smaller split used to identify hyperparameters and `env_x_in` is for training and final evaluation. Hope this helps.

Yes, @abalhomaid is correct. This is a design decision, and including the full dataset could make checkpoint selection difficult for some algorithms (you need some validation data for this). Closing. Feel free to re-open if there is still confusion.
Repository: facebookresearch/DomainBed
Issue: Implementation of WILDS-FMoW dataset in DomainBed
Root cause: Documentation Deficiency

Hi @lopezpaz,

I was wondering what do [Region 0...5 ](https://github.com/facebookresearch/DomainBed/blob/93ebb6bccdb46d40471ce0713a165aaf7b5e21b6/domainbed/datasets.py#L358C53-L358C53) proposed in the definition of WILDS-FMoW stands for? Are they mapped to the regions proposed in WILDS such as Asia, Europe, Americas, Africa, Oceania and Others? If so, in which order are they mapped?

Thanks.

Regards,
Sara A. Al-Emadi

Hi, I looked into  the WILDS code (https://github.com/p-lambda/wilds/blob/472677590de351857197a9bf24958838c39c272b/wilds/datasets/fmow_dataset.py#L90).

it looks like it is 

```{0:'Asia', 1:'Europe', 2:'Africa', 3:'Americas', 4:'Oceania', 5:'Other'}```

Hi @piotr-teterwak,

Thanks for your reply. As far as I understand, FMoW-WILDS code defines domains in terms of years to address the DG problem (as shown in WILDS code https://github.com/p-lambda/wilds/blob/472677590de351857197a9bf24958838c39c272b/wilds/datasets/fmow_dataset.py#L77C13-L77C45). Whereas, regions are used to display the subpopulation shift (without explicitly splitting the data with respect to regions as described here: https://github.com/p-lambda/wilds/issues/138).

Therefore, could you kindly confirm if your implementation is aligned with FMoW-WILDS's definition for DG (i.e. domains are defined in terms of years)?

Hi @saraalemadi , the DomainBed implementation does not use the years as domains, and instead uses regions. However, if you'd like to add another FMoW-WILDS-years dataset to the codebase and issue a pull request, it would be a great contribution!
Repository: facebookresearch/DomainBed
Issue: DANN Gradient Reversal Layer Missing

I am interested in the DANN model by Ganin. In the paper, they use a Gradient Reversal Layer(GRL)  to reach the saddle point using single SGD updates. I don't see any mention of GRL in this repository so interested in how you implemented DANN.

I think the implementation uses a hparam `lambda` for this, which defaults to 1. In this case, the expression becomes `classifier_loss - disc_loss`, which has the same behavior as GRL.

https://github.com/facebookresearch/DomainBed/blob/93ebb6bccdb46d40471ce0713a165aaf7b5e21b6/domainbed/algorithms.py#L287-L288

@Bogay is correct, closing. Feel free to re-open if there are further questions.
Repository: facebookresearch/DomainBed
Issue: Mistake in collect_results.py with --single_test_envs sweep option
Root cause: Validation Gap

# Summary
model_selection.py:90 uses test environment numbers in `test_in_acc_key = 'env{}_in_acc'.format(test_env)` but if we have two environments: env0 and env1, the error `KeyError: 'env3_in_acc'` is produced.


# Replication steps
- Pick a dataset with two domains
- Run sweep with as few trials as possible like the following 
  - <code>python -m domainbed.scripts.sweep launch\
       --data_dir=./domainbed/data/\
       --output_dir=./my/sweep/<two_domain_dataset>\
       --algorithms CORAL\
       --datasets <two_domain_dataset>\
       --n_hparams 1\
       --n_trials 1\
       --seed 0\
       --holdout_fraction 0.1\
       --command_launcher local\
       --skip_confirmation
       --single_test_envs</code>
- Sweep produces 4 directories under `./my/sweep/<two_domain_dataset>`
- Each directory has out.txt with the following accuracies: `env0_in_acc   env0_out_acc  env1_in_acc   env1_out_acc`
  - _**test environments**_ are numbered with [0], [1], [2], [3]
- collect_results.py:170 uses model_selection IIDAccuracySelectionMethod
- model_selection.py:90 uses the _**test environment**_ numbers in `test_in_acc_key = 'env{}_in_acc'.format(test_env)` but we only have env0 and env1

Hence the error `KeyError: 'env3_in_acc'` is produced.

The mistake is on my side, I specified 4 environments under datasets.py while I trained on two. Closing the issue
Repository: facebookresearch/DomainBed
Issue: about cmnist results

Why is the result on dataset CMNIST so low, only about 50%?



The spurious correlation of color with class label is apparently very easy to learn, therefore reducing performance. If you have a source which would suggest a higher performance, I can look into the differences. 

Closing due to inactivity/age. Feel free to re-open if needed.
Repository: facebookresearch/DomainBed
Issue: StopIteration error when training on OfficeHome

I am trying to train on OfficeHome but get this error for some of the trials (I am running 16 hparams x 3 seeds). 

`python -m domainbed.scripts.train --algorithm ERM --data_dir /home/datasets --dataset OfficeHome --holdout_fraction 0.2 --hparams_seed 9 --output_dir output/OfficeHome_ERM/e788b9ef90401eb7d1aab8c0e5408488 --seed 1556291245 --task domain_generalization --test_envs 0 3 --trial_seed 2`

```
Traceback (most recent call last):
  File "/home/users/.conda/envs/domainbed/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/users/.conda/envs/domainbed/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/data/projects/domainbed/scripts/train.py", line 225, in <module>
    for x,y in next(train_minibatches_iterator)]
StopIteration

```

train_minibatches_iterator appears to be empty for some reason. Would appreciate any help to debug this! Thanks.

Apologies, the issue was due to how I had downloaded the OfficeHome dataset. It was not due to DomainBed.
Repository: facebookresearch/DomainBed
Issue: Adding Learning Rate Schedules

Since optimizers are currently defined in `algorithms.py` and used in the `update` method, where would one add a learning rate schedule e.g. `torch.optim.lr_scheduler.ReduceLROnPlateau` to keep the design patterns?

The cleanest would be creating a class `Optimizer(parameters, hparams)` in a new file `optimizers.py`, and extend `hparams_registry.py` such that learning rate scheduler flags are active when necessary. Feel free to PR in such a way that all default values are kept and schedulers are deactivated by default.

Sounds good, I'm on it
Repository: facebookresearch/DomainBed
Issue: 如何对Camelyon17的处理
Root cause: Documentation Deficiency

看到您datasets.py里面对Camelyon17的处理代码：
class WILDSCamelyon(WILDSDataset):
    ENVIRONMENTS = [ "hospital_0", "hospital_1", "hospital_2", "hospital_3",
            "hospital_4"]
    def __init__(self, root, test_envs, hparams):
        dataset = Camelyon17Dataset(root_dir=root)
        super().__init__(
            dataset, "hospital", test_envs, hparams['data_augmentation'], hparams)
我下载下来的Camelyon17的数据集没有hospital_0", "hospital_1", "hospital_2", "hospital_3",
            "hospital_4"，是要运行datasets.py来对之进行划分吗

Hi @yuu-Wang , is this a duplicate of https://github.com/facebookresearch/DomainBed/issues/151?
Repository: facebookresearch/DomainBed
Issue: Mixup algorithm data flow problem
Root cause: Unimplemented Feature Gap

Thanks for ur excellent and laborious work!
I was looking for the code of Mixup, and got to this respository
I have a question for the data flow: In the setting of domain adaptation, the 'uda_device' means the data without labels in target domain in ur file 'train.py'L211-L212. While if using the algorightm Mixup which inherits from ERM, the method 'update' of both class donot use the unlabeled data which is exactly the 'uda_device', the original paper indicated both domains(source and target) are used to produce new data.
Is there something wrong with my understanding?
Thanks for ur work again and Looking for ur reply

Hi, as noticed, the code base has scaffolding for domain adaptation but it is not yet enabled for any algorithm. If you would like to enable this, feel free to submit a PR. Closing for now.
Repository: facebookresearch/DomainBed
Issue: Unable to download dataset
Root cause: Documentation Deficiency

Hello, I can't download the data by this command, what is the reason, looking forward to your answer, thank you!
‘python3 -m domainbed.scripts.download \--data_dir=./domainbed/data’

Hi, what is the error? 

Hi,the command was executed with no response·



Sent&nbsp;from&nbsp;my&nbsp;iPhone


------------------ Original ------------------
From: Piotr Teterwak ***@***.***&gt;
Date: Thu,Dec 12,2024 0:09 AM
To: facebookresearch/DomainBed ***@***.***&gt;
Cc: ChenJing036-hub ***@***.***&gt;, Author ***@***.***&gt;
Subject: Re: [facebookresearch/DomainBed] Unable to download dataset (Issue#157)





 
Hi, what is the error?
 
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

![3c21225358090ae266eb51029fbbfc7](https://github.com/user-attachments/assets/4c55677c-e1c6-43c0-b626-9a3d18c20cb3)
Hi,the command was executed with no response.

Is there anything in  the `./domainbed/data` directory?
Repository: facebookresearch/DomainBed
Issue: Are the MTL embeddings being overwritten?
Root cause: Algorithmic Error

In the `MTL.predict()` function, the method `self.update_embeddings_(features, env)` is called, and it returns a weighted average of the features. However, immediately afterward, the .normal_() function is called without specifying the mean or standard deviation. This causes the embeddings returned by self.update_embeddings_() to be overwritten with a standard normal distribution, effectively replacing the previously computed weighted average. I don't believe this is intended.

Original
```python
def predict(self, x, env=None):
    features = self.featurizer(x)
    embedding = self.update_embeddings_(features, env).normal_()
    return self.classifier(torch.cat((features, embedding), 1))
```

Assuming the intent is to build a distribution around embeddings, here are two possible solutions (assuming std=1)
```python
# Allow update_embeddings_() to be differentiable
def predict(self, x, env=None):
    features = self.featurizer(x)
    embedding = self.update_embeddings_(features, env)
    embedding = embedding + torch.randn_like(embedding)
    return self.classifier(torch.cat((features, embedding), 1))

# Don't allow update_embeddings_() to be differentiable
def predict(self, x, env=None):
    features = self.featurizer(x)
    embedding = self.update_embeddings_(features, env)
    embedding = torch.normal(embedding, 1)
    return self.classifier(torch.cat((features, embedding), 1))
```

Thanks for the benchmark, by the way!

Great catch! I'm currently a bit busy but will fix it soon (or, if you want to give a pull request and run unit tests, that would be amazing!) 

I'd be happy to help with a pull request. However, I struggled to fully grasp the 'Domain Generalization by Marginal Transfer Learning' paper, and the current MTL implementation seems quite different from what I expected. If you could clarify the intended behavior of the embeddings, I'd be happy to take a crack at the fix. However, if it’s quicker for you to address it directly, feel free to go ahead.
Repository: facebookresearch/DomainBed
Issue: Guarantee that sampled hyperparameters are the same for future commits
Root cause: Architectural Constraint

Just opening this here to keep track of the issue mentioned in #12:

"We will have to find a solution to guarantee that the same random parameters are sampled in `hparams_registry` given the same random seed, for all future commits. Right now, adding algorithms or conditional logic in `hparams_registry` changes the random hyperparameters sampled for a given algorithm, dataset, and seed." 

Took a shot at fixing this. Let me know if you have thoughts.

Once this has been fixed it is probably a good idea to run a new sweep. From that point onwards adding a new algorithm should only require running the new algorithm on all datasets since sampled hparams stay consistent for the others.

After 5 years I see this 🥴🫶
Repository: facebookresearch/DomainBed
Issue: The next steps after DomainBed

Hello, my question is about next steps after setting up the Domain Bed Benchmark.

I have a custom Dataset, with a variety of algorithms and networks tested I am able to achieve satisfactory results. I understand the relation between domains and my dataset classes. My question is regarding the next steps, there are a plethora of options but I could use some guidance.

Thanks for your insights
Repository: facebookresearch/DomainBed
Issue: Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

              Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

_Originally posted by @facebook-github-bot in https://github.com/facebookresearch/DomainBed/issues/160#issuecomment-2785930677_
Repository: facebookresearch/DomainBed
Issue: Leave-one-out method doesn't handle test envs correctly
Root cause: Algorithmic Error

Two serious bugs in LOO model selection method:

1. On line 138 `test_records = get_test_records(records)` is called, which returns `None` when number of test environments is not 1 which is incompatible with the usage as can be seen on line 149 where number of test envs is required to be 2.
2. More serious, it assumes on line 142 that the test environment is always the first one in `test_envs` entry, which is incorrect: the method is called once for the first entry and second time for the second one. Problem is that as it currently stands no indication which environment of the pair is the test one is unavailable: only the test record with `test_envs` field is.

I have a patches availabe, if you want, so let me know if you want a pull.

PS It's also very inconvenient that it assumes that all environments are provided as validation ones on line 153. But that's a different issue subject of extending sweep to support explicitly specified test environment, which I have patches for too. So, let me know if you want a pull with them too.
Repository: facebookresearch/DomainBed
Issue: Fail to download datasets because of expired server or access blocked
Root cause: External Dependency Breakage

Currently, downloading datasets through datasets.py is failed.
I think it's because the server has expired or the drive is locked.
Would you mind updating the server or release the access permission?

Thank you.
Best regards.

I share my copy.
PACS https://drive.google.com/file/d/1WWi9tuUNFmUiw7Eirq_JT_hxYZv3J-Vv/view?usp=sharing  
OfficeHome  https://drive.google.com/file/d/1USXqK4nCvOAR5Ty4iwKcuCOrRJoRuDyU/view?usp=drive_link 
VLCS  https://drive.google.com/file/d/1yDuLrJdPSvMbTmpq-7XmwG0HnJengD-T/view?usp=sharing
Repository: facebookresearch/DomainBed
Issue: Just opening this here to keep track of the issue mentioned in #12:
Root cause: Algorithmic Error

Just opening this here to keep track of the issue mentioned in #12:

"We will have to find a solution to guarantee that the same random parameters are sampled in `hparams_registry` given the same random seed, for all future commits. Right now, adding algorithms or conditional logic in `hparams_registry` changes the random hyperparameters sampled for a given algorithm, dataset, and seed." 

_Originally posted by @SirRob1997 in https://github.com/facebookresearch/DomainBed/issues/16_
Repository: facebookresearch/DomainBed
Issue: Why are there no proper epochs?

As of now, the training loop just uses continuous update steps and hence the checkpoints usually occur in the middle of epochs as reported by the floats produced in `'epoch': step / steps_per_epoch` .

For me, this doesn't make too much sense. Shouldn't we count proper epochs and then define `checkpoint_freq` in terms of the epochs instead of the steps such that we report validation accuracies for every `checkpoint_freq` COMPLETED epochs?

This way we could also follow proper design for learning rate schedules since `scheduler.step(val_loss)` is usually invoked after every epoch and for `torch.optim.lr_scheduler.ReduceLROnPlateau` the learning rate gets adapted based on the change of the validation loss over multiple consecutive epochs.

This seems closed now, but for reference doing a proper epoch loop breaks IRM since it relies on independent examples in each minibatch and sampling without replacement introduces a slight correlation between examples. The way this manifests is that if you train for long enough, the penalty term becomes negative. I think you should be able to invoke `scheduler.step(val_loss)` every checkpoint_freq steps.

Separately, we found it more practical to define `checkpoint_freq` and other such things in terms of steps rather than epochs because we're dealing with many datasets of very different sizes. As one example, training for 5000 steps is a reasonable default, whereas training for 30 epochs would simultaneously be too short for MNIST and too long for DomainNet.

Yeah, came to the same conclusion. Closed due to the mentioned reasons.