zhiminy/EvalEng huggingface dataset:
Repository: facebookresearch/DomainBed
Issue: About BN layer

https://github.com/facebookresearch/DomainBed/blob/fef79d51f5f15286b91e93615a058791a4825c71/domainbed/networks.py#L80

Hi, I noticed that the above line was commented out. I would like to consult in which of the following two settings are the results in the paper obtained?

(1) all BN layers in ResNet50 are removed
(2) all BN layers in ResNet50 are set to ".eval()", i.e. running mean/variance is fixed, weights and biases are trainable



(2)
Repository: facebookresearch/DomainBed
Issue: IRM implementation

I wonder about your IRM implementation (below).
```
    def _irm_penalty(logits, y):
        device = "cuda" if logits[0][0].is_cuda else "cpu"
        scale = torch.tensor(1.).to(device).requires_grad_()
        loss_1 = F.cross_entropy(logits[::2] * scale, y[::2])
        loss_2 = F.cross_entropy(logits[1::2] * scale, y[1::2])
        grad_1 = autograd.grad(loss_1, [scale], create_graph=True)[0]
        grad_2 = autograd.grad(loss_2, [scale], create_graph=True)[0]
        result = torch.sum(grad_1 * grad_2)
        return result
```


IRM paper indicates that penalty term is gradient's square.
<img width="395" alt="image" src="https://user-images.githubusercontent.com/32060985/148762328-23e93090-fef3-43a7-9fdc-72764d2ae3ef.png">

Also, IRM penalty from 'https://github.com/facebookresearch/InvariantRiskMinimization' is the below code.
```
  def penalty(logits, y):
    scale = torch.tensor(1.).cuda().requires_grad_()
    loss = mean_nll(logits * scale, y)
    grad = autograd.grad(loss, [scale], create_graph=True)[0]
    return torch.sum(grad**2)
```

I do not think your implementation is wrong because you are the authors of the IRM paper.

However, ... 
Could you teach me why you exploit the dot product between gradient_odd and gradient_even instead of gradient's square?
Could you please explain if I have misunderstood something?


It's not a bug; see section 3.2 of the IRM paper. In short, splitting the minibatch like this gives an unbiased estimator of the full-dataset squared gradient norm; squaring the minibatch gradient would be biased.

I understand!
Thanks for your kind explanation.
Repository: facebookresearch/DomainBed
Issue: about the result

Could I ask a question?
What's the difference between the result "env0_in" and "env0_out"?

If environment 0 is a training domain:
* `env_0_in` is the training error at environment 0 (larger split),
* `env_1_out` is the validation error at environment 0 (smaller split).

If environment 0 is a testing domain
* `env_0_in` is a test error reported in tables (larger split),
* `env_1_out` is a test error used to do Oracle model selection (smaller split).

See Appendix of: https://openreview.net/forum?id=lQdXeXDoWtI

@lopezpaz what about env_0_out. I can understand the training set and the test set( --test_env 0,0=test_domain). What I am failing to understand is what includes the validation set for env_0_out

Also if I do not pass --test_env flag and train with ERM algorithm. What do the results mean then? Please correct me if I am wrong
Repository: facebookresearch/DomainBed
Issue: reproduced results show much larger fluctuation
Root cause: Environment Incompatibility

Hi, when I reproduce ERM and CORAL, the results show much larger fluctuation, i.e. much larger "std" when running "collect_results.py". Then mean ACC is also a little lower than claimed, especially on PACS, TerraIncognita, and VLCS. What could be the reason for this? Thanks!

To find out the cause, I git clone the latest code and rerun ERM and CORAL on TerraIncognita with "n_hparams=20", "n_trials=3". The code remains unchanged. Performance with the model selection method of "training-domain validation set" is as follows:

-------- Dataset: TerraIncognita, model selection method: training-domain validation set
Algorithm             L100                  L38                   L43                   L46                   Avg                  
ERM                   51.9 +/- 1.4          42.9 +/- 1.4          56.3 +/- 0.5          37.0 +/- 0.8          47.0
CORAL                 52.1 +/- 2.6          41.9 +/- 2.2          56.2 +/- 0.5          37.1 +/- 1.7          46.8                 

-------- Averages, model selection method: training-domain validation set
Algorithm                  TerraIncognita             Avg                       
ERM                        47.0 +/- 0.7               47.0
CORAL                      46.8 +/- 0.8               46.8

The reproduced results are much different from the results in DomainBed paper. I am not sure where the problem is. Thanks for helping me with this!

An example of implementations is as follows:
Environment:
	Python: 3.6.13
	PyTorch: 1.10.1
	Torchvision: 0.11.2
	CUDA: 11.3
	CUDNN: 8200
	NumPy: 1.19.2
	PIL: 8.3.1
Args:
	algorithm: CORAL
	checkpoint_freq: None
	data_dir: ../../../datasets
	dataset: TerraIncognita
	holdout_fraction: 0.2
	hparams: None
	hparams_seed: 13
	output_dir: TI_CORAL_output/0c34459f16b9139dd5cd1551d848107f
	save_model_every_checkpoint: False
	seed: 641058751
	skip_model_save: False
	steps: None
	task: domain_generalization
	test_envs: [2]
	trial_seed: 0
	uda_holdout_fraction: 0
HParams:
	batch_size: 40
	class_balanced: False
	data_augmentation: True
	lr: 0.0001653813153854724
	mmd_gamma: 0.6095584318401025
	nonlinear_classifier: False
	resnet18: False
	resnet_dropout: 0.5
	weight_decay: 2.7643974709171963e-05

Unfortunately there are many factors at play that one should consider, down to the hardware specifications, pytorch version (as well as other libraries), irreducible indeterminism in some operations, multiprocessing, and so on. We are trying to secure the resources to do an entire sweep ourselves, but we cannot promise that amount of compute by now.
Repository: facebookresearch/DomainBed
Issue: Why use freeze_bn during the training?

Thanks a lot for your wonderful job! 

Would you tell me why you are using freeze_bn during the training phase?

As described in Appendix D from [our manuscript](https://arxiv.org/abs/2007.01434):

> Observing that batch normalization interferes with domain generalization
> algorithms (as different minibatches follow different distributions), we freeze all batch normalization
> layers before fine-tuning.
Repository: facebookresearch/DomainBed
Issue: Hyperparameter settings for the reported results
Root cause: Documentation Deficiency

Hi, I'm wondering if you can also share the selected hyperparameters settings for the reported results? Since it's pretty expensive to run the whole hyperparameter search, it would be nice if we can directly reproduce the results using the optimal hyperparameter settings you found. Also, it would be nice to get more understanding of what hyperparameters work the best for each model and dataset.

Thanks,
Xinyi

Hi! Full sweep logs available [here](https://drive.google.com/file/d/16VFQWTble6-nB5AdXBtQpQFwjEC7CChM/view?usp=sharing).
Repository: facebookresearch/DomainBed
Issue: MLDG domain split bug
Root cause: Algorithmic Error

In the update rule of the function "random_pairs_of_minibatches" which extract pairs for meta-train and meta-test, you use all possible pairs permutation before using optimizer.step. This is not as the formula defined in the paper where you need to split the domains to V meta test and S-V meta train, aggregate and do the optimization step, while domains are split.

@shakedpe @Dsyforever mind sending fix?

Sure
I also enabled choosing how many meta test you want by adding a parameter to the function hparams_registery.
This is the function im using instead:

`

    def split_meta_train_test(minibatches, num_meta_test=1):
        n_domains = len(minibatches)
        perm = torch.randperm(n_domains).tolist()
        pairs = []
        meta_train = perm[:(n_domains-num_meta_test)]
        meta_test = perm[-num_meta_test:]

        for i,j in zip(meta_train, cycle(meta_test)):
             xi, yi = minibatches[i][0], minibatches[i][1]
             xj, yj = minibatches[j][0], minibatches[j][1]

             min_n = min(len(xi), len(xj))
             pairs.append(((xi[:min_n], yi[:min_n]), (xj[:min_n], yj[:min_n])))

        return pairs
`

Fixed by #95
Repository: facebookresearch/DomainBed
Issue: Bug in domain adaptation
Root cause: Algorithmic Error

In the `train.py` script, the chunk for creating uda_loader is as follows:

```
    uda_loaders = [InfiniteDataLoader(
        dataset=env,
        weights=env_weights,
        batch_size=hparams['batch_size'],
        num_workers=dataset.N_WORKERS)
        for i, (env, env_weights) in enumerate(uda_splits)
        if i in args.test_envs]
```

But the `if` statement check here is a logical bug. The `uda_split` is only appended in the lines above for test environments, as seen in line 140-141. So, for the first test environment, the index when enumerating over `uda_splits` would be 0, and the uda split for this environment will NOT be included in training. Instead, we should directly include the uda splits in the loader without a check for test environment, as this check has already been done while creating the uda_split. 

Happy to create a pull request for it if that helps.

Best,
Spandan

A PR would help! Thanks :)
Repository: google-deepmind/meltingpot
Issue: Module Not Found Error in Mac
Root cause: Environment Incompatibility

I installed by first cloning the repo, then using a venv, I did "sudo pip3 install ." in meltingpot directory.
Got the success message, but in interpreter I get module not found error for importing dmlab2d.  The test command also shows failed.
```
Building wheels for collected packages: dm-meltingpot
Installing collected packages: dm-meltingpot
  Attempting uninstall: dm-meltingpot
    Found existing installation: dm-meltingpot 1.0.0
    Uninstalling dm-meltingpot-1.0.0:
      Successfully uninstalled dm-meltingpot-1.0.0
Successfully installed dm-meltingpot-1.0.0
```

Error message when running the test.
```
======================================================================
ERROR: harvest_finished (unittest.loader._FailedTest)
----------------------------------------------------------------------
ImportError: Failed to import test module: harvest_finished
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "/Users/nantha/Projc/my_projects/Alab/meltingpotrepo/meltingpot/examples/tutorial/harvest/configs/environment/harvest_finished.py", line 17, in <module>
    from meltingpot.python.utils.substrates import shapes
  File "/Users/nantha/Projc/my_projects/Alab/meltingpotrepo/meltingpot/meltingpot/python/__init__.py", line 17, in <module>
    from meltingpot.python import scenario
  File "/Users/nantha/Projc/my_projects/Alab/meltingpotrepo/meltingpot/meltingpot/python/scenario.py", line 25, in <module>
    from meltingpot.python import substrate as substrate_factory
  File "/Users/nantha/Projc/my_projects/Alab/meltingpotrepo/meltingpot/meltingpot/python/substrate.py", line 22, in <module>
    from meltingpot.python.utils.substrates import builder
  File "/Users/nantha/Projc/my_projects/Alab/meltingpotrepo/meltingpot/meltingpot/python/utils/substrates/builder.py", line 26, in <module>
    import dmlab2d
ModuleNotFoundError: No module named 'dmlab2d'
```

Hello,

Did you download and install DeepMind Lab2D? You can do that by downloading a release wheel and installing it. e.g.

```
readonly PYTHON_VERSION="$(python3 --version | sed 's/^Python \([0-9]\)\.\([0-9]\+\)\..*$/\1\2/g')"
readonly DMLAB_WHEEL_ROOT='https://github.com/deepmind/lab2d/releases/download/release_candidate_2021-07-13'
readonly DMLAB_WHEEL="${DMLAB_WHEEL_ROOT}/dmlab2d-1.0-cp${PYTHON_VERSION}-cp${PYTHON_VERSION}-macosx_10_15_x86_64.whl"
pip install "${DMLAB_WHEEL}"
```

If that fails, you will have to build DMLab2D from source.

Let me know if this worked.
Repository: google-deepmind/meltingpot
Issue: Questions about applying PopArt on OPRE
Root cause: Documentation Deficiency

Hi,

I noticed that the PopArt layer is applied on the value head of the models in Meltingpot v2.0. I was able to implement it on IMPALA successfully, but when applying it to OPRE, there seems to be some issue, and the model never converges.
I implemented it using the following reference: https://github.com/deepmind/rlax/blob/master/examples/pop_art.py.

The structure of OPRE uses a mixture of n different models where n​ is the options dimension. Given that we have only one return value, how exactly are the weights of those n​ different value heads being handled?

**Option 1:**
- apply popart_update on n​ different models n​ times
- but the challenge here is that if I use n​ separate popart states for all n​ different models, which values should be used to normalize the final targets
- and if we use a single popart state, it'll get updated n​ different times while updating the weights of n​ models

**Option 2:**
- use a single feed-forward layer with n​ different outputs
- apply popart_update normally as usually done on layers with a single output

Tried both methods, and the loss values go to nan​ in both cases.
How exactly are you guys implementing this?

Thanks
Kinal

The way to apply popArt to OPRE, in my experience, is to simply apply it to the resulting mixture (the final sum).
I think in the original paper the fiddle with the linear layer that produces the baseline values, but you can simply "whiten" the value function by learning the shift, the scale and the second moment. 

Thanks for the response. 

I implemented the version you mentioned, and it is working fine. 
Also, I figured that the issue of `nan` was because I used a smaller batch size than the one mentioned in the paper. Tuning the $\epsilon$ parameter for RMSprop and setting it to `1e-2` instead of `1e-5` helped stabilize the learning. 

Closing this as it seems to be solved now.
Repository: google-deepmind/meltingpot
Issue: Adjustment for Ray in self_play_train.py
Root cause: External Dependency Breakage

Hi,

I had to change `config.num_workers = num_rollout_workers` to `config.num_rollout_workers = num_rollout_workers` [here](https://github.com/deepmind/meltingpot/blob/main/examples/rllib/self_play_train.py#L60) in order to make my code run smoothly on Mac OS M1 chip under ray version 2.2.0. 

More information: [link to ray issue](https://github.com/ray-project/ray/issues/31964).

nvm just saw the comment
Repository: google-deepmind/meltingpot
Issue: [RLLib] Issues running RLLib example script in Melting Pot 2.0
Root cause: External Dependency Breakage

Hi,

I'm running into errors when I try running [self_play_train.py.](https://github.com/deepmind/meltingpot/blob/main/examples/rllib/self_play_train.py):

The most recent fail is
```
File "/Users/nell/Documents/GitHub/norm-games/examples/rllib/self_play_train.py", line 155, in main
    assert results.num_errors == 0
AssertionError
```

However, there are several (maybe relevant) warnings occurring on the way:
```
2023-01-30 13:47:52,454	INFO algorithm_config.py:2503 -- Your framework setting is 'tf', meaning you are using static-graph mode. Set framework='tf2' to enable eager execution with tf2.x. You may also then want to set eager_tracing=True in order to reach similar execution speed as with static-graph mode.
2023-01-30 13:47:52,455	INFO algorithm_config.py:2503 -- Your framework setting is 'tf', meaning you are using static-graph mode. Set framework='tf2' to enable eager execution with tf2.x. You may also then want to set eager_tracing=True in order to reach similar execution speed as with static-graph mode.
```

```
(pid=46962)   if (distutils.version.LooseVersion(tf.__version__) <
(PPO pid=46962) 2023-01-30 13:47:56,215	WARNING algorithm_config.py:488 -- Cannot create PPOConfig from given `config_dict`! Property __stdout_file__ not supported.
```


```
(RolloutWorker pid=46974) 2023-01-30 13:48:00,643	DEBUG rollout_worker.py:1932 -- Creating policy for agent_0
(RolloutWorker pid=46974) 2023-01-30 13:48:00,645	DEBUG worker_set.py:938 -- Creating TF session {'intra_op_parallelism_threads': 2, 'inter_op_parallelism_threads': 2, 'gpu_options': {'allow_growth': True}, 'log_device_placement': False, 'device_count': {'CPU': 1}, 'allow_soft_placement': True}
(RolloutWorker pid=46975) /Users/nell/miniforge3/lib/python3.10/site-packages/gym/spaces/box.py:155: UserWarning: WARN: Casting input x to numpy array.
(RolloutWorker pid=46975)   logger.warn("Casting input x to numpy array.")
```

[This is a link](https://wtools.io/paste-code/bJoD) to the ray error message. I've also tried altering the substrates but all give the same error. Apart from this, I haven't changed anything.

Versions:
Mac OS M1 chip
Python 3.10.8
Ray version 2.2.0
Melting Pot 2.0.0

Does this error seem familiar to you? Let me know if you need any more context.

We currently don't support Ray 2.2.0, please use 2.0.0 instead.

That is what our setup uses: https://github.com/deepmind/meltingpot/blob/005ef2bffc1ad5fadd40998bae2dbf385e34c412/setup.py#L74

Ah, that makes sense. Thanks a lot!
Repository: google-deepmind/meltingpot
Issue: Problems running examples/rllib/view_models.py
Root cause: Architectural Constraint

Hi,

I've tried training examples on different environments, e.g., `commons_harvest__open` and `clean_up` but I am running into the following error when trying to execute `view_models.py`

```
File "/Users/nell/Documents/GitHub/social-contracts/examples/rllib/utils.py", line 183, in initial_state
    return self._model.get_policy(self._policy_id).get_initial_state()
AttributeError: 'NoneType' object has no attribute 'get_initial_state'
```

As `self._policy_id` is coming out as `"av"`, I've also tried setting it to the current version of ray's [`sample_batch.py`](https://github.com/ray-project/ray/blob/80e982b17aa20ad9f6be16704a0a654ba72d4218/rllib/policy/sample_batch.py#L27) `"default_policy"` but nothing changed.

It's running on
MacOs M1 chip
Python 3.10.8
Ray version 2.0.0

Thank you in advance!

This is because view_models is loading a hardcoded policy "av". Now that self_play_train.py has changed to name policies `f"agent_{i}"`, where i is a range over `substrate.get_config(substrate_name).default_player_roles,` this line will need to be updated to reflect that change.
Repository: google-deepmind/meltingpot
Issue: Problem installing dmlab2d for M2 arm64
Root cause: Environment Incompatibility

The README states that there is no appropriate wheel for the Apple M1/M2 chipset. It provides an example bash script for installing and setting up the wheel, but this results in an error when trying to import dmlab2d_pybind.so, saying that the version installed was still 'x86_64' rather than 'arm64'. I'm unsure what to change about the bash script to get the right version installed.

I can't help I'm afraid: I think this is an issue with your setup. The bash script works fine on my arm64 mac, and in our tests. My guess is that you have the `x86_64` version of Bazel and not the `arm64` version (or XCode or some other part of your toolchain).

Here's what my mac says:
```
$ bazel version
WARNING: Invoking Bazel in batch mode since it is not invoked from within a workspace (below a directory having a WORKSPACE file).
Build label: 5.3.2
Build target: bazel-out/darwin_arm64-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Wed Oct 19 18:35:48 2022 (1666204548)
Build timestamp: 1666204548
Build timestamp as int: 1666204548
```
Repository: google-deepmind/meltingpot
Issue: ImportError: cannot import name 'BaseParallelWrapper' from 'pettingzoo.utils'
Root cause: External Dependency Breakage

After following the installation instructions, I encounter the error in the title when trying to run the `sb3_train.py` script. I've tried multiple Python versions (3.9 & 3.10), reinstalling pettingzoo, etc, but nothing seems to work. Any help would be appreciated. Thanks!

Turns out it's a typo (`BaseParallelWrapper` vs `BaseParallelWraper`) upstream in the PettingZoo repo. For anyone else who encounters this, it looks like someone is working on fixing it, so you can either wait for just go fix the typo yourself.

I have the same question. Have you solved the problem

As a quick and dirty fix, running `find . -type f -exec sed -i 's/BaseParallelWraper/BaseParallelWrapper/g' {} +` in the root directory of the project appears to do the trick. It's a bit hacky but it works until FaramaFoundation fixes PettingZoo.
Edit: this is of course for *nix systems. If that's not what you're using, sorry and good luck!
Repository: google-deepmind/meltingpot
Issue: Setting a random seed
Root cause: Configuration Error

Hi!

I'm trying to find a place to set a random seed for the initial substrate setting. However, all places I've tried so far seem to do it, and I was wondering if an interface exists anywhere in the code.

Things I've tried:
- [`meltingpot/lua/modules/api_factory.lua`](https://github.com/deepmind/meltingpot/blob/df1be36229c00a8e2933f85de77fe827f26f2ffa/meltingpot/lua/modules/api_factory.lua#L38) + removing `random:seed()` calls
- [`meltingpot/python/configs/substrates/clean_up.py`](https://github.com/deepmind/meltingpot/blob/df1be36229c00a8e2933f85de77fe827f26f2ffa/meltingpot/python/configs/substrates/clean_up.py#L842) (as an example) + adding an `env_seed` to the `build()` function.
- [`meltingpot/python/substrate.py`](https://github.com/deepmind/meltingpot/blob/main/meltingpot/python/substrate.py#L32) + adding an `env_seed` to the build function + all dependencies.

Thanks in advance!

There are multiple random seeds:

1. `dmlab2d.Lab2d` (pybind C++) takes a seed in the `lab2d_settings_dict` that is passed to Lua (see [here](https://github.com/deepmind/meltingpot/blob/main/meltingpot/python/utils/substrates/builder.py)).
2. [`dmlab2d.Environment`](https://github.com/deepmind/lab2d/blob/main/dmlab2d/__init__.py) takes a seed for it's Python/C++
3. Stuff in the scenarios and the bots that I don't think a relevant here.

1 & 2 should be controllable by the [`env_seed` argument](https://github.com/deepmind/meltingpot/blob/main/meltingpot/python/utils/substrates/builder.py). However, [this isn't wired in](https://github.com/deepmind/meltingpot/blob/main/meltingpot/python/utils/substrates/substrate.py).

I've discussed this at https://github.com/deepmind/meltingpot/issues/113#issuecomment-1462212082 there may be more relevant info there on what solution to take (e.g. `seed` in a config, as an override to `build`, as an argument to `reset()`).

Amazing, thank you! Wired it in.