zhiminy/EvalEng huggingface dataset:
Repository: HumanCompatibleAI/overcooked_ai
Issue: Pass action into env.step for gym env
Root cause: Validation Gap

The action should be a tuple with the joint action of the primary and secondary agents in index format. I tried the following code:

mdp = OvercookedGridworld.from_layout_name("cramped_room")
base_env = OvercookedEnv.from_mdp(mdp, horizon=500)
env = gym.make("Overcooked-v0",base_env = base_env, featurize_fn =base_env.featurize_state_mdp)
env.reset()
env.step((0,0))


I got the following error:


AttributeError                            Traceback (most recent call last)
Cell In[5], line 1
----> 1 env.step((0,0))

File ~/.pyenv/versions/3.9.1/envs/myvenv/lib/python3.9/site-packages/gym/wrappers/order_enforcing.py:37, in OrderEnforcing.step(self, action)
     35 if not self._has_reset:
     36     raise ResetNeeded("Cannot call env.step() before calling env.reset()")
---> 37 return self.env.step(action)

File ~/.pyenv/versions/3.9.1/envs/myvenv/lib/python3.9/site-packages/gym/wrappers/step_api_compatibility.py:52, in StepAPICompatibility.step(self, action)
     43 def step(self, action):
     44     """Steps through the environment, returning 5 or 4 items depending on `new_step_api`.
     45 
     46     Args:
   (...)
     50         (observation, reward, terminated, truncated, info) or (observation, reward, done, info)
     51     """
---> 52     step_returns = self.env.step(action)
     53     if self.new_step_api:
     54         return step_to_new_api(step_returns)

File ~/.pyenv/versions/3.9.1/envs/myvenv/lib/python3.9/site-packages/gym/wrappers/env_checker.py:39, in PassiveEnvChecker.step(self, action)
     37     return env_step_passive_checker(self.env, action)
     38 else:
...
-> 1851 for obj in state.objects.values():
   1852     if obj.position in counters_considered:
   1853         counter_objects_dict[obj.name].append(obj.position)

AttributeError: 'OvercookedGridworld' object has no attribute 'objects'


I tried many alternatives but none of them worked. Can you share an example of what I should pass in the env.step function?

Hi,

Thanks for bringing this issue up. I looked into this briefly and it looks like it is a hidden bug in the code. There is a quick one-line fix for this issue. If you go to [overcooked_env.py](https://github.com/HumanCompatibleAI/overcooked_ai/blob/2398fb4be3aaec75e6f7cccc042b6524344b617d/src/overcooked_ai_py/mdp/overcooked_env.py#L753), you just need to change 

`        ob_p0, ob_p1 = self.featurize_fn(self.mdp, next_state)
`

to 

`        ob_p0, ob_p1 = self.featurize_fn(next_state)
`

I apologize for this issue and will push a fix asap. The gym environment is not very well maintained and is mostly there for reference. If you run into any other problems please let me know! 


Thanks a lot @jyan1999!
Repository: HumanCompatibleAI/overcooked_ai
Issue: Clean up Human-AI / Human-human data collection

Uncover what's going on in the Human-AI data collection repo

[This repo](https://github.com/micahcarroll/overcooked-interactive-exp) was used for HH data collection, and has the newer changes from Nathan.

I think once you understand HH data collection repo, it might be easiest to just have a H-AI data collection modality within it (maybe taking inspiration from the repo below on how to structure).

[This repo](https://github.com/HumanCompatibleAI/overcooked-hAI-exp) was used for the original H-AI data collection, and is out of date.

The goal is to have only one repo that allows you to do either. One challenge is that psiturk might not work anymore (idk the state of development), but let's cross that bridge when we get there (if it doesn't work). Sandbox psiturk mode and testing things in localhost are your friends.
Repository: HumanCompatibleAI/overcooked_ai
Issue: How can I visualize agents' trajectory just given a sequence of actions
Root cause: Documentation Deficiency

Hello, authors. The agents' initial positions seem to be fixed. If given a fixed action sequence, the trajectory should be fixed. I want to know how to visualize the trajectory if I have the action sequences of two agents? Thanks.

Hi,

Have you checked out [this Colab notebook](https://colab.research.google.com/drive/1AAVP2P-QQhbx6WTOnIG54NXLXFbO7y6n#scrollTo=Z1RBlqADnTDw)? It shows an example of how you can see the agents in action given a sequence of joined actions. 

Thanks for your advice. I solved it.
Repository: HumanCompatibleAI/overcooked_ai
Issue: Feature request: Gymnasium & PettingZoo support
Root cause: Unimplemented Feature Gap

Hi,  would it be possible to add [PettingZoo](https://github.com/Farama-Foundation/PettingZoo) support in the future? The current setup.py also uses OpenAI gym, which has not been maintained for a few years now, [Gymnasium](https://github.com/Farama-Foundation/Gymnasium) is the maintained version of it. 

Gymnasium and PettingZoo are compatible with current RL training libraries ([rllib](https://github.com/ray-project/ray/tree/master/rllib), [tianshou](https://github.com/thu-ml/tianshou) and [CleanRL](https://github.com/vwxyzjn/cleanrl) have already migrated, and [stable-baselines3](https://github.com/DLR-RM/stable-baselines3/blob/e5deeed16efb57c34ccdcb14692439154d970527/docs/guide/install.rst#bleeding-edge-version) will soon) as well as other tools such as Comet and WandB.

For information about upgrading and compatibility, see [migration guide](https://gymnasium.farama.org/content/migration-guide/) and [gym compatibility](https://gymnasium.farama.org/content/gym_compatibility/). The main difference is the API has switched to returning `truncated` and `terminated`, rather than `done`, in order to give more information and mitigate edge case issues. 

Adding PettingZoo support would be a bit more complicated, but if needed I would be happy to help look over code or answer any questions. It would be really helpful for future researchers, and we would be excited to list it in the third party environments lists ([PettingZoo](https://pettingzoo.farama.org/environments/third_party_envs/), [Gymnasium](https://gymnasium.farama.org/environments/third_party_environments/)).

Hi @elliottower, does [this](https://github.com/HumanCompatibleAI/overcooked_ai/pull/123/files#diff-8f1a35244d81486e07eecac44167715a54afa3c2daf194a362a478b22a83bb24) look sufficient to you / what you had in mind?

> Hi @elliottower, does [this](https://github.com/HumanCompatibleAI/overcooked_ai/pull/123/files#diff-8f1a35244d81486e07eecac44167715a54afa3c2daf194a362a478b22a83bb24) look sufficient to you / what you had in mind?

Left some comments on that PR but just looked it over again and posting here as well, looks good to me.
Repository: HumanCompatibleAI/overcooked_ai
Issue: How to repeat the Near-optimal model-based planner and Planning experiments(in old dynamics))？
Root cause: Documentation Deficiency

I am wondering how to use the deprecated MediumLevelPlanner to repeat. I met this problem
<img width="557" alt="image" src="https://github.com/HumanCompatibleAI/overcooked_ai/assets/106607865/0df0428c-b683-47f0-90ab-621349770861">
<img width="463" alt="image" src="https://github.com/HumanCompatibleAI/overcooked_ai/assets/106607865/a029249b-7505-4169-81c3-395264b67631">

and many unfound functions in class Heuristic

Hi,

If you have specific needs for this function, I'd recommend going back and using an older commit before this method was deprecated and following the instructions included in that commit. For example the [DELIVERY_REW_THRES](https://github.com/HumanCompatibleAI/overcooked_ai/blob/62a0f2bf7efc280c5ed87c08ea58bdfc02188b90/src/overcooked_ai_py/planning/planners.py#L23) can be found in this commit. Note we are unlikely to be able to offer support for deprecated methods and classes. 

thank you， I will take a try

Just to add on to that, if you want to reproduce the old experiments exactly as in the paper I'd use [this tagged version](https://github.com/HumanCompatibleAI/overcooked_ai/releases/tag/neurips2019)
Repository: HumanCompatibleAI/overcooked_ai
Issue: PettingZoo

Does the author use PettingZoo Library to implement OverCooked Environment?

What do you mean? We don't use PettingZoo, but if I remember correctly PettingZoo tried to incorporate Overcooked in it at some point.

If you go on [overcooked_ai](https://github.com/HumanCompatibleAI/overcooked_ai/tree/master)/[src](https://github.com/HumanCompatibleAI/overcooked_ai/tree/master/src)/[overcooked_ai_py](https://github.com/HumanCompatibleAI/overcooked_ai/tree/master/src/overcooked_ai_py)/[mdp](https://github.com/HumanCompatibleAI/overcooked_ai/tree/master/src/overcooked_ai_py/mdp)
on line 672 - from pettingzoo.utils.env import ParallelEnv 

Why do you import that?

The environment was created due to a feature request for PettingZoo support (details can be found in this [issue](https://github.com/HumanCompatibleAI/overcooked_ai/issues/120)). This environment is not used anywhere in the codebase and exists merely as an optional supplement. 

Ah, right! Thanks @jyan1999!

Thank you!
Repository: HumanCompatibleAI/overcooked_ai
Issue: Step Info Ordering - Bug or Documentation Gap 
Root cause: Documentation Deficiency

When the environment is initialized or reset, [the assignment of agents to player positions in the environment is randomized](https://github.com/HumanCompatibleAI/overcooked_ai/blob/b7dfc012e3291d0835e270688fbbb71d75b8b795/src/overcooked_ai_py/mdp/overcooked_env.py#L901). However, this operation applies only to the ordering of actions and observations in the `step` interface - it does not affect the environment info returned by the `step` interface, including shaped and sparse rewards. As a result, intuitive use of the interface will lead to misattribution of rewards 50% of the time.

The `agent_idx` flag is [returned with the environment info](https://github.com/HumanCompatibleAI/overcooked_ai/blob/b7dfc012e3291d0835e270688fbbb71d75b8b795/src/overcooked_ai_py/mdp/overcooked_env.py#L878), which seems like it's intended to give the user a workaround to this problem by allowing them to reinterpret the order of the environment info outside of the environment interface. However, this does not appear to be documented, so users have no way of knowing they need to do this. Users therefore have to reverse engineer source code to understand why their rewards are misattributed, if they catch the issue at all.

Ideas for improving this:

- reorder the environment info, so that the shuffle operation applies to the entire `step` interface.
- introduce an environment parameter to enable/disable shuffling of agents, and issue a warning if using the shuffling feature while it does not apply completely to the `step` interface.
- write thorough docs/demos so users have a reasonable way to understand and avoid this pitfall.

Hi @bgiddens, thanks for pointing this out – you're totally right that it's tough to realize that this is happening under the hood right now.

> reorder the environment info, so that the shuffle operation applies to the entire step interface.

This would not be backwards compatible, so maybe this is the change that I'd be most worried about.

---

Regarding the other two solutions, both seem like good ideas. I currently don't have time to implement them, but given that there seems to be some interest in this issue, if anyone wants to take a stab at doing either of those (or both), I'll happily review a PR!
Repository: HumanCompatibleAI/overcooked_ai
Issue: Errors related to module 'ray'
Root cause: Environment Incompatibility

I'm trying to build the environment from source, following the guidance in README.md.
After running
```
conda create -n overcooked_ai python=3.7
conda activate overcooked_ai
git clone https://github.com/HumanCompatibleAI/overcooked_ai.git
cd overcooked_ai
pip install -e .
python testing/overcooked_test.py
```
The code raised an error:
```
ModuleNotFoundError: No module named 'ray'
```

I tried to solve it using
```
pip install -U "ray[rllib]"
```
The error message changed to
```
ModuleNotFoundError: No module named 'ray.rllib.agents'
```
The version of my python is 3.7.12 and the version of ray is 2.7.1
How can I solve this problem?

Hey there, 

Could you try running this command [here](https://github.com/HumanCompatibleAI/overcooked_ai#:~:text=If%20you%20also%20need%20the%20DRL%20implementations%3A) : `pip install -e overcooked_ai[harl]`, instead of `pip install -e .`

This command installs several other dependencies specific for training and some other tasks, and the code uses an old version of ray so it probably has some mismatch with newer versions of ray. 

> Hey there,
> 
> Could you try running this command [here](https://github.com/HumanCompatibleAI/overcooked_ai#:~:text=If%20you%20also%20need%20the%20DRL%20implementations%3A) : `pip install -e overcooked_ai[harl]`, instead of `pip install -e .`
> 
> This command installs several other dependencies specific for training and some other tasks, and the code uses an old version of ray so it probably has some mismatch with newer versions of ray.

Thanks for your help, but where should I run this command?
I tried running it in the "overcooked_ai" directory and its parent directory, but got the same error as follow:
```
zsh: no matches found: overcooked_ai[harl]
```

You should run the command in the directory where you cloned the repo. If you have cd'ed into the repo directory, you should do `pip install -e .[harl]`. Let me know if this works. 

> You should run the command in the directory where you cloned the repo. If you have cd'ed into the repo directory, you should do `pip install -e .[harl]`. Let me know if this works.

I got a similar error as before
```
zsh: no matches found: .[harl]
```

hmmm weird, I haven't seen this error before in this context. Maybe try quoting it like `pip install -e "overcooked_ai[harl]"`, or `pip install -e ".[harl]"`.  It looks like your shell is somehow interpreting the brackets as pattern-matching expression. 

Wow! It works!
I've passed all two tests mentioned in README.md
Many thanks!
Repository: HumanCompatibleAI/overcooked_ai
Issue: How to train human proxy model?
Root cause: Documentation Deficiency

This might be a silly question.
But I can't find the source code for training a human proxy model.
I noticed that `overcooked_ai/src/human_aware_rl/imitation/reproduce_bc.py` can train bc models which are then stored in `imitation/bc_runs/train`.
But the `overcooked_ai/src/human_aware_rl/ppo/evaluate.py` requires models stored in `imitation/bc_runs/test`.
Which file can I run to generate corresponding hp models?

If I remember correctly, you just need to manually change the output directory and training dataset in `overcooked_ai/src/human_aware_rl/imitation/reproduce_bc.py` so it uses `CLEAN_2019_HUMAN_DATA_TEST` and saves the output model to the `imitation/bc_runs/test` dir. The human proxy model is basically another behavior cloning model trained with the held-out data to evaluate against other models at test time. 

@jyan1999 Thanks for the help, I've successfully obtained the human proxy models following your instructions.
Repository: HumanCompatibleAI/overcooked_ai
Issue: Installation verification shows error
Root cause: Environment Incompatibility

After following the instructions as per https://github.com/HumanCompatibleAI/overcooked_ai,
when verifying the installation was checked, the test fails. 
I have tried installing the program in both windows and ubuntu and the error still persists. 
Attached with this comment is the error.
![failed_overcooked_installation](https://github.com/HumanCompatibleAI/overcooked_ai/assets/54328611/d9d59ea1-e860-467e-afb0-c49903cfcfec)




Hi there, thanks for opening the issue!

I just [reran the Github actions workflow](https://github.com/HumanCompatibleAI/overcooked_ai/actions/runs/7622113394/job/20759601770) and it seemed to work fine.

I also wasn't able to reproduce the issue locally, which is surprising if you got the same error on two separate machines (is that the case?). In any case, I'm not entirely sure what the issue is caused by without being able to reproduce it.

@sunamatya Hi, I have the same issue, Were you able to solve it? I am running on windows. Here is my full log. 
[log (2).txt](https://github.com/HumanCompatibleAI/overcooked_ai/files/14046981/log.2.txt)


Hi @boshenzh @sunamatya I was able to reproduce this issue on windows, and the problem is the weird default Windows encoding standard not being UTF-8, and when testing we print to a file that contains the upward arrow key, which cannot be recognized by the windows encoding scheme and was causing an error. The permission error is related as we were manually opening and closing the log file before, so when writing failed the file was never closed.

I have pushed a fix to address that. Thx for catching this! 
(Pro tip: tbh just use Mac or Linux)

@jyan1999 Thank you for following up! That solves the issue. (also thank you for your tips ;) )


Thanks so much @jyan1999 for figuring this out!
Repository: HumanCompatibleAI/overcooked_ai
Issue: Train 2 agents simultaneously

Hi,
Could you explain me if it's possible to train 2 PPO agents simultaneously? And if it's so how to do it.

Thanks

I would check out the PPO tests [here](https://github.com/HumanCompatibleAI/overcooked_ai/blob/master/src/human_aware_rl/run_tests.sh). Training with self-play is essentially training 2 PPO agents simultaneously.
Repository: HumanCompatibleAI/overcooked_ai
Issue:  Some problems during installation
Root cause: Environment Incompatibility

Dear author,when I run the following script "./run_tests.sh" from the src/human_aware_rl directory, there is an errors I don't understand what happenning and don't how to solve it. The error is as follows:
FileNotFoundError: [Errno 2] No such file or directory: '/home/file2/ray_results/PPO_cramped_room_False_nw=16_vf=0.000100_es=0.200000_en=0.000500_kl=0.200000_11_2024-04-12_14-13-38v6uj1upa'

There are the information of the directory I used:(overcooked_ai) file2@hciubuntu:~/overcooked_ai/src/human_aware_rl$ 


I'm not able to reproduce the error. I would try to run the individual commands in `./run_tests.sh` to try to isolate the issue further. Feel free to share a stack trace here
Repository: HumanCompatibleAI/overcooked_ai
Issue: Value Error - Overcooked_ai 1.1.0
Root cause: Environment Incompatibility

Hi, I am currenly using the overcooked_ai 1.1.0 version and when I am running the testing suite, I am getting a value error from the test_mdp_serialization . I have not changed any lines of code besides changing np.int to int as np.int is deprecated. 
<img width="1330" alt="Screenshot 2024-05-11 at 3 34 04 PM" src="https://github.com/HumanCompatibleAI/overcooked_ai/assets/111927365/95f8bc28-075b-48e8-89cb-844d290e2247">


I believe the test passes on the Github Actions, suggesting that maybe there is a discrepancy with your local setup. That being said, if you manage to figure out the fix and it's something in the code, feel free to make a PR!
Repository: HumanCompatibleAI/overcooked_ai
Issue: Can I reproduce the results of NeuIPS 2019 using this code?

Hi, I'm currently trying to conduct a new research using this envrionment.

However, when I go through these sample codes in "human_aware_rl", I have a feeling that there are some crucial changes are made from the original NeulIPS 2019 version.

For example, all the parameters defined in "src/human_aware_rl/ppo/run_experiments.sh" and the characteristics of training curve (E.g. the figure attached) seems to be different.

Have I missed something or do I just have to use the deprecated old repository to reproduce & test a new proposing method for my research? (As most of the prior works are based on the old repository)

![wandb_cramped_room](https://github.com/HumanCompatibleAI/overcooked_ai/assets/47871855/855caf64-06de-416a-871b-ffab1cf51d78)

Hi there. I believe the differences you see are due to the fact that the reward in the figure above may be the dense shaped reward, rather than the sparse reward, which is what is reported in the original paper and in the plots [here](https://github.com/HumanCompatibleAI/overcooked_ai/tree/master/src/human_aware_rl), which were generated using the `src/human_aware_rl/ppo/run_experiments.sh` script. 

There have been various changes since the NeurIPS 19 version, but the plots at the link above were our attempt to show that the changes do not significantly affect the final results.

As long as you are able to reproduce the results from our new figure in the README (which you should be able to by using the `src/human_aware_rl/ppo/run_experiments.sh` script), I encourage you to use the newer version of the code, as it will be nicer to work with than the neurips2019 branch.

Let me know if you have any additional questions, or run into any issues.
Repository: HumanCompatibleAI/overcooked_ai
Issue: New Training and Evaluating Platform

We implement a training and evaluating platform using Overcooked-ai, https://github.com/sjtu-marl/ZSC-Eval. 
ZSC-Eval includes 7 strong algorithms and novel evaluation partners, and a human experiment platform.
Would it be added to the "Research Papers using Overcooked AI" section for reference?

Yes, absolutely, if you make a PR I will merge it!

Yes, I make a PR #145 .
Repository: HumanCompatibleAI/overcooked_ai
Issue: Test with BC Model
Root cause: Documentation Deficiency

Hello! I want to evaluate the trained agents with the behavior cloning model. First question is do I have to train the BC agent locally, or is there a version available to use? 

I believe you have to train them, but it should be quite easy to do so following the instructions [here](https://github.com/HumanCompatibleAI/overcooked_ai/tree/master/src/human_aware_rl)
Repository: HumanCompatibleAI/overcooked_ai
Issue: Replace instances of np.Inf to np.inf following Numpy 2.0 release
Root cause: External Dependency Breakage

Since Numpy 2.0 release in June 2024, `np.Inf` is deprecated (See 'Highlights' in the [release note](https://github.com/numpy/numpy/releases/tag/v2.0.0)). Would it be possible to have instances of `np.Inf` replaced with `np.inf` so we can use Numpy 2.0 when using this envrionment?

Creating the issue directly because I don't think there is a formal contribution guide. Please feel free to close this and link one in the comments if there is one.

Thank you for pointing this our @luke9kim8!

Is np.inf compatible with older versions of Numpy? If so, feel free to find and replace all the instances, and make a PR!
Repository: HumanCompatibleAI/overcooked_ai
Issue: Awkward gym dependencies
Root cause: External Dependency Breakage

I noticed that this repo depends both on the deprecated openai [gym](https://github.com/openai/gym?tab=readme-ov-file) as well as the farama's actively maintained [gymnasium](https://github.com/Farama-Foundation/Gymnasium).

Would it be OK to remove the open-ai gym dependencies and ensure the env is registered with `gymnasium`? Happy to put up a PR. 

Yes that would be great!