zhiminy/EvalEng huggingface dataset:
Repository: HumanCompatibleAI/overcooked_ai
Issue: Default Recipe config
Root cause: Validation Gap

Currently when recipe config is not specified user gets an error `ValueError: Recipe class must be configured before recipes can be created`.
The fix for this is to insert `Recipe.configure ({})` somewhere in the code that does not add any additional information except the will of the user to prevent error. I would like to do something that won't force user to add `Recipe.configure ({})` in the code without sacrificing the positives of the current solution.
Why this check was added? There is anything wrong with assuming that if recipes are not configured user chose to have a default configuration (maybe with adding some warning print)?

So this is also a problem when it comes to serialising and unserializing the environment for python multi-processing. Would be keen to understand how to get past this too.

The best person for this would be @nathan-miller23, but I don't know if he'll see this (as he's not actively involved in maintenance anymore). I currently don't have a lot of time to look into this, but (hopefully) soon we'll have an engineer working on the repo, and I'll direct them to this.

@akbir if you find a fix in the meantime, feel free to make a PR. I'm guessing that since the env works with rllib, they have some way of doing the serializing that works?

This issue is stale because it has been open for 30 days with no activity. It will be closed in 14 days unless there is some new activity

This issue is stale because it has been open for 30 days with no activity. It will be closed in 14 days unless there is some new activity
Repository: HumanCompatibleAI/overcooked_ai
Issue: Put overcooked-demo inside of overcooked_ai repo, run overcooked-demo code in test workflow
Root cause: Unimplemented Feature Gap

To prevent making overcooked-demo code outdated. Starting a container should be big improvement here (now there is no warning that changes here broke overcooked-demo i.e. its not even starting).

This is a great idea!

@nathan-miller23 @mesutyang97 one thing me and Mesut were talking about is that it might make sense to turn overcooked-demo into a submodule of overcooked-ai.

The main change it would require would be to also have a way to run the overcooked-demo docker commands (e.g. `./up.sh`) from any location (without being in the repo directory itself): otherwise, if one installs `overcooked-ai` with `pip` and then attempts to use `overcooked-demo`, they would have to find the folder where overcooked-demo was installed by pip, and run the docker commands there, that seems quite absurd.

If this is something that you would want to work on @bmielnicki, this would be really useful! I think making overcooked-demo a submodule might also help (or at least is relevant) for completing this specific issue.

I would gladly do this! Doing this would make solving this issue quite easily.

This seems doable in those steps:
1. Put overcooked-demo into a subdirectory of overcooked_ai.
2. Add to package_data om setup.py overcooked-demo directory.
3. By having a directory package in static.py we can get the path of up.sh. Now we can have a python script that runs up.sh `python overcooked-demo.py up.sh` or return path to up.sh `python overcooked-demo-path.py up.sh`. The former can be tricky as by default sudo is required to run docker container (user need to copy one line of bash (for more detailed info see https://askubuntu.com/questions/477551/how-can-i-use-docker-without-sudo ) that allows running docker without sudo), later is easy, but inconvenient (you need to run python script to copy sh file path and run it with sudo). It seems not that time consuming to do both.

Sounds good to me! However, I would wait on @nathan-miller23 to confirm that this way of approaching the merge makes most sense – I just talked with @mesutyang97 and we seem to be on the same page.

One thing to figure out here is how to integrate everything into one repo without bloating it excessively in terms of installation (or even cloning it): we don't want people to have to download hundreds of megabytes for a simple environment, or have to install the DRL code (with all it's dependencies) unless they choose to do so.

Some libraries that I know of that seem to handle this kind of usage are `ray` (which you can install with or without `rllib` for example). We might want to copy their structure, or generally compare with other approaches that exist out there.

Note that any approach we use here will likely be the same approach we use [for this issue](https://github.com/HumanCompatibleAI/human_aware_rl/issues/26), so it's important for this approach to be scalable to that.
Repository: HumanCompatibleAI/overcooked_ai
Issue: Have only one single player?
Root cause: Unimplemented Feature Gap

Thanks for the great work. Is it possible to configure the environment to have just one player instead of multiple? So the collaboration component will be removed and it becomes a standard RL task. 

Yes, I believe the environment side should handle this fine. However, it might require some changes to the DRL framework, which I expect would be quite easy to make. 

If you want to test the one-player setting in a preliminary manner right now without making any changes, I would encourage using a StayAgent as the second agent, and placing them in a location of the layout that is not important for the completion of the task – this should effectively be equivalent to training a single agent (but would require placing the StayAgent in the same position at test time too).

Gotcha. Thank you!
Repository: HumanCompatibleAI/overcooked_ai
Issue: Native python visualization - trajectory slider
Root cause: Unimplemented Feature Gap

Taken from @micahcarroll comment in https://github.com/HumanCompatibleAI/overcooked_ai/issues/45
Steps:
1. Convert entire trajectory into images in a single function.
2. Combine trajectory from 1. and ipythonwidgets to flick between images of the same trajectory

This task is not blocking native python visualization PR https://github.com/HumanCompatibleAI/overcooked_ai/pull/53 as core visualization functionality (convert OvercookedState to jpeg) is already there.

Code is pushed to https://github.com/HumanCompatibleAI/overcooked_ai/pull/53
Repository: HumanCompatibleAI/overcooked_ai
Issue: Metrics for layout difficulty
Root cause: Unimplemented Feature Gap

Summary and more detailed description of the layout metrics task discussed with @mesutyang97  so they are open to discussion with everyone.

1. Score difference between agents self-play and play with other agents:
    - function (or method) input:
        - layout
        - agents list
        - number of games for every agent pair
    - brief description of implementation:
        - every agent from the list play with every agent (including itself), the final score is calculated
    - result:
        - avg of non-self-play scores divided by avg of self-play scores

2. Measuring how much irrational play can lower score (how much play is fragile to random perturbations):
    - function (or method) input:
        - layout
        - agents pairs
        - probability of doing random actions for every agent
        - number of games for every agent pair
    - brief description of implementation:
        - probably use SampleAgent recently added to https://github.com/HumanCompatibleAI/overcooked_ai/pull/53 (see https://github.com/HumanCompatibleAI/overcooked_ai/blob/2bd97b2966213d69c91a1eae63c9d82964ecfe2d/src/overcooked_ai_py/agents/agent.py) with added optional weights argument (so agent can sample action from random agent some fraction of the time)
        - every agent pair plays in a randomized and non-randomized version
    - result:
        - avg of randomized play scores divided by avg of non-randomized play scores
3. Calculate the number of corridors tiles (tiles that are part of paths wide enough for only 1 chef) to detect layout with possible coordination of movement problems. It is tricky to figure out how to find __relevant__ corridors, possibly by measuring how often they are used by list of agents.
4. How many counters can be used to pass an object and how much time it can save - @mesutyang97  said he implemented something like that, but the code is currently outdated. Anyway finding relevant counters used for passes can also be tricky, possible testing which counters are used by self-play agents could reveal some of them (as passing the item by counter requires usually high coordination).
5. Measuring how much actions of the agent can infer his intent - I've discussed details with @mesutyang97 , by forgot exact details here.


Besides metrics, another thing to do is using them somewhere in overcooked repositories. My ideas are:
a) Adding metrics to trajectory dicts (to allow evaluation of how well agents does on layouts with certain metrics)
b) Generate layouts until there is one that passes some metrics requirements.

1,2, a), and b) seems straightforward to implement and I will code them first. Then probably 3 and 4 using agents for finding relevant terrain tiles.

If you have some trained agents that do well (or at least not very poorly) on the generated random layouts let me know - they would be useful for defaults used in agent-related metrics.
Repository: HumanCompatibleAI/overcooked_ai
Issue: Agent reset during activation overrides agent's index and mdp
Root cause: Architectural Constraint

Hello Micah and Nathan, 
I found out that my unability to run a GreedyHumanModel agent in overcooked demo comes from the agent being resetted during game's activation in overcooked demo [game.py](https://github.com/HumanCompatibleAI/overcooked-demo/blob/a9c68a93f705939bed41c3eec50333658e919d47/server/game.py#L547). I don't know how to handle that without breaking everything, though for now commenting out the "super" [here ](https://github.com/HumanCompatibleAI/overcooked_ai/blob/48d05dbbdb702d34c6769f31182ca619a3338ca0/src/overcooked_ai_py/agents/agent.py#L274) makes the agent run. Thanks for your help !

Closing this as stale
Repository: HumanCompatibleAI/overcooked_ai
Issue: event_infos tag bug
Root cause: Algorithmic Error

Reported via email:

-----------------

I’m looking to use the ‘event_infos’ tags as propositional labels for my particular application, but it seems that sometimes these tags don’t get correctly updated.

For example, in the following (state, action, new_state) triple:


X       X       P       X       X       

O                       →1      T       

Xo              →0              X       

X       D       X       S       X 

(Action.INTERACT, Action.INTERACT)

X       X       P       X       X       

O                       →1t     T       

Xo              →0              X       

X       D       X       S       X


I can see, by inspecting the relevant variables, that `state.players[1].held_object` has value `None` and `new_state.players[1].held_object` has value `tomato@(3, 1)`, but that if I look at:

_, info = env.mdp.get_state_transition(state, (Action.INTERACT, Action.INTERACT,), False, env.mp)

Then the value of `info['event_infos']['tomato_pickup’]` is `[False,False]`, when I believe it should be `[False,True]`. This may well happen with other info tags as well, but I haven’t dug into that yet. Do you have any idea what might be causing this problem and/or how to fix it?

For the record, I’m using whatever version of the repo is installed when running `pip install overcooked-ai`.

This issue is stale because it has been open for 30 days with no activity. It will be closed in 14 days unless there is some new activity

This issue is stale because it has been open for 30 days with no activity. It will be closed in 14 days unless there is some new activity
Repository: HumanCompatibleAI/overcooked_ai
Issue: Layout compatibility
Root cause: Documentation Deficiency

I've came across the agent documentation located here https://github.com/HumanCompatibleAI/overcooked-demo/tree/master/server/static/assets/agents/RllibSelfPlay_CrampedRoom

Regarding the layout compatibility part, it says that an agent trained on layout X can only be run on layout X/will achieve poor performance.
So does that mean I can't test an agent on an unseen layout ? 

In other words, I want to train a custom agent on one OR more randomly generated layouts (with the same constraints) and test the agent's performance on a new unseen layout. Is that possible ?

Thanks in advance!
Repository: HumanCompatibleAI/overcooked_ai
Issue: Invalid Value (NaNs) in Overcooked Class 
Root cause: Algorithmic Error

https://github.com/HumanCompatibleAI/overcooked_ai/blob/bdbf0232adde1412db87975d0ec2357093775ef4/src/overcooked_ai_py/mdp/overcooked_env.py#L549

This line was added to fix the "soup_cooking_time" issue. But now it returns:
"RuntimeWarning: invalid value encountered in multiply
  gym.spaces.Box(high * 0, high, dtype=np.float32)".
  
 Simple printing shows that variable _high_ is an array of Infs, and thus high * 0 becomes an array of NaNs.

Thanks for reaching out!

As a temporary solution, I've [changed the value to a large number](https://github.com/HumanCompatibleAI/overcooked_ai/commit/9395f2d4dea856d830c9a337bf629dffc5e93377). We'll figure a better solution for this soon.  

> Thanks for reaching out!
> 
> As a temporary solution, I've [changed the value to a large number](https://github.com/HumanCompatibleAI/overcooked_ai/commit/9395f2d4dea856d830c9a337bf629dffc5e93377). We'll figure a better solution for this soon.

It actually has a simpler solution than I thought. The issue is due to the next line actually:
`return gym.spaces.Box(high * 0, high, dtype=np.float32)`

The 0*high is the problem. This solves it:
```
high = np.ones(obs_shape) * float("inf")
low = np.zeros(obs_shape)
return gym.spaces.Box(low, high, dtype=np.float32)
```


Ah, I see, you're right that this is much nicer 😅 Thanks so much for your response!
Repository: HumanCompatibleAI/overcooked_ai
Issue: broken link

This link https://github.com/HumanCompatibleAI/human_aware_rl/blob/master/human_aware_rl/overcooked_interactive.py  is not exist.




This is the file you're looking for: https://github.com/HumanCompatibleAI/human_aware_rl/blob/neurips2019/human_aware_rl/overcooked_interactive.py

Where did you find this link linked?

Here:
https://github.com/HumanCompatibleAI/overcooked_ai/blob/master/README.md#python-visualizations-
Repository: HumanCompatibleAI/overcooked_ai
Issue: Cannot build from source
Root cause: Validation Gap

Hey. I tried to follow the instructions to install the repo from source but I'm getting the following error (through pip everything worked fine):
```
Obtaining file:///C:/Users/Andre.LAPTOP01/overcooked_ai
    ERROR: Command errored out with exit status 1:
     command: 'C:\Users\Andre.LAPTOP01\anaconda3\envs\overcooked_ai\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Andre.LAPTOP01\\overcooked_ai\\setup.py'"'"'; __file__='"'"'C:\\Users\\Andre.LAPTOP01\\overcooked_ai\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\ANDRE~1.LAP\AppData\Local\Temp\pip-pip-egg-info-5295cu1v'
         cwd: C:\Users\Andre.LAPTOP01\overcooked_ai\
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\Andre.LAPTOP01\overcooked_ai\setup.py", line 6, in <module>
        long_description = fh.read()
      File "C:\Users\Andre.LAPTOP01\anaconda3\envs\overcooked_ai\lib\encodings\cp1252.py", line 23, in decode
        return codecs.charmap_decode(input,self.errors,decoding_table)[0]
    UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 843: character maps to <undefined>
    ----------------------------------------
WARNING: Discarding file:///C:/Users/Andre.LAPTOP01/overcooked_ai. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
```

Thanks for reporting this! I'm guessing it might have to do with Windows installations. We'll be looking into this shortly.

As a note to self: in addition to solving the root cause of this bug, we should add a github actions test that tests installation on a windows computer.
Repository: HumanCompatibleAI/overcooked_ai
Issue: Switch to pytest
GitHub labels: enhancement

stale

Switch repo to use `pytest` for tests, instead of our current `unittesting` framework that is kind of janky and requires running shell scripts. This is something that should be done across all repos, or only on `overcooked_ai` once we have merged them.

This issue is stale because it has been open for 30 days with no activity. It will be closed in 14 days unless there is some new activity

This issue is stale because it has been open for 30 days with no activity. It will be closed in 14 days unless there is some new activity
Repository: HumanCompatibleAI/overcooked_ai
Issue: should be onions all the way through this line?
GitHub labels: stale

https://github.com/HumanCompatibleAI/overcooked_ai/blob/df2fc6ceaa19203250aa68755b861602d8ff586b/src/overcooked_ai_py/mdp/overcooked_mdp.py#L2961

Just spotted it, looks wrong

This issue is stale because it has been open for 30 days with no activity. It will be closed in 14 days unless there is some new activity

This issue is stale because it has been open for 30 days with no activity. It will be closed in 14 days unless there is some new activity
Repository: HumanCompatibleAI/overcooked_ai
Issue: trajectory replay code

Hi,
I have been trying to investigate how to get the individual frames for each time step from the trajectory. Therefore, I was looking for the code that replays the trajectory json file. 
That is the code that does this: https://humancompatibleai.github.io/overcooked-demo/replay
I am not able to understand how a 30s game play is equivalent to 204 steps, whereas a 60s game play is equivalent to 404 steps. 

It would be great if some guidance could be given regarding this.

Thank you! 

I think there might be some bug in the replay functionality – it had always been a bit finicky. If you find it, you're more than welcome to submit a PR. 

In general, for greatest reliability, I would try to process trajectories (and view them) in python as much as possible.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [API] generate_qa_docs in trial_tasks.py

              generate_qa_docs in trial_tasks.py

_Originally posted by @hongsw in https://github.com/Marker-Inc-Korea/AutoRAG/issues/987#issuecomment-2497547394_
            

![Image](https://github.com/user-attachments/assets/d1da4a9f-7b0d-4727-afa1-afe502ec0999)


close at #1005
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] Create multiple questions
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
I find the QA generation feature very useful and am eager to make full use of it. I have a question I'd like to ask.

Currently, it seems that only one question is generated per row in the corpus. Is there a feature that allows multiple questions to be generated for a single row? I couldn’t find this feature in the official documentation, so if I’ve missed something, I would appreciate your guidance.

Thank you for developing such a great library!


https://github.com/Marker-Inc-Korea/AutoRAG/pull/1009#issuecomment-2503759822
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] It seems that the validation logic is not only performing validation.
Root cause: Architectural Constraint

**Is your feature request related to a problem? Please describe.**
When running AutoRAG, I expected the evaluation logic to be triggered once, but it seems to occur twice. 
![Image](https://github.com/user-attachments/assets/5aa2461b-8ead-4811-a823-5fb960ab5207)
The code related to this is in the validate function of the Validator class.
```python
# start Evaluate at temp project directory
with (
	tempfile.NamedTemporaryFile(suffix=".parquet", delete=False) as qa_path,
	tempfile.NamedTemporaryFile(suffix=".parquet", delete=False) as corpus_path,
	tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as temp_project_dir,
):
	sample_qa_df.to_parquet(qa_path.name, index=False)
	sample_corpus_df.to_parquet(corpus_path.name, index=False)

	evaluator = Evaluator(
		qa_data_path=qa_path.name,
		corpus_data_path=corpus_path.name,
		project_dir=temp_project_dir,
	)
	evaluator.start_trial(yaml_path, skip_validation=True)
	qa_path.close()
	corpus_path.close()
	os.unlink(qa_path.name)
	os.unlink(corpus_path.name)
```

Calling the evaluation twice results in two API requests, causing the user to be charged twice. 
How about performing the validation only for the form?


@e7217 
When we execute validation, we slice the data to the small pieces. Just 5 QAs. So it does not call many APIs. But, still loading and experimenting all features so can check the system quite well.
So it is intended feature.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Feature Request] AutoRAG RaaS Clinet
Root cause: Unimplemented Feature Gap

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**


# AutoRAGClient: RAG Pipeline Management Client

## Purpose
AutoRAGClient is a client library designed to simplify the creation and management of RAG (Retrieval-Augmented Generation) pipelines through a REST API interface.

## Key Features
1. **Project Management**
   - Create new RAG projects
   - Manage documents within projects
   - Project-level configuration

2. **Document Processing**
   - Support for multiple file formats (PDF, TXT, CSV, MD)
   - Batch file uploads with pattern matching
   - File tracking and management

3. **Embedding Management**
   - Generate vector embeddings for documents
   - Flexible vector storage options
   - Automatic embedding model selection

4. **RAG Pipeline Operations**
   - Create and manage RAG pipelines
   - Question-answering capabilities
   - Context retrieval and results management

## Technical Details
- Asynchronous HTTP client (using aiohttp)
- Context manager support (`async with` syntax)
- Environment-based configuration
- Integrated logging system
- Error handling and API error abstractions

## Usage Example



**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.
Repository: Marker-Inc-Korea/AutoRAG
Issue: [Suggestion] How about adding AutoRAG to the LLM Evaluation section of Awesome-LLM?

https://github.com/Hannibal046/Awesome-LLM

https://github.com/Hannibal046/Awesome-LLM/pull/186
I made an PR for it!!! 
@e7217 

@e7217 AutoRAG is now in the Awesome-LLM
Thanks for your suggestion!!