zhiminy/EvalEng huggingface dataset:
Repository: Farama-Foundation/Metaworld
Issue: Questions regarding the simulator
Root cause: Documentation Deficiency

Hello,

Awesome work! I have some questions about how the meta world simulator takes care of certain cases. 

 - How does meta-world simulate grasping? Is the object assumed to be grasped if near the arm and the grip action is taken? Is there any simulation for contact or friction while grasping? Some info about the assumptions made would be very useful. 

-  As far as I understand, the action space is of size 4 (3 for the location of the arm and 1 for gripper). Am I correct in my observation? Does the orientation of the arm have no impact on the environment? Some info about the inverse kinematics used could be very useful here. Also, if the pose of the robotic arm matters, how can one control it?

Best regards,
Ankit
 

Hi thanks for the question!

- The grasping is simulated fully by mujoco. We did not assume the object is grasped if it is near the arm/gripper and there is simulation going on during this process.

- Yes, your observation is correct. I don't think the pose of the arm matters a lot as long as a certain pose of the gripper can be achieved. We were using mujoco for inverse kinematics. 

@tianheyu927 could you please confirm?

Hi @zhanpenghe ,
Thanks for the response. I had some quick follow-ups. 
- Since all the tasks can be solved when training in the single task setup, am I correct in assuming grasping objects (per task wise) is not very difficult in the current setup (given assumptions on object sizes, the gripper, and friction).
- I may have misworded my second question before. What I meant was how much the orientation of the gripper affects the environment as no orientation info is currently present in the observation or action space.
Thanks again!

@imankgoyal 

1) In our experience object grasping behavior (based on MuJoCo's friction simulation) is fairly robust though not perfect. Please feel free to report significant regressions or instabilities if you find them.

2) The gripper orientation is fixed in a palm-down configuration to make exploration easier, so no info on orientation is needed on the observation. All environments are solvable without changing the orientation (see the scripted policies in `metaworld.policies` for proof)
Repository: Farama-Foundation/Metaworld
Issue: Inconsistency between GIF on github and implementation
Root cause: Documentation Deficiency

https://github.com/rlworkgroup/metaworld/blob/b556580204bf704dabcd0b5e558850db7a4335c4/metaworld/envs/mujoco/env_dict.py#L158-L211

I'm currently working on a project and we are using ML45 GIF from [here](https://meta-world.github.io/) as a standard of our task suite. I've spotted some tasks in the GIF that are not in the dictionary above. Our goal is to include stack, unstack and open_box, which appear in the GIF, to our task suite. 

I also found these environments in the dictionary that are not in the GIF: hand_insert, assembly_pug, pick_out_of_hole, shelf_remove, and sweep_tool. Additionally, shelf_remove and sweep_tool are imported in the file but they are not in the dictionary.

One last thing I would like to point out here is that environments [pick_place](https://github.com/rlworkgroup/metaworld/blob/master/metaworld/envs/mujoco/sawyer_xyz/sawyer_pick_and_place.py) and [pick_place_wsg](https://github.com/rlworkgroup/metaworld/blob/master/metaworld/envs/mujoco/sawyer_xyz/sawyer_pick_and_place_wsg.py) exist but are not used in the ML45 dictionary.

I'm just hoping that I don't miss the environments of interest in my task suite(stack, unstack, open_box)

@cl519 

The GIF was likely generated at a time when the final set of environments was still in flux. The authoritative list is that in the paper and the dictionary above. Some environments were added/removed to better balance the benchmark, because they didn't work well, etc.

Note that the only public API in this package is `metaworld.benchmarks`. You are welcome to use any code you find here, of course, because it's open source.
Repository: Farama-Foundation/Metaworld
Issue: Static Task IDs for environments
Root cause: Unimplemented Feature Gap

For training scenarios where task one-hot ids are necessary for training, static task IDs are necessary across tasks for training and evaluation. An example of this is in Multitask RL algorithms where the inputs to the policy are task one-hot ids

@avnishn are you working on this?

Yes @ryanjulian

Fixed by #72
Repository: Farama-Foundation/Metaworld
Issue: Observations are incorrectly padded with zeros
Root cause: Algorithmic Error

in `MultiClassMultiTaskEnv` a use case that routinely has to be dealt with is when multiple environments have observation spaces that are of different sizes. In ML45/MT50 the observation space is (6,) and (9,) for some spaces. The observation spaces that are of shape (6,) are supposed to be padded with zeros in order to make the observation spaces that are size( 6,) size (9,).

The issue is that where that is currently done, if a static_task_id, or one hot id is appended to the observation, then the one hot id is appended to the observation before any zeros are padded onto the observation. This leads to the one-hot id being incorrectly shifted inside the returned observation depending on the observaton_space shape of the underlying environment.

I agree this isn't ideal and probably unintended, but does not constitute a bug unless it's inconsistent between iterations of the algorithm. 

Is the onehot always in the same place?

@avnishn this is closed right?
Repository: Farama-Foundation/Metaworld
Issue: Benchmarks must be constructed before task names can be accessed
Root cause: Architectural Constraint

Currently in order to access all of the task names that belong to a benchmark's train/test set, the benchmark and all of its environments must first be constructed.

This is wasteful in the case that one needs to construct a benchmark's environments using the from task_api. The proper behavior should be that one should be able to access the names without constructing all of the underlying environments.

Can't you access all of the names from `EASY_MODE_CLS_DICT`, `MEDIUM_MODE_CLS_DICT`, and `HARD_MODE_CLS_DICT`? The `ML1` class actually returns these through its `available_tasks` class method.

None of those dicts are part of the public API, which consists only of "metaworld.benchmarks"

they should probably be prefixed with _ to make them private e.g. `_EASY_MODE_CLS_DICT`

If all `Benchmark` subclasses had `available_tasks` methods that returned the private dicts, would that be a satisfactory fix?

See https://github.com/rlworkgroup/metaworld/pull/85/files#diff-75dbfe903b808c91e2dd503cf761b531 for the right way to fix this.

@avnishn is this still true?
Repository: Farama-Foundation/Metaworld
Issue: ML10/45 Goal Variation
Root cause: Documentation Deficiency

Hello,

First, thank you for providing us this benchmark and for helping us to understand its details.

I have two questions about ML10/45 and paper results.

1) So, in these benchmarks, is there goal variation across episodes?
As an example, lets say I sample reach-v1 task in ML10. In the official benchmark, should I expect to have a constant goal during all episodes OR the goal can change after each reset?

2) In the paper, specifically in Figure 6 ("Single Task Success Rate"), is the goal constant for each task? Or is there goal variation during learning as well?

Thanks!

1. See my response on your other issue
2. Single-task success rate fixed the goal and start-state, as would be typical in single-task RL. If we don't do this, the problem is no longer an MDP (it's a POMDP), and we should not expect a single-task RL algorithm to be able to solve a POMDP.
Repository: Farama-Foundation/Metaworld
Issue: Paper and results

Hi,

Thanks for open sourcing this.
I was wondering when the paper for Meta-World will be out? If the paper is not ready yet and will be released in the far future, is there any plan to release your evaluation results on these tasks for those 6 meta-rl and multi-task learning algorithms mentioned in the website? 

Hey there Rasool -- thanks for your interest. We are working hard on verifying our final results, and you should expect a public version of them within a week or two.
Repository: Farama-Foundation/Metaworld
Issue: Do you plan to publish the codes (MAML, RL^2, PEARL) to reproduce the paper?

It would be very helpful if you could make it public.

Hi @emuemuJP, you can find code for the algorithms that you have listed in 
https://github.com/rlworkgroup/garage , along with working examples on metaworld.

Please note that the above is not the exact code originally used in the metaworld paper, which is unfortunately not available.

Thank you for letting me know!
Repository: Farama-Foundation/Metaworld
Issue: Regarding the done signal at max_path_length
Root cause: Unimplemented Feature Gap

Hello!

I read #45 and I understand why you decided to remove it, so sorry for referencing again this issue. However, I believe that as @krzentner [said](https://github.com/rlworkgroup/metaworld/pull/45#issuecomment-594911269), there are many cases that it is helpful to have this functionality. 

I also understand the [decision](https://github.com/rlworkgroup/metaworld/pull/45#issuecomment-594959879) that whoever wants to change this, will need to implement it themselves. But if there is a lot of interest of having this functionality, would it be possible to add this as an option when creating the environment? 

In the same style that there are the arguments `sample_all, sample_goals` etc, there could be an argument that would toggle between sending `done=True` when reaching `max_path_length` or not. Personally this is not an issue as I have already implement this functionality in my code as you suggest so it would be more of a future enhancement if there is a demand for it.

Would love to hear your thoughts on it!

@Kostis-S-Z As you have probably read in the issues, we removed `done=True` on `max_path_length` because it can cause severe performance degradations in off-policy algorithms, misrepresenting their true performance.

It's also a pretty subtle and hard-to-detect bug, so we regarded it as a dangerous design choice for a benchmark which is designed to be run by authors without much attention to the details. We prefer creating a little more hassle for rare use cases over causing someone to accidentally publish inaccurate results.

I'm not in favor of adding an option, because an important feature of benchmarks is that they shouldn't have any options. In reported paper results, how are we to distinguish Metaworld-with-done-is-true-on-max-path-length from Metaworld-without-done-is-true-on-max-path-length? We are actually in the process of removing the other options you mention (`sample_all`, etc.) for exactly this reason.

Unlike on-policy algos, off-policy algorithms distinguish between episodes which terminate due to the MDP reaching a true termination condition (`done=True`), and those which terminate due to a fixed horizon (`max_path_length`). Truncated but non-terminal (`max_path_length`) episodes need to be bootstrapped to account for the time steps the agent did not experience (e.g. `Q(s, a) ~ R + discount * max_{a'} Q(s', a')` given `(s, a, r, s')`) whereas true terminal episodes (`done=True`) do not need bootstrapping (e.g. `Q(s, a) ~ R`), because in the terminal case the return `R` is known to be the true, full-horizon return.

Conflating termination `done=True` with a fixed horizon actually makes the RL problem unlearnable (theoretically speaking) for the off-policy RL algorithm, because the MDP is no longer fully-observable (it can be considered a POMDP with time as a hidden variable, because you need to know time to estimate the true expected return). 

Thank you for such a detailed answer! I see your point and it makes sense.
Keep up the good work!
Repository: Farama-Foundation/Metaworld
Issue: Scripted policies for each environment
Root cause: Unimplemented Feature Gap


If you are going to start a 50-part issue, I recommend you either add a checklist to this issue or generate a new issue for each environment. 

It is probably a bad idea to try making one giant PR with all 50 scripted policies.

Environments for which testing scripts have been completed (or not):
Status | Task | Notes
:------: | :--------: | ----------
[x] | Reach | Aim for `env._state_goal` instead of `env.goal`
[x] | Push | See Reach notes. Only successful 90% of the time
[x] | Pick Place | See Reach notes. Only successful 80% of the time
[x] | Door Open |
[x] | Drawer Open |
[x] | Drawer Close |
[ ] | Button Press Top Down |
[ ] | Peg Insert Side |
[x] | Window Open |
[x] | Window Close |

_update this list as test scripts are completed_

Can you open a separate PR for each environment? Merging a PR with 50 scripted policies will be impossible.

Also note that your scripted policy can only use information in the observation. If accessing `env._state_goal` or `env.goal` is necessary to solve it, the environment has a problem.

Yeah I can open separate pull requests.

It looks like `env._state_goal` is returned by the `_get_obs_dict()` function but not by `_get_obs()`. Does that mean I can use it? Or am I looking in the wrong place?

You can only use `obs, r, done, info = env.step(action)`
Repository: Farama-Foundation/Metaworld
Issue: Scripted Policies for ML10/MT10 Environments

ML10/MT10 environments:
- [x] reach
- [x] push
- [x] pick-place
- [x] door-open
- [x] drawer-close
- [x] button-press-topdown
- [x] peg-insert-side
- [x] window-open
- [x] sweep
- [x] basketball
- [x] drawer-open
- [x] door-close
- [x] shelf-place
- [x] sweep-into
- [x] lever-pull


I updated the list above to reflect what's on master.

I'm going to also leave here a check list of policies that have open pull requests:

ML10/MT10 environments:
- [x] reach
- [x] push
- [x] pick-place
- [x] door-open
- [x] drawer-close
- [x] button-press-topdown
- [x] peg-insert-side
- [x] window-open
- [x] sweep
- [x] basketball
- [x] drawer-open
- [x] door-close
- [x] shelf-place
- [x] sweep-into
- [x] lever-pull
Repository: Giskard-AI/giskard-oss
Issue: [GSK-1109] Object of type int64 is not JSON serializable issue
Root cause: Interface Contract Mismatch

solved in [https://github.com/Giskard-AI/giskard/pull/1022](https://github.com/Giskard-AI/giskard/pull/1022)

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-1109](https://linear.app/giskard/issue/GSK-1109/object-of-type-int64-is-not-json-serializable-issue)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-1124] Add the possibility to set tag for tests in the test suite
Root cause: Unimplemented Feature Gap


![image.png](https://uploads.linear.app/5cb5c438-2552-41c0-89ac-b08d19e80d42/ee6a2213-ab0c-4d68-b78a-6d56309fc33e/50bb8b53-2410-440b-9099-854c25c524ee)

* Display the tags as in the figma mockup (use color generated by hashes)
* Add a programmatic way to do it (`suite.add_test(test, tags)` 
* Add a way to add tests from the UI

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-1124](https://linear.app/giskard/issue/GSK-1124/add-the-possibility-to-set-tag-for-tests-in-the-test-suite)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-1125] Overlapping UI Elements

### Issue Type

Bug

### Source

docker

### Giskard Library Version

1.9.1

### Giskard Server Version

1.5.0

### OS Platform and Distribution

macOS Ventura Version 13.4 (22F66)

### Python version

3.9

### Installed python packages

_No response_

### Current Behaviour?

```shell
Current UI elements are overlapping such as:
* Download
* Refresh
* Invite
```


### Standalone code OR list down the steps to reproduce the issue

```shell
Excepted to have non-overlapping application UI elements
```


### Relevant log output

_No response_

<sub>[GSK-1125](https://linear.app/giskard/issue/GSK-1125/[bug]-overlapping-ui-elements)</sub>

<img width="1438" alt="Screenshot 2023-05-26 at 7 53 16 PM" src="https://github.com/Giskard-AI/giskard/assets/69671407/59476730-6b1f-4027-b3ca-6f7ee34d429f">


This issue has been fixed with the new UI released on the 2.0 alpha version

<!-- From kevin on Linear -->
Repository: Giskard-AI/giskard-oss
Issue: [GSK-1275] Importance of metrics calculated on partial data slice
Root cause: Algorithmic Error

User KD_A on reddit pointed out that

> And does the # samples refer to the # samples in the data slice? It should be the denominator of the metric for the slice. For example, if that second row's recall of 0.111 is 1 predicted positive / 9 true positive, it's debatable whether to flag that.

This is right. We may have 1000 samples in our data slice, but to calculate for example the recall we only use the positive samples, which may be just a few samples out of the total, making the detection a false positive.

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-1275](https://linear.app/giskard/issue/GSK-1275/importance-of-metrics-calculated-on-partial-data-slice)</sub>

on hold

<!-- From Matteo Dora on Linear -->
Repository: Giskard-AI/giskard-oss
Issue: [GSK-1390] doc: fixing a spelling mistake in CONTRIBUTING.md file

Hello team, there is a small spelling mistake in CONTRIBUTING.md file in the lines mentioned below in the image 👇 
The word 'staring' should be 'starring'.
![image](https://github.com/Giskard-AI/giskard/assets/86348193/f739f9c2-167e-41da-8a69-d92e2e06a61f)

I'm creating this issue to address the same and simultaneously creating a  PR to rectify the issue ! 😃 
Thank you.

<sub>[GSK-1390](https://linear.app/giskard/issue/GSK-1390/doc-fixing-a-spelling-mistake-in-contributingmd-file)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-1417] Dataframes with numeric column names are not correctly supported
Root cause: Validation Gap

Trying to wrap a `pandas.DataFrame` that has numeric column names causes validation errors that prevent the creation of a `giskard.Dataset`:

1. Not possible to specify the `target` column, because the validation converts it to string, then compares to integers, not finding it.
2. Not possible to specify column types or correctly slice the dataset, because of casting to string, e.g. “Please make sure that the column names in \`column_types\` refers to existing columns in your dataset”.

How to reproduce:

```
import giskard
import numpy as np
import pandas as pd

df = pd.DataFrame(np.ones((10, 3)), columns=[1, 2, 3])

giskard.Dataset(df, target=2). # ERROR

ds = giskard.Dataset(df, column_types={1: "number"})
# UserWarning: The provided keys ['1'] in 'column_types' are not part of your dataset 'columns'. Please make sure that the column names in `column_types` refers to existing columns in your dataset.
```

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-1417](https://linear.app/giskard/issue/GSK-1417/dataframes-with-numeric-column-names-are-not-correctly-supported)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-1440] Failed WebSocket connection after  and infinite retry
Root cause: Validation Gap

### Issue Type

Bug

### Source

source

### Giskard Library Version

2.0.0b10

### Giskard Server Version

2.0.0b10

### OS Platform and Distribution

_No response_

### Python version

_No response_

### Installed python packages

_No response_

### Current Behaviour?

```shell
WebSocket connection keeps failed after JWT expiration.

Error info:

ERROR
message:Failed to send message to ExecutorSubscribableChannel[clientInboundChannel]; nested exception is io.jsonwebtoken.ExpiredJwtException\c JWT expired at 2023-07-19T18\c42\c02Z. Current time\c 2023-07-19T18\c42\c48Z, a difference of 46285 milliseconds.  Allowed clock skew\c 0 milliseconds.
content-length:0
```
```


### Standalone code OR list down the steps to reproduce the issue

```shell
1. Open frontend and login
2. Wait for JWT expiration (or manually expire it)
3. Do not refresh the frontend, and restart the Backend
4. After re-login, the WebSocket connection cannot be established using the previous JWT, which is acquired during the first login
```


### Relevant log output

_No response_

<sub>[GSK-1440](https://linear.app/giskard/issue/GSK-1440/failed-websocket-connection-after-and-infinite-retry)</sub>
Repository: Giskard-AI/giskard-oss
Issue: [GSK-1640] [GSK-1639] Set minimum python version to 3.10
Root cause: Unimplemented Feature Gap

to support Optional type

3\.10 is the default version of Colab

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-1639](https://linear.app/giskard/issue/GSK-1639/set-minimum-python-version-to-310)</sub>

let's return it to the backlog, it's not the top priority ATM

<!-- From Andrey Avtomonov on Linear. LinearCommentId:173b4f0e-4371-48d7-b899-ff500cef3dc0: -->

Hey @Hartorn - Is this issue still relevant or should we close it?

Can be closed, optional is available in 3.9, it's the `| None` pattern that's not available
Repository: Giskard-AI/giskard-oss
Issue: [GSK-1643] Run Backend tests with testcontainers profile during the nightly build

Here are the commands to run tests on H2 and Postgres:

* `../gradlew :backend:test :backend:integrationTest`
* `../gradlew :backend:test :backend:integrationTest -Ptestcontainers`

`testcontainers` profile will create a temporary postgres database per test, so it's slower, but closer to real prod conditions

We should run backend tests with and without `-Ptestcontainers` during the night build

<sub>From [SyncLinear.com](https://synclinear.com) | [GSK-1643](https://linear.app/giskard/issue/GSK-1643/run-backend-tests-with-testcontainers-profile-during-the-nightly-build)</sub>
Repository: Giskard-AI/giskard-oss
Issue: Giskard setup using Docker Compose fails
Root cause: Documentation Deficiency

### Issue Type

Bug

### Source

docker

### Giskard Library Version

n/a

### Giskard Server Version

Latest based on docker definitions

### OS Platform and Distribution

Linux Ubuntu 20x

### Python version

_No response_

### Installed python packages

```shell
n/a
```


### Current Behaviour?

```shell
Running "docker-compose build" should have successfully build all the docker images and components which did not happen. This is the error

:~/giskard# docker-compose build
[+] Building 0.0s (0/0)
could not find /root/giskard/python-client/docker: stat /root/giskard/python-client/docker: no such file or directory
```


### Standalone code OR list down the steps to reproduce the issue

```shell
1) Clone the Giskard repo from https://github.com/Giskard-AI/giskard.git
2) cd giskard
3) Run docker-compose up

Error:
failed to solve: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount2532142894/Dockerfile: no such file or directory
```


### Relevant log output

```shell
Seems a number of DockerFiles are missing in the cloned repo from https://github.com/Giskard-AI/giskard.git
e.g. No Dockerfile in in the path mentioned in docker-compose at giskard/python-client/
```


HI, I would like to work on this issue, can you please assign it to me?

@PAARTH2608 As of today we provide a single docker image [giskardai/giskard](https://hub.docker.com/r/giskardai/giskard).

The docker compose is not working since the image used aren't build anymore. I think we can simply cleanup by deleting those files.

@kevinmessiaen can you please eloborate on which files are to be cleaned up as I cant find any docker or docker-compose files in the cloned repo?

@PAARTH2608 My bad, those files has been removed from this repository already...
Repository: Giskard-AI/giskard-oss
Issue: Documentation does not include how to compare model
Root cause: Documentation Deficiency

### Issue Type

Documentation Bug

### Source

source

### Giskard Library Version

2.0.0b14

### Giskard Server Version

2.0.0b11

### OS Platform and Distribution

_No response_

### Python version

_No response_

### Installed python packages

_No response_

### Current Behaviour?

```shell
Nothing much, just the documentation for comparing machine learning model is empty.
```


### Standalone code OR list down the steps to reproduce the issue

```shell
Empty web page
```


### Relevant log output

_No response_