zhiminy/EvalEng huggingface dataset:
Repository: stepjam/RLBench
Issue: Question - Robot arm keeps curling up after implementing reward signal.
Root cause: Algorithmic Error

Hi,

I have a question regarding the implementation of dense reward signal. I was able to implement the reward signal but my test results always lead to the arm curling up or trying to go back to it's initial starting position for some reason. I trained on the reach_target task and was able to reach the target but the episode does not terminate but instead the arm tries to return to its initial starting position after reaching the target. Is this a bug on RLBench or could the problem be with the algorithm on my end? Attached is my dense reward code and the test result.

    def reward(self,dense) -> Union[float,None]:
        if dense == 0:
            return None
        if dense:
            
            dist_reward = self.distance_to_target()


            success, terminate = self.success()
            
            if success:
                reward = dist_reward + 1.0
            else:
                reward = dist_reward

            return reward
    
    def distance_to_target(self):    
        x1, y1, z1 = self.agent_ee_tip.get_position()
        x2, y2, z2 = self.target.get_position()
        reward = -np.sqrt((x1-x2) ** 2 + (y1-y2) **2 + (z1-z2) ** 2)
        
        return reward
![recording](https://user-images.githubusercontent.com/89533315/147035760-d40da693-7a39-4878-8305-add690d3b22e.gif)


Hi Shane. 
What action more are you using?
I've trained dense-reward policies with RLBench without issue, so it's likely an issue with your algorithm/reward.

Hi James,

The action mode is the default action mode.
I tried running the reward with DDPG,TRPO and Option Critic and it all ends
up with the arm curling up. I'm using a reward function which was able to
solve the task with 2 DOF. Mine has 6 DOF being the joints. Could this be
the reason?

On Tue, Feb 15, 2022 at 12:36 AM Stephen James ***@***.***>
wrote:

> Hi Shane.
> What action more are you using?
> I've trained dense-reward policies with RLBench without issue, so it's
> likely an issue with your algorithm/reward.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/stepjam/RLBench/issues/150#issuecomment-1039299313>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AVLCXAYTZNDG54QY5S3O45TU3EVPXANCNFSM5KRVLJHQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
Repository: stepjam/RLBench
Issue: Error Running Pour from Cup to Cup task for data collection
Root cause: Resource Mishandling

I'm using RLbench for generating a dataset and wanted to use the PourFromCuptoCup task for my work. When I run the code it errors out and says this shouldn't happen. 

I've included an image of the error and the script that was run (had to upload as txt file)

![error_pourcuptocupt](https://user-images.githubusercontent.com/10580307/151844479-d1805b87-9217-47df-901c-0e3e77c7ff7a.png)

[gen_rl_bench_data.txt](https://github.com/stepjam/RLBench/files/7972921/gen_rl_bench_data.txt)

I was running in a conda environment running python 3.9.7 using an Nvidia RTX 3080 graphics card on ubuntu 21.10 
 

Thanks for raising. How soon after launching the dataset_generation does the error appear?

it varies I think from run to run. Sometimes it is completely fine for pouring into certain positions then when the environment resets to a new setting, it will crash. I think as is the script I sent SHOULD work as is, but I think the most minimal way (hopefully) to reproduce is:

```
from rlbench.environment import Environment
from rlbench.action_modes import ArmActionMode, ActionMode
from rlbench.observation_config import ObservationConfig
from rlbench.tasks import ReachTarget, PourFromCupToCup

obs_config = ObservationConfig()
obs_config.set_all(True)
action_mode = ActionMode(ArmActionMode.ABS_JOINT_VELOCITY)
env = Environment(
      action_mode, '', obs_config, False)
env.launch()


task = env.get_task(PourFromCupToCup)

for i in range(200):
       demo = task.get_demos(1, live_demos=True)[0]
        
```
just as a disclaimer, this might not run as is since I was refactoring from the script originally included.
Repository: stepjam/RLBench
Issue: Prevent the panda robot from lowering itself step by step
Root cause: Configuration Error

Dear developer,

right now I am working with RLBench as a framework to collect demonstrations for a DRL-algortihm. Instead of going straight from waypoint to waypoint I am using a "teleoperation tool" to create user flexible demonstrations (control every movement step based on the endeffector). After creating my specific tasks via the task_buliding tool I open the files and the Panda robot starts to lowering itself step by step without any settings in the ttt file or in the ObservationConfig class. It also aims no specific point as the waypoint, it just folds on itself. I can see the joint_targets change every step although I haven't assigned any command. How can I prevent the Robot from lowering itself? I need to have the control about every movement via the teleoperation otherwise the robot need to be static.

About your quick help I would be very happy! This is a problem that I could not solve despite the search of all PyRep and RLBench files.

Cheers,
Marcel

Hi,
Thanks for the question. I'll need a little more information to be able to help. Please post a minimal reproducible example. Thanks!

Hey Stephen,

thanks for your reply. I created a folder for the important files and information, including a video where I show you the lowering of the robot and the overwriting of the target position of the joints. Don't get distracted from the movements between the lowering, because I change the position via teleoperation. In general I am using CEILing (also available in github) approach with the RLBench task "CloseMicrowave" by calling teleoperation_env.py. The environment is setup with custom_env.py in the src folder, so I guess the lines from line 69 to 151 should be the important for the lowering. I could not refer the lines or any configuration in RLBench to the lowering problem. Just opening the scene and play it does not produce the lowering, that's why I am confused. Also the author of the algorithm (CEILing) leads the lowering back to RLBench but without any guess. 
Also as you can see in the custom_env file I changed the imports and usages of the action_mode module. Another question, did I come up with the right solution for RLBench 1.2 to exchange 

`action_mode = MoveArmThenGripper(
            arm_action_mode=EndEffectorPoseViaIK(False, 'end effector'),
            gripper_action_mode=Discrete())`
with the old
`action_mode = ActionMode(ArmActionMode.EE_POSE_EE_FRAME) `
I just want to get back the Change in the endeffector as EE_POSE_EE_FRAME is used before.

Find the data attached via:

https://we.tl/t-lyqWwcDYEn 

renewed link because of the run out.

https://we.tl/t-4StuoZTyQc 

Cheers and thank you so much for your help,
Marcel

Hey Stephen,

have you had a change to have a look on my prob? Thank you so far!

Cheers
Marcel

Can you upload the file as Pull request or a branch in your repo? Can you show video as a youtube link? the we transfer expires

Same problems like me, which refers to https://github.com/stepjam/PyRep/issues/370
Repository: stepjam/RLBench
Issue: Errors since update 5 days ago
Root cause: External Dependency Breakage

Hello RLBench developers,

since your update five days ago these several errors appearing:

`ImportError: cannot import name 'InvalidActionError from 'rlbench.task_environment (python3.8/site-packages/rlbench/task_environment.py)
File "/home/anaconda3/envs/lib/python3.8/site-packages/rlbench/environment.py", line 10, in <module>
    from rlbench.action_modes.action_mode import ActionMode
ModuleNotFoundError: No module named 'rlbench.action_modes'
`
Is there any chance to get the old folders/files to make it executable again.

Cheers,
Marcel 

In version 1.2, there are a few slight changes to the API for action modes. Pease take a look at the examples folder to see how to update your existing code.
Repository: stepjam/RLBench
Issue: Installation Paths
Root cause: Documentation Deficiency

Hi,

Instruction say the repository should be installed as `pip install .` from the same folder as the `requirements.txt`. This results in import error when running `import rlbench` from any location different than the RLbench folder: 

```bash
<MY LOCAL PATH>/.local/lib/python3.6/site-packages/rlbench/environment.py", line 10, in <module>
    from rlbench.action_modes.action_mode import ActionMode
ModuleNotFoundError: No module named 'rlbench.action_modes'
```
I managed to resolve this issue by installing with `pip install -e .` instead.

Note that this was not an issue with PyRep, just RLbench
Repository: stepjam/RLBench
Issue: Target object info missing in the Gym wrapper?
Root cause: Architectural Constraint

Currently the gym wrapper resets an episode and simply discards the task descriptions which encode the key information about the target object for a task. 

```python
def reset(self) -> Dict[str, np.ndarray]:
        descriptions, obs = self.task.reset()
        del descriptions  # Not used.
        return self._extract_obs(obs)
```

For example, in the `pick_up_cup.py` task, there is actually a variation about which cup to pick up and the cup's color might be different each episode. 

This target information is not in image inputs. Also in the default ``get_low_dim_state()``, I don't see it. The only place I can see this is in the returned `descriptions`. One exception is ``reach_target.py`` which defines its own ``get_low_dim_state()`` and returns the target object position. Am I missing something here? 

(Edit: upon further inspection, I found that ``self._variation_number`` seems not set (always 0))
Repository: stepjam/RLBench
Issue: It seems get_low_dim_data() method has bug inside
Root cause: Validation Gap

Hi, I'm really enjoying this fascinating environment you guys have released to facilitate the RL robotics research.
While I'm implementing some algorithms that is going to be tested on your environment, I encountered an error related to get_low_dim_data() method of Observation() class.
In
 `        low_dim_data = [] if self.gripper_open is None else [self.gripper_open]
        for data in [self.joint_velocities, self.joint_positions,
                     self.joint_forces,
                     self.gripper_pose, self.gripper_joint_positions,
                     self.gripper_touch_forces, self.task_low_dim_state]:
            if data is not None:
                low_dim_data.extend(data)
        return np.concatenate(low_dim_data)`,

the np.concatenate does not work since the gripper_open attribute is float, while others are np.ndarray, resulting the python complaining as

>  return np.concatenate(low_dim_data)
ValueError: zero-dimensional arrays cannot be concatenated 

Is this a minor bug or am I using it in the wrong way?

Thanks in advance!

   

`low_dim_data = [] if obs.gripper_open is None else [np.array(self.gripper_open).reshape(-1)]
      for data in [self.joint_velocities, self.joint_positions,
                    self.joint_forces,
                    self.gripper_pose, self.gripper_joint_positions,
                    self.gripper_touch_forces, self.task_low_dim_state]:
          if data is not None:
              low_dim_data.append(data)
      for data in low_dim_data:
      return np.concatenate(low_dim_data)`

I changed to code as above and it seems like working fine!

Thanks for pointing out. Fixed in #15.
Repository: stepjam/RLBench
Issue: ModuleNotFoundError: No module named 'rlbench.action_modes'
Root cause: Validation Gap

Dear developer,

I installed the package according to the tutorial. Then I tried to run the example(Few-Shot Learning and Meta Learning) in the tutorial. 
When I run 'from rlbench.action_modes.action_mode import MoveArmThenGripper', there were some errors.

ModuleNotFoundError: No module named 'rlbench.action_modes'.

After I checked  the installation path, there was no action_modes submodule there. Then I checked the setup.py file, there was no rlbench.action_modes item in the packages list. So, edit the packages list in the setup.py file could be a solution.



I've just experienced the same problem. Could anyone please answer this issue?

> I've just experienced the same problem. Could anyone please answer this issue?

Modify the packages list in the setup.py file as follows.
```python
packages=[
            'rlbench',
            'rlbench.action_modes',
            'rlbench.backend',
            'rlbench.tasks',
            'rlbench.task_ttms',
            'rlbench.robot_ttms',
            'rlbench.sim2real',
            'rlbench.assets',
            'rlbench.gym'
      ]
```
Then install the package again.
`pip install .`
Repository: stepjam/RLBench
Issue: Can not make two gym enviroments
Root cause: Documentation Deficiency

Dear Developers，
Thanks for your contributions.
In rlbench_gym.py, when I tyied to generate two gym environments as follows:
    env1 = gym.make('reach_target-state-v0', render_mode='human')
    env2 = gym.make('reach_target-state-v0', render_mode='human')
there have been some errors:
    qt.qpa.xcb: QXcbConnection: XCB error: 145 (Unknown), sequence: 177, resource id: 0, major code: 139 (Unknown), minor code: 20
    Attribute Qt::AA_UseDesktopOpenGL must be set before QCoreApplication is created.
    WARNING: QApplication was not created in the main() thread.
    (process:18254): GLib-CRITICAL **: 16:19:58.776: g_main_context_push_thread_default: assertion 'acquired_context' failed
We successfully make env1, but then the code  stays at env2 all the time.
Do you know how to resolve this problem?
Thanks again. 




Yes, but they need to be in separate processes. Please see here: https://github.com/stepjam/PyRep#running-multiple-pyrep-instances

Hi, Stephen, I tried to make two gym environments and saw your provided example, but it's a bit difficult for me to implement, could you show me more details about making two gym environments?
Repository: stepjam/RLBench
Issue: How to obtain Depth Images from the robot
Root cause: Documentation Deficiency

Hi everyone!
Before starting, thanks for your awesome work, it's really interesting!
I'm working with RLBench making an Imitation Learning project and I wanna obtain the depth images of the robot. I saw that in the paper we can get this type of image: 
![image](https://user-images.githubusercontent.com/49028287/162960479-65451f8c-a1b5-4bf2-b536-91ec70424fc6.png)
The problem is when the image is made, because I get this other kind of images:
![image](https://user-images.githubusercontent.com/49028287/162960645-53961403-0852-4a62-b426-2ab69b26ee0b.png)

How can I get the depth images like the ones that are shown on the paper?

Thank you again for your time!


I have the same problem

Hi,
The saved depth images are encoded. Please load the dataset using the built in data loading (env.get_demo; this will do the depth decoding), and then you can visualise the depth as you see fit (e.g. by accessing `obs.front_depth`).
Repository: stepjam/RLBench
Issue: Handle cam_over_shoulder_left_mask does not exist.
Root cause: Documentation Deficiency

python 3.6

reproduce:

1. use RLBench to open CoppeliaSim

```
python3 task_builder.py
```

2. make a small change in CoppeliaSim such as adding a sphere, then close CoppeliaSim. 

```
Ubuntu:~/Documents/RLBench/tools$ git status
On branch master
Your branch is up to date with 'origin/master'.

Changes not staged for commit:
	modified:   ../rlbench/task_design.ttt

no changes added to commit (use "git add" and/or "git commit -a")
```

3. reopen CoppeliaSim, get the error 'Handle cam_over_shoulder_left_mask does not exist'.

```
Ubuntu:~/Documents/RLBench/tools$ python3 task_builder.py 
Traceback (most recent call last):
  File "task_builder.py", line 289, in <module>
    scene = Scene(pr, robot, obs_config)
  File "/home/Documents/RLBench/tools/../rlbench/backend/scene.py", line 51, in __init__
    'cam_over_shoulder_left_mask')
  File "/home/miniconda3/envs/panda/lib/python3.6/site-packages/pyrep/objects/vision_sensor.py", line 14, in __init__
    super().__init__(name_or_handle)
  File "/home/miniconda3/envs/panda/lib/python3.6/site-packages/pyrep/objects/object.py", line 24, in __init__
    self._handle = sim.simGetObjectHandle(name_or_handle)
  File "/home/miniconda3/envs/panda/lib/python3.6/site-packages/pyrep/backend/sim.py", line 94, in simGetObjectHandle
    raise RuntimeError('Handle %s does not exist.' % objectName)
RuntimeError: Handle cam_over_shoulder_left_mask does not exist.
QObject::~QObject: Timers cannot be stopped from another thread
QMutex: destroying locked mutex
```

You probably edited the scene (and saved) when using task building.

If you are using task_builder, you should never save the scene in CoppeliaSim, and only save using the saving function within the task_builder. If you plan to change the scene, then please open it up in CoppeliaSim (not through task_builder) and then save it that way. 

To fix your problem, revert `task_design.ttt`, and follow advice above :)
Repository: stepjam/RLBench
Issue: Just interesting Question: PyRep? or RLBench for enviroment for learning

Hello, I'm Yuya, a first-year graduate student from Japan.

I'm interested in applying reinforcement learning to robots.
I'm trying to understand RLBench. I've read the paper.
I would like to confirm one thing.

Is it easy to set up environment for learning other robots, not arms, using RLBench?
I would like to apply reinforcement learning to other robots such as snake robots and mobile robots, using RLBench.
I have applied machine learning in PyRep before though.

Now, I guess it is easier to do it in PyRep rather than RLBench.
What do you think? What benefit does RLBench have?
I would like to know which is easier and more convenient for applying machine learning to various robots.

Thank you for your attention.

HI im a new too,have you applied a new robot in pyrep, already?

@working236 
Yes. I did it in pyrep for a test.
I'm going to apply other robots in pyrep.
I just made the environment, so I need to design a problem to solve in order to make the learning work well.

@yuya-0411 
Have you been able to use a mobile robot with or without an arm in RLBench?

@jtgenova 
Actually, I decided to use PyRep instead of RLBench because I can't find special advantage of RLBench.
You can create envrionment of Reinforcement Learning for any robots including your custom robots.
If you want to realize it, I can recommend combining Gym & Sable Baseline3 & PyRep.
You can get tips from CoppeliaSimRL (https://github.com/chauby/CoppeliaSimRL).

Thank you.

@yuya-0411 
Hi!
Have you ever used pyrep and ros together
Repository: stepjam/RLBench
Issue: Handle <Robot_Name> does not exist.
Root cause: Environment Incompatibility

Hi Stephen,

I have added my custom robot, (Wendy), PyRep by following the tutorial. The first picture shown is running the unit test, which shows a warning but at the end, it said okay. I did not do a pull request, not sure if this is needed.

![image](https://user-images.githubusercontent.com/104475121/166404592-52d172b1-904c-4283-ac9e-ff642a9d16fc.png)

Next, I changed the task_builder.py file to use Wendy instead of Panda. The second picture showed the error I'm getting. Any help would be appreciated.

![image](https://user-images.githubusercontent.com/104475121/166404956-f90ad9df-3d49-4270-905a-d324bd42bf21.png)

Thanks!



hello,i have encountered the same problem,do you solve it?

Hello, i can not even find the handle panda,. do you solve it?

@Victor-Ultimate I was using the latest version, CoppeliaSim_Edu_V4_7_0_rev4_Ubuntu20_04.tar.xz, but encountered the 'Handle Panda does not exist' error. I then switched to CoppeliaSim_Edu_V4_1_0_Ubuntu20_04.tar.xz, which resolved the issue.
Repository: stepjam/RLBench
Issue: Effect or set_joint_positions with disable dynamics=True.
Root cause: Unimplemented Feature Gap

Hi, thanks for maintaining this amazing project!

I have question for solving ik via Jacobian that is conducted via rlbench API.

in  [this line of code](https://github.com/stepjam/RLBench/blob/31c7bd96df1cdc06959d2c33f5751e5539925b27/rlbench/backend/scene.py#L155)
we disable dynamics for resetting the agent's position to the initial pose, no matter how the agent is posed at current state.

Meanwhile, if we set the action space to be Carteian space delta, the IK solved action is set to joint position as 
'set_joint_target_position' where the dynamics are computed.

I assume this setting yields tremendous errors in doing reinforcement learning with random exploration.

However, in the dm-control environment based on the Mujoco, the mocap based Cartesian space actions seem to work well with
the random exploration.

Therefore, I guess we can mimic such behavior of mocap motion in mujoco with set_joint_positions with disable dynamics=True.
I want to hear from the author about this remedy.

Thanks!
Repository: stepjam/RLBench
Issue: How to get Object Positions (x,y,z)?
Root cause: Documentation Deficiency

Hi!
I'm trying to get the positions of the objects of my scene to calculate the distance between the robot Gripper and the Red Ball that I need to catch with it. How can I get it? I would need the position of the Gripper and the Ball.

Thanks you for your time! Have a good day to everyone!
Repository: stepjam/RLBench
Issue: task.reset() performance question
Root cause: Environment Incompatibility

Hi! Thank you for RLBench!

I am running `few_shot_rl.py` example (using xvfb) and `task.reset()` takes a full minute to execute. It does not look like the correct behavior to me, so if you could give some advice on what the problem may be (or how to detect it), it would be very nice.

I tried setting OPENGL instead of OPENGL3 in all env cameras, but it did not help.

PC spec:
Ubuntu 18.04
Intel(R) Core(TM) i7-6700K
RAM: 60 Gb
GPUs: GTX 1080 (x2) and Titan V

After installing
```bash
sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libgl1-mesa-dev
```

It works much faster

You can now do GPU rendering: https://github.com/stepjam/RLBench/blob/master/README.md#running-headless

Thank you sooo much! You save my life. :) @Guitaricet 

> After installing
> 
> ```shell
> sudo apt-get install libavcodec-dev libavformat-dev libswscale-dev libgl1-mesa-dev
> ```
> 
> It works much faster