Repository: Farama-Foundation/Metaworld
Issue: Custom Camera Views
Root cause: Unimplemented Feature Gap

Hello,

Thank you for the excellent code.

I'm trying to use the setup to collect simulated RGB data for a project but I haven't been able to find a way to set custom camera angle while rendering. All I can see is some predefined modes like `topview`. Anyway I can setup custom camera views for the tasks?

Thanks! 

@karttikeya we haven't extensively tested metaworld for image-based learning yet, which is why you're finding that these features are either half-baked or lacking.

Image-based learning is a feature area we plan on improving in future versions of metaworld, but I can't give you any hard commitments on when you'll see better support for this other than "some time in (early) 2020."

It sounds like you have both the motivation and the expertise to add or improve the features you need for your project. If you do make improvements, we'd love it if you take a look at `CONTRIBUTING.md` and send us a pull request with your improvements.

**Note:**
The only public API in metaworld right now is the `metaworld.benchmarks` package. You should consider the APIs for the individual environment classes private, and thus highly unstable. 

Okay, thanks for letting me know!
I see that there aren't any fully functional camera setup methods implemented yet, however is it possible to share the configurations (or how to do them) for the limited case of camera positions in this [GIF](https://meta-world.github.io/figures/ml45-1080p.gif)? 

Of course, if I end up implementing the full functionality, I would be happy to put in a PR as well :) 



Here's a tiny hack meanwhile. 

I modified `viewer_setup()` method in [`mujoco_env.py`](https://github.com/rlworkgroup/metaworld/blob/master/metaworld/envs/mujoco/mujoco_env.py#L77) with the following statements

```python
...

def viewer_setup(self):
  self.viewer.cam.azimuth = -20
  self.viewer.cam.elevation = -20

...
```

Gets you this perspective
![sawyer_push](https://user-images.githubusercontent.com/3909933/71597315-82ebc680-2af7-11ea-8685-7a01e906cd1c.gif)


So overwriting the `viewer_setup` API lets you change perspective when you do `env.render('human')`. But when you use `env.get_image` API, it still uses the default view. Btw `env.render('rgb_array')` crashes for me for some reason. 
What's the fundamental difference between setting up rendering via accessing the `viewers` vs calling `self.sim.render` ? 

Edit : I found that setting the device id to `-1` in  `self.viewer = mujoco_py.MjRenderContextOffscreen(self.sim, -1)` inside `mujoco_env.py` fixes the rendering issue in `rgb` mode.

Yes, modifying the `viewer_setup()` api does the trick. In my case (headless remote server, No GLFW) to use `env.sim.render` instead of `env.render` using the following snippet to directly create and adjust the viewer 
```
env.viewer = mujoco_py.MjRenderContextOffscreen(env.sim, -1) 
env.viewer.cam.azimuth =  205
env.viewer.cam.elevation = -170
env.viewer.cam.distance = 2.3
env.viewer.cam.lookat[0] = 1.1
env.viewer.cam.lookat[1] = 1.1 
env.viewer.cam.lookat[2] = -0.1
```
renders a nice GIF-esque angle. 

![image](https://user-images.githubusercontent.com/10197226/72766766-2cc5ec80-3ba6-11ea-8b3e-a31ff0604044.png)

On a related note, would there be a way to render this "effector ghost", ie same as usual except without the sawyer arm. 

As Meta-World ownership has been transferred from the RLWorkGroup to [Farama Foundation](https://farama.org/), I am closing this issue. If there are any questions or requests for features please join our [Discord](https://discord.gg/PfR7a79FpQ)

> 

Can you please share the code how you managed to do env.render("rgb_array")? I am using V2
Thanks!

It's been a while since I ran this but back then I modified the `def _get_viewer(self, mode):` as mentioned above. The code is [here](https://github.com/melfm/metaworld/blob/ibit/metaworld/envs/mujoco/mujoco_env.py#L186). Not sure if this would still work with the recent versions.

For anyone who need modify the camera with V2 version, here is one solution.

from gymnasium.envs.mujoco.mujoco_rendering import MujocoRenderer

DEFAULT_CAMERA_CONFIG = {
    "distance": 2,
    "azimuth": 215,
    "elevation": -20.0,
    "lookat": np.array([0, 0.5, 0.0]),
}

env.mujoco_renderer = MujocoRenderer(env.model, env.data, DEFAULT_CAMERA_CONFIG)

Image.fromarray(env.render())
![side_view](https://github.com/Farama-Foundation/Metaworld/assets/93953048/4543293e-7da4-4ccf-bd8b-c27b4f23b7db)



@XZDang13 I ran this code and it seemed to work, except that setting any of the 3 indices in lookat would actually set all indices to that value. Ex. If lookat was [0, 0, 0] and I did lookat[1] = 1, lookat would become [1, 1, 1]. Does this happen in your example?

You should be able to do `env.make(..., default_camera_config=YOUR_CAMERA_CONFIG_HERE)`, 
not sure if it is available in the current latest release, but should be for the next one.

> @XZDang13 I ran this code and it seemed to work, except that setting any of the 3 indices in lookat would actually set all indices to that value. Ex. If lookat was [0, 0, 0] and I did lookat[1] = 1, lookat would become [1, 1, 1]. Does this happen in your example?

@joshuasmith44 

No. I can give you a demo how to set it with a wrapper.

```python
import numpy as np
from PIL import Image
import gymnasium
from gymnasium.core import Env
from gymnasium.envs.mujoco.mujoco_rendering import MujocoRenderer
from metaworld.envs import ALL_V2_ENVIRONMENTS_GOAL_OBSERVABLE


DEFAULT_CAMERA_CONFIG = {
    "distance": 1.25,
    "azimuth": 145,
    "elevation": -25.0,
    "lookat": np.array([0.0, 0.65, 0.0]),
    }

DEFAULT_SIZE=224

class CameraWrapper(gymnasium.Wrapper):
    def __init__(self, env: Env, seed:int):
        super().__init__(env)

        self.unwrapped.model.vis.global_.offwidth = DEFAULT_SIZE
        self.unwrapped.model.vis.global_.offheight = DEFAULT_SIZE
        self.unwrapped.mujoco_renderer = MujocoRenderer(env.model, env.data, DEFAULT_CAMERA_CONFIG, DEFAULT_SIZE, DEFAULT_SIZE)

        # Hack: enable random reset
        self.unwrapped._freeze_rand_vec = False
        self.unwrapped.seed(seed)

    def reset(self):
        obs, info = super().reset()
        
        return obs, info

    def step(self, action):
        next_obs, reward, done, truncate, info = self.env.step(action) 
        
        return next_obs, reward, done, truncate, info

def setup_metaworld_env(task_name:str, seed:int):
    env_cls = ALL_V2_ENVIRONMENTS_GOAL_OBSERVABLE[task_name]
    
    env = CameraWrapper(env_cls(render_mode="rgb_array"), seed)
    
    return env

env = setup_metaworld_env('button-press-v2-goal-observable', 10)

obs, _ = env.reset()
img_array = env.render()
Image.fromarray(img_array)
```

When I set the "lookat": np.array([0.0, 0.65, 0.0]), the image is like this
![img1](https://github.com/user-attachments/assets/5a7dc2e4-2819-44bb-9065-c7cdba63ca68)

If I change it to np.array([0.65, 0.65, 0.0]), the image is like this
![img2](https://github.com/user-attachments/assets/3e5f625e-2afa-4fd3-bb20-d736bca3d3db)