Issues when setting sim_backend='gpu' and shader='rt'

[shaido987]

When running the test code below with the PickClutterYCB-v1 environment, the robot collapses at the origin in the recorded video. This issue occurs specifically when using both sim_backend='gpu' and shader='rt'. Other combinations of sim_backend and shader seem ok.

Additionally, when changing render_mode from 'sensors' to 'human' in this setup, a runtime error is raised, indicating that modifying a scene is not allowed after creating the batched render system.

Test code:

import gymnasium as gym
from mani_skill.envs.sapien_env import BaseEnv
from mani_skill.envs import PickClutterYCBEnv
from mani_skill.utils.wrappers import RecordEpisode

env_id = 'PickClutterYCB-v1'
shader = 'rt'
record_dir = 'record_test'

# noinspection PyTypeChecker
env: BaseEnv = gym.make(
    env_id,
    obs_mode='rgb',
    render_mode='sensors',
    num_envs=1,
    sensor_configs=dict(shader_pack=shader),
    human_render_camera_configs=dict(shader_pack=shader),
    viewer_camera_configs=dict(shader_pack=shader),
    sim_backend='gpu',
)

record_dir = record_dir.format(env_id=env_id)
env: RecordEpisode = RecordEpisode(env, record_dir, info_on_video=False, save_trajectory=False)

obs, info = env.reset()

for _ in range(100):
    action = env.action_space.sample()
    obs, reward, terminated, truncated, info = env.step(action)
env.close()

print("done")

Screenshot of resulting video:

When changing render_mode to 'human' the error is:

Traceback (most recent call last):
  File ".../demos/test.py", line 38, in <module>
    obs, reward, terminated, truncated, info = env.step(action)
                                               ^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/mani_skill/utils/wrappers/record.py", line 467, in step
    obs, rew, terminated, truncated, info = super().step(action)
                                            ^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/gymnasium/core.py", line 461, in step
    return self.env.step(action)
           ^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/mani_skill/utils/registration.py", line 161, in step
    observation, reward, terminated, truncated, info = self.env.step(action)
                                                       ^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/gymnasium/wrappers/order_enforcing.py", line 56, in step
    return self.env.step(action)
           ^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/mani_skill/envs/sapien_env.py", line 935, in step
    obs = self.get_obs(info)
          ^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/mani_skill/envs/sapien_env.py", line 500, in get_obs
    obs = self._get_obs_with_sensor_data(info)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/mani_skill/envs/sapien_env.py", line 573, in _get_obs_with_sensor_data
    sensor_data=self._get_obs_sensor_data(apply_texture_transforms),
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../lib/python3.11/site-packages/mani_skill/envs/sapien_env.py", line 545, in _get_obs_sensor_data
    self.scene.update_render(update_sensors=True, update_human_render_cameras=False)
  File ".../lib/python3.11/site-packages/mani_skill/envs/scene.py", line 391, in update_render
    self._sapien_update_render(
  File ".../lib/python3.11/site-packages/mani_skill/envs/scene.py", line 414, in _sapien_update_render
    self.render_system_group.update_render()
RuntimeError: Modifying a scene (add/remove object/camera) is not allowed after creating the batched render system.

[StoneT2000]

• Using the viewer with a visual observation mode with the physx_cuda backend is currently not supported.
• GPU parallelized sim does not support the ray tracing shader. high quality RT shader is limited to one per python process at the moment and you can only use the physx_cpu backend, not "gpu" (now called physx_cuda)

Since you are using just one env anyway you should just use the physx_cpu backend which will run faster.

[shaido987]

Thanks for the quick answer. The single env is actually only for testing, I plan to run with a larger number of envs later on.

Using the viewer with a visual observation mode with the physx_cuda backend is currently not supported.

Maybe some error or warning should be added to BaseEnv to highlight that this is not supported?

GPU parallelized sim does not support the ray tracing shader. high quality RT shader is limited to one per python process at the moment and you can only use the physx_cpu backend, not "gpu" (now called physx_cuda)

I saw there is a check in BaseEnv when using RT shader with more than 1 env, but it does not mention anything about the GPU right now. If the RT shader is not supported at all, even with a single env, then this check could be updated.

ManiSkill/mani_skill/envs/sapien_env.py

Lines 246 to 249 in 3922a6f

	if shader_dir is not None:
	if "rt" == shader_dir[:2]:
	if num_envs > 1 and parallel_in_single_scene == False:
	raise RuntimeError("""Currently you cannot run ray-tracing on more than one environment in a single process""")

---

Also, as a side note to this check: it will only check if the shader_dir argument is set, but if this argument is used then a warning is raised here to use sensor_configs and human_render_camera_configs instead (viewer_camera_configs is not mentioned but should probably be included in the warning). However, following this recommendation, the check above that uses shader_dir is skipped.

I could help add a PR if these settings are not supported right now and it is just the checks that should be updated.

[StoneT2000]

A PR would be appreciated, these checks are for sure outdated. We need an updated warning/error safeguard around these options mixing.

Although possibly the error might be better handled in the cameras.py file which creates the camera and checks the shader used. It should be able to reference the maniskill scene object and check what the sim backend is and if it is a parallelized sim then complain if shader is RT as well.

Given the number of considerations I might just handle this tomorrow though, there's quite a few edge cases and I also need to add a new dataclass to mark which sim_backends are "single python process but parallelized" (just physx_cuda atm).