Skip to content

K
KincoActuatorIsaacLab
Commit 7997bfec authored by Kelly Guo's avatar Kelly Guo Committed by David Hoeller
Browse files
Options

Adds Shadow Hand Vision environment (#107) 

Adds Shadow Hand environment with vision and proprioception
observations. Image observations are used to train a CNN with RGB,
depth, and segmentation images, regressing on keypoint states that are
then combined with proprioception observations to be fed into an
asymmetric actor-critic policy.

- New feature (non-breaking change which adds functionality)
- This change requires a documentation update

![image](https://github.com/user-attachments/assets/d67c966b-2931-4ee4-84d1-8bc004471a21)

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there
parent f09f8645
Hide whitespace changes
Inline Side-by-side
Showing with 678 additions and 108 deletions
docs/source/features/environments.rst
View file @ 7997bfec
......@@ -83,9 +83,10 @@ for the reach environment:
| |cube-allegro| | |cube-allegro-link| | In-hand reorientation of a cube using Allegro hand |
| | |allegro-direct-link| | |
+----------------+---------------------------+-----------------------------------------------------------------------------+
| |cube-shadow| | |cube-shadow-link| | In-hand reorientation of a cube using Shadow hand |
| | |cube-shadow-ff-link| | |
| | |cube-shadow-lstm-link| | |
| |cube-shadow| | | |cube-shadow-link| | In-hand reorientation of a cube using Shadow hand |
| | | |cube-shadow-ff-link| | |
| | | |cube-shadow-lstm-link| | |
| | | |cube-shadow-vis-link| | |
+----------------+---------------------------+-----------------------------------------------------------------------------+
.. |reach-franka| image:: ../_static/tasks/manipulation/franka_reach.jpg
......@@ -108,6 +109,7 @@ for the reach environment:
.. |cube-shadow-link| replace:: `Isaac-Repose-Cube-Shadow-Direct-v0 <https://github.com/isaac-sim/IsaacLab/blob/main/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/shadow_hand/shadow_hand_env_cfg.py>`__
.. |cube-shadow-ff-link| replace:: `Isaac-Repose-Cube-Shadow-OpenAI-FF-Direct-v0 <https://github.com/isaac-sim/IsaacLab/blob/main/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/shadow_hand/shadow_hand_env_cfg.py>`__
.. |cube-shadow-lstm-link| replace:: `Isaac-Repose-Cube-Shadow-OpenAI-LSTM-Direct-v0 <https://github.com/isaac-sim/IsaacLab/blob/main/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/shadow_hand/shadow_hand_env_cfg.py>`__
.. |cube-shadow-vis-link| replace:: `Isaac-Repose-Cube-Shadow-Vision-Direct-v0 <https://github.com/isaac-sim/IsaacLab/blob/main/source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/shadow_hand/shadow_hand_vision_env.py>`__
Locomotion
~~~~~~~~~~
......@@ -285,106 +287,108 @@ Comprehensive List of Environments
.. table::
:widths: 33 33 19 25
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Task Name | Inference Task Name | Workflow | RL Library |
+================================================+============================================+===============+=============================+
| Isaac-Repose-Cube-Allegro-Direct-v0 | | Direct | rsl_rl, rl_games |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Ant-Direct-v0 | | Direct | rsl_rl, rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Anymal-C-Direct-v0 | | Direct | rsl_rl, rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Anymal-C-Direct-v0 | | Direct | rsl_rl, rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Cart-Double-Pendulum-Direct-v0 | | Direct | rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Cartpole-Direct-v0 | | Direct | rsl_rl, rl_games, skrl, sb3 |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Cartpole-RGB-Camera-Direct-v0 | | Direct | rl_games |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Cartpole-Depth-Camera-Direct-v0 | | Direct | rl_games |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Franka-Cabinet-Direct-v0 | | Direct | rsl_rl, rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Humanoid-Direct-v0 | | Direct | rsl_rl, rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Repose-Cube-Shadow-Direct-v0 | | Direct | rsl_rl, rl_games |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Repose-Cube-Shadow-OpenAI-FF-Direct-v0 | | Direct | rsl_rl, rl_games |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Repose-Cube-Shadow-OpenAI-LSTM-Direct-v0 | | Direct | rl_games |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Quadcopter-Direct-v0 | | Direct | rsl_rl, rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Shadow-Hand-Over-Direct-v0 | | Direct | rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Humanoid-v0 | | Manager Based | rsl_rl, rl_games, skrl, sb3 |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Ant-v0 | | Manager Based | rsl_rl, rl_games, skrl, sb3 |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Cartpole-v0 | | Manager Based | rsl_rl, rl_games, skrl, sb3 |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Unitree-A1-v0 | Isaac-Velocity-Flat-Unitree-A1-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Unitree-A1-v0 | Isaac-Velocity-Rough-Unitree-A1-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Anymal-B-v0 | Isaac-Velocity-Flat-Anymal-B-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Anymal-B-v0 | Isaac-Velocity-Rough-Anymal-B-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Anymal-C-v0 | Isaac-Velocity-Flat-Anymal-C-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Anymal-C-v0 | Isaac-Velocity-Rough-Anymal-C-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Anymal-D-v0 | Isaac-Velocity-Flat-Anymal-D-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Anymal-D-v0 | Isaac-Velocity-Rough-Anymal-D-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Cassie-v0 | Isaac-Velocity-Flat-Cassie-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Cassie-v0 | Isaac-Velocity-Rough-Cassie-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-G1-v0 | Isaac-Velocity-Rough-G1-Play-v0 | Manager Based | rsl_rl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-G1-v0 | Isaac-Velocity-Flat-G1-Play-v0 | Manager Based | rsl_rl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Unitree-Go1-v0 | Isaac-Velocity-Flat-Unitree-Go1-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Unitree-Go1-v0 | Isaac-Velocity-Rough-Unitree-Go1-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Unitree-Go2-v0 | Isaac-Velocity-Flat-Unitree-Go2-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Unitree-Go2-v0 | Isaac-Velocity-Rough-Unitree-Go2-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-H1-v0 | Isaac-Velocity-Rough-H1-Play-v0 | Manager Based | rsl_rl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-H1-v0 | Isaac-Velocity-Flat-H1-Play-v0 | Manager Based | rsl_rl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Spot-v0 | Isaac-Velocity-Flat-Spot-Play-v0 | Manager Based | rsl_rl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Open-Drawer-Franka-v0 | Isaac-Open-Drawer-Franka-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Open-Drawer-Franka-IK-Abs-v0 | | Manager Based | |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Open-Drawer-Franka-IK-Rel-v0 | | Manager Based | |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Repose-Cube-Allegro-v0 | Isaac-Repose-Cube-Allegro-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Repose-Cube-Allegro-NoVelObs-v0 | Isaac-Repose-Cube-Allegro-NoVelObs-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Lift-Cube-Franka-v0 | Isaac-Lift-Cube-Franka-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Lift-Cube-Franka-IK-Abs-v0 | | Manager Based | |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Lift-Cube-Franka-IK-Rel-v0 | | Manager Based | |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Reach-Franka-v0 | Isaac-Reach-Franka-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Reach-Franka-IK-Abs-v0 | | Manager Based | |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Reach-Franka-IK-Rel-v0 | | Manager Based | |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Reach-UR10-v0 | Isaac-Reach-UR10-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
| Isaac-Navigation-Flat-Anymal-C-v0 | Isaac-Navigation-Flat-Anymal-C-Play-v0 | Manager Based | rsl_rl |
+------------------------------------------------+--------------------------------------------+---------------+-----------------------------+
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Task Name | Inference Task Name | Workflow | RL Library |
+================================================+================================================+===============+=============================+
| Isaac-Repose-Cube-Allegro-Direct-v0 | | Direct | rsl_rl, rl_games |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Ant-Direct-v0 | | Direct | rsl_rl, rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Anymal-C-Direct-v0 | | Direct | rsl_rl, rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Anymal-C-Direct-v0 | | Direct | rsl_rl, rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Cart-Double-Pendulum-Direct-v0 | | Direct | rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Cartpole-Direct-v0 | | Direct | rsl_rl, rl_games, skrl, sb3 |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Cartpole-RGB-Camera-Direct-v0 | | Direct | rl_games |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Cartpole-Depth-Camera-Direct-v0 | | Direct | rl_games |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Franka-Cabinet-Direct-v0 | | Direct | rsl_rl, rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Humanoid-Direct-v0 | | Direct | rsl_rl, rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Quadcopter-Direct-v0 | | Direct | rsl_rl, rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Repose-Cube-Shadow-Direct-v0 | | Direct | rsl_rl, rl_games |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Repose-Cube-Shadow-OpenAI-FF-Direct-v0 | | Direct | rsl_rl, rl_games |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Repose-Cube-Shadow-OpenAI-LSTM-Direct-v0 | | Direct | rl_games |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Repose-Cube-Shadow-Vision-Direct-v0 | Isaac-Repose-Cube-Shadow-Vision-Direct-Play-v0 | Direct | rsl_rl, rl_games |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Shadow-Hand-Over-Direct-v0 | | Direct | rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Humanoid-v0 | | Manager Based | rsl_rl, rl_games, skrl, sb3 |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Ant-v0 | | Manager Based | rsl_rl, rl_games, skrl, sb3 |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Cartpole-v0 | | Manager Based | rsl_rl, rl_games, skrl, sb3 |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Unitree-A1-v0 | Isaac-Velocity-Flat-Unitree-A1-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Unitree-A1-v0 | Isaac-Velocity-Rough-Unitree-A1-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Anymal-B-v0 | Isaac-Velocity-Flat-Anymal-B-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Anymal-B-v0 | Isaac-Velocity-Rough-Anymal-B-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Anymal-C-v0 | Isaac-Velocity-Flat-Anymal-C-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Anymal-C-v0 | Isaac-Velocity-Rough-Anymal-C-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Anymal-D-v0 | Isaac-Velocity-Flat-Anymal-D-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Anymal-D-v0 | Isaac-Velocity-Rough-Anymal-D-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Cassie-v0 | Isaac-Velocity-Flat-Cassie-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Cassie-v0 | Isaac-Velocity-Rough-Cassie-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-G1-v0 | Isaac-Velocity-Rough-G1-Play-v0 | Manager Based | rsl_rl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-G1-v0 | Isaac-Velocity-Flat-G1-Play-v0 | Manager Based | rsl_rl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Unitree-Go1-v0 | Isaac-Velocity-Flat-Unitree-Go1-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Unitree-Go1-v0 | Isaac-Velocity-Rough-Unitree-Go1-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Unitree-Go2-v0 | Isaac-Velocity-Flat-Unitree-Go2-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-Unitree-Go2-v0 | Isaac-Velocity-Rough-Unitree-Go2-Play-v0 | Manager Based | rsl_rl, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Rough-H1-v0 | Isaac-Velocity-Rough-H1-Play-v0 | Manager Based | rsl_rl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-H1-v0 | Isaac-Velocity-Flat-H1-Play-v0 | Manager Based | rsl_rl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Velocity-Flat-Spot-v0 | Isaac-Velocity-Flat-Spot-Play-v0 | Manager Based | rsl_rl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Open-Drawer-Franka-v0 | Isaac-Open-Drawer-Franka-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Open-Drawer-Franka-IK-Abs-v0 | | Manager Based | |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Open-Drawer-Franka-IK-Rel-v0 | | Manager Based | |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Repose-Cube-Allegro-v0 | Isaac-Repose-Cube-Allegro-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Repose-Cube-Allegro-NoVelObs-v0 | Isaac-Repose-Cube-Allegro-NoVelObs-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Lift-Cube-Franka-v0 | Isaac-Lift-Cube-Franka-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Lift-Cube-Franka-IK-Abs-v0 | | Manager Based | |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Lift-Cube-Franka-IK-Rel-v0 | | Manager Based | |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Reach-Franka-v0 | Isaac-Reach-Franka-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Reach-Franka-IK-Abs-v0 | | Manager Based | |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Reach-Franka-IK-Rel-v0 | | Manager Based | |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Reach-UR10-v0 | Isaac-Reach-UR10-Play-v0 | Manager Based | rsl_rl, rl_games, skrl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
| Isaac-Navigation-Flat-Anymal-C-v0 | Isaac-Navigation-Flat-Anymal-C-Play-v0 | Manager Based | rsl_rl |
+------------------------------------------------+------------------------------------------------+---------------+-----------------------------+
source/extensions/omni.isaac.lab_tasks/docs/CHANGELOG.rst
View file @ 7997bfec
Changelog
---------
0.10.4 (2024-09-10)
~~~~~~~~~~~~~~~~~~~
Added
^^^^^
* Added ``Isaac-Repose-Cube-Shadow-Vision-Direct-v0`` environment with heterogeneous proprioception and vision observations.
0.10.3 (2024-09-05)
~~~~~~~~~~~~~~~~~~~
......
source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/inhand_manipulation/inhand_manipulation_env.py
View file @ 7997bfec
......@@ -9,6 +9,7 @@ from __future__ import annotations
import numpy as np
import torch
from collections.abc import Sequence
from typing import TYPE_CHECKING
import omni.isaac.lab.sim as sim_utils
from omni.isaac.lab.assets import Articulation, RigidObject
......@@ -17,8 +18,9 @@ from omni.isaac.lab.markers import VisualizationMarkers
from omni.isaac.lab.sim.spawners.from_files import GroundPlaneCfg, spawn_ground_plane
from omni.isaac.lab.utils.math import quat_conjugate, quat_from_angle_axis, quat_mul, sample_uniform, saturate
from omni.isaac.lab_tasks.direct.allegro_hand import AllegroHandEnvCfg
from omni.isaac.lab_tasks.direct.shadow_hand import ShadowHandEnvCfg
if TYPE_CHECKING:
from omni.isaac.lab_tasks.direct.allegro_hand import AllegroHandEnvCfg
from omni.isaac.lab_tasks.direct.shadow_hand import ShadowHandEnvCfg
class InHandManipulationEnv(DirectRLEnv):
......
source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/shadow_hand/__init__.py
View file @ 7997bfec
......@@ -11,6 +11,7 @@ import gymnasium as gym
from . import agents
from .shadow_hand_env_cfg import ShadowHandEnvCfg, ShadowHandOpenAIEnvCfg
from .shadow_hand_vision_env import ShadowHandVisionEnvCfg, ShadowHandVisionEnvPlayCfg
##
# Register Gym environments.
......@@ -47,3 +48,27 @@ gym.register(
"rl_games_cfg_entry_point": f"{agents.__name__}:rl_games_ppo_lstm_cfg.yaml",
},
)
### Vision
gym.register(
id="Isaac-Repose-Cube-Shadow-Vision-Direct-v0",
entry_point="omni.isaac.lab_tasks.direct.shadow_hand.shadow_hand_vision_env:ShadowHandVisionEnv",
disable_env_checker=True,
kwargs={
"env_cfg_entry_point": ShadowHandVisionEnvCfg,
"rsl_rl_cfg_entry_point": f"{agents.__name__}.rsl_rl_ppo_cfg:ShadowHandVisionFFPPORunnerCfg",
"rl_games_cfg_entry_point": f"{agents.__name__}:rl_games_ppo_vision_cfg.yaml",
},
)
gym.register(
id="Isaac-Repose-Cube-Shadow-Vision-Direct-Play-v0",
entry_point="omni.isaac.lab_tasks.direct.shadow_hand.shadow_hand_vision_env:ShadowHandVisionEnv",
disable_env_checker=True,
kwargs={
"env_cfg_entry_point": ShadowHandVisionEnvPlayCfg,
"rsl_rl_cfg_entry_point": f"{agents.__name__}.rsl_rl_ppo_cfg:ShadowHandVisionFFPPORunnerCfg",
"rl_games_cfg_entry_point": f"{agents.__name__}:rl_games_ppo_vision_cfg.yaml",
},
)
source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/shadow_hand/agents/rl_games_ppo_vision_cfg.yaml 0 → 100644
View file @ 7997bfec
params:
seed: 42
# environment wrapper clipping
env:
# added to the wrapper
clip_observations: 5.0
# can make custom wrapper?
clip_actions: 1.0
algo:
name: a2c_continuous
model:
name: continuous_a2c_logstd
network:
name: actor_critic
separate: False
space:
continuous:
mu_activation: None
sigma_activation: None
mu_init:
name: default
sigma_init:
name: const_initializer
val: 0
fixed_sigma: True
mlp:
units: [1024, 512, 512, 256, 128]
activation: elu
d2rl: False
initializer:
name: default
regularizer:
name: None
load_checkpoint: False # flag which sets whether to load the checkpoint
load_path: '' # path to the checkpoint to load
config:
name: shadow_hand_vision
env_name: rlgpu
device: 'cuda:0'
device_name: 'cuda:0'
multi_gpu: False
ppo: True
mixed_precision: False
normalize_input: True
normalize_value: True
num_actors: -1
reward_shaper:
scale_value: 0.01
normalize_advantage: True
gamma: 0.99
tau: 0.95
learning_rate: 5e-4
lr_schedule: adaptive
schedule_type: standard
kl_threshold: 0.01
score_to_win: 100000
max_epochs: 50000
save_best_after: 100
save_frequency: 200
print_stats: True
grad_norm: 1.0
entropy_coef: 0.0
truncate_grads: True
e_clip: 0.2
horizon_length: 64
minibatch_size: 19600
mini_epochs: 5
critic_coef: 4
clip_value: True
seq_length: 4
bounds_loss_coef: 0.0001
central_value_config:
minibatch_size: 19600
mini_epochs: 5
learning_rate: 5e-4
lr_schedule: adaptive
schedule_type: standard
kl_threshold: 0.01
clip_value: True
normalize_input: True
truncate_grads: True
network:
name: actor_critic
central_value: True
mlp:
units: [1024, 512, 512, 256, 128]
activation: elu
d2rl: False
initializer:
name: default
regularizer:
name: None
player:
deterministic: True
games_num: 100000
print_stats: True
source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/shadow_hand/agents/rsl_rl_ppo_cfg.py
View file @ 7997bfec
......@@ -68,3 +68,32 @@ class ShadowHandAsymFFPPORunnerCfg(RslRlOnPolicyRunnerCfg):
desired_kl=0.01,
max_grad_norm=1.0,
)
@configclass
class ShadowHandVisionFFPPORunnerCfg(RslRlOnPolicyRunnerCfg):
num_steps_per_env = 64
max_iterations = 50000
save_interval = 250
experiment_name = "shadow_hand_vision"
empirical_normalization = True
policy = RslRlPpoActorCriticCfg(
init_noise_std=1.0,
actor_hidden_dims=[1024, 512, 512, 256, 128],
critic_hidden_dims=[1024, 512, 512, 256, 128],
activation="elu",
)
algorithm = RslRlPpoAlgorithmCfg(
value_loss_coef=1.0,
use_clipped_value_loss=True,
clip_param=0.2,
entropy_coef=0.005,
num_learning_epochs=5,
num_mini_batches=4,
learning_rate=5.0e-4,
schedule="adaptive",
gamma=0.99,
lam=0.95,
desired_kl=0.01,
max_grad_norm=1.0,
)
source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/shadow_hand/feature_extractor.py 0 → 100644
View file @ 7997bfec
# Copyright (c) 2022-2024, The Isaac Lab Project Developers.
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
import glob
import os
import torch
import torch.nn as nn
import torchvision
from omni.isaac.lab.sensors import save_images_to_file
from omni.isaac.lab.utils import configclass
class FeatureExtractorNetwork(nn.Module):
"""CNN architecture used to regress keypoint positions of the in-hand cube from image data."""
def __init__(self):
super().__init__()
num_channel = 7
self.cnn = nn.Sequential(
nn.Conv2d(num_channel, 16, kernel_size=6, stride=2, padding=0),
nn.ReLU(),
nn.LayerNorm([16, 58, 58]),
nn.Conv2d(16, 32, kernel_size=4, stride=2, padding=0),
nn.ReLU(),
nn.LayerNorm([32, 28, 28]),
nn.Conv2d(32, 64, kernel_size=4, stride=2, padding=0),
nn.ReLU(),
nn.LayerNorm([64, 13, 13]),
nn.Conv2d(64, 128, kernel_size=3, stride=2, padding=0),
nn.ReLU(),
nn.LayerNorm([128, 6, 6]),
nn.AvgPool2d(6),
)
self.linear = nn.Sequential(
nn.Linear(128, 27),
)
self.data_transforms = torchvision.transforms.Compose([
torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])
def forward(self, x):
x = x.permute(0, 3, 1, 2)
x[:, 0:3, :, :] = self.data_transforms(x[:, 0:3, :, :])
x[:, 4:7, :, :] = self.data_transforms(x[:, 4:7, :, :])
cnn_x = self.cnn(x)
out = self.linear(cnn_x.view(-1, 128))
return out
@configclass
class FeatureExtractorCfg:
"""Configuration for the feature extractor model."""
train: bool = True
"""If True, the feature extractor model is trained during the rollout process. Default is False."""
load_checkpoint: bool = False
"""If True, the feature extractor model is loaded from a checkpoint. Default is False."""
write_image_to_file: bool = False
"""If True, the images from the camera sensor are written to file. Default is False."""
class FeatureExtractor:
"""Class for extracting features from image data.
It uses a CNN to regress keypoint positions from normalized RGB, depth, and segmentation images.
If the train flag is set to True, the CNN is trained during the rollout process.
"""
def __init__(self, cfg: FeatureExtractorCfg, device: str):
"""Initialize the feature extractor model.
Args:
cfg (FeatureExtractorCfg): Configuration for the feature extractor model.
device (str): Device to run the model on.
"""
self.cfg = cfg
self.device = device
# Feature extractor model
self.feature_extractor = FeatureExtractorNetwork()
self.feature_extractor.to(self.device)
self.step_count = 0
self.log_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), "logs")
if not os.path.exists(self.log_dir):
os.makedirs(self.log_dir)
if self.cfg.load_checkpoint:
list_of_files = glob.glob(self.log_dir + "/*.pth")
latest_file = max(list_of_files, key=os.path.getctime)
checkpoint = os.path.join(self.log_dir, latest_file)
print(f"[INFO]: Loading feature extractor checkpoint from {checkpoint}")
self.feature_extractor.load_state_dict(torch.load(checkpoint, weights_only=True))
if self.cfg.train:
self.optimizer = torch.optim.Adam(self.feature_extractor.parameters(), lr=1e-4)
self.l2_loss = nn.MSELoss()
self.feature_extractor.train()
else:
self.feature_extractor.eval()
def _preprocess_images(
self, rgb_img: torch.Tensor, depth_img: torch.Tensor, segmentation_img: torch.Tensor
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
"""Preprocesses the input images.
Args:
rgb_img (torch.Tensor): RGB image tensor. Shape: (N, H, W, 3).
depth_img (torch.Tensor): Depth image tensor. Shape: (N, H, W, 1).
segmentation_img (torch.Tensor): Segmentation image tensor. Shape: (N, H, W, 3)
Returns:
tuple[torch.Tensor, torch.Tensor, torch.Tensor]: Preprocessed RGB, depth, and segmentation
"""
rgb_img = rgb_img / 255.0
# process depth image
depth_img[depth_img == float("inf")] = 0
depth_img /= 5.0
depth_img /= torch.max(depth_img)
# process segmentation image
segmentation_img = segmentation_img / 255.0
mean_tensor = torch.mean(segmentation_img, dim=(1, 2), keepdim=True)
segmentation_img -= mean_tensor
return rgb_img, depth_img, segmentation_img
def _save_images(self, rgb_img: torch.Tensor, depth_img: torch.Tensor, segmentation_img: torch.Tensor):
"""Writes image buffers to file.
Args:
rgb_img (torch.Tensor): RGB image tensor. Shape: (N, H, W, 3).
depth_img (torch.Tensor): Depth image tensor. Shape: (N, H, W, 1).
segmentation_img (torch.Tensor): Segmentation image tensor. Shape: (N, H, W, 3).
"""
save_images_to_file(rgb_img, "shadow_hand_rgb.png")
save_images_to_file(depth_img, "shadow_hand_depth.png")
save_images_to_file(segmentation_img, "shadow_hand_segmentation.png")
def step(
self, rgb_img: torch.Tensor, depth_img: torch.Tensor, segmentation_img: torch.Tensor, gt_pose: torch.Tensor
) -> tuple[torch.Tensor, torch.Tensor]:
"""Extracts the features using the images and trains the model if the train flag is set to True.
Args:
rgb_img (torch.Tensor): RGB image tensor. Shape: (N, H, W, 3).
depth_img (torch.Tensor): Depth image tensor. Shape: (N, H, W, 1).
segmentation_img (torch.Tensor): Segmentation image tensor. Shape: (N, H, W, 3).
gt_pose (torch.Tensor): Ground truth pose tensor (position and corners). Shape: (N, 27).
Returns:
tuple[torch.Tensor, torch.Tensor]: Pose loss and predicted pose.
"""
rgb_img, depth_img, segmentation_img = self._preprocess_images(rgb_img, depth_img, segmentation_img)
if self.cfg.write_image_to_file:
self._save_images(rgb_img, depth_img, segmentation_img)
if self.cfg.train:
with torch.enable_grad():
with torch.inference_mode(False):
img_input = torch.cat((rgb_img, depth_img, segmentation_img), dim=-1)
self.optimizer.zero_grad()
predicted_pose = self.feature_extractor(img_input).squeeze()
pose_loss = self.l2_loss(predicted_pose, gt_pose.clone()) * 100
pose_loss.backward()
self.optimizer.step()
self.step_count += 1
if self.step_count % 50000 == 0:
torch.save(
self.feature_extractor.state_dict(),
os.path.join(self.log_dir, f"cnn_{self.step_count}_{pose_loss.detach().cpu().numpy()}.pth"),
)
return pose_loss, predicted_pose
else:
img_input = torch.cat((rgb_img, depth_img, segmentation_img), dim=-1)
predicted_pose = self.feature_extractor(img_input).squeeze()
return None, predicted_pose
source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/direct/shadow_hand/shadow_hand_vision_env.py 0 → 100644
View file @ 7997bfec
# Copyright (c) 2022-2024, The Isaac Lab Project Developers.
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
from __future__ import annotations
import torch
import omni.usd
from pxr import Semantics
import omni.isaac.lab.sim as sim_utils
from omni.isaac.lab.assets import Articulation, RigidObject
from omni.isaac.lab.scene import InteractiveSceneCfg
from omni.isaac.lab.sensors import TiledCamera, TiledCameraCfg
from omni.isaac.lab.utils import configclass
from omni.isaac.lab.utils.math import quat_apply
from omni.isaac.lab_tasks.direct.inhand_manipulation.inhand_manipulation_env import InHandManipulationEnv, unscale
from .feature_extractor import FeatureExtractor, FeatureExtractorCfg
from .shadow_hand_env_cfg import ShadowHandEnvCfg
@configclass
class ShadowHandVisionEnvCfg(ShadowHandEnvCfg):
# scene
scene: InteractiveSceneCfg = InteractiveSceneCfg(num_envs=1225, env_spacing=2.0, replicate_physics=True)
# camera
tiled_camera: TiledCameraCfg = TiledCameraCfg(
prim_path="/World/envs/env_.*/Camera",
offset=TiledCameraCfg.OffsetCfg(pos=(0, -0.35, 1.0), rot=(0.7071, 0.0, 0.7071, 0.0), convention="world"),
data_types=["rgb", "depth", "semantic_segmentation"],
spawn=sim_utils.PinholeCameraCfg(
focal_length=24.0, focus_distance=400.0, horizontal_aperture=20.955, clipping_range=(0.1, 20.0)
),
width=120,
height=120,
)
feature_extractor = FeatureExtractorCfg()
# env
num_observations = 188 + 27 # state observation + vision CNN embedding
num_states = 187 + 27 # asymettric states + vision CNN embedding
@configclass
class ShadowHandVisionEnvPlayCfg(ShadowHandVisionEnvCfg):
# scene
scene: InteractiveSceneCfg = InteractiveSceneCfg(num_envs=64, env_spacing=2.0, replicate_physics=True)
# inference for CNN
feature_extractor = FeatureExtractorCfg(train=False, load_checkpoint=True)
class ShadowHandVisionEnv(InHandManipulationEnv):
cfg: ShadowHandVisionEnvCfg
def __init__(self, cfg: ShadowHandVisionEnvCfg, render_mode: str | None = None, **kwargs):
super().__init__(cfg, render_mode, **kwargs)
self.feature_extractor = FeatureExtractor(self.cfg.feature_extractor, self.device)
# hide goal cubes
self.goal_pos[:, :] = torch.tensor([-0.2, 0.1, 0.6], device=self.device)
# keypoints buffer
self.gt_keypoints = torch.ones(self.num_envs, 8, 3, dtype=torch.float32, device=self.device)
self.goal_keypoints = torch.ones(self.num_envs, 8, 3, dtype=torch.float32, device=self.device)
def _setup_scene(self):
# add hand, in-hand object, and goal object
self.hand = Articulation(self.cfg.robot_cfg)
self.object = RigidObject(self.cfg.object_cfg)
self._tiled_camera = TiledCamera(self.cfg.tiled_camera)
# get stage
stage = omni.usd.get_context().get_stage()
# add semantics for in-hand cube
prim = stage.GetPrimAtPath("/World/envs/env_0/object")
sem = Semantics.SemanticsAPI.Apply(prim, "Semantics")
sem.CreateSemanticTypeAttr()
sem.CreateSemanticDataAttr()
sem.GetSemanticTypeAttr().Set("class")
sem.GetSemanticDataAttr().Set("cube")
# clone and replicate (no need to filter for this environment)
self.scene.clone_environments(copy_from_source=False)
# add articultion to scene - we must register to scene to randomize with EventManager
self.scene.articulations["robot"] = self.hand
self.scene.rigid_objects["object"] = self.object
self.scene.sensors["tiled_camera"] = self._tiled_camera
# add lights
light_cfg = sim_utils.DomeLightCfg(intensity=2000.0, color=(0.75, 0.75, 0.75))
light_cfg.func("/World/Light", light_cfg)
# hide default lights when running with viewer
if stage.GetPrimAtPath("/Environment/defaultLight"):
stage.GetPrimAtPath("/Environment/defaultLight").GetAttribute("visibility").Set("invisible")
def _compute_image_observations(self):
# generate ground truth keypoints for in-hand cube
compute_keypoints(pose=torch.cat((self.object_pos, self.object_rot), dim=1), out=self.gt_keypoints)
object_pose = torch.cat([self.object_pos, self.gt_keypoints.view(-1, 24)], dim=-1)
# train CNN to regress on keypoint positions
pose_loss, embeddings = self.feature_extractor.step(
self._tiled_camera.data.output["rgb"],
self._tiled_camera.data.output["depth"],
self._tiled_camera.data.output["semantic_segmentation"][..., :3],
object_pose,
)
self.embeddings = embeddings.clone().detach()
# compute keypoints for goal cube
compute_keypoints(
pose=torch.cat((torch.zeros_like(self.goal_pos), self.goal_rot), dim=-1), out=self.goal_keypoints
)
obs = torch.cat(
(
self.embeddings,
self.goal_keypoints.view(-1, 24),
),
dim=-1,
)
# log pose loss from CNN training
if "log" not in self.extras:
self.extras["log"] = dict()
self.extras["log"]["pose_loss"] = pose_loss
return obs
def _compute_proprio_observations(self):
"""Proprioception observations from physics."""
obs = torch.cat(
(
# hand
unscale(self.hand_dof_pos, self.hand_dof_lower_limits, self.hand_dof_upper_limits),
self.cfg.vel_obs_scale * self.hand_dof_vel,
# goal
self.in_hand_pos,
self.goal_rot,
# fingertips
self.fingertip_pos.view(self.num_envs, self.num_fingertips * 3),
self.fingertip_rot.view(self.num_envs, self.num_fingertips * 4),
self.fingertip_velocities.view(self.num_envs, self.num_fingertips * 6),
# actions
self.actions,
),
dim=-1,
)
return obs
def _compute_states(self):
"""Asymmetric states for the critic."""
sim_states = self.compute_full_state()
state = torch.cat((sim_states, self.embeddings), dim=-1)
return state
def _get_observations(self) -> dict:
# proprioception observations
state_obs = self._compute_proprio_observations()
# vision observations from CMM
image_obs = self._compute_image_observations()
obs = torch.cat((state_obs, image_obs), dim=-1)
# asymmetric critic states
self.fingertip_force_sensors = self.hand.root_physx_view.get_link_incoming_joint_force()[:, self.finger_bodies]
state = self._compute_states()
observations = {"policy": obs, "critic": state}
return observations
@torch.jit.script
def compute_keypoints(
pose: torch.Tensor,
num_keypoints: int = 8,
size: tuple[float, float, float] = (2 * 0.03, 2 * 0.03, 2 * 0.03),
out: torch.Tensor | None = None,
):
"""Computes positions of 8 corner keypoints of a cube.
Args:
pose: Position and orientation of the center of the cube. Shape is (N, 7)
num_keypoints: Number of keypoints to compute. Default = 8
size: Length of X, Y, Z dimensions of cube. Default = [0.06, 0.06, 0.06]
out: Buffer to store keypoints. If None, a new buffer will be created.
"""
num_envs = pose.shape[0]
if out is None:
out = torch.ones(num_envs, num_keypoints, 3, dtype=torch.float32, device=pose.device)
else:
out[:] = 1.0
for i in range(num_keypoints):
# which dimensions to negate
n = [((i >> k) & 1) == 0 for k in range(3)]
corner_loc = ([(1 if n[k] else -1) * s / 2 for k, s in enumerate(size)],)
corner = torch.tensor(corner_loc, dtype=torch.float32, device=pose.device) * out[:, i, :]
# express corner position in the world frame
out[:, i, :] = pose[:, :3] + quat_apply(pose[:, 3:7], corner)
return out
source/extensions/omni.isaac.lab_tasks/test/test_environments.py
View file @ 7997bfec
......@@ -21,6 +21,8 @@ import unittest
import carb
import omni.usd
from omni.isaac.lab.envs import ManagerBasedRLEnvCfg
import omni.isaac.lab_tasks # noqa: F401
from omni.isaac.lab_tasks.utils.parse_cfg import parse_env_cfg
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment