Commit 0224a373 authored by peterd-NV's avatar peterd-NV Committed by Kelly Guo

Updates pick place env to use steering wheel asset (#447)

<!--
Thank you for your interest in sending a pull request. Please make sure
to check the contribution guidelines.

Link:
https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html
-->

This change updates the Isaac-PickPlace-GR1T2-Abs-v0 env to use a
steering wheel asset instead of a metal rod. The steering wheel asset is
easier to grasp.

Changes:

1. Update Isaac-PickPlace-GR1T2-Abs-v0 to use steering wheel asset
2. Update Mimic docs to reflect the change

Fixes # (issue)

Fixes poor graspability of the object once it falls in the
Isaac-PickPlace-GR1T2-Abs-v0 env.

<!-- As you go through the list, delete the ones that are not
applicable. -->

- New feature (non-breaking change which adds functionality)
- This change requires a documentation update

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->

---------
Signed-off-by: 's avatarKelly Guo <kellyguo123@hotmail.com>
Co-authored-by: 's avatarKelly Guo <kellyg@nvidia.com>
parent 34b604ba
...@@ -136,7 +136,7 @@ Pre-recorded demonstrations ...@@ -136,7 +136,7 @@ Pre-recorded demonstrations
^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^
We provide a pre-recorded ``dataset.hdf5`` containing 10 human demonstrations for ``Isaac-Stack-Cube-Franka-IK-Rel-v0`` We provide a pre-recorded ``dataset.hdf5`` containing 10 human demonstrations for ``Isaac-Stack-Cube-Franka-IK-Rel-v0``
`here <https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0/Isaac/IsaacLab/Mimic/dataset.hdf5>`_. `here <https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0/Isaac/IsaacLab/Mimic/franka_stack_datasets/dataset.hdf5>`_.
This dataset may be downloaded and used in the remaining tutorial steps if you do not wish to collect your own demonstrations. This dataset may be downloaded and used in the remaining tutorial steps if you do not wish to collect your own demonstrations.
.. note:: .. note::
...@@ -307,10 +307,10 @@ By inferencing using the generated model, we can visualize the results of the po ...@@ -307,10 +307,10 @@ By inferencing using the generated model, we can visualize the results of the po
--checkpoint /PATH/TO/desired_model_checkpoint.pth --checkpoint /PATH/TO/desired_model_checkpoint.pth
Demo: Data Generation and Policy Training for a Humanoid Robot Demo 1: Data Generation and Policy Training for a Humanoid Robot
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/gr-1_pick_place.gif .. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/gr-1_steering_wheel_pick_place.gif
:width: 100% :width: 100%
:align: center :align: center
:alt: GR-1 humanoid robot performing a pick and place task :alt: GR-1 humanoid robot performing a pick and place task
...@@ -352,11 +352,11 @@ This setup allows Isaac Lab Mimic to interpolate the right hand's trajectory acc ...@@ -352,11 +352,11 @@ This setup allows Isaac Lab Mimic to interpolate the right hand's trajectory acc
Therefore, avoid moving the right hand while the left hand picks up the object and brings it to a stable position. Therefore, avoid moving the right hand while the left hand picks up the object and brings it to a stable position.
.. |good_demo| image:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/gr1_pick_place_good_human_demo.gif .. |good_demo| image:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/gr-1_steering_wheel_pick_place_good_demo.gif
:width: 49% :width: 49%
:alt: GR-1 humanoid robot performing a good pick and place demonstration :alt: GR-1 humanoid robot performing a good pick and place demonstration
.. |bad_demo| image:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/gr1_pick_place_bad_human_demo.gif .. |bad_demo| image:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/gr-1_steering_wheel_pick_place_bad_demo.gif
:width: 49% :width: 49%
:alt: GR-1 humanoid robot performing a bad pick and place demonstration :alt: GR-1 humanoid robot performing a bad pick and place demonstration
...@@ -459,7 +459,7 @@ Generate the dataset ...@@ -459,7 +459,7 @@ Generate the dataset
^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^
If you skipped the prior collection and annotation step, download the pre-recorded annotated dataset ``dataset_annotated_gr1.hdf5`` from If you skipped the prior collection and annotation step, download the pre-recorded annotated dataset ``dataset_annotated_gr1.hdf5`` from
`here <https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0/Isaac/IsaacLab/Mimic/dataset_annotated_gr1.hdf5>`_. `here <https://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/5.0/Isaac/IsaacLab/Mimic/pick_place_datasets/dataset_annotated_gr1.hdf5>`_.
Place the file under ``IsaacLab/datasets`` and run the following command to generate a new dataset with 1000 demonstrations. Place the file under ``IsaacLab/datasets`` and run the following command to generate a new dataset with 1000 demonstrations.
.. code:: bash .. code:: bash
...@@ -499,7 +499,7 @@ Visualize the results of the trained policy by running the following command, us ...@@ -499,7 +499,7 @@ Visualize the results of the trained policy by running the following command, us
--enable_pinocchio \ --enable_pinocchio \
--task Isaac-PickPlace-GR1T2-Abs-v0 \ --task Isaac-PickPlace-GR1T2-Abs-v0 \
--num_rollouts 50 \ --num_rollouts 50 \
--horizon 250 \ --horizon 400 \
--norm_factor_min <NORM_FACTOR_MIN> \ --norm_factor_min <NORM_FACTOR_MIN> \
--norm_factor_max <NORM_FACTOR_MAX> \ --norm_factor_max <NORM_FACTOR_MAX> \
--checkpoint /PATH/TO/desired_model_checkpoint.pth --checkpoint /PATH/TO/desired_model_checkpoint.pth
...@@ -507,7 +507,7 @@ Visualize the results of the trained policy by running the following command, us ...@@ -507,7 +507,7 @@ Visualize the results of the trained policy by running the following command, us
.. note:: .. note::
Change the ``NORM_FACTOR`` in the above command with the values generated in the training step. Change the ``NORM_FACTOR`` in the above command with the values generated in the training step.
.. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/gr-1_pick_place_policy.gif .. figure:: https://download.isaacsim.omniverse.nvidia.com/isaaclab/images/gr-1_steering_wheel_pick_place_policy.gif
:width: 100% :width: 100%
:align: center :align: center
:alt: GR-1 humanoid robot performing a pick and place task :alt: GR-1 humanoid robot performing a pick and place task
...@@ -516,8 +516,8 @@ Visualize the results of the trained policy by running the following command, us ...@@ -516,8 +516,8 @@ Visualize the results of the trained policy by running the following command, us
The trained policy performing the pick and place task in Isaac Lab. The trained policy performing the pick and place task in Isaac Lab.
Demo: Visuomotor Policy for a Humanoid Robot Demo 2: Visuomotor Policy for a Humanoid Robot
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Download the Dataset Download the Dataset
^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^
...@@ -526,6 +526,52 @@ Download the pre-generated dataset from `here <https://download.isaacsim.omniver ...@@ -526,6 +526,52 @@ Download the pre-generated dataset from `here <https://download.isaacsim.omniver
The dataset contains 1000 demonstrations of a humanoid robot performing a pouring/placing task that was The dataset contains 1000 demonstrations of a humanoid robot performing a pouring/placing task that was
generated using Isaac Lab Mimic for the ``Isaac-NutPour-GR1T2-Pink-IK-Abs-Mimic-v0`` task. generated using Isaac Lab Mimic for the ``Isaac-NutPour-GR1T2-Pink-IK-Abs-Mimic-v0`` task.
.. hint::
If desired, data collection, annotation, and generation can be done using the same commands as the prior examples.
**Note that the following commands are only for your reference and are not required for this demo.**
To collect demonstrations:
.. code:: bash
./isaaclab.sh -p scripts/tools/record_demos.py \
--device cpu \
--task Isaac-NutPour-GR1T2-Pink-IK-Abs-v0 \
--teleop_device handtracking \
--dataset_file ./datasets/dataset_gr1_nut_pouring.hdf5 \
--num_demos 5 --enable_pinocchio
Since this is a visuomotor environment, the ``--enable_cameras`` flag must be added to the annotation and data generation commands.
To annotate the demonstrations:
.. code:: bash
./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/annotate_demos.py \
--device cpu \
--enable_cameras \
--rendering_mode balanced \
--task Isaac-NutPour-GR1T2-Pink-IK-Abs-Mimic-v0 \
--input_file ./datasets/dataset_gr1_nut_pouring.hdf5 \
--output_file ./datasets/dataset_annotated_gr1_nut_pouring.hdf5 --enable_pinocchio
To generate the dataset:
.. code:: bash
./isaaclab.sh -p scripts/imitation_learning/isaaclab_mimic/generate_dataset.py \
--device cpu \
--headless \
--enable_pinocchio \
--enable_cameras \
--rendering_mode balanced \
--task Isaac-NutPour-GR1T2-Pink-IK-Abs-Mimic-v0 \
--generation_num_trials 1000 \
--num_envs 5 \
--input_file ./datasets/dataset_annotated_gr1_nut_pouring.hdf5 \
--output_file ./datasets/generated_dataset_gr1_nut_pouring.hdf5
Train a policy Train a policy
^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
......
[package] [package]
# Note: Semantic Versioning is used: https://semver.org/ # Note: Semantic Versioning is used: https://semver.org/
version = "0.10.38" version = "0.10.39"
# Description # Description
title = "Isaac Lab Environments" title = "Isaac Lab Environments"
......
Changelog Changelog
--------- ---------
0.10.38 (2025-06-26) 0.10.39 (2025-06-26)
~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
Fixed Fixed
...@@ -10,7 +10,7 @@ Fixed ...@@ -10,7 +10,7 @@ Fixed
* Relaxed upper range pin for protobuf python dependency for more permissive installation. * Relaxed upper range pin for protobuf python dependency for more permissive installation.
0.10.37 (2025-05-22) 0.10.38 (2025-05-22)
~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
Fixed Fixed
...@@ -19,7 +19,7 @@ Fixed ...@@ -19,7 +19,7 @@ Fixed
* Fixed redundant body_names assignment in rough_env_cfg.py for H1 robot. * Fixed redundant body_names assignment in rough_env_cfg.py for H1 robot.
0.10.36 (2025-06-16) 0.10.37 (2025-06-16)
~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
Changed Changed
...@@ -28,7 +28,7 @@ Changed ...@@ -28,7 +28,7 @@ Changed
* Show available RL library configs on error message when an entry point key is not available for a given task. * Show available RL library configs on error message when an entry point key is not available for a given task.
0.10.35 (2025-05-15) 0.10.36 (2025-05-15)
~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
Added Added
...@@ -38,7 +38,7 @@ Added ...@@ -38,7 +38,7 @@ Added
implements assembly tasks to insert pegs into their corresponding sockets. implements assembly tasks to insert pegs into their corresponding sockets.
0.10.34 (2025-05-21) 0.10.35 (2025-05-21)
~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
Added Added
...@@ -48,6 +48,15 @@ Added ...@@ -48,6 +48,15 @@ Added
can be pushed to a visualization dashboard to track improvements or regressions. can be pushed to a visualization dashboard to track improvements or regressions.
0.10.34 (2025-05-22)
~~~~~~~~~~~~~~~~~~~~
Changed
^^^^^^^
* Changed ``Isaac-PickPlace-GR1T2-Abs-v0`` object asset to a steering wheel.
0.10.33 (2025-05-12) 0.10.33 (2025-05-12)
~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
......
...@@ -26,15 +26,15 @@ if TYPE_CHECKING: ...@@ -26,15 +26,15 @@ if TYPE_CHECKING:
from isaaclab.envs import ManagerBasedRLEnv from isaaclab.envs import ManagerBasedRLEnv
def task_done( def task_done_pick_place(
env: ManagerBasedRLEnv, env: ManagerBasedRLEnv,
object_cfg: SceneEntityCfg = SceneEntityCfg("object"), object_cfg: SceneEntityCfg = SceneEntityCfg("object"),
right_wrist_max_x: float = 0.26, right_wrist_max_x: float = 0.26,
min_x: float = 0.30, min_x: float = 0.40,
max_x: float = 0.95, max_x: float = 0.85,
min_y: float = 0.25, min_y: float = 0.35,
max_y: float = 0.66, max_y: float = 0.60,
min_height: float = 1.13, max_height: float = 1.10,
min_vel: float = 0.20, min_vel: float = 0.20,
) -> torch.Tensor: ) -> torch.Tensor:
"""Determine if the object placement task is complete. """Determine if the object placement task is complete.
...@@ -53,7 +53,7 @@ def task_done( ...@@ -53,7 +53,7 @@ def task_done(
max_x: Maximum x position of the object for task completion. max_x: Maximum x position of the object for task completion.
min_y: Minimum y position of the object for task completion. min_y: Minimum y position of the object for task completion.
max_y: Maximum y position of the object for task completion. max_y: Maximum y position of the object for task completion.
min_height: Minimum height (z position) of the object for task completion. max_height: Maximum height (z position) of the object for task completion.
min_vel: Minimum velocity magnitude of the object for task completion. min_vel: Minimum velocity magnitude of the object for task completion.
Returns: Returns:
...@@ -63,10 +63,10 @@ def task_done( ...@@ -63,10 +63,10 @@ def task_done(
object: RigidObject = env.scene[object_cfg.name] object: RigidObject = env.scene[object_cfg.name]
# Extract wheel position relative to environment origin # Extract wheel position relative to environment origin
wheel_x = object.data.root_pos_w[:, 0] - env.scene.env_origins[:, 0] object_x = object.data.root_pos_w[:, 0] - env.scene.env_origins[:, 0]
wheel_y = object.data.root_pos_w[:, 1] - env.scene.env_origins[:, 1] object_y = object.data.root_pos_w[:, 1] - env.scene.env_origins[:, 1]
wheel_height = object.data.root_pos_w[:, 2] - env.scene.env_origins[:, 2] object_height = object.data.root_pos_w[:, 2] - env.scene.env_origins[:, 2]
wheel_vel = torch.abs(object.data.root_vel_w) object_vel = torch.abs(object.data.root_vel_w)
# Get right wrist position relative to environment origin # Get right wrist position relative to environment origin
robot_body_pos_w = env.scene["robot"].data.body_pos_w robot_body_pos_w = env.scene["robot"].data.body_pos_w
...@@ -74,15 +74,15 @@ def task_done( ...@@ -74,15 +74,15 @@ def task_done(
right_wrist_x = robot_body_pos_w[:, right_eef_idx, 0] - env.scene.env_origins[:, 0] right_wrist_x = robot_body_pos_w[:, right_eef_idx, 0] - env.scene.env_origins[:, 0]
# Check all success conditions and combine with logical AND # Check all success conditions and combine with logical AND
done = wheel_x < max_x done = object_x < max_x
done = torch.logical_and(done, wheel_x > min_x) done = torch.logical_and(done, object_x > min_x)
done = torch.logical_and(done, wheel_y < max_y) done = torch.logical_and(done, object_y < max_y)
done = torch.logical_and(done, wheel_y > min_y) done = torch.logical_and(done, object_y > min_y)
done = torch.logical_and(done, wheel_height < min_height) done = torch.logical_and(done, object_height < max_height)
done = torch.logical_and(done, right_wrist_x < right_wrist_max_x) done = torch.logical_and(done, right_wrist_x < right_wrist_max_x)
done = torch.logical_and(done, wheel_vel[:, 0] < min_vel) done = torch.logical_and(done, object_vel[:, 0] < min_vel)
done = torch.logical_and(done, wheel_vel[:, 1] < min_vel) done = torch.logical_and(done, object_vel[:, 1] < min_vel)
done = torch.logical_and(done, wheel_vel[:, 2] < min_vel) done = torch.logical_and(done, object_vel[:, 2] < min_vel)
return done return done
...@@ -108,6 +108,7 @@ def task_done_nut_pour( ...@@ -108,6 +108,7 @@ def task_done_nut_pour(
This function checks whether all success conditions for the task have been met: This function checks whether all success conditions for the task have been met:
1. The factory nut is in the sorting bowl 1. The factory nut is in the sorting bowl
2. The sorting beaker is in the sorting bin
3. The sorting bowl is placed on the sorting scale 3. The sorting bowl is placed on the sorting scale
Args: Args:
......
...@@ -31,7 +31,7 @@ from isaaclab.managers import TerminationTermCfg as DoneTerm ...@@ -31,7 +31,7 @@ from isaaclab.managers import TerminationTermCfg as DoneTerm
from isaaclab.scene import InteractiveSceneCfg from isaaclab.scene import InteractiveSceneCfg
from isaaclab.sim.spawners.from_files.from_files_cfg import GroundPlaneCfg, UsdFileCfg from isaaclab.sim.spawners.from_files.from_files_cfg import GroundPlaneCfg, UsdFileCfg
from isaaclab.utils import configclass from isaaclab.utils import configclass
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR, ISAACLAB_NUCLEUS_DIR
from . import mdp from . import mdp
...@@ -54,24 +54,13 @@ class ObjectTableSceneCfg(InteractiveSceneCfg): ...@@ -54,24 +54,13 @@ class ObjectTableSceneCfg(InteractiveSceneCfg):
), ),
) )
# Object
object = RigidObjectCfg( object = RigidObjectCfg(
prim_path="{ENV_REGEX_NS}/Object", prim_path="{ENV_REGEX_NS}/Object",
init_state=RigidObjectCfg.InitialStateCfg(pos=[-0.35, 0.40, 1.1691], rot=[1, 0, 0, 0]), init_state=RigidObjectCfg.InitialStateCfg(pos=[-0.45, 0.45, 0.9996], rot=[1, 0, 0, 0]),
spawn=sim_utils.CylinderCfg( spawn=UsdFileCfg(
radius=0.018, usd_path=f"{ISAACLAB_NUCLEUS_DIR}/Mimic/pick_place_task/pick_place_assets/steering_wheel.usd",
height=0.35, scale=(0.75, 0.75, 0.75),
rigid_props=sim_utils.RigidBodyPropertiesCfg(), rigid_props=sim_utils.RigidBodyPropertiesCfg(),
mass_props=sim_utils.MassPropertiesCfg(mass=0.3),
collision_props=sim_utils.CollisionPropertiesCfg(),
visual_material=sim_utils.PreviewSurfaceCfg(diffuse_color=(0.15, 0.15, 0.15), metallic=1.0),
physics_material=sim_utils.RigidBodyMaterialCfg(
friction_combine_mode="max",
restitution_combine_mode="min",
static_friction=0.9,
dynamic_friction=0.9,
restitution=0.0,
),
), ),
) )
...@@ -300,7 +289,7 @@ class TerminationsCfg: ...@@ -300,7 +289,7 @@ class TerminationsCfg:
func=mdp.root_height_below_minimum, params={"minimum_height": 0.5, "asset_cfg": SceneEntityCfg("object")} func=mdp.root_height_below_minimum, params={"minimum_height": 0.5, "asset_cfg": SceneEntityCfg("object")}
) )
success = DoneTerm(func=mdp.task_done) success = DoneTerm(func=mdp.task_done_pick_place)
@configclass @configclass
...@@ -314,8 +303,8 @@ class EventCfg: ...@@ -314,8 +303,8 @@ class EventCfg:
mode="reset", mode="reset",
params={ params={
"pose_range": { "pose_range": {
"x": [-0.05, 0.0], "x": [-0.01, 0.01],
"y": [0.0, 0.05], "y": [-0.01, 0.01],
}, },
"velocity_range": {}, "velocity_range": {},
"asset_cfg": SceneEntityCfg("object"), "asset_cfg": SceneEntityCfg("object"),
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment