-
njawale42 authored
## Description This PR introduces the SkillGen framework to Isaac Lab, integrating GPU motion planning with skill-segmented data generation. It enables efficient, high-quality dataset creation with robust collision handling, visualization, and reproducibility. **Note:** - Please look at the cuRobo usage license  - Please look at updated isaacsim license  ### Technical Implementation: **Annotation Framework:** - Manual subtask start annotations to cleanly separate skill execution from motion-planning segments - Consistent trajectory segmentation for downstream dataset consumers **Motion Planning:** - **Base Motion Planner (Extensible):** - Introduces a reusable planner interface for uniform integration: - `source/isaaclab_mimic/isaaclab_mimic/motion_planners/base_motion_planner.py` - Defines a minimal, consistent API for planners: - `update_world_and_plan_motion(...)`, `get_planned_poses(...)`, etc. - The cuRobo planner inherits from this base class. - New planners can be added by subclassing the base class and implementing the same API, enabling drop-in replacement without changes to the SkillGen pipeline. - **CuRobo Planner** (GPU-accelerated, collision-aware): - Multi-phase planning: approach → contact → retreat - Dynamic object attach/detach and contact-aware sphere management - Real-time world synchronization between Isaac Lab and cuRobo - Configurable collision filtering for contact phases - **Tests**: - `source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_cube_stack.py` - `source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_franka.py` - `source/isaaclab_mimic/test/test_generate_dataset_skillgen.py` **Data Generation Pipeline:** - Automated dataset generation with precise skill-based segmentation - Integrates with existing observation/action spaces - Supports multi-env parallel collection with cuRobo-backed planning **Visualization and Debugging:** - Rerun-based 3D visualization for trajectory/collision inspection - Real-time sphere visualization for collision boundaries and contact phases ### Dependencies: - **cuRobo**: motion planning and collision checking - **Rerun**: 3D visualization and debugging ### Integration: This extends the existing mimic pipeline and remains backward compatible. It integrates into the manager-based environment structure and existing observation/action interfaces without breaking current users. ## Type of change - [x] New feature (non-breaking change which adds functionality) - [x] This change requires a documentation update ## Screenshot ### SkillGen Data Generation <table> <tr> <td align="center"><strong>Cube Stacking SkillGen Data Generation</strong></td> <td align="center"><strong>Bin Cube Stacking SkillGen Data Generation (Using Vanilla Cube Stacking Source Demos)</strong></td> </tr> <tr> <td> <img src="https://github.com/user-attachments/assets/de240b89-e670-4035-84ae-4101a4f70dae" alt="Cube Stacking Data Generation" style="width: 480px; height: 270px; object-fit: contain;"> </td> <td> <img src="https://github.com/user-attachments/assets/dd94e0a6-ad1b-4366-80c4-7ff96cabeb3e" alt="Bin Cube Stacking Data Generation" style="width: 480px; height: 270px; object-fit: contain;"> </td> </tr> </table> ### Bin Cube Stacking Behavior Cloned Policy  ### Rerun Integration  ### Motion Planner Tests <table> <tr> <td align="center"><strong>Obstacle Avoidance (cuRobo)</strong></td> <td align="center"><strong>Cube Stack End-to-End (cuRobo)</strong></td> </tr> <tr> <td> <img src="https://github.com/user-attachments/assets/a022342f-4c4e-42ea-a48c-cb1ea65c94db" alt="Obstacle Avoidance cuRobo" style="width: 480px; height: 270px; object-fit: contain;"> </td> <td> <img src="https://github.com/user-attachments/assets/7e6290b6-8322-4702-ae2f-f363a87badde" alt="Cube Stack End-to-End cuRobo" style="width: 480px; height: 270px; object-fit: contain;"> </td> </tr> </table> ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./isaaclab.sh --format` - [x] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [x] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [x] I have added my name to the `CONTRIBUTORS.md` or my name already exists there --------- Signed-off-by:
Kelly Guo <kellyg@nvidia.com>
Co-authored-by:
Kelly Guo <kellyg@nvidia.com>
Co-authored-by:
Pascal Roth <57946385+pascal-roth@users.noreply.github.com>
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| config | ||
| docs | ||
| isaaclab_mimic | ||
| test | ||
| pyproject.toml | ||
| setup.py |