• njawale42's avatar
    Adds SkillGen framework to Isaac Lab with cuRobo support (#3303) · 994979c2
    njawale42 authored
    ## Description
    
    This PR introduces the SkillGen framework to Isaac Lab, integrating GPU
    motion planning with skill-segmented data generation. It enables
    efficient, high-quality dataset creation with robust collision handling,
    visualization, and reproducibility.
    
    **Note:** 
    - Please look at the cuRobo usage license
    ![here](docs/licenses/dependencies/cuRobo-license.txt)
    - Please look at updated isaacsim license
    ![here](docs/licenses/dependencies/isaacsim-license.txt)
    
    ### Technical Implementation:
    
    **Annotation Framework:**
    - Manual subtask start annotations to cleanly separate skill execution
    from motion-planning segments
    - Consistent trajectory segmentation for downstream dataset consumers
    
    **Motion Planning:**
    - **Base Motion Planner (Extensible):**
       - Introduces a reusable planner interface for uniform integration:
    -
    `source/isaaclab_mimic/isaaclab_mimic/motion_planners/base_motion_planner.py`
       - Defines a minimal, consistent API for planners:
    - `update_world_and_plan_motion(...)`, `get_planned_poses(...)`, etc.
       - The cuRobo planner inherits from this base class.
    - New planners can be added by subclassing the base class and
    implementing the same API, enabling drop-in replacement without changes
    to the SkillGen pipeline.
     
    - **CuRobo Planner** (GPU-accelerated, collision-aware):
      - Multi-phase planning: approach → contact → retreat
      - Dynamic object attach/detach and contact-aware sphere management
      - Real-time world synchronization between Isaac Lab and cuRobo
      - Configurable collision filtering for contact phases
      - **Tests**:
    -
    `source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_cube_stack.py`
    -
    `source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_franka.py`
        - `source/isaaclab_mimic/test/test_generate_dataset_skillgen.py`
    
    **Data Generation Pipeline:**
    - Automated dataset generation with precise skill-based segmentation
    - Integrates with existing observation/action spaces
    - Supports multi-env parallel collection with cuRobo-backed planning
    
    **Visualization and Debugging:**
    - Rerun-based 3D visualization for trajectory/collision inspection
    - Real-time sphere visualization for collision boundaries and contact
    phases
    
    ### Dependencies:
    - **cuRobo**: motion planning and collision checking
    - **Rerun**: 3D visualization and debugging
    
    ### Integration:
    This extends the existing mimic pipeline and remains backward
    compatible. It integrates into the manager-based environment structure
    and existing observation/action interfaces without breaking current
    users.
    
    ## Type of change
    - [x] New feature (non-breaking change which adds functionality)
    - [x] This change requires a documentation update
    
    ## Screenshot
    
    ### SkillGen Data Generation
    
    <table>
      <tr>
    <td align="center"><strong>Cube Stacking SkillGen Data
    Generation</strong></td>
    <td align="center"><strong>Bin Cube Stacking SkillGen Data Generation
    (Using Vanilla Cube Stacking Source Demos)</strong></td>
      </tr>
      <tr>
        <td>
    <img
    src="https://github.com/user-attachments/assets/de240b89-e670-4035-84ae-4101a4f70dae"
               alt="Cube Stacking Data Generation"
               style="width: 480px; height: 270px; object-fit: contain;">
        </td>
        <td>
    <img
    src="https://github.com/user-attachments/assets/dd94e0a6-ad1b-4366-80c4-7ff96cabeb3e"
               alt="Bin Cube Stacking Data Generation"
               style="width: 480px; height: 270px; object-fit: contain;">
        </td>
      </tr>
    </table>
    
    ### Bin Cube Stacking Behavior Cloned Policy
    
    ![bin_cube_stack_bc_policy](https://github.com/user-attachments/assets/d577d726-d623-4b27-90e5-a047cd67e4f9)
    
    ### Rerun Integration
    
    ![rerun_skillgen](https://github.com/user-attachments/assets/9c469bc4-d3f6-465a-8ca6-0ddfd85c6ad0)
    
    ### Motion Planner Tests
    
    <table>
      <tr>
        <td align="center"><strong>Obstacle Avoidance (cuRobo)</strong></td>
    <td align="center"><strong>Cube Stack End-to-End (cuRobo)</strong></td>
      </tr>
      <tr>
        <td>
    <img
    src="https://github.com/user-attachments/assets/a022342f-4c4e-42ea-a48c-cb1ea65c94db"
               alt="Obstacle Avoidance cuRobo"
               style="width: 480px; height: 270px; object-fit: contain;">
        </td>
        <td>
    <img
    src="https://github.com/user-attachments/assets/7e6290b6-8322-4702-ae2f-f363a87badde"
               alt="Cube Stack End-to-End cuRobo"
               style="width: 480px; height: 270px; object-fit: contain;">
        </td>
      </tr>
    </table>
    
    ## Checklist
    
    - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
    `./isaaclab.sh --format`
    - [x] I have made corresponding changes to the documentation
    - [x] My changes generate no new warnings
    - [x] I have added tests that prove my fix is effective or that my
    feature works
    - [x] I have updated the changelog and the corresponding version in the
    extension's `config/extension.toml` file
    - [x] I have added my name to the `CONTRIBUTORS.md` or my name already
    exists there
    
    ---------
    Signed-off-by: 's avatarKelly Guo <kellyg@nvidia.com>
    Co-authored-by: 's avatarKelly Guo <kellyg@nvidia.com>
    Co-authored-by: 's avatarPascal Roth <57946385+pascal-roth@users.noreply.github.com>
    994979c2
Name
Last commit
Last update
.github Loading commit data...
.vscode Loading commit data...
apps Loading commit data...
docker Loading commit data...
docs Loading commit data...
scripts Loading commit data...
source Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.flake8 Loading commit data...
.gitattributes Loading commit data...
.gitignore Loading commit data...
.pre-commit-config.yaml Loading commit data...
CITATION.cff Loading commit data...
CONTRIBUTING.md Loading commit data...
CONTRIBUTORS.md Loading commit data...
LICENSE Loading commit data...
LICENSE-mimic Loading commit data...
README.md Loading commit data...
SECURITY.md Loading commit data...
VERSION Loading commit data...
environment.yml Loading commit data...
isaaclab.bat Loading commit data...
isaaclab.sh Loading commit data...
pyproject.toml Loading commit data...
pytest.ini Loading commit data...