• njawale42's avatar
    Adds SkillGen framework to Isaac Lab with cuRobo support (#3303) · 994979c2
    njawale42 authored
    ## Description
    
    This PR introduces the SkillGen framework to Isaac Lab, integrating GPU
    motion planning with skill-segmented data generation. It enables
    efficient, high-quality dataset creation with robust collision handling,
    visualization, and reproducibility.
    
    **Note:** 
    - Please look at the cuRobo usage license
    ![here](docs/licenses/dependencies/cuRobo-license.txt)
    - Please look at updated isaacsim license
    ![here](docs/licenses/dependencies/isaacsim-license.txt)
    
    ### Technical Implementation:
    
    **Annotation Framework:**
    - Manual subtask start annotations to cleanly separate skill execution
    from motion-planning segments
    - Consistent trajectory segmentation for downstream dataset consumers
    
    **Motion Planning:**
    - **Base Motion Planner (Extensible):**
       - Introduces a reusable planner interface for uniform integration:
    -
    `source/isaaclab_mimic/isaaclab_mimic/motion_planners/base_motion_planner.py`
       - Defines a minimal, consistent API for planners:
    - `update_world_and_plan_motion(...)`, `get_planned_poses(...)`, etc.
       - The cuRobo planner inherits from this base class.
    - New planners can be added by subclassing the base class and
    implementing the same API, enabling drop-in replacement without changes
    to the SkillGen pipeline.
     
    - **CuRobo Planner** (GPU-accelerated, collision-aware):
      - Multi-phase planning: approach → contact → retreat
      - Dynamic object attach/detach and contact-aware sphere management
      - Real-time world synchronization between Isaac Lab and cuRobo
      - Configurable collision filtering for contact phases
      - **Tests**:
    -
    `source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_cube_stack.py`
    -
    `source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_franka.py`
        - `source/isaaclab_mimic/test/test_generate_dataset_skillgen.py`
    
    **Data Generation Pipeline:**
    - Automated dataset generation with precise skill-based segmentation
    - Integrates with existing observation/action spaces
    - Supports multi-env parallel collection with cuRobo-backed planning
    
    **Visualization and Debugging:**
    - Rerun-based 3D visualization for trajectory/collision inspection
    - Real-time sphere visualization for collision boundaries and contact
    phases
    
    ### Dependencies:
    - **cuRobo**: motion planning and collision checking
    - **Rerun**: 3D visualization and debugging
    
    ### Integration:
    This extends the existing mimic pipeline and remains backward
    compatible. It integrates into the manager-based environment structure
    and existing observation/action interfaces without breaking current
    users.
    
    ## Type of change
    - [x] New feature (non-breaking change which adds functionality)
    - [x] This change requires a documentation update
    
    ## Screenshot
    
    ### SkillGen Data Generation
    
    <table>
      <tr>
    <td align="center"><strong>Cube Stacking SkillGen Data
    Generation</strong></td>
    <td align="center"><strong>Bin Cube Stacking SkillGen Data Generation
    (Using Vanilla Cube Stacking Source Demos)</strong></td>
      </tr>
      <tr>
        <td>
    <img
    src="https://github.com/user-attachments/assets/de240b89-e670-4035-84ae-4101a4f70dae"
               alt="Cube Stacking Data Generation"
               style="width: 480px; height: 270px; object-fit: contain;">
        </td>
        <td>
    <img
    src="https://github.com/user-attachments/assets/dd94e0a6-ad1b-4366-80c4-7ff96cabeb3e"
               alt="Bin Cube Stacking Data Generation"
               style="width: 480px; height: 270px; object-fit: contain;">
        </td>
      </tr>
    </table>
    
    ### Bin Cube Stacking Behavior Cloned Policy
    
    ![bin_cube_stack_bc_policy](https://github.com/user-attachments/assets/d577d726-d623-4b27-90e5-a047cd67e4f9)
    
    ### Rerun Integration
    
    ![rerun_skillgen](https://github.com/user-attachments/assets/9c469bc4-d3f6-465a-8ca6-0ddfd85c6ad0)
    
    ### Motion Planner Tests
    
    <table>
      <tr>
        <td align="center"><strong>Obstacle Avoidance (cuRobo)</strong></td>
    <td align="center"><strong>Cube Stack End-to-End (cuRobo)</strong></td>
      </tr>
      <tr>
        <td>
    <img
    src="https://github.com/user-attachments/assets/a022342f-4c4e-42ea-a48c-cb1ea65c94db"
               alt="Obstacle Avoidance cuRobo"
               style="width: 480px; height: 270px; object-fit: contain;">
        </td>
        <td>
    <img
    src="https://github.com/user-attachments/assets/7e6290b6-8322-4702-ae2f-f363a87badde"
               alt="Cube Stack End-to-End cuRobo"
               style="width: 480px; height: 270px; object-fit: contain;">
        </td>
      </tr>
    </table>
    
    ## Checklist
    
    - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
    `./isaaclab.sh --format`
    - [x] I have made corresponding changes to the documentation
    - [x] My changes generate no new warnings
    - [x] I have added tests that prove my fix is effective or that my
    feature works
    - [x] I have updated the changelog and the corresponding version in the
    extension's `config/extension.toml` file
    - [x] I have added my name to the `CONTRIBUTORS.md` or my name already
    exists there
    
    ---------
    Signed-off-by: 's avatarKelly Guo <kellyg@nvidia.com>
    Co-authored-by: 's avatarKelly Guo <kellyg@nvidia.com>
    Co-authored-by: 's avatarPascal Roth <57946385+pascal-roth@users.noreply.github.com>
    994979c2
Name
Last commit
Last update
..
augmented_imitation.rst Loading commit data...
index.rst Loading commit data...
skillgen.rst Loading commit data...
teleop_imitation.rst Loading commit data...