• njawale42's avatar
    Adds SkillGen framework to Isaac Lab with cuRobo support (#3303) · 994979c2
    njawale42 authored
    ## Description
    
    This PR introduces the SkillGen framework to Isaac Lab, integrating GPU
    motion planning with skill-segmented data generation. It enables
    efficient, high-quality dataset creation with robust collision handling,
    visualization, and reproducibility.
    
    **Note:** 
    - Please look at the cuRobo usage license
    ![here](docs/licenses/dependencies/cuRobo-license.txt)
    - Please look at updated isaacsim license
    ![here](docs/licenses/dependencies/isaacsim-license.txt)
    
    ### Technical Implementation:
    
    **Annotation Framework:**
    - Manual subtask start annotations to cleanly separate skill execution
    from motion-planning segments
    - Consistent trajectory segmentation for downstream dataset consumers
    
    **Motion Planning:**
    - **Base Motion Planner (Extensible):**
       - Introduces a reusable planner interface for uniform integration:
    -
    `source/isaaclab_mimic/isaaclab_mimic/motion_planners/base_motion_planner.py`
       - Defines a minimal, consistent API for planners:
    - `update_world_and_plan_motion(...)`, `get_planned_poses(...)`, etc.
       - The cuRobo planner inherits from this base class.
    - New planners can be added by subclassing the base class and
    implementing the same API, enabling drop-in replacement without changes
    to the SkillGen pipeline.
     
    - **CuRobo Planner** (GPU-accelerated, collision-aware):
      - Multi-phase planning: approach → contact → retreat
      - Dynamic object attach/detach and contact-aware sphere management
      - Real-time world synchronization between Isaac Lab and cuRobo
      - Configurable collision filtering for contact phases
      - **Tests**:
    -
    `source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_cube_stack.py`
    -
    `source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_franka.py`
        - `source/isaaclab_mimic/test/test_generate_dataset_skillgen.py`
    
    **Data Generation Pipeline:**
    - Automated dataset generation with precise skill-based segmentation
    - Integrates with existing observation/action spaces
    - Supports multi-env parallel collection with cuRobo-backed planning
    
    **Visualization and Debugging:**
    - Rerun-based 3D visualization for trajectory/collision inspection
    - Real-time sphere visualization for collision boundaries and contact
    phases
    
    ### Dependencies:
    - **cuRobo**: motion planning and collision checking
    - **Rerun**: 3D visualization and debugging
    
    ### Integration:
    This extends the existing mimic pipeline and remains backward
    compatible. It integrates into the manager-based environment structure
    and existing observation/action interfaces without breaking current
    users.
    
    ## Type of change
    - [x] New feature (non-breaking change which adds functionality)
    - [x] This change requires a documentation update
    
    ## Screenshot
    
    ### SkillGen Data Generation
    
    <table>
      <tr>
    <td align="center"><strong>Cube Stacking SkillGen Data
    Generation</strong></td>
    <td align="center"><strong>Bin Cube Stacking SkillGen Data Generation
    (Using Vanilla Cube Stacking Source Demos)</strong></td>
      </tr>
      <tr>
        <td>
    <img
    src="https://github.com/user-attachments/assets/de240b89-e670-4035-84ae-4101a4f70dae"
               alt="Cube Stacking Data Generation"
               style="width: 480px; height: 270px; object-fit: contain;">
        </td>
        <td>
    <img
    src="https://github.com/user-attachments/assets/dd94e0a6-ad1b-4366-80c4-7ff96cabeb3e"
               alt="Bin Cube Stacking Data Generation"
               style="width: 480px; height: 270px; object-fit: contain;">
        </td>
      </tr>
    </table>
    
    ### Bin Cube Stacking Behavior Cloned Policy
    
    ![bin_cube_stack_bc_policy](https://github.com/user-attachments/assets/d577d726-d623-4b27-90e5-a047cd67e4f9)
    
    ### Rerun Integration
    
    ![rerun_skillgen](https://github.com/user-attachments/assets/9c469bc4-d3f6-465a-8ca6-0ddfd85c6ad0)
    
    ### Motion Planner Tests
    
    <table>
      <tr>
        <td align="center"><strong>Obstacle Avoidance (cuRobo)</strong></td>
    <td align="center"><strong>Cube Stack End-to-End (cuRobo)</strong></td>
      </tr>
      <tr>
        <td>
    <img
    src="https://github.com/user-attachments/assets/a022342f-4c4e-42ea-a48c-cb1ea65c94db"
               alt="Obstacle Avoidance cuRobo"
               style="width: 480px; height: 270px; object-fit: contain;">
        </td>
        <td>
    <img
    src="https://github.com/user-attachments/assets/7e6290b6-8322-4702-ae2f-f363a87badde"
               alt="Cube Stack End-to-End cuRobo"
               style="width: 480px; height: 270px; object-fit: contain;">
        </td>
      </tr>
    </table>
    
    ## Checklist
    
    - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
    `./isaaclab.sh --format`
    - [x] I have made corresponding changes to the documentation
    - [x] My changes generate no new warnings
    - [x] I have added tests that prove my fix is effective or that my
    feature works
    - [x] I have updated the changelog and the corresponding version in the
    extension's `config/extension.toml` file
    - [x] I have added my name to the `CONTRIBUTORS.md` or my name already
    exists there
    
    ---------
    Signed-off-by: 's avatarKelly Guo <kellyg@nvidia.com>
    Co-authored-by: 's avatarKelly Guo <kellyg@nvidia.com>
    Co-authored-by: 's avatarPascal Roth <57946385+pascal-roth@users.noreply.github.com>
    994979c2
Name
Last commit
Last update
..
Farama-Notifications-license.txt Loading commit data...
GitPython-license.txt Loading commit data...
InquirerPy-license.txt Loading commit data...
absl-py-license.txt Loading commit data...
accessible-pygments-license.txt Loading commit data...
alabaster-license.txt Loading commit data...
antlr4_python3_runtime-license.txt Loading commit data...
anyio-license.txt Loading commit data...
anytree-license.txt Loading commit data...
assimp-license.txt Loading commit data...
asttokens-license.txt Loading commit data...
attrs-license Loading commit data...
autodocsumm-license.txt Loading commit data...
babel-license.txt Loading commit data...
beautifulsoup4-license.txt Loading commit data...
black-license.txt Loading commit data...
boost-license.txt Loading commit data...
botocore-license.txt Loading commit data...
certifi-license.txt Loading commit data...
charset-normalizer-license.txt Loading commit data...
click-license.txt Loading commit data...
cloudpickle-license.txt Loading commit data...
cmeel-license.txt Loading commit data...
codespell-license.txt Loading commit data...
colorama-license.txt Loading commit data...
comm-license.txt Loading commit data...
console-bridge-license.txt Loading commit data...
contourpy-license.txt Loading commit data...
cuRobo-license.txt Loading commit data...
cycler-license.txt Loading commit data...
decorator-license.txt Loading commit data...
dex-retargeting-license.txt Loading commit data...
docker-pycreds-license.txt Loading commit data...
egl_probe-license.txt Loading commit data...
eigenpy-license.txt Loading commit data...
einops-license.txt Loading commit data...
executing-license.txt Loading commit data...
filelock-license.txt Loading commit data...
flake8-license.txt Loading commit data...
flaky-license.txt Loading commit data...
flatdict-license.txt Loading commit data...
fonttools-license.txt Loading commit data...
fsspec-license.txt Loading commit data...
gitdb-license.txt Loading commit data...
grpcio-license.txt Loading commit data...
gym-license.txt Loading commit data...
gym-notices-license.txt Loading commit data...
gymnasium-license.txt Loading commit data...
h5py-license.txt Loading commit data...
hf-xet-license.txt Loading commit data...
hidapi-license.txt Loading commit data...
hpp-fcl-license.txt Loading commit data...
huggingface-hub-license.txt Loading commit data...
hydra-license.txt Loading commit data...
idna-license.txt Loading commit data...
imageio-license.txt Loading commit data...
imageio_ffmpeg-license.txt Loading commit data...
imagesize-license.txt Loading commit data...
iniconfig-license.txt Loading commit data...
ipython-license.txt Loading commit data...
ipywidgets-license.txt Loading commit data...
isaacsim-license.txt Loading commit data...
isort-license.txt Loading commit data...
jinja2-license.txt Loading commit data...
jmespath-license.txt Loading commit data...
joblib-license.txt Loading commit data...
jsonschema-license Loading commit data...
jsonschema-specifications-license Loading commit data...
junitparser-license.txt Loading commit data...
jupyterlab_widgets-license.txt Loading commit data...
kiwisolver-license.txt Loading commit data...
latexcodec-license.txt Loading commit data...
legged-gym-license.txt Loading commit data...
loop-rate-limiters-license.txt Loading commit data...
markdown-it-py-license.txt Loading commit data...
markdown-license.txt Loading commit data...
markupsafe-license.txt Loading commit data...
matplotlib-inline-license.txt Loading commit data...
matplotlib-license.txt Loading commit data...
mdit-py-plugins-license.txt Loading commit data...
mdurl-license.txt Loading commit data...
moviepy-license.txt Loading commit data...
mpmath-license.txt Loading commit data...
myst-parser-license.txt Loading commit data...
networkx-license.txt Loading commit data...
newton-license.txt Loading commit data...
nlopt-license.txt Loading commit data...
numpy-license.txt Loading commit data...
octomap-license.txt Loading commit data...
omegaconf-license.txt Loading commit data...
onnx-license.txt Loading commit data...
open3d-license.txt Loading commit data...
opencv-python-license.txt Loading commit data...
packaging-license.txt Loading commit data...
pandas-license.txt Loading commit data...
pathtools-license.txt Loading commit data...
pexpect-license.txt Loading commit data...
pillow-license.txt Loading commit data...
pin-license.txt Loading commit data...
pin-pink-license.txt Loading commit data...
pluggy-license.txt Loading commit data...
pre-commit-hooks-license.txt Loading commit data...
pre-commit-license.txt Loading commit data...
prettytable-license.txt Loading commit data...
proglog-license.txt Loading commit data...
protobuf-license.txt Loading commit data...
psutil-license.txt Loading commit data...
ptyprocess-license.txt Loading commit data...
pure_eval-license.txt Loading commit data...
pybtex-docutils-license.txt Loading commit data...
pydata-sphinx-theme-license.txt Loading commit data...
pygame-license Loading commit data...
pyglet-license.txt Loading commit data...
pygrep-hooks-license.txt Loading commit data...
pyparsing-license.txt Loading commit data...
pyperclip-license.txt Loading commit data...
pyright-license.txt Loading commit data...
pytest-license.txt Loading commit data...
pytest-mock-license.txt Loading commit data...
python-dateutil-license.txt Loading commit data...
python-dotenv-license.txt Loading commit data...
pytorch3d-license.txt Loading commit data...
pytransform3d-license.txt Loading commit data...
pytz-license.txt Loading commit data...
pyupgrade-license.txt Loading commit data...
pyyaml-license.txt Loading commit data...
qhull-license.txt Loading commit data...
qpsolvers-license.txt Loading commit data...
quadprog-license.txt Loading commit data...
ray-license.txt Loading commit data...
referencing-license Loading commit data...
regex-license.txt Loading commit data...
requests-license.txt Loading commit data...
rich-license.txt Loading commit data...
rl_games-license.txt Loading commit data...
robomimc-license.txt Loading commit data...
rsl_rl-license.txt Loading commit data...
safetensors-license.txt Loading commit data...
scikit-learn-license.txt Loading commit data...
sentry-sdk-license.txt Loading commit data...
setproctitle-license.txt Loading commit data...
setuptools-license.txt Loading commit data...
shortuuid-license.txt Loading commit data...
six-license.txt Loading commit data...
skrl-license.txt Loading commit data...
smmap-license.txt Loading commit data...
sniffio-license.txt Loading commit data...
snowballstemmer-license.txt Loading commit data...
soupsieve-license.txt Loading commit data...
sphinx-book-theme-license.txt Loading commit data...
sphinx-copybutton-license.txt Loading commit data...
sphinx-design-license.txt Loading commit data...
sphinx-icon-license.txt Loading commit data...
sphinx-license.txt Loading commit data...
sphinx-multiversion-license.txt Loading commit data...
sphinx-tabs-license.txt Loading commit data...
sphinxcontrib-applehelp-license.txt Loading commit data...
sphinxcontrib-bibtex-license.txt Loading commit data...
sphinxcontrib-devhelp-license.txt Loading commit data...
sphinxcontrib-htmlhelp-license.txt Loading commit data...
sphinxcontrib-jsmath-license.txt Loading commit data...
sphinxcontrib-qthelp-license.txt Loading commit data...
sphinxcontrib-serializinghtml-license.txt Loading commit data...
sphinxemoji-license.txt Loading commit data...
stable-baselines3-license.txt Loading commit data...
stack-data-license.txt Loading commit data...
sympy-license.txt Loading commit data...
tensorboard-license.txt Loading commit data...
tensorboardx-license.txt Loading commit data...
tensordict-license.txt Loading commit data...
termcolor-license.txt Loading commit data...
threadpoolctl-license.txt Loading commit data...
tinyxml-license.txt Loading commit data...
tokenizers-license.txt Loading commit data...
toml-license.txt Loading commit data...
torch-license.txt Loading commit data...
torchvision-license.txt Loading commit data...
tqdm-license.txt Loading commit data...
traitlets-license.txt Loading commit data...
transformers-license.txt Loading commit data...
trimesh-license.txt Loading commit data...
triton-license.txt Loading commit data...
typing-extensions-license.txt Loading commit data...
typing-inspection-license Loading commit data...
tzdata-license.txt Loading commit data...
urdfdom-license.txt Loading commit data...
urllib3-license.txt Loading commit data...
walk-these-ways-license.txt Loading commit data...
wandb-license.txt Loading commit data...
warp-license.txt Loading commit data...
werkzeug-license.txt Loading commit data...
widgetsnbextension-license.txt Loading commit data...
zipp-license Loading commit data...
zlib-license.txt Loading commit data...