Adds SkillGen framework to Isaac Lab with cuRobo support (#3303)

## Description

This PR introduces the SkillGen framework to Isaac Lab, integrating GPU
motion planning with skill-segmented data generation. It enables
efficient, high-quality dataset creation with robust collision handling,
visualization, and reproducibility.

**Note:** 
- Please look at the cuRobo usage license
![here](docs/licenses/dependencies/cuRobo-license.txt)
- Please look at updated isaacsim license
![here](docs/licenses/dependencies/isaacsim-license.txt)

### Technical Implementation:

**Annotation Framework:**
- Manual subtask start annotations to cleanly separate skill execution
from motion-planning segments
- Consistent trajectory segmentation for downstream dataset consumers

**Motion Planning:**
- **Base Motion Planner (Extensible):**
   - Introduces a reusable planner interface for uniform integration:
-
`source/isaaclab_mimic/isaaclab_mimic/motion_planners/base_motion_planner.py`
   - Defines a minimal, consistent API for planners:
- `update_world_and_plan_motion(...)`, `get_planned_poses(...)`, etc.
   - The cuRobo planner inherits from this base class.
- New planners can be added by subclassing the base class and
implementing the same API, enabling drop-in replacement without changes
to the SkillGen pipeline.
 
- **CuRobo Planner** (GPU-accelerated, collision-aware):
  - Multi-phase planning: approach → contact → retreat
  - Dynamic object attach/detach and contact-aware sphere management
  - Real-time world synchronization between Isaac Lab and cuRobo
  - Configurable collision filtering for contact phases
  - **Tests**:
-
`source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_cube_stack.py`
-
`source/isaaclab_mimic/isaaclab_mimic/motion_planners/curobo/test/test_curobo_planner_franka.py`
    - `source/isaaclab_mimic/test/test_generate_dataset_skillgen.py`

**Data Generation Pipeline:**
- Automated dataset generation with precise skill-based segmentation
- Integrates with existing observation/action spaces
- Supports multi-env parallel collection with cuRobo-backed planning

**Visualization and Debugging:**
- Rerun-based 3D visualization for trajectory/collision inspection
- Real-time sphere visualization for collision boundaries and contact
phases

### Dependencies:
- **cuRobo**: motion planning and collision checking
- **Rerun**: 3D visualization and debugging

### Integration:
This extends the existing mimic pipeline and remains backward
compatible. It integrates into the manager-based environment structure
and existing observation/action interfaces without breaking current
users.

## Type of change
- [x] New feature (non-breaking change which adds functionality)
- [x] This change requires a documentation update

## Screenshot

### SkillGen Data Generation

<table>
  <tr>
<td align="center"><strong>Cube Stacking SkillGen Data
Generation</strong></td>
<td align="center"><strong>Bin Cube Stacking SkillGen Data Generation
(Using Vanilla Cube Stacking Source Demos)</strong></td>
  </tr>
  <tr>
    <td>
<img
src="https://github.com/user-attachments/assets/de240b89-e670-4035-84ae-4101a4f70dae"
           alt="Cube Stacking Data Generation"
           style="width: 480px; height: 270px; object-fit: contain;">
    </td>
    <td>
<img
src="https://github.com/user-attachments/assets/dd94e0a6-ad1b-4366-80c4-7ff96cabeb3e"
           alt="Bin Cube Stacking Data Generation"
           style="width: 480px; height: 270px; object-fit: contain;">
    </td>
  </tr>
</table>

### Bin Cube Stacking Behavior Cloned Policy

![bin_cube_stack_bc_policy](https://github.com/user-attachments/assets/d577d726-d623-4b27-90e5-a047cd67e4f9)

### Rerun Integration

![rerun_skillgen](https://github.com/user-attachments/assets/9c469bc4-d3f6-465a-8ca6-0ddfd85c6ad0)

### Motion Planner Tests

<table>
  <tr>
    <td align="center"><strong>Obstacle Avoidance (cuRobo)</strong></td>
<td align="center"><strong>Cube Stack End-to-End (cuRobo)</strong></td>
  </tr>
  <tr>
    <td>
<img
src="https://github.com/user-attachments/assets/a022342f-4c4e-42ea-a48c-cb1ea65c94db"
           alt="Obstacle Avoidance cuRobo"
           style="width: 480px; height: 270px; object-fit: contain;">
    </td>
    <td>
<img
src="https://github.com/user-attachments/assets/7e6290b6-8322-4702-ae2f-f363a87badde"
           alt="Cube Stack End-to-End cuRobo"
           style="width: 480px; height: 270px; object-fit: contain;">
    </td>
  </tr>
</table>

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

---------
Signed-off-by: Kelly Guo <kellyg@nvidia.com>
Co-authored-by: Kelly Guo <kellyg@nvidia.com>
Co-authored-by: Pascal Roth <57946385+pascal-roth@users.noreply.github.com>

.github/actions/docker-build/action.yml

@@ -18,7 +18,7 @@ inputs:
     required: true
   dockerfile-path:
     description: 'Path to Dockerfile'
-    default: 'docker/Dockerfile.base'
+    default: 'docker/Dockerfile.curobo'
     required: false
   context-path:
     description: 'Build context path'

CONTRIBUTORS.md

@@ -101,6 +101,7 @@ Guidelines for modifications:
 * Miguel Alonso Jr
 * Mingyu Lee
 * Muhong Guo
+* Neel Anand Jawale
 * Nicola Loi
 * Norbert Cygiert
 * Nuoyan Chen (Alvin)

README.md

@@ -202,6 +202,8 @@ dependencies and assets are present in the [`docs/licenses`](docs/licenses) dire
 
 Note that Isaac Lab requires Isaac Sim, which includes components under proprietary licensing terms. Please see the [Isaac Sim license](docs/licenses/dependencies/isaacsim-license.txt) for information on Isaac Sim licensing.
 
+Note that the `isaaclab_mimic` extension requires cuRobo, which has proprietary licensing terms that can be found in [`docs/licenses/dependencies/cuRobo-license.txt`](docs/licenses/dependencies/cuRobo-license.txt).
+
 ## Acknowledgement
 
 Isaac Lab development initiated from the [Orbit](https://isaac-orbit.github.io/) framework. We would appreciate if

docker/Dockerfile.curobo

+# Copyright (c) 2022-2025, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+# Nvidia Dockerfiles: https://github.com/NVIDIA-Omniverse/IsaacSim-dockerfiles
+# Please check above link for license information.
+
+# Base image
+ARG ISAACSIM_BASE_IMAGE_ARG
+ARG ISAACSIM_VERSION_ARG
+FROM ${ISAACSIM_BASE_IMAGE_ARG}:${ISAACSIM_VERSION_ARG} AS base
+ENV ISAACSIM_VERSION=${ISAACSIM_VERSION_ARG}
+
+# Set default RUN shell to bash
+SHELL ["/bin/bash", "-c"]
+
+# Adds labels to the Dockerfile
+LABEL version="2.1.1"
+LABEL description="Dockerfile for building and running the Isaac Lab framework inside Isaac Sim container image."
+
+# Arguments
+# Path to Isaac Sim root folder
+ARG ISAACSIM_ROOT_PATH_ARG
+ENV ISAACSIM_ROOT_PATH=${ISAACSIM_ROOT_PATH_ARG}
+# Path to the Isaac Lab directory
+ARG ISAACLAB_PATH_ARG
+ENV ISAACLAB_PATH=${ISAACLAB_PATH_ARG}
+# Home dir of docker user, typically '/root'
+ARG DOCKER_USER_HOME_ARG
+ENV DOCKER_USER_HOME=${DOCKER_USER_HOME_ARG}
+
+# Set environment variables
+ENV LANG=C.UTF-8
+ENV DEBIAN_FRONTEND=noninteractive
+
+USER root
+
+# Install dependencies and remove cache
+RUN --mount=type=cache,target=/var/cache/apt \
+    apt-get update && apt-get install -y --no-install-recommends \
+    build-essential \
+    cmake \
+    git \
+    libglib2.0-0 \
+    ncurses-term \
+    wget && \
+    apt -y autoremove && apt clean autoclean && \
+    rm -rf /var/lib/apt/lists/*
+
+# Detect Ubuntu version and install CUDA 12.8 via NVIDIA network repo (cuda-keyring)
+RUN set -euo pipefail && \
+    . /etc/os-release && \
+    case "$ID" in \
+      ubuntu) \
+        case "$VERSION_ID" in \
+          "20.04") cuda_repo="ubuntu2004";; \
+          "22.04") cuda_repo="ubuntu2204";; \
+          "24.04") cuda_repo="ubuntu2404";; \
+          *) echo "Unsupported Ubuntu $VERSION_ID"; exit 1;; \
+        esac ;; \
+      *) echo "Unsupported base OS: $ID"; exit 1 ;; \
+    esac && \
+    apt-get update && apt-get install -y --no-install-recommends wget gnupg ca-certificates && \
+    wget -q https://developer.download.nvidia.com/compute/cuda/repos/${cuda_repo}/x86_64/cuda-keyring_1.1-1_all.deb && \
+    dpkg -i cuda-keyring_1.1-1_all.deb && \
+    rm -f cuda-keyring_1.1-1_all.deb && \
+    wget -q https://developer.download.nvidia.com/compute/cuda/repos/${cuda_repo}/x86_64/cuda-${cuda_repo}.pin && \
+    mv cuda-${cuda_repo}.pin /etc/apt/preferences.d/cuda-repository-pin-600 && \
+    apt-get update && \
+    apt-get install -y --no-install-recommends cuda-toolkit-12-8 && \
+    apt-get -y autoremove && apt-get clean && rm -rf /var/lib/apt/lists/*
+
+
+ENV CUDA_HOME=/usr/local/cuda-12.8
+ENV PATH=${CUDA_HOME}/bin:${PATH}
+ENV LD_LIBRARY_PATH=${CUDA_HOME}/lib64:${LD_LIBRARY_PATH}
+ENV TORCH_CUDA_ARCH_LIST=8.0+PTX
+
+# Copy the Isaac Lab directory (files to exclude are defined in .dockerignore)
+COPY ../ ${ISAACLAB_PATH}
+
+# Ensure isaaclab.sh has execute permissions
+RUN chmod +x ${ISAACLAB_PATH}/isaaclab.sh
+
+# Set up a symbolic link between the installed Isaac Sim root folder and _isaac_sim in the Isaac Lab directory
+RUN ln -sf ${ISAACSIM_ROOT_PATH} ${ISAACLAB_PATH}/_isaac_sim
+
+# Install toml dependency
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install toml
+
+# Install apt dependencies for extensions that declare them in their extension.toml
+RUN --mount=type=cache,target=/var/cache/apt \
+    ${ISAACLAB_PATH}/isaaclab.sh -p ${ISAACLAB_PATH}/tools/install_deps.py apt ${ISAACLAB_PATH}/source && \
+    apt -y autoremove && apt clean autoclean && \
+    rm -rf /var/lib/apt/lists/*
+
+# for singularity usage, have to create the directories that will binded
+RUN mkdir -p ${ISAACSIM_ROOT_PATH}/kit/cache && \
+    mkdir -p ${DOCKER_USER_HOME}/.cache/ov && \
+    mkdir -p ${DOCKER_USER_HOME}/.cache/pip && \
+    mkdir -p ${DOCKER_USER_HOME}/.cache/nvidia/GLCache &&  \
+    mkdir -p ${DOCKER_USER_HOME}/.nv/ComputeCache && \
+    mkdir -p ${DOCKER_USER_HOME}/.nvidia-omniverse/logs && \
+    mkdir -p ${DOCKER_USER_HOME}/.local/share/ov/data && \
+    mkdir -p ${DOCKER_USER_HOME}/Documents
+
+# for singularity usage, create NVIDIA binary placeholders
+RUN touch /bin/nvidia-smi && \
+    touch /bin/nvidia-debugdump && \
+    touch /bin/nvidia-persistenced && \
+    touch /bin/nvidia-cuda-mps-control && \
+    touch /bin/nvidia-cuda-mps-server && \
+    touch /etc/localtime && \
+    mkdir -p /var/run/nvidia-persistenced && \
+    touch /var/run/nvidia-persistenced/socket
+
+# installing Isaac Lab dependencies
+# use pip caching to avoid reinstalling large packages
+RUN --mount=type=cache,target=${DOCKER_USER_HOME}/.cache/pip \
+    ${ISAACLAB_PATH}/isaaclab.sh --install
+
+# Install cuRobo from source (pinned commit); needs CUDA env and Torch
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip install --no-build-isolation \
+    "nvidia-curobo @ git+https://github.com/NVlabs/curobo.git@ebb71702f3f70e767f40fd8e050674af0288abe8"
+
+# HACK: Remove install of quadprog dependency
+RUN ${ISAACLAB_PATH}/isaaclab.sh -p -m pip uninstall -y quadprog
+
+# aliasing isaaclab.sh and python for convenience
+RUN echo "export ISAACLAB_PATH=${ISAACLAB_PATH}" >> ${HOME}/.bashrc && \
+    echo "alias isaaclab=${ISAACLAB_PATH}/isaaclab.sh" >> ${HOME}/.bashrc && \
+    echo "alias python=${ISAACLAB_PATH}/_isaac_sim/python.sh" >> ${HOME}/.bashrc && \
+    echo "alias python3=${ISAACLAB_PATH}/_isaac_sim/python.sh" >> ${HOME}/.bashrc && \
+    echo "alias pip='${ISAACLAB_PATH}/_isaac_sim/python.sh -m pip'" >> ${HOME}/.bashrc && \
+    echo "alias pip3='${ISAACLAB_PATH}/_isaac_sim/python.sh -m pip'" >> ${HOME}/.bashrc && \
+    echo "alias tensorboard='${ISAACLAB_PATH}/_isaac_sim/python.sh ${ISAACLAB_PATH}/_isaac_sim/tensorboard'" >> ${HOME}/.bashrc && \
+    echo "export TZ=$(date +%Z)" >> ${HOME}/.bashrc && \
+    echo "shopt -s histappend" >> /root/.bashrc && \
+    echo "PROMPT_COMMAND='history -a'" >> /root/.bashrc
+
+# make working directory as the Isaac Lab directory
+# this is the default directory when the container is run
+WORKDIR ${ISAACLAB_PATH}

docs/source/overview/imitation-learning/index.rst

@@ -9,3 +9,4 @@ with Isaac Lab.
 
   augmented_imitation
   teleop_imitation
+  skillgen

pyproject.toml

@@ -95,6 +95,6 @@ reportPrivateUsage = "warning"
 skip = '*.usd,*.svg,*.png,_isaac_sim*,*.bib,*.css,*/_build'
 quiet-level = 0
 # the world list should always have words in lower case
-ignore-words-list = "haa,slq,collapsable,buss"
+ignore-words-list = "haa,slq,collapsable,buss,reacher"
 # todo: this is hack to deal with incorrect spelling of "Environment" in the Isaac Sim grid world asset
 exclude-file = "source/isaaclab/isaaclab/sim/spawners/from_files/from_files.py"

scripts/imitation_learning/isaaclab_mimic/generate_dataset.py

@@ -39,6 +39,12 @@ parser.add_argument(
     default=False,
     help="Enable Pinocchio.",
 )
+parser.add_argument(
+    "--use_skillgen",
+    action="store_true",
+    default=False,
+    help="use skillgen to generate motion trajectories",
+)
 # append AppLauncher cli args
 AppLauncher.add_app_launcher_args(parser)
 # parse the arguments
@@ -96,27 +102,55 @@ def main():
         generation_num_trials=args_cli.generation_num_trials,
     )
 
-    # create environment
+    # Create environment
     env = gym.make(env_name, cfg=env_cfg).unwrapped
 
     if not isinstance(env, ManagerBasedRLMimicEnv):
         raise ValueError("The environment should be derived from ManagerBasedRLMimicEnv")
 
-    # check if the mimic API from this environment contains decprecated signatures
+    # Check if the mimic API from this environment contains decprecated signatures
     if "action_noise_dict" not in inspect.signature(env.target_eef_pose_to_action).parameters:
         omni.log.warn(
             f'The "noise" parameter in the "{env_name}" environment\'s mimic API "target_eef_pose_to_action", '
             "is deprecated. Please update the API to take action_noise_dict instead."
         )
 
-    # set seed for generation
+    # Set seed for generation
     random.seed(env.cfg.datagen_config.seed)
     np.random.seed(env.cfg.datagen_config.seed)
     torch.manual_seed(env.cfg.datagen_config.seed)
 
-    # reset before starting
+    # Reset before starting
     env.reset()
 
+    motion_planners = None
+    if args_cli.use_skillgen:
+        from isaaclab_mimic.motion_planners.curobo.curobo_planner import CuroboPlanner
+        from isaaclab_mimic.motion_planners.curobo.curobo_planner_cfg import CuroboPlannerCfg
+
+        # Create one motion planner per environment
+        motion_planners = {}
+        for env_id in range(num_envs):
+            print(f"Initializing motion planner for environment {env_id}")
+            # Create a config instance from the task name
+            planner_config = CuroboPlannerCfg.from_task_name(env_name)
+
+            # Ensure visualization is only enabled for the first environment
+            # If not, sphere and plan visualization will be too slow in isaac lab
+            # It is efficient to visualize the spheres and plan for the first environment in rerun
+            if env_id != 0:
+                planner_config.visualize_spheres = False
+                planner_config.visualize_plan = False
+
+            motion_planners[env_id] = CuroboPlanner(
+                env=env,
+                robot=env.scene["robot"],
+                config=planner_config,  # Pass the config object
+                env_id=env_id,  # Pass environment ID
+            )
+
+        env.cfg.datagen_config.use_skillgen = True
+
     # Setup and run async data generation
     async_components = setup_async_generation(
         env=env,
@@ -124,6 +158,7 @@ def main():
         input_file=args_cli.input_file,
         success_term=success_term,
         pause_subtask=args_cli.pause_subtask,
+        motion_planners=motion_planners,  # Pass the motion planners dictionary
     )
 
     try:
@@ -147,6 +182,14 @@ def main():
             print("Remaining async tasks cancelled and cleaned up.")
         except Exception as e:
             print(f"Error cancelling remaining async tasks: {e}")
+        # Cleanup of motion planners and their visualizers
+        if motion_planners is not None:
+            for env_id, planner in motion_planners.items():
+                if getattr(planner, "plan_visualizer", None) is not None:
+                    print(f"Closing plan visualizer for environment {env_id}")
+                    planner.plan_visualizer.close()
+                    planner.plan_visualizer = None
+            motion_planners.clear()
 
 
 if __name__ == "__main__":
@@ -154,5 +197,5 @@ if __name__ == "__main__":
         main()
     except KeyboardInterrupt:
         print("\nProgram interrupted by user. Exiting...")
-    # close sim app
+    # Close sim app
     simulation_app.close()

source/isaaclab/isaaclab/envs/manager_based_rl_mimic_env.py

@@ -117,6 +117,22 @@ class ManagerBasedRLMimicEnv(ManagerBasedRLEnv):
             )
         return object_pose_matrix
 
+    def get_subtask_start_signals(self, env_ids: Sequence[int] | None = None) -> dict[str, torch.Tensor]:
+        """
+        Gets a dictionary of start signal flags for each subtask in a task. The flag is 1
+        when the subtask has started and 0 otherwise. The implementation of this method is
+        required if intending to enable automatic subtask start signal annotation when running the
+        dataset annotation tool. This method can be kept unimplemented if intending to use manual
+        subtask start signal annotation.
+
+        Args:
+            env_ids: Environment indices to get the start signals for. If None, all envs are considered.
+
+        Returns:
+            A dictionary start signal flags (False or True) for each subtask.
+        """
+        raise NotImplementedError
+
     def get_subtask_term_signals(self, env_ids: Sequence[int] | None = None) -> dict[str, torch.Tensor]:
         """
         Gets a dictionary of termination signal flags for each subtask in a task. The flag is 1

source/isaaclab/isaaclab/envs/mimic_env_cfg.py

@@ -73,6 +73,9 @@ class DataGenConfig:
     generation_interpolate_from_last_target_pose: bool = True
     """Whether to interpolate from last target pose."""
 
+    use_skillgen: bool = False
+    """Whether to use skillgen to generate motion trajectories."""
+
 
 @configclass
 class SubTaskConfig:
@@ -115,6 +118,12 @@ class SubTaskConfig:
     first_subtask_start_offset_range: tuple = (0, 0)
     """Range for start offset of the first subtask."""
 
+    subtask_start_offset_range: tuple = (0, 0)
+    """Range for start offset of the subtask (only used if use_skillgen is True)
+
+    Note: This value overrides the first_subtask_start_offset_range when skillgen is enabled
+    """
+
     subtask_term_offset_range: tuple = (0, 0)
     """Range for offsetting subtask termination."""
 

source/isaaclab_mimic/config/extension.toml

 [package]
 
 # Semantic Versioning is used: https://semver.org/
-version = "1.0.13"
+version = "1.0.14"
 
 # Description
 category = "isaaclab"

source/isaaclab_mimic/docs/CHANGELOG.rst

 Changelog
 ---------
 
+1.0.14 (2025-09-08)
+~~~~~~~~~~~~~~~~~~~
+
+Added
+^^^^^
+
+* Added SkillGen integration for automated demonstration generation using cuRobo; enable via ``--use_skillgen`` in ``scripts/imitation_learning/isaaclab_mimic/generate_dataset.py``.
+* Added cuRobo motion planner interface (:class:`CuroboPlanner`, :class:`CuroboPlannerCfg`)
+* Added manual subtask start boundary annotation for SkillGen; enable via ``--annotate_subtask_start_signals`` in ``scripts/imitation_learning/isaaclab_mimic/annotate_demos.py``.
+* Added Rerun integration for motion plan visualization and debugging; enable via ``visualize_plan = True`` in :class:`CuroboPlannerCfg`.
+
+
 1.0.13 (2025-08-14)
 ~~~~~~~~~~~~~~~~~~~
 

source/isaaclab_mimic/isaaclab_mimic/datagen/datagen_info.py

@@ -20,6 +20,7 @@ class DatagenInfo:
     Core Elements:
     - **eef_pose**: Captures the current 6 dimensional poses of the robot's end-effector.
     - **object_poses**: Captures the 6 dimensional poses of relevant objects in the scene.
+    - **subtask_start_signals**: Captures subtask start signals. Used by skillgen to identify the precise start of a subtask from a demonstration.
     - **subtask_term_signals**: Captures subtask completions signals.
     - **target_eef_pose**: Captures the target 6 dimensional poses for robot's end effector at each time step.
     - **gripper_action**:  Captures the gripper's state.
@@ -30,6 +31,7 @@ class DatagenInfo:
         eef_pose=None,
         object_poses=None,
         subtask_term_signals=None,
+        subtask_start_signals=None,
         target_eef_pose=None,
         gripper_action=None,
     ):
@@ -38,6 +40,9 @@ class DatagenInfo:
             eef_pose (torch.Tensor or None): robot end effector poses of shape [..., 4, 4]
             object_poses (dict or None): dictionary mapping object name to object poses
                 of shape [..., 4, 4]
+            subtask_start_signals (dict or None): dictionary mapping subtask name to a binary
+                indicator (0 or 1) on whether subtask has started. This is required when using skillgen.
+                Each value in the dictionary could be an int, float, or torch.Tensor of shape [..., 1].
             subtask_term_signals (dict or None): dictionary mapping subtask name to a binary
                 indicator (0 or 1) on whether subtask has been completed. Each value in the
                 dictionary could be an int, float, or torch.Tensor of shape [..., 1].
@@ -53,6 +58,17 @@ class DatagenInfo:
         if object_poses is not None:
             self.object_poses = {k: object_poses[k] for k in object_poses}
 
+        # When using skillgen, demonstrations must be annotated with subtask start signals.
+        self.subtask_start_signals = None
+        if subtask_start_signals is not None:
+            self.subtask_start_signals = dict()
+            for k in subtask_start_signals:
+                if isinstance(subtask_start_signals[k], (float, int)):
+                    self.subtask_start_signals[k] = subtask_start_signals[k]
+                else:
+                    # Only create torch tensor if value is not a single value
+                    self.subtask_start_signals[k] = subtask_start_signals[k]
+
         self.subtask_term_signals = None
         if subtask_term_signals is not None:
             self.subtask_term_signals = dict()
@@ -80,6 +96,8 @@ class DatagenInfo:
             ret["eef_pose"] = self.eef_pose
         if self.object_poses is not None:
             ret["object_poses"] = deepcopy(self.object_poses)
+        if self.subtask_start_signals is not None:
+            ret["subtask_start_signals"] = deepcopy(self.subtask_start_signals)
         if self.subtask_term_signals is not None:
             ret["subtask_term_signals"] = deepcopy(self.subtask_term_signals)
         if self.target_eef_pose is not None:

source/isaaclab_mimic/isaaclab_mimic/datagen/generation.py

@@ -31,6 +31,7 @@ async def run_data_generator(
     data_generator: DataGenerator,
     success_term: TerminationTermCfg,
     pause_subtask: bool = False,
+    motion_planner: Any = None,
 ):
     """Run mimic data generation from the given data generator in the specified environment index.
 
@@ -42,6 +43,7 @@ async def run_data_generator(
         data_generator: The data generator instance to use.
         success_term: The success termination term to use.
         pause_subtask: Whether to pause the subtask during generation.
+        motion_planner: The motion planner to use.
     """
     global num_success, num_failures, num_attempts
     while True:
@@ -51,6 +53,7 @@ async def run_data_generator(
             env_reset_queue=env_reset_queue,
             env_action_queue=env_action_queue,
             pause_subtask=pause_subtask,
+            motion_planner=motion_planner,
         )
         if bool(results["success"]):
             num_success += 1
@@ -190,7 +193,12 @@ def setup_env_config(
 
 
 def setup_async_generation(
-    env: Any, num_envs: int, input_file: str, success_term: Any, pause_subtask: bool = False
+    env: Any,
+    num_envs: int,
+    input_file: str,
+    success_term: Any,
+    pause_subtask: bool = False,
+    motion_planners: Any = None,
 ) -> dict[str, Any]:
     """Setup async data generation tasks.
 
@@ -200,6 +208,7 @@ def setup_async_generation(
         input_file: Path to input dataset file
         success_term: Success termination condition
         pause_subtask: Whether to pause after subtasks
+        motion_planners: Motion planner instances for all environments
 
     Returns:
         List of asyncio tasks for data generation
@@ -216,9 +225,17 @@ def setup_async_generation(
     data_generator = DataGenerator(env=env, src_demo_datagen_info_pool=shared_datagen_info_pool)
     data_generator_asyncio_tasks = []
     for i in range(num_envs):
+        env_motion_planner = motion_planners[i] if motion_planners else None
         task = asyncio_event_loop.create_task(
             run_data_generator(
-                env, i, env_reset_queue, env_action_queue, data_generator, success_term, pause_subtask=pause_subtask
+                env,
+                i,
+                env_reset_queue,
+                env_action_queue,
+                data_generator,
+                success_term,
+                pause_subtask=pause_subtask,
+                motion_planner=env_motion_planner,
             )
         )
         data_generator_asyncio_tasks.append(task)

source/isaaclab_mimic/isaaclab_mimic/envs/__init__.py

@@ -7,11 +7,13 @@
 
 import gymnasium as gym
 
+from .franka_bin_stack_ik_rel_mimic_env_cfg import FrankaBinStackIKRelMimicEnvCfg
 from .franka_stack_ik_abs_mimic_env import FrankaCubeStackIKAbsMimicEnv
 from .franka_stack_ik_abs_mimic_env_cfg import FrankaCubeStackIKAbsMimicEnvCfg
 from .franka_stack_ik_rel_blueprint_mimic_env_cfg import FrankaCubeStackIKRelBlueprintMimicEnvCfg
 from .franka_stack_ik_rel_mimic_env import FrankaCubeStackIKRelMimicEnv
 from .franka_stack_ik_rel_mimic_env_cfg import FrankaCubeStackIKRelMimicEnvCfg
+from .franka_stack_ik_rel_skillgen_env_cfg import FrankaCubeStackIKRelSkillgenEnvCfg
 from .franka_stack_ik_rel_visuomotor_cosmos_mimic_env_cfg import FrankaCubeStackIKRelVisuomotorCosmosMimicEnvCfg
 from .franka_stack_ik_rel_visuomotor_mimic_env_cfg import FrankaCubeStackIKRelVisuomotorMimicEnvCfg
 from .galbot_stack_rmp_abs_mimic_env import RmpFlowGalbotCubeStackAbsMimicEnv
@@ -74,6 +76,28 @@ gym.register(
 )
 
 
+##
+# SkillGen
+##
+
+gym.register(
+    id="Isaac-Stack-Cube-Franka-IK-Rel-Skillgen-v0",
+    entry_point="isaaclab_mimic.envs:FrankaCubeStackIKRelMimicEnv",
+    kwargs={
+        "env_cfg_entry_point": franka_stack_ik_rel_skillgen_env_cfg.FrankaCubeStackIKRelSkillgenEnvCfg,
+    },
+    disable_env_checker=True,
+)
+
+gym.register(
+    id="Isaac-Stack-Cube-Bin-Franka-IK-Rel-Mimic-v0",
+    entry_point="isaaclab_mimic.envs:FrankaCubeStackIKRelMimicEnv",
+    kwargs={
+        "env_cfg_entry_point": franka_bin_stack_ik_rel_mimic_env_cfg.FrankaBinStackIKRelMimicEnvCfg,
+    },
+    disable_env_checker=True,
+)
+
 ##
 # Galbot Stack Cube with RmpFlow - Relative Pose Control
 ##
@@ -99,6 +123,7 @@ gym.register(
 ##
 # Galbot Stack Cube with RmpFlow - Absolute Pose Control
 ##
+
 gym.register(
     id="Isaac-Stack-Cube-Galbot-Left-Arm-Gripper-RmpFlow-Abs-Mimic-v0",
     entry_point="isaaclab_mimic.envs:RmpFlowGalbotCubeStackAbsMimicEnv",

source/isaaclab_mimic/isaaclab_mimic/envs/franka_bin_stack_ik_rel_mimic_env_cfg.py

+# Copyright (c) 2024-2025, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
+# All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+from isaaclab.envs.mimic_env_cfg import MimicEnvCfg, SubTaskConfig
+from isaaclab.utils import configclass
+
+from isaaclab_tasks.manager_based.manipulation.stack.config.franka.bin_stack_ik_rel_env_cfg import FrankaBinStackEnvCfg
+
+
+@configclass
+class FrankaBinStackIKRelMimicEnvCfg(FrankaBinStackEnvCfg, MimicEnvCfg):
+    """
+    Isaac Lab Mimic environment config class for Franka Cube Stack IK Rel env.
+    """
+
+    def __post_init__(self):
+
+        # post init of parents
+        super().__post_init__()
+
+        # Override the existing values
+        self.datagen_config.name = "demo_src_stack_isaac_lab_task_D0"
+        self.datagen_config.generation_guarantee = True
+        self.datagen_config.generation_keep_failed = True
+        self.datagen_config.generation_num_trials = 10
+        self.datagen_config.generation_select_src_per_subtask = True
+        self.datagen_config.generation_transform_first_robot_pose = False
+        self.datagen_config.generation_interpolate_from_last_target_pose = True
+        self.datagen_config.generation_relative = True
+        self.datagen_config.max_num_failures = 25
+        self.datagen_config.seed = 1
+
+        # The following are the subtask configurations for the stack task.
+        subtask_configs = []
+        subtask_configs.append(
+            SubTaskConfig(
+                object_ref="cube_2",
+                subtask_term_signal="grasp_1",
+                subtask_term_offset_range=(0, 0),
+                selection_strategy="nearest_neighbor_object",
+                selection_strategy_kwargs={"nn_k": 3},
+                action_noise=0.03,
+                num_interpolation_steps=0,
+                num_fixed_steps=0,
+                apply_noise_during_interpolation=False,
+                description="Grasp red cube",
+                next_subtask_description="Stack red cube on top of blue cube",
+            )
+        )
+        subtask_configs.append(
+            SubTaskConfig(
+                object_ref="cube_1",
+                subtask_term_signal="stack_1",
+                subtask_term_offset_range=(0, 0),
+                selection_strategy="nearest_neighbor_object",
+                selection_strategy_kwargs={"nn_k": 3},
+                action_noise=0.03,
+                num_interpolation_steps=0,
+                num_fixed_steps=0,
+                apply_noise_during_interpolation=False,
+                next_subtask_description="Grasp green cube",
+            )
+        )
+        subtask_configs.append(
+            SubTaskConfig(
+                object_ref="cube_3",
+                subtask_term_signal="grasp_2",
+                subtask_term_offset_range=(0, 0),
+                selection_strategy="nearest_neighbor_object",
+                selection_strategy_kwargs={"nn_k": 3},
+                action_noise=0.03,
+                num_interpolation_steps=0,
+                num_fixed_steps=0,
+                apply_noise_during_interpolation=False,
+                next_subtask_description="Stack green cube on top of red cube",
+            )
+        )
+        subtask_configs.append(
+            SubTaskConfig(
+                object_ref="cube_2",
+                subtask_term_signal="stack_2",
+                subtask_term_offset_range=(0, 0),
+                selection_strategy="nearest_neighbor_object",
+                selection_strategy_kwargs={"nn_k": 3},
+                action_noise=0.03,
+                num_interpolation_steps=0,
+                num_fixed_steps=0,
+                apply_noise_during_interpolation=False,
+            )
+        )
+        self.subtask_configs["franka"] = subtask_configs

source/isaaclab_mimic/isaaclab_mimic/envs/franka_stack_ik_rel_mimic_env.py

@@ -163,3 +163,26 @@ class FrankaCubeStackIKRelMimicEnv(ManagerBasedRLMimicEnv):
         signals["stack_1"] = subtask_terms["stack_1"][env_ids]
         # final subtask is placing cubeC on cubeA (motion relative to cubeA) - but final subtask signal is not needed
         return signals
+
+    def get_expected_attached_object(self, eef_name: str, subtask_index: int, env_cfg) -> str | None:
+        """
+        (SkillGen) Return the expected attached object for the given EEF/subtask.
+
+        Assumes 'stack' subtasks place the object grasped in the preceding 'grasp' subtask.
+        Returns None for 'grasp' (or others) at subtask start.
+        """
+        if eef_name not in env_cfg.subtask_configs:
+            return None
+
+        subtask_configs = env_cfg.subtask_configs[eef_name]
+        if not (0 <= subtask_index < len(subtask_configs)):
+            return None
+
+        current_cfg = subtask_configs[subtask_index]
+        # If stacking, expect we are holding the object grasped in the prior subtask
+        if "stack" in str(current_cfg.subtask_term_signal).lower():
+            if subtask_index > 0:
+                prev_cfg = subtask_configs[subtask_index - 1]
+                if "grasp" in str(prev_cfg.subtask_term_signal).lower():
+                    return prev_cfg.object_ref
+        return None

source/isaaclab_mimic/isaaclab_mimic/envs/franka_stack_ik_rel_skillgen_env_cfg.py

+# Copyright (c) 2024-2025, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
+# All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+from isaaclab.envs.mimic_env_cfg import MimicEnvCfg, SubTaskConfig
+from isaaclab.utils import configclass
+
+from isaaclab_tasks.manager_based.manipulation.stack.config.franka.stack_ik_rel_env_cfg_skillgen import (
+    FrankaCubeStackSkillgenEnvCfg,
+)
+
+
+@configclass
+class FrankaCubeStackIKRelSkillgenEnvCfg(FrankaCubeStackSkillgenEnvCfg, MimicEnvCfg):
+    """
+    Isaac Lab Mimic environment config class for Franka Cube Stack IK Rel env.
+    """
+
+    def __post_init__(self):
+        # post init of parents
+        super().__post_init__()
+        # # TODO: Figure out how we can move this to the MimicEnvCfg class
+        # # The __post_init__() above only calls the init for FrankaCubeStackEnvCfg and not MimicEnvCfg
+        # # https://stackoverflow.com/questions/59986413/achieving-multiple-inheritance-using-python-dataclasses
+
+        # Override the existing values
+        self.datagen_config.name = "demo_src_stack_isaac_lab_task_D0"
+        self.datagen_config.generation_guarantee = True
+        self.datagen_config.generation_keep_failed = True
+        self.datagen_config.generation_num_trials = 10
+        self.datagen_config.generation_select_src_per_subtask = True
+        self.datagen_config.generation_transform_first_robot_pose = False
+        self.datagen_config.generation_interpolate_from_last_target_pose = True
+        self.datagen_config.generation_relative = True
+        self.datagen_config.max_num_failures = 25
+        self.datagen_config.seed = 1
+
+        # The following are the subtask configurations for the stack task.
+        subtask_configs = []
+        subtask_configs.append(
+            SubTaskConfig(
+                # Each subtask involves manipulation with respect to a single object frame.
+                object_ref="cube_2",
+                # This key corresponds to the binary indicator in "datagen_info" that signals
+                # when this subtask is finished (e.g., on a 0 to 1 edge).
+                subtask_term_signal="grasp_1",
+                # Specifies time offsets for data generation when splitting a trajectory into
+                # subtask segments. Random offsets are added to the termination boundary.
+                subtask_term_offset_range=(0, 0),  # (10, 20),
+                # Selection strategy for the source subtask segment during data generation
+                selection_strategy="nearest_neighbor_object",
+                # Optional parameters for the selection strategy function
+                selection_strategy_kwargs={"nn_k": 3},
+                # Amount of action noise to apply during this subtask
+                action_noise=0.03,
+                # Number of interpolation steps to bridge to this subtask segment
+                num_interpolation_steps=0,  # 5,
+                # Additional fixed steps for the robot to reach the necessary pose
+                num_fixed_steps=0,
+                # If True, apply action noise during the interpolation phase and execution
+                apply_noise_during_interpolation=False,
+                description="Grasp red cube",
+                next_subtask_description="Stack red cube on top of blue cube",
+            )
+        )
+        subtask_configs.append(
+            SubTaskConfig(
+                # Each subtask involves manipulation with respect to a single object frame.
+                object_ref="cube_1",
+                # Corresponding key for the binary indicator in "datagen_info" for completion
+                subtask_term_signal="stack_1",
+                # Time offsets for data generation when splitting a trajectory
+                subtask_term_offset_range=(0, 0),  # (10, 20),
+                # Selection strategy for source subtask segment
+                selection_strategy="nearest_neighbor_object",
+                # Optional parameters for the selection strategy function
+                selection_strategy_kwargs={"nn_k": 3},
+                # Amount of action noise to apply during this subtask
+                action_noise=0.03,
+                # Number of interpolation steps to bridge to this subtask segment
+                num_interpolation_steps=0,  # 5,
+                # Additional fixed steps for the robot to reach the necessary pose
+                num_fixed_steps=0,
+                # If True, apply action noise during the interpolation phase and execution
+                apply_noise_during_interpolation=False,
+                next_subtask_description="Grasp green cube",
+            )
+        )
+        subtask_configs.append(
+            SubTaskConfig(
+                # Each subtask involves manipulation with respect to a single object frame.
+                object_ref="cube_3",
+                # Corresponding key for the binary indicator in "datagen_info" for completion
+                subtask_term_signal="grasp_2",
+                # Time offsets for data generation when splitting a trajectory
+                subtask_term_offset_range=(0, 0),  # (10, 20),
+                # Selection strategy for source subtask segment
+                selection_strategy="nearest_neighbor_object",
+                # Optional parameters for the selection strategy function
+                selection_strategy_kwargs={"nn_k": 3},
+                # Amount of action noise to apply during this subtask
+                action_noise=0.03,
+                # Number of interpolation steps to bridge to this subtask segment
+                num_interpolation_steps=0,  # 5,
+                # Additional fixed steps for the robot to reach the necessary pose
+                num_fixed_steps=0,
+                # If True, apply action noise during the interpolation phase and execution
+                apply_noise_during_interpolation=False,
+                next_subtask_description="Stack green cube on top of red cube",
+            )
+        )
+        subtask_configs.append(
+            SubTaskConfig(
+                # Each subtask involves manipulation with respect to a single object frame.
+                object_ref="cube_2",
+                # End of final subtask does not need to be detected for MimicGen
+                # Needs to be detected for SkillGen
+                # Setting this doesn't affect the data generation for MimicGen
+                subtask_term_signal="stack_2",
+                # No time offsets for the final subtask
+                subtask_term_offset_range=(0, 0),
+                # Selection strategy for source subtask segment
+                selection_strategy="nearest_neighbor_object",
+                # Optional parameters for the selection strategy function
+                selection_strategy_kwargs={"nn_k": 3},
+                # Amount of action noise to apply during this subtask
+                action_noise=0.03,
+                # Number of interpolation steps to bridge to this subtask segment
+                num_interpolation_steps=0,  # 5,
+                # Additional fixed steps for the robot to reach the necessary pose
+                num_fixed_steps=0,
+                # If True, apply action noise during the interpolation phase and execution
+                apply_noise_during_interpolation=False,
+            )
+        )
+        self.subtask_configs["franka"] = subtask_configs

source/isaaclab_mimic/isaaclab_mimic/motion_planners/motion_planner_base.py

+# Copyright (c) 2024-2025, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
+# All rights reserved.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+import torch
+from abc import ABC, abstractmethod
+from typing import Any
+
+from isaaclab.assets import Articulation
+from isaaclab.envs.manager_based_env import ManagerBasedEnv
+
+
+class MotionPlannerBase(ABC):
+    """Abstract base class for motion planners.
+
+    This class defines the public interface that all motion planners must implement.
+    It focuses on the essential functionality that users interact with, while leaving
+    implementation details to specific planner backends.
+
+    The core workflow is:
+    1. Initialize planner with environment and robot
+    2. Call update_world_and_plan_motion() to plan to a target
+    3. Execute plan using has_next_waypoint() and get_next_waypoint_ee_pose()
+
+    Example:
+        >>> from isaaclab_mimic.motion_planners.curobo.curobo_planner import CuroboPlanner
+        >>> from isaaclab_mimic.motion_planners.curobo.curobo_planner_cfg import CuroboPlannerCfg
+        >>> config = CuroboPlannerCfg.franka_config()
+        >>> planner = CuroboPlanner(env, robot, config)
+        >>> success = planner.update_world_and_plan_motion(target_pose)
+        >>> if success:
+        >>>     while planner.has_next_waypoint():
+        >>>         action = planner.get_next_waypoint_ee_pose()
+        >>>         obs, info = env.step(action)
+    """
+
+    def __init__(
+        self, env: ManagerBasedEnv, robot: Articulation, env_id: int = 0, debug: bool = False, **kwargs
+    ) -> None:
+        """Initialize the motion planner.
+
+        Args:
+            env: The environment instance
+            robot: Robot articulation to plan motions for
+            env_id: Environment ID (0 to num_envs-1)
+            debug: Whether to print detailed debugging information
+            **kwargs: Additional planner-specific arguments
+        """
+        self.env = env
+        self.robot = robot
+        self.env_id = env_id
+        self.debug = debug
+
+    @abstractmethod
+    def update_world_and_plan_motion(self, target_pose: torch.Tensor, **kwargs: Any) -> bool:
+        """Update collision world and plan motion to target pose.
+
+        This is the main entry point for motion planning. It should:
+        1. Update the planner's internal world representation
+        2. Plan a collision-free path to the target pose
+        3. Store the plan internally for execution
+
+        Args:
+            target_pose: Target pose to plan motion to (4x4 transformation matrix)
+            **kwargs: Planner-specific arguments (e.g., retiming, contact planning)
+
+        Returns:
+            bool: True if planning succeeded, False otherwise
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def has_next_waypoint(self) -> bool:
+        """Check if there are more waypoints in current plan.
+
+        Returns:
+            bool: True if there are more waypoints, False otherwise
+        """
+        raise NotImplementedError
+
+    @abstractmethod
+    def get_next_waypoint_ee_pose(self) -> Any:
+        """Get next waypoint's end-effector pose from current plan.
+
+        This method should only be called after checking has_next_waypoint().
+
+        Returns:
+            Any: End-effector pose for the next waypoint in the plan.
+        """
+        raise NotImplementedError
+
+    def get_planned_poses(self) -> list[Any]:
+        """Get all planned poses from current plan.
+
+        Returns:
+            list[Any]: List of planned poses.
+
+        Note:
+            Default implementation iterates through waypoints.
+            Child classes can override for a more efficient implementation.
+        """
+        planned_poses = []
+        # Create a copy of the planner state to not affect the original plan execution
+        # This is a placeholder and may need to be implemented by child classes
+        # if they manage complex internal state.
+        # For now, we assume the planner can be reset and we can iterate through the plan.
+        # A more robust solution might involve a dedicated method to get the full plan.
+        self.reset_plan()
+        while self.has_next_waypoint():
+            pose = self.get_next_waypoint_ee_pose()
+            planned_poses.append(pose)
+        return planned_poses
+
+    @abstractmethod
+    def reset_plan(self) -> None:
+        """Reset the current plan and execution state.
+
+        This should clear any stored plan and reset the execution index or iterator.
+        """
+        raise NotImplementedError
+
+    def get_planner_info(self) -> dict[str, Any]:
+        """Get information about the planner.
+
+        Returns:
+            dict: Information about the planner (name, version, capabilities, etc.)
+        """
+        return {
+            "name": self.__class__.__name__,
+            "env_id": self.env_id,
+            "debug": self.debug,
+        }

source/isaaclab_tasks/config/extension.toml

 [package]
 
 # Note: Semantic Versioning is used: https://semver.org/
-version = "0.10.50"
+version = "0.10.51"
 
 # Description
 title = "Isaac Lab Environments"

source/isaaclab_tasks/isaaclab_tasks/manager_based/manipulation/stack/config/franka/stack_joint_pos_env_cfg.py

@@ -59,6 +59,7 @@ class EventCfg:
 
 @configclass
 class FrankaCubeStackEnvCfg(StackEnvCfg):
+
     def __post_init__(self):
         # post init of parent
         super().__post_init__()