Unverified Commit 8608fbce authored by Pascal Roth's avatar Pascal Roth Committed by GitHub

Fixes template docs and restructure imitation learning docs (#3283)

# Description

There have been two places where the template documentation has been
placed (under Developers Guide and Workthrough), this PR unifies them
into a new structure (see image below).

Furthermore, the imitation learning examples were missing a grouping,
this PR introduces a structure similar to the section about
reinforcement learning

Also some general docs fixes are included.

## Type of change

- Bug fix (non-breaking change which fixes an issue)

## Screenshots

<img width="588" height="1323" alt="image"
src="https://github.com/user-attachments/assets/a17a6328-a9e3-44cd-a299-6aa62e0e422e"
/>

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there
parent a80fa53b
...@@ -89,6 +89,7 @@ Table of Contents ...@@ -89,6 +89,7 @@ Table of Contents
:titlesonly: :titlesonly:
source/setup/quickstart source/setup/quickstart
source/overview/own-project/index
source/setup/walkthrough/index source/setup/walkthrough/index
source/tutorials/index source/tutorials/index
source/how-to/index source/how-to/index
...@@ -104,8 +105,7 @@ Table of Contents ...@@ -104,8 +105,7 @@ Table of Contents
source/overview/core-concepts/index source/overview/core-concepts/index
source/overview/environments source/overview/environments
source/overview/reinforcement-learning/index source/overview/reinforcement-learning/index
source/overview/teleop_imitation source/overview/imitation-learning/index
source/overview/augmented_imitation
source/overview/showroom source/overview/showroom
source/overview/simple_agents source/overview/simple_agents
......
...@@ -18,7 +18,7 @@ properties of an :class:`~assets.Articulation` in Isaac Lab. ...@@ -18,7 +18,7 @@ properties of an :class:`~assets.Articulation` in Isaac Lab.
We will use the Cartpole example to demonstrate how to create an :class:`~assets.ArticulationCfg`. We will use the Cartpole example to demonstrate how to create an :class:`~assets.ArticulationCfg`.
The Cartpole is a simple robot that consists of a cart with a pole attached to it. The cart The Cartpole is a simple robot that consists of a cart with a pole attached to it. The cart
is free to move along a rail, and the pole is free to rotate about the cart. The file for this configuration example is is free to move along a rail, and the pole is free to rotate about the cart. The file for this configuration example is
``source/isaaclab_assets/isaaclab_assets/robots/cartpole.py``. ``source/isaaclab_assets/isaaclab_assets/robots/cartpole.py``.
.. dropdown:: Code for Cartpole configuration .. dropdown:: Code for Cartpole configuration
:icon: code :icon: code
......
...@@ -13,4 +13,3 @@ using VSCode. ...@@ -13,4 +13,3 @@ using VSCode.
VS Code <vs_code> VS Code <vs_code>
repo_structure repo_structure
development development
template
...@@ -83,7 +83,7 @@ Example usage for the cube stacking task: ...@@ -83,7 +83,7 @@ Example usage for the cube stacking task:
Running Cosmos for Visual Augmentation Running Cosmos for Visual Augmentation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
After converting the demonstrations to MP4 format, you can use a `Cosmos <https://github.com/NVIDIA/Cosmos?tab=readme-ov-file>`_ model to visually augment the videos. Follow the Cosmos documentation for details on the augmentation process. Visual augmentation can include changes to lighting, textures, backgrounds, and other visual elements while preserving the essential task-relevant features. After converting the demonstrations to MP4 format, you can use a `Cosmos`_ model to visually augment the videos. Follow the Cosmos documentation for details on the augmentation process. Visual augmentation can include changes to lighting, textures, backgrounds, and other visual elements while preserving the essential task-relevant features.
We use the RGB, depth and shaded segmentation videos from the previous step as input to the Cosmos model as seen below: We use the RGB, depth and shaded segmentation videos from the previous step as input to the Cosmos model as seen below:
...@@ -99,7 +99,7 @@ We provide an example augmentation output from `Cosmos Transfer1 <https://github ...@@ -99,7 +99,7 @@ We provide an example augmentation output from `Cosmos Transfer1 <https://github
:align: center :align: center
:alt: Cosmos Transfer1 augmentation output :alt: Cosmos Transfer1 augmentation output
We recommend using the `Cosmos Transfer1 <https://github.com/nvidia-cosmos/cosmos-transfer1/tree/e4055e39ee9c53165e85275bdab84ed20909714a>`_ model for visual augmentation as we found it to produce the best results in the form of a highly diverse dataset with a wide range of visual variations. You can refer to the installation instructions `here <https://github.com/nvidia-cosmos/cosmos-transfer1/blob/e4055e39ee9c53165e85275bdab84ed20909714a/INSTALL.md#environment-setup>`_, the checkpoint download instructions `here <https://github.com/nvidia-cosmos/cosmos-transfer1/blob/e4055e39ee9c53165e85275bdab84ed20909714a/examples/inference_cosmos_transfer1_7b.md#download-checkpoints>`_ and `this example <https://github.com/nvidia-cosmos/cosmos-transfer1/blob/e4055e39ee9c53165e85275bdab84ed20909714a/examples/inference_cosmos_transfer1_7b.md#example-2-multimodal-control>`_ for reference on how to use Transfer1 for this usecase. We further recommend the following settings to be used with the Transfer1 model for this task: We recommend using the `Cosmos Transfer1 <https://github.com/nvidia-cosmos/cosmos-transfer1/tree/e4055e39ee9c53165e85275bdab84ed20909714a>`_ model for visual augmentation as we found it to produce the best results in the form of a highly diverse dataset with a wide range of visual variations. You can refer to the `installation instructions <https://github.com/nvidia-cosmos/cosmos-transfer1/blob/e4055e39ee9c53165e85275bdab84ed20909714a/INSTALL.md#environment-setup>`_, the `checkpoint download instructions <https://github.com/nvidia-cosmos/cosmos-transfer1/blob/e4055e39ee9c53165e85275bdab84ed20909714a/examples/inference_cosmos_transfer1_7b.md#download-checkpoints>`_ and `this example <https://github.com/nvidia-cosmos/cosmos-transfer1/blob/e4055e39ee9c53165e85275bdab84ed20909714a/examples/inference_cosmos_transfer1_7b.md#example-2-multimodal-control>`_ for reference on how to use Transfer1 for this usecase. We further recommend the following settings to be used with the Transfer1 model for this task:
.. rubric:: Hyperparameters .. rubric:: Hyperparameters
......
Imitation Learning
==================
In this section, we show existing scripts for running imitation learning
with Isaac Lab.
.. toctree::
:maxdepth: 1
augmented_imitation
teleop_imitation
...@@ -413,7 +413,7 @@ Annotations denote the end of a subtask. For the pick and place task, this means ...@@ -413,7 +413,7 @@ Annotations denote the end of a subtask. For the pick and place task, this means
Each demo requires a single annotation between the first and second subtask of the right arm. This annotation ("S" button press) should be done when the right robot arm finishes the "idle" subtask and begins to Each demo requires a single annotation between the first and second subtask of the right arm. This annotation ("S" button press) should be done when the right robot arm finishes the "idle" subtask and begins to
move towards the target object. An example of a correct annotation is shown below: move towards the target object. An example of a correct annotation is shown below:
.. figure:: ../_static/tasks/manipulation/gr-1_pick_place_annotation.jpg .. figure:: ../../_static/tasks/manipulation/gr-1_pick_place_annotation.jpg
:width: 100% :width: 100%
:align: center :align: center
......
.. _own-project:
Build your Own Project or Task
==============================
To get started, first create a new project or task with the template generator :ref:`template-generator`.
For more detailed information on how your project is structured, see :ref:`project-structure`.
.. toctree::
:maxdepth: 1
:titlesonly:
template
project_structure
.. _walkthrough_project_setup: .. _project-structure:
Isaac Lab Project Setup
========================
The best way to create a new project is to use the :ref:`Template Generator<template-generator>`. Generating the template
for this tutorial series is done by calling the ``isaaclab`` script from the root directory of the repository
.. code-block:: bash
./isaaclab.sh --new
Be sure to select ``External`` and ``Direct | single agent``. For the frameworks, select ``skrl`` and both ``PPO`` and ``AMP`` on the following menu. You can
select other frameworks if you like, but this tutorial will detail ``skrl`` specifically. The configuration process for other frameworks is similar. You
can get a copy of this code directly by checking out the `initial branch of the tutorial repository <https://github.com/isaac-sim/IsaacLabTutorial/tree/initial>`_!
This will create an extension project with the specified name at the chosen path. For this tutorial, we chose the name ``isaac_lab_tutorial``.
.. note::
The template generator expects the project name to respect "snake_case": all lowercase with underscores separating words. However, we have renamed the
sample project to "IsaacLabTutorial" to more closely match the naming convention GitHub and our other projects. If you are following along with the example
repository, note this minor difference as some superficial path names may change. If you are following along by building the project yourself, then you can ignore this note.
Next, we must install the project as a python module. Navigate to the directory that was just created
(it will contain the ``source`` and ``scripts`` directories for the project) and then run the following to install the module.
.. code-block:: bash
python -m pip install -e source/isaac_lab_tutorial
To verify that things have been setup properly, run
.. code-block:: bash
python scripts/list_envs.py
from the root directory of your new project. This should generate a table that looks something like the following
.. code-block:: bash
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Available Environments in Isaac Lab |
+--------+---------------------------------------+-----------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
| S. No. | Task Name | Entry Point | Config |
+--------+---------------------------------------+-----------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
| 1 | Template-Isaac-Lab-Tutorial-Direct-v0 | isaac_lab_tutorial.tasks.direct.isaac_lab_tutorial.isaac_lab_tutorial_env:IsaacLabTutorialEnv | isaac_lab_tutorial.tasks.direct.isaac_lab_tutorial.isaac_lab_tutorial_env_cfg:IsaacLabTutorialEnvCfg |
+--------+---------------------------------------+-----------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------+
We can now use the task name to run the environment.
.. code-block:: bash
python scripts/skrl/train.py --task=Template-Isaac-Lab-Tutorial-Direct-v0
and by default, this should start a cartpole training environment.
Let the training finish and then run the following command to see the trained policy in action!
.. code-block:: bash
python scripts/skrl/play.py --task=Template-Isaac-Lab-Tutorial-Direct-v0
Notice that you did not need to specify the path for the checkpoint file! This is because Isaac Lab handles much of the minute details
like checkpoint saving, loading, and logging. In this case, the ``train.py`` script will create two directories: **logs** and **output**, which are
used as the default output directories for tasks run by this project.
Project Structure Project Structure
----------------- =================
There are four nested structures you need to be aware of when working in the direct workflow with an Isaac Lab template There are four nested structures you need to be aware of when working in the direct workflow with an Isaac Lab template
project: the **Project**, the **Extension**, the **Modules**, and the **Task**. project: the **Project**, the **Extension**, the **Modules**, and the **Task**.
......
.. _template-generator: .. _template-generator:
Build your Own Project or Task
============================== Create new project or task
==========================
Traditionally, building new projects that utilize Isaac Lab's features required creating your own Traditionally, building new projects that utilize Isaac Lab's features required creating your own
extensions within the Isaac Lab repository. However, this approach can obscure project visibility and extensions within the Isaac Lab repository. However, this approach can obscure project visibility and
......
...@@ -24,7 +24,7 @@ reference frame is what defines the world. ...@@ -24,7 +24,7 @@ reference frame is what defines the world.
"Above" the world in structure is the **Sim**\ ulation and the **App**\ lication. The **Application** is "the thing responsible for "Above" the world in structure is the **Sim**\ ulation and the **App**\ lication. The **Application** is "the thing responsible for
everything else": It governs all resource management as well as launching and destroying the simulation when we are done with it. everything else": It governs all resource management as well as launching and destroying the simulation when we are done with it.
When we :ref:`launched training with the template<walkthrough_project_setup>`, the window that appears with the viewport of cartpoles When we :ref:`launched training with the template<template-generator>`, the window that appears with the viewport of cartpoles
training is the Application window. The application is not defined by the GUI however, and even when running in headless mode all training is the Application window. The application is not defined by the GUI however, and even when running in headless mode all
simulations have an application that governs them. simulations have an application that governs them.
......
...@@ -17,7 +17,6 @@ represents a different stage of modifying the default template project to achiev ...@@ -17,7 +17,6 @@ represents a different stage of modifying the default template project to achiev
:maxdepth: 1 :maxdepth: 1
:titlesonly: :titlesonly:
project_setup
concepts_env_design concepts_env_design
api_env_design api_env_design
technical_env_design technical_env_design
......
...@@ -125,7 +125,7 @@ Added ...@@ -125,7 +125,7 @@ Added
0.44.12 (2025-08-12) 0.44.12 (2025-08-12)
~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
Fixed Fixed
^^^^^ ^^^^^
...@@ -135,7 +135,7 @@ Fixed ...@@ -135,7 +135,7 @@ Fixed
0.44.11 (2025-08-11) 0.44.11 (2025-08-11)
~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
Fixed Fixed
^^^^^ ^^^^^
...@@ -144,7 +144,7 @@ Fixed ...@@ -144,7 +144,7 @@ Fixed
0.44.10 (2025-08-06) 0.44.10 (2025-08-06)
~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
Fixed Fixed
^^^^^ ^^^^^
......
...@@ -393,6 +393,7 @@ class RigidObject(AssetBase): ...@@ -393,6 +393,7 @@ class RigidObject(AssetBase):
all the external wrenches will be applied in the frame specified by the last call. all the external wrenches will be applied in the frame specified by the last call.
.. code-block:: python .. code-block:: python
# example of setting external wrench in the global frame # example of setting external wrench in the global frame
asset.set_external_force_and_torque(forces=torch.ones(1, 1, 3), env_ids=[0], is_global=True) asset.set_external_force_and_torque(forces=torch.ones(1, 1, 3), env_ids=[0], is_global=True)
# example of setting external wrench in the link frame # example of setting external wrench in the link frame
......
...@@ -15,7 +15,7 @@ ...@@ -15,7 +15,7 @@
"inputs": [ "inputs": [
{ {
"id": "isaac_path", "id": "isaac_path",
"description": "Absolute path to the current Isaac Sim installation. Can be skipped if Isaac Sim installed from pip.", "description": "Absolute path to the current Isaac Sim installation. If you installed IsaacSim from pip, the import of it failed. Please make sure you run the task with the correct python environment. As fallback, you can directly execute the python script by running: ``python.sh <path-to-your-project>/.vscode/tools/setup_vscode.py``",
{% if platform == "win32" %} {% if platform == "win32" %}
"default": "C:/isaacsim", "default": "C:/isaacsim",
{% else %} {% else %}
......
...@@ -49,11 +49,11 @@ except EOFError: ...@@ -49,11 +49,11 @@ except EOFError:
# check if the isaac-sim directory exists # check if the isaac-sim directory exists
if not os.path.exists(isaacsim_dir): if not os.path.exists(isaacsim_dir):
raise FileNotFoundError( raise FileNotFoundError(
f"Could not find the isaac-sim directory: {isaacsim_dir}. There are two possible reasons for this:" f"Could not find the isaac-sim directory: {isaacsim_dir}. There are two possible reasons for this:\n\t1. The"
"\n\t1. The Isaac Sim directory does not exist as provided CLI path." " Isaac Sim directory does not exist as provided CLI path.\n\t2. The script couldn't import the 'isaacsim'"
"\n\t2. The script could import the 'isaacsim' package. This could be due to the 'isaacsim' package not being " " package. This could be due to the 'isaacsim' package not being installed in the Python"
"installed in the Python environment.\n" " environment.\n\nPlease make sure that the Isaac Sim directory exists or that the 'isaacsim' package is"
"\nPlease make sure that the Isaac Sim directory exists or that the 'isaacsim' package is installed." " installed."
) )
ISAACSIM_DIR = isaacsim_dir ISAACSIM_DIR = isaacsim_dir
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment