Commit 3aba0d13 authored by cosmith-nvidia's avatar cosmith-nvidia Committed by Kelly Guo

Adds initial documentation for CloudXR Teleoperation. (#296)

# Description

Add initial documentation for CloudXR Teleoperation.

## Type of change

- Documentation

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

---------
Signed-off-by: 's avatarKelly Guo <kellyguo123@hotmail.com>
Co-authored-by: 's avatarKelly Guo <kellyg@nvidia.com>
parent 464631fa
This diff is collapsed.
...@@ -113,3 +113,15 @@ additional resources that help you use Omniverse features in Isaac Lab. ...@@ -113,3 +113,15 @@ additional resources that help you use Omniverse features in Isaac Lab.
:maxdepth: 1 :maxdepth: 1
master_omniverse master_omniverse
Setting up CloudXR Teleoperation
--------------------------------
This guide explains how to use CloudXR and Apple Vision Pro for immersive streaming and
teleoperation in Isaac Lab.
.. toctree::
:maxdepth: 1
cloudxr_teleoperation
.. _teleoperation-imitation-learning:
Teleoperation and Imitation Learning Teleoperation and Imitation Learning
==================================== ====================================
...@@ -33,6 +35,17 @@ For smoother operation and off-axis operation, we recommend using a SpaceMouse a ...@@ -33,6 +35,17 @@ For smoother operation and off-axis operation, we recommend using a SpaceMouse a
Only compatible with the SpaceMouse Wireless and SpaceMouse Compact models from 3Dconnexion. Only compatible with the SpaceMouse Wireless and SpaceMouse Compact models from 3Dconnexion.
For tasks that benefit from the use of an extended reality (XR) device with hand tracking, Isaac Lab supports using NVIDIA CloudXR to immersively stream the scene to compatible XR devices for teleoperation.
.. code:: bash
./isaaclab.sh -p scripts/environments/teleoperation/teleop_se3_agent.py --task Isaac-Stack-Cube-Franka-IK-Abs-v0 --num_envs 1 --teleop_device handtracking --device cpu
.. note::
See :ref:`cloudxr-teleoperation` to learn more about using CloudXR with Isaac Lab.
The script prints the teleoperation events configured. For keyboard, The script prints the teleoperation events configured. For keyboard,
these are as follows: these are as follows:
...@@ -78,7 +91,7 @@ To collect demonstrations with teleoperation for the environment ``Isaac-Stack-C ...@@ -78,7 +91,7 @@ To collect demonstrations with teleoperation for the environment ``Isaac-Stack-C
# step a: create folder for datasets # step a: create folder for datasets
mkdir -p datasets mkdir -p datasets
# step b: collect data with a selected teleoperation device. Replace <teleop_device> with your preferred input device. # step b: collect data with a selected teleoperation device. Replace <teleop_device> with your preferred input device.
# Available options: spacemouse, keyboard # Available options: spacemouse, keyboard, handtracking
./isaaclab.sh -p scripts/tools/record_demos.py --task Isaac-Stack-Cube-Franka-IK-Rel-v0 --teleop_device <teleop_device> --dataset_file ./datasets/dataset.hdf5 --num_demos 10 ./isaaclab.sh -p scripts/tools/record_demos.py --task Isaac-Stack-Cube-Franka-IK-Rel-v0 --teleop_device <teleop_device> --dataset_file ./datasets/dataset.hdf5 --num_demos 10
# step a: replay the collected dataset # step a: replay the collected dataset
./isaaclab.sh -p scripts/tools/replay_demos.py --task Isaac-Stack-Cube-Franka-IK-Rel-v0 --dataset_file ./datasets/dataset.hdf5 ./isaaclab.sh -p scripts/tools/replay_demos.py --task Isaac-Stack-Cube-Franka-IK-Rel-v0 --dataset_file ./datasets/dataset.hdf5
...@@ -88,6 +101,10 @@ To collect demonstrations with teleoperation for the environment ``Isaac-Stack-C ...@@ -88,6 +101,10 @@ To collect demonstrations with teleoperation for the environment ``Isaac-Stack-C
The order of the stacked cubes should be blue (bottom), red (middle), green (top). The order of the stacked cubes should be blue (bottom), red (middle), green (top).
.. note::
When using the ``handtracking`` device, we suggest collecting demonstrations with the ``Isaac-Stack-Cube-Frank-IK-Abs-v0`` version of the task, which controls the end effector using the absolute position of the hand.
About 10 successful demonstrations are required in order for the following steps to succeed. About 10 successful demonstrations are required in order for the following steps to succeed.
Here are some tips to perform demonstrations that lead to successful policy training: Here are some tips to perform demonstrations that lead to successful policy training:
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment