Adds documentation for openxr device and retargeters (#302)

# Description
Initial documentation for openxr device and retargeters
<!--
Thank you for your interest in sending a pull request. Please make sure
to check the contribution guidelines.

Link:
https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html
-->

Please include a summary of the change and which issue is fixed. Please
also include relevant motivation and context.
List any dependencies that are required for this change.

Fixes # (issue)

<!-- As a practice, it is recommended to open an issue to have
discussions on the proposed pull request.
This makes it easier for the community to keep track of what is being
developed or added, and if a given feature
is demanded by more than one party. -->

## Type of change

<!-- As you go through the list, delete the ones that are not
applicable. -->

- Bug fix (non-breaking change which fixes an issue)
- New feature (non-breaking change which adds functionality)
- Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- This change requires a documentation update

## Screenshots

Please attach before and after screenshots of the change if applicable.

<!--
Example:

| Before | After |
| ------ | ----- |
| _gif/png before_ | _gif/png after_ |

To upload images to a PR -- simply drag and drop an image while in edit
mode and it should upload the image directly. You can then paste that
source into the above before/after sections.
-->

## Checklist

- [ ] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [ ] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->

---------
Signed-off-by: rwiltz <165190220+rwiltz@users.noreply.github.com>
Co-authored-by: Ashwin Varghese Kuruttukulam <123109010+ashwinvkNV@users.noreply.github.com>

docs/source/api/lab/isaaclab.devices.rst

@@ -8,12 +8,24 @@
   .. autosummary::
 
     DeviceBase
+    RetargeterBase
     Se2Gamepad
     Se3Gamepad
     Se2Keyboard
     Se3Keyboard
+    Se2SpaceMouse
     Se3SpaceMouse
-    Se3SpaceMouse
+    OpenXRDevice
+    isaaclab.devices.openxr.retargeters.GripperRetargeter
+    isaaclab.devices.openxr.retargeters.Se3AbsRetargeter
+    isaaclab.devices.openxr.retargeters.Se3RelRetargeter
+    isaaclab.devices.openxr.retargeters.GR1T2Retargeter
+
+  .. rubric:: Modules
+
+  .. autosummary::
+
+    isaaclab.devices.openxr.retargeters
 
 Device Base
 -----------
@@ -21,6 +33,12 @@ Device Base
 .. autoclass:: DeviceBase
     :members:
 
+Retargeter Base
+---------------
+
+.. autoclass:: RetargeterBase
+    :members:
+
 Game Pad
 --------
 
@@ -59,3 +77,34 @@ Space Mouse
     :members:
     :inherited-members:
     :show-inheritance:
+
+OpenXR
+------
+
+.. autoclass:: OpenXRDevice
+    :members:
+    :inherited-members:
+    :show-inheritance:
+
+Retargeters
+-----------
+
+.. autoclass:: isaaclab.devices.openxr.retargeters.GripperRetargeter
+    :members:
+    :inherited-members:
+    :show-inheritance:
+
+.. autoclass:: isaaclab.devices.openxr.retargeters.Se3AbsRetargeter
+    :members:
+    :inherited-members:
+    :show-inheritance:
+
+.. autoclass:: isaaclab.devices.openxr.retargeters.Se3RelRetargeter
+    :members:
+    :inherited-members:
+    :show-inheritance:
+
+.. autoclass:: isaaclab.devices.openxr.retargeters.GR1T2Retargeter
+    :members:
+    :inherited-members:
+    :show-inheritance:

docs/source/how-to/cloudxr_teleoperation.rst

@@ -49,6 +49,9 @@ This guide will walk you through how to:
 * :ref:`develop-xr-isaac-lab`, including how to :ref:`run-isaac-lab-with-xr`,
   :ref:`configure-scene-placement`, and :ref:`optimize-xr-performance`.
 
+* :ref:`control-robot-with-xr`, including the :ref:`openxr-device-architecture`,
+  :ref:`control-robot-with-xr-retargeters`, and how to implement :ref:`control-robot-with-xr-callbacks`.
+
 As well as :ref:`xr-known-issues`.
 
 
@@ -361,7 +364,7 @@ Back on your Apple Vision Pro:
 .. _develop-xr-isaac-lab:
 
 Develop for XR in Isaac Lab
-----------------------------
+---------------------------
 
 This section will walk you through how to develop XR environments in Isaac Lab for building
 teleoperation workflows.
@@ -460,6 +463,144 @@ Optimize XR Performance
    latency when only a single environment is present in the simulation.
 
 
+.. _control-robot-with-xr:
+
+Control the Robot with XR Device Inputs
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Isaac Lab provides a flexible architecture for using XR tracking data to control
+simulated robots. This section explains the components of this architecture and how they work together.
+
+.. _openxr-device-architecture:
+
+OpenXR Device
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The :class:`isaaclab.devices.OpenXRDevice` is the core component that enables XR-based teleoperation in Isaac Lab.
+This device interfaces with CloudXR to receive tracking data from the XR headset and transform it into robot control
+commands.
+
+At its heart, XR teleoperation requires mapping (or "retargeting") user inputs, such as hand movements and poses,
+into robot control signals. Isaac Lab makes this straightforward through its OpenXRDevice and Retargeter architecture.
+The OpenXRDevice captures hand tracking data via Isaac Sim's OpenXR API, then passes this data through one or more
+Retargeters that convert it into robot actions.
+
+The OpenXRDevice also integrates with the XR device's user interface when using CloudXR, allowing users to trigger
+simulation events directly from their XR environment.
+
+.. _control-robot-with-xr-retargeters:
+
+Retargeting Architecture
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Retargeters are specialized components that convert raw tracking data into meaningful control signals
+for robots. They implement the :class:`isaaclab.devices.RetargeterBase` interface and are passed to
+the OpenXRDevice during initialization.
+
+Isaac Lab provides three main retargeters for hand tracking:
+
+.. dropdown:: Se3RelRetargeter (:class:`isaaclab.devices.openxr.retargeters.Se3RelRetargeter`)
+
+   * Generates incremental robot commands from relative hand movements
+   * Best for precise manipulation tasks
+
+.. dropdown:: Se3AbsRetargeter (:class:`isaaclab.devices.openxr.retargeters.Se3AbsRetargeter`)
+
+   * Maps hand position directly to robot end-effector position
+   * Enables 1:1 spatial control
+
+.. dropdown:: GripperRetargeter (:class:`isaaclab.devices.openxr.retargeters.GripperRetargeter`)
+
+   * Controls gripper state based on thumb-index finger distance
+   * Used alongside position retargeters for full robot control
+
+.. dropdown:: GR1T2Retargeter (:class:`isaaclab.devices.openxr.retargeters.GR1T2Retargeter`)
+
+   * Retargets OpenXR hand tracking data to GR1T2 hand end-effector commands
+   * Handles both left and right hands, converting hand poses to joint angles for the GR1T2 robot's hands
+   * Supports visualization of tracked hand joints
+
+Retargeters can be combined to control different robot functions simultaneously.
+
+Using Retargeters with Hand Tracking
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Here's an example of setting up hand tracking:
+
+.. code-block:: python
+
+   from isaaclab.devices import OpenXRDevice
+   from isaaclab.devices.openxr.retargeters import Se3AbsRetargeter, GripperRetargeter
+
+   # Create retargeters
+   position_retargeter = Se3AbsRetargeter(
+       zero_out_xy_rotation=True,
+       use_wrist_position=False  # Use pinch position (thumb-index midpoint) instead of wrist
+   )
+   gripper_retargeter = GripperRetargeter()
+
+   # Create OpenXR device with hand tracking and both retargeters
+   device = OpenXRDevice(
+       env_cfg.xr,
+       hand=OpenXRDevice.Hand.RIGHT,
+       retargeters=[position_retargeter, gripper_retargeter],
+   )
+
+   # Main control loop
+   while True:
+       # Get the latest commands from the XR device
+       commands = device.advance()
+       if commands is None:
+           continue
+
+       # Apply the commands to the environment
+       obs, reward, terminated, truncated, info = env.step(commands)
+
+       if terminated or truncated:
+           break
+
+
+Extending the Retargeting System
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The retargeting system is designed to be extensible. You can create custom retargeters by extending
+the :class:`isaaclab.devices.RetargeterBase` class and implementing the ``retarget`` method that
+processes the incoming tracking data:
+
+.. code-block:: python
+
+   from isaaclab.devices.retargeter_base import RetargeterBase
+
+   class MyCustomRetargeter(RetargeterBase):
+       def retarget(self, data: dict[str, np.ndarray]) -> Any:
+           # Process the tracking data
+           # Return control commands in appropriate format
+           ...
+
+As the OpenXR capabilities expand beyond hand tracking to include head tracking and other features,
+additional retargeters can be developed to map this data to various robot control paradigms.
+
+.. _control-robot-with-xr-callbacks:
+
+Adding Callbacks for XR UI Events
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The OpenXRDevice can handle events triggered by user interactions with XR UI elements like buttons and menus.
+When a user interacts with these elements, the device triggers registered callback functions:
+
+.. code-block:: python
+
+   # Register callbacks for teleop control events
+   device.add_callback("RESET", reset_callback)
+   device.add_callback("START", start_callback)
+   device.add_callback("STOP", stop_callback)
+
+When the user interacts with the XR UI, these callbacks will be triggered to control the simulation
+or recording process. You can also add custom messages from the client side using custom keys that will
+trigger these callbacks, allowing for programmatic control of the simulation alongside direct user interaction.
+The custom keys can be any string value that matches the callback registration.
+
+
 .. _xr-known-issues:
 
 Known Issues

source/isaaclab/isaaclab/devices/device_base.py

@@ -87,8 +87,8 @@ class DeviceBase(ABC):
         2. Override this method completely for custom command processing
 
         Returns:
-            If no retargeters: raw device data in its original format
-            If retargeters are provided: tuple with output from each retargeter
+            Raw device data if no retargeters are configured.
+            When retargeters are configured, returns a tuple containing each retargeter's processed output.
         """
         raw_data = self._get_raw_data()
 

source/isaaclab/isaaclab/devices/openxr/openxr_device.py

@@ -3,6 +3,8 @@
 #
 # SPDX-License-Identifier: BSD-3-Clause
 
+"""OpenXR-powered device for teleoperation and interaction."""
+
 import contextlib
 import numpy as np
 from collections.abc import Callable
@@ -25,13 +27,6 @@ from isaacsim.core.prims import SingleXFormPrim
 
 
 class OpenXRDevice(DeviceBase):
-    class Hand(Enum):
-        LEFT = 0
-        RIGHT = 1
-        BOTH = 2
-
-    TELEOP_COMMAND_EVENT_TYPE = "teleop_command"
-
     """An OpenXR-powered device for teleoperation and interaction.
 
     This device tracks hand joints using OpenXR and makes them available as:
@@ -54,6 +49,21 @@ class OpenXRDevice(DeviceBase):
     poses are transformed into robot control commands suitable for teleoperation.
     """
 
+    class Hand(Enum):
+        """Enum class specifying which hand(s) to track with OpenXR.
+
+        Attributes:
+            LEFT: Track only the left hand
+            RIGHT: Track only the right hand
+            BOTH: Track both hands simultaneously
+        """
+
+        LEFT = 0
+        RIGHT = 1
+        BOTH = 2
+
+    TELEOP_COMMAND_EVENT_TYPE = "teleop_command"
+
     def __init__(
         self,
         xr_cfg: XrCfg | None,
@@ -63,8 +73,10 @@ class OpenXRDevice(DeviceBase):
         """Initialize the hand tracking device.
 
         Args:
-            hand: Which hand to track (left or right)
-            retargeters: List of retargeters to transform hand tracking data
+            xr_cfg: Configuration object for OpenXR settings. If None, default settings are used.
+            hand: Which hand(s) to track (LEFT, RIGHT, or BOTH)
+            retargeters: List of retargeters to transform hand tracking data into robot commands.
+                        If None or empty list, raw joint poses will be returned.
         """
         super().__init__(retargeters)
         self._openxr = OpenXR()
@@ -152,6 +164,13 @@ class OpenXRDevice(DeviceBase):
         self._previous_joint_poses_right = np.full((26, 7), [0, 0, 0, 1, 0, 0, 0], dtype=np.float32)
 
     def add_callback(self, key: str, func: Callable):
+        """Add additional functions to bind to client messages.
+
+        Args:
+            key: The message type to bind to. Valid values are "START", "STOP", and "RESET".
+            func: The function to call when the message is received. The callback function should not
+                take any arguments.
+        """
         self._additional_callbacks[key] = func
 
     def _get_raw_data(self) -> Any:

source/isaaclab/isaaclab/devices/openxr/retargeters/__init__.py

@@ -4,6 +4,7 @@
 # SPDX-License-Identifier: BSD-3-Clause
 """Retargeters for mapping input device data to robot commands."""
 
+from .humanoid.fourier.gr1t2_retargeter import GR1T2Retargeter
 from .manipulator.gripper_retargeter import GripperRetargeter
 from .manipulator.se3_abs_retargeter import Se3AbsRetargeter
 from .manipulator.se3_rel_retargeter import Se3RelRetargeter

source/isaaclab/isaaclab/devices/openxr/retargeters/humanoid/fourier/gr1t2_retargeter.py

@@ -3,6 +3,7 @@
 #
 # SPDX-License-Identifier: BSD-3-Clause
 
+import contextlib
 import numpy as np
 import torch
 
@@ -12,7 +13,9 @@ from isaaclab.devices import OpenXRDevice
 from isaaclab.devices.retargeter_base import RetargeterBase
 from isaaclab.markers import VisualizationMarkers, VisualizationMarkersCfg
 
-from .gr1_t2_dex_retargeting_utils import GR1TR2DexRetargeting
+# This import exception is suppressed because gr1_t2_dex_retargeting_utils depends on pinocchio which is not available on windows
+with contextlib.suppress(Exception):
+    from .gr1_t2_dex_retargeting_utils import GR1TR2DexRetargeting
 
 
 class GR1T2Retargeter(RetargeterBase):