Skip to content

K
KincoActuatorIsaacLab
Commit 5df26d99 authored by rwiltz's avatar rwiltz Committed by Kelly Guo
Browse files
Options

Adds documentation for teleop config changes. (#427) 

# Description

Updated documentation for the teleop config changes. Includes
documentation for:
- Teleop device config inside the env config
- How to add new devices
- How to add new retargeters
- Updates to script params for handtracking

<!--
Thank you for your interest in sending a pull request. Please make sure
to check the contribution guidelines.

Link:
https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html
-->

Please include a summary of the change and which issue is fixed. Please
also include relevant motivation and context.
List any dependencies that are required for this change.

Fixes # (issue)

<!-- As a practice, it is recommended to open an issue to have
discussions on the proposed pull request.
This makes it easier for the community to keep track of what is being
developed or added, and if a given feature
is demanded by more than one party. -->

## Type of change

<!-- As you go through the list, delete the ones that are not
applicable. -->

- This change requires a documentation update

## Screenshots

Please attach before and after screenshots of the change if applicable.

<!--
Example:

| Before | After |
| ------ | ----- |
| _gif/png before_ | _gif/png after_ |

To upload images to a PR -- simply drag and drop an image while in edit
mode and it should upload the image directly. You can then paste that
source into the above before/after sections.
-->

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->
parent 2b921b30
Show whitespace changes
Inline Side-by-side
Showing with 280 additions and 21 deletions
docs/source/how-to/cloudxr_teleoperation.rst
View file @ 5df26d99
......@@ -541,7 +541,7 @@ Here's an example of setting up hand tracking:
.. code-block:: python
from isaaclab.devices import OpenXRDevice
from isaaclab.devices import OpenXRDevice, OpenXRDeviceCfg
from isaaclab.devices.openxr.retargeters import Se3AbsRetargeter, GripperRetargeter
# Create retargeters
......@@ -554,7 +554,7 @@ Here's an example of setting up hand tracking:
# Create OpenXR device with hand tracking and both retargeters
device = OpenXRDevice(
env_cfg.xr,
OpenXRDeviceCfg(xr_cfg=env_cfg.xr),
retargeters=[position_retargeter, gripper_retargeter],
)
......@@ -571,21 +571,161 @@ Here's an example of setting up hand tracking:
if terminated or truncated:
break
.. _control-robot-with-xr-callbacks:
Adding Callbacks for XR UI Events
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The OpenXRDevice can handle events triggered by user interactions with XR UI elements like buttons and menus.
When a user interacts with these elements, the device triggers registered callback functions:
.. code-block:: python
# Register callbacks for teleop control events
device.add_callback("RESET", reset_callback)
device.add_callback("START", start_callback)
device.add_callback("STOP", stop_callback)
When the user interacts with the XR UI, these callbacks will be triggered to control the simulation
or recording process. You can also add custom messages from the client side using custom keys that will
trigger these callbacks, allowing for programmatic control of the simulation alongside direct user interaction.
The custom keys can be any string value that matches the callback registration.
Teleop Environment Configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
XR-based teleoperation can be integrated with Isaac Lab's environment configuration system using the
``teleop_devices`` field in your environment configuration:
.. code-block:: python
from dataclasses import field
from isaaclab.envs import ManagerBasedEnvCfg
from isaaclab.devices import DevicesCfg, OpenXRDeviceCfg
from isaaclab.devices.openxr import XrCfg
from isaaclab.devices.openxr.retargeters import Se3AbsRetargeterCfg, GripperRetargeterCfg
@configclass
class MyEnvironmentCfg(ManagerBasedEnvCfg):
"""Configuration for a teleoperation-enabled environment."""
# Add XR configuration with custom anchor position
xr: XrCfg = XrCfg(
anchor_pos=[0.0, 0.0, 0.0],
anchor_rot=[1.0, 0.0, 0.0, 0.0]
)
# Define teleoperation devices
teleop_devices: DevicesCfg = field(default_factory=lambda: DevicesCfg(
# Configuration for hand tracking with absolute position control
handtracking=OpenXRDeviceCfg(
xr_cfg=None, # Will use environment's xr config
retargeters=[
Se3AbsRetargeterCfg(
bound_hand=0, # HAND_LEFT enum value
zero_out_xy_rotation=True,
use_wrist_position=False,
),
GripperRetargeterCfg(bound_hand=0),
]
),
# Add other device configurations as needed
))
Teleop Device Factory
^^^^^^^^^^^^^^^^^^^^^
To create a teleoperation device from your environment configuration, use the ``create_teleop_device`` factory function:
.. code-block:: python
from isaaclab.devices import create_teleop_device
from isaaclab.envs import ManagerBasedEnv
# Create environment from configuration
env_cfg = MyEnvironmentCfg()
env = ManagerBasedEnv(env_cfg)
# Define callbacks for teleop events
callbacks = {
"RESET": lambda: print("Reset simulation"),
"START": lambda: print("Start teleoperation"),
"STOP": lambda: print("Stop teleoperation"),
}
# Create teleop device from configuration with callbacks
device_name = "handtracking" # Must match a key in teleop_devices
device = create_teleop_device(
device_name,
env_cfg.teleop_devices,
callbacks=callbacks
)
# Use device in control loop
while True:
# Get the latest commands from the device
commands = device.advance()
if commands is None:
continue
# Apply commands to environment
obs, reward, terminated, truncated, info = env.step(commands)
Extending the Retargeting System
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The retargeting system is designed to be extensible. You can create custom retargeters by extending
the :class:`isaaclab.devices.RetargeterBase` class and implementing the ``retarget`` method that
processes the incoming tracking data:
The retargeting system is designed to be extensible. You can create custom retargeters by following these steps:
1. Create a configuration dataclass for your retargeter:
.. code-block:: python
from dataclasses import dataclass
from isaaclab.devices.retargeter_base import RetargeterCfg
@dataclass
class MyCustomRetargeterCfg(RetargeterCfg):
"""Configuration for my custom retargeter."""
scaling_factor: float = 1.0
filter_strength: float = 0.5
# Add any other configuration parameters your retargeter needs
2. Implement your retargeter class by extending the RetargeterBase:
.. code-block:: python
from isaaclab.devices.retargeter_base import RetargeterBase
from isaaclab.devices import OpenXRDevice
import torch
from typing import Any
class MyCustomRetargeter(RetargeterBase):
def retarget(self, data: dict)-> Any:
"""A custom retargeter that processes OpenXR tracking data."""
def __init__(self, cfg: MyCustomRetargeterCfg):
"""Initialize retargeter with configuration.
Args:
cfg: Configuration object for retargeter settings.
"""
super().__init__()
self.scaling_factor = cfg.scaling_factor
self.filter_strength = cfg.filter_strength
# Initialize any other required attributes
def retarget(self, data: dict) -> Any:
"""Transform raw tracking data into robot control commands.
Args:
data: Dictionary containing tracking data from OpenXRDevice.
Keys are TrackingTarget enum values, values are joint pose dictionaries.
Returns:
Any: The transformed control commands for the robot.
"""
# Access hand tracking data using TrackingTarget enum
right_hand_data = data[OpenXRDevice.TrackingTarget.HAND_RIGHT]
......@@ -597,32 +737,151 @@ processes the incoming tracking data:
# Access head tracking data
head_pose = data[OpenXRDevice.TrackingTarget.HEAD]
# Process the tracking data
# Process the tracking data and apply your custom logic
# ...
# Return control commands in appropriate format
...
return torch.tensor([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]) # Example output
3. Register your retargeter with the factory by adding it to the ``RETARGETER_MAP``:
.. code-block:: python
# Import your retargeter at the top of your module
from my_package.retargeters import MyCustomRetargeter, MyCustomRetargeterCfg
# Add your retargeter to the factory
from isaaclab.devices.teleop_device_factory import RETARGETER_MAP
# Register your retargeter type with its constructor
RETARGETER_MAP[MyCustomRetargeterCfg] = MyCustomRetargeter
4. Now you can use your custom retargeter in teleop device configurations:
.. code-block:: python
from isaaclab.devices import OpenXRDeviceCfg, DevicesCfg
from isaaclab.devices.openxr import XrCfg
from my_package.retargeters import MyCustomRetargeterCfg
# Create XR configuration for proper scene placement
xr_config = XrCfg(anchor_pos=[0.0, 0.0, 0.0], anchor_rot=[1.0, 0.0, 0.0, 0.0])
# Define teleop devices with custom retargeter
teleop_devices = DevicesCfg(
handtracking=OpenXRDeviceCfg(
xr_cfg=xr_config,
retargeters=[
MyCustomRetargeterCfg(
scaling_factor=1.5,
filter_strength=0.7,
),
]
),
)
As the OpenXR capabilities expand beyond hand tracking to include head tracking and other features,
additional retargeters can be developed to map this data to various robot control paradigms.
.. _control-robot-with-xr-callbacks:
Adding Callbacks for XR UI Events
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Creating Custom Teleop Devices
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The OpenXRDevice can handle events triggered by user interactions with XR UI elements like buttons and menus.
When a user interacts with these elements, the device triggers registered callback functions:
You can create and register your own custom teleoperation devices by following these steps:
1. Create a configuration dataclass for your device:
.. code-block:: python
# Register callbacks for teleop control events
device.add_callback("RESET", reset_callback)
device.add_callback("START", start_callback)
device.add_callback("STOP", stop_callback)
from dataclasses import dataclass
from isaaclab.devices import DeviceCfg
When the user interacts with the XR UI, these callbacks will be triggered to control the simulation
or recording process. You can also add custom messages from the client side using custom keys that will
trigger these callbacks, allowing for programmatic control of the simulation alongside direct user interaction.
The custom keys can be any string value that matches the callback registration.
@dataclass
class MyCustomDeviceCfg(DeviceCfg):
"""Configuration for my custom device."""
sensitivity: float = 1.0
invert_controls: bool = False
# Add any other configuration parameters your device needs
2. Implement your device class by inheriting from DeviceBase:
.. code-block:: python
from isaaclab.devices import DeviceBase
import torch
class MyCustomDevice(DeviceBase):
"""A custom teleoperation device."""
def __init__(self, cfg: MyCustomDeviceCfg):
"""Initialize the device with configuration.
Args:
cfg: Configuration object for device settings.
"""
super().__init__()
self.sensitivity = cfg.sensitivity
self.invert_controls = cfg.invert_controls
# Initialize any other required attributes
self._device_input = torch.zeros(7) # Example: 6D pose + gripper
def reset(self):
"""Reset the device state."""
self._device_input.zero_()
# Reset any other state variables
def add_callback(self, key: str, func):
"""Add callback function for a button/event.
Args:
key: Button or event name.
func: Callback function to be called when event occurs.
"""
# Implement callback registration
pass
def advance(self) -> torch.Tensor:
"""Get the latest commands from the device.
Returns:
torch.Tensor: Control commands (e.g., delta pose + gripper).
"""
# Update internal state based on device input
# Return command tensor
return self._device_input
3. Register your device with the teleoperation device factory by adding it to the ``DEVICE_MAP``:
.. code-block:: python
# Import your device at the top of your module
from my_package.devices import MyCustomDevice, MyCustomDeviceCfg
# Add your device to the factory
from isaaclab.devices.teleop_device_factory import DEVICE_MAP
# Register your device type with its constructor
DEVICE_MAP[MyCustomDeviceCfg] = MyCustomDevice
4. Now you can use your custom device in environment configurations:
.. code-block:: python
from dataclasses import field
from isaaclab.envs import ManagerBasedEnvCfg
from isaaclab.devices import DevicesCfg
from my_package.devices import MyCustomDeviceCfg
@configclass
class MyEnvironmentCfg(ManagerBasedEnvCfg):
"""Environment configuration with custom teleop device."""
teleop_devices: DevicesCfg = field(default_factory=lambda: DevicesCfg(
my_custom_device=MyCustomDeviceCfg(
sensitivity=1.5,
invert_controls=True,
),
))
.. _xr-known-issues:
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment