Adds Haply device API with force feedback and teleoperation demo (#3873)

# Description
Add Haply haptic device teleoperation support for robotic manipulation
with force feedback.

## Type of change

- New feature (non-breaking change which adds functionality)

1.  Add haply devices API for teleoperation with force feedback
2. Real-time teleoperation with force feedback demo via Haply Inverse3
handle

[Haply device documentation](https://docs.haply.co/docs/quick-start) and
[usage](https://docs.haply.co/inverseSDK/)

**Usage (make sure your Haply device is connected):**
```bash
./isaaclab.sh -p scripts/demos/haply_teleoperation.py --websocket_uri ws://localhost:10001 --pos_sensitivity 1.65
```

## Screenshots

![haply-frankal](https://github.com/user-attachments/assets/f2c0572e-83d6-4c10-b95d-6ea083b90f62)

## Checklist

- [x] I have read and understood the [contribution
guidelines](https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html)
- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

---------
Signed-off-by: mingxueg-nv <mingxueg@nvidia.com>
Signed-off-by: mingxueg <mingxueg@nvidia.com>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Kelly Guo <kellyg@nvidia.com>

CONTRIBUTORS.md

@@ -100,6 +100,7 @@ Guidelines for modifications:
 * Michael Noseworthy
 * Michael Lin
 * Miguel Alonso Jr
+* Mingxue Gu
 * Mingyu Lee
 * Muhong Guo
 * Narendra Dahile

docs/source/api/lab/isaaclab.devices.rst

@@ -15,6 +15,7 @@
     Se3Keyboard
     Se2SpaceMouse
     Se3SpaceMouse
+    HaplyDevice
     OpenXRDevice
     ManusVive
     isaaclab.devices.openxr.retargeters.GripperRetargeter
@@ -79,6 +80,14 @@ Space Mouse
     :inherited-members:
     :show-inheritance:
 
+Haply
+-----
+
+.. autoclass:: HaplyDevice
+    :members:
+    :inherited-members:
+    :show-inheritance:
+
 OpenXR
 ------
 

docs/source/how-to/haply_teleoperation.rst

+.. _haply-teleoperation:
+
+Setting up Haply Teleoperation
+===============================
+
+.. currentmodule:: isaaclab
+
+`Haply Devices`_ provides haptic devices that enable intuitive robot teleoperation with
+directional force feedback. The Haply Inverse3 paired with the VerseGrip creates an
+end-effector control system with force feedback capabilities.
+
+Isaac Lab supports Haply devices for teleoperation workflows that require precise spatial
+control with haptic feedback. This enables operators to feel contact forces during manipulation
+tasks, improving control quality and task performance.
+
+This guide explains how to set up and use Haply devices with Isaac Lab for robot teleoperation.
+
+.. _Haply Devices: https://haply.co/
+
+
+Overview
+--------
+
+Using Haply with Isaac Lab involves the following components:
+
+* **Isaac Lab** simulates the robot environment and streams contact forces back to the operator
+
+* **Haply Inverse3** provides 3-DOF position tracking and force feedback in the operator's workspace
+
+* **Haply VerseGrip** adds orientation sensing and button inputs for gripper control
+
+* **Haply SDK** manages WebSocket communication between Isaac Lab and the Haply hardware
+
+This guide will walk you through:
+
+* :ref:`haply-system-requirements`
+* :ref:`haply-installation`
+* :ref:`haply-device-setup`
+* :ref:`haply-running-demo`
+* :ref:`haply-troubleshooting`
+
+
+.. _haply-system-requirements:
+
+System Requirements
+-------------------
+
+Hardware Requirements
+~~~~~~~~~~~~~~~~~~~~~
+
+* **Isaac Lab Workstation**
+
+  * Ubuntu 22.04 or Ubuntu 24.04
+  * Hardware requirements for 200Hz physics simulation:
+
+    * CPU: 8-Core Intel Core i7 or AMD Ryzen 7 (or higher)
+    * Memory: 32GB RAM (64GB recommended)
+    * GPU: RTX 3090 or higher
+
+  * Network: Same local network as Haply devices for WebSocket communication
+
+* **Haply Devices**
+
+  * Haply Inverse3 - Haptic device for position tracking and force feedback
+  * Haply VerseGrip - Wireless controller for orientation and button inputs
+  * Both devices must be powered on and connected to the Haply SDK
+
+Software Requirements
+~~~~~~~~~~~~~~~~~~~~~
+
+* Isaac Lab (follow the :ref:`installation guide <isaaclab-installation-root>`)
+* Haply SDK (provided by Haply Robotics)
+* Python 3.10+
+* ``websockets`` Python package (automatically installed with Isaac Lab)
+
+
+.. _haply-installation:
+
+Installation
+------------
+
+1. Install Isaac Lab
+~~~~~~~~~~~~~~~~~~~~
+
+Follow the Isaac Lab :ref:`installation guide <isaaclab-installation-root>` to set up your environment.
+
+The ``websockets`` dependency is automatically included in Isaac Lab's requirements.
+
+2. Install Haply SDK
+~~~~~~~~~~~~~~~~~~~~
+
+Download the Haply SDK from the `Haply Devices`_ website.
+Install the SDK software and configure the devices.
+
+3. Verify Installation
+~~~~~~~~~~~~~~~~~~~~~~
+
+Test that your Haply devices are detected by the Haply Device Manager.
+You should see both Inverse3 and VerseGrip as connected.
+
+
+.. _haply-device-setup:
+
+Device Setup
+------------
+
+1. Physical Setup
+~~~~~~~~~~~~~~~~~
+
+* Place the Haply Inverse3 on a stable surface
+* Ensure the VerseGrip is charged and paired
+* Position yourself comfortably to reach the Inverse3 workspace
+* Keep the workspace clear of obstacles
+
+2. Start Haply SDK
+~~~~~~~~~~~~~~~~~~
+
+Launch the Haply SDK according to Haply's documentation. The SDK typically:
+
+* Runs a WebSocket server on ``localhost:10001``
+* Streams device data at 200Hz
+* Displays connection status for both devices
+
+3. Test Communication
+~~~~~~~~~~~~~~~~~~~~~
+
+You can test the WebSocket connection using the following Python script:
+
+.. code:: python
+
+   import asyncio
+   import websockets
+   import json
+
+   async def test_haply():
+       uri = "ws://localhost:10001"
+       async with websockets.connect(uri) as ws:
+           response = await ws.recv()
+           data = json.loads(response)
+           print("Inverse3:", data.get("inverse3", []))
+           print("VerseGrip:", data.get("wireless_verse_grip", []))
+
+   asyncio.run(test_haply())
+
+You should see device data streaming from both Inverse3 and VerseGrip.
+
+
+.. _haply-running-demo:
+
+Running the Demo
+----------------
+
+The Haply teleoperation demo showcases robot manipulation with force feedback using
+a Franka Panda arm.
+
+Basic Usage
+~~~~~~~~~~~
+
+.. tab-set::
+   :sync-group: os
+
+   .. tab-item:: :icon:`fa-brands fa-linux` Linux
+      :sync: linux
+
+      .. code:: bash
+
+         # Ensure Haply SDK is running
+         ./isaaclab.sh -p scripts/demos/haply_teleoperation.py --websocket_uri ws://localhost:10001 --pos_sensitivity 1.65
+
+   .. tab-item:: :icon:`fa-brands fa-windows` Windows
+      :sync: windows
+
+      .. code:: batch
+
+         REM Ensure Haply SDK is running
+         isaaclab.bat -p scripts\demos\haply_teleoperation.py --websocket_uri ws://localhost:10001 --pos_sensitivity 1.65
+
+The demo will:
+
+1. Connect to the Haply devices via WebSocket
+2. Spawn a Franka Panda robot and a cube in simulation
+3. Map Haply position to robot end-effector position
+4. Stream contact forces back to the Inverse3 for haptic feedback
+
+Controls
+~~~~~~~~
+
+* **Move Inverse3**: Controls the robot end-effector position
+* **VerseGrip Button A**: Open gripper
+* **VerseGrip Button B**: Close gripper
+* **VerseGrip Button C**: Rotate end-effector by 60°
+
+Advanced Options
+~~~~~~~~~~~~~~~~
+
+Customize the demo with command-line arguments:
+
+.. code:: bash
+
+   # Use custom WebSocket URI
+   ./isaaclab.sh -p scripts/demos/haply_teleoperation.py \
+       --websocket_uri ws://192.168.1.100:10001
+
+   # Adjust position sensitivity (default: 1.0)
+   ./isaaclab.sh -p scripts/demos/haply_teleoperation.py \
+        --websocket_uri ws://localhost:10001 \
+        --pos_sensitivity 2.0
+
+Demo Features
+~~~~~~~~~~~~~
+
+* **Workspace Mapping**: Haply workspace is mapped to robot reachable space with safety limits
+* **Inverse Kinematics**: Inverse Kinematics (IK) computes joint positions for desired end-effector pose
+* **Force Feedback**: Contact forces from end-effector sensors are sent to Inverse3 for haptic feedback
+
+
+.. _haply-troubleshooting:
+
+Troubleshooting
+---------------
+
+No Haptic Feedback
+~~~~~~~~~~~~~~~~~~
+
+**Problem**: No haptic feedback felt on Inverse3
+
+Solutions:
+
+* Verify Inverse3 is the active device in Haply SDK
+* Check contact forces are non-zero in simulation (try grasping the cube)
+* Ensure ``limit_force`` is not set too low (default: 2.0N)
+
+
+Next Steps
+----------
+
+* **Customize the demo**: Modify the workspace mapping or add custom button behaviors
+* **Implement your own controller**: Use :class:`~isaaclab.devices.HaplyDevice` in your own scripts
+
+For more information on device APIs, see :class:`~isaaclab.devices.HaplyDevice` in the API documentation.

docs/source/how-to/index.rst

@@ -149,6 +149,18 @@ teleoperation in Isaac Lab.
     cloudxr_teleoperation
 
 
+Setting up Haply Teleoperation
+------------------------------
+
+This guide explains how to use Haply Inverse3 and VerseGrip devices for robot teleoperation
+with directional force feedback in Isaac Lab.
+
+.. toctree::
+    :maxdepth: 1
+
+    haply_teleoperation
+
+
 Understanding Simulation Performance
 ------------------------------------
 

docs/source/overview/showroom.rst

@@ -228,6 +228,41 @@ A few quick showroom scripts to run and checkout:
 
 
 
+-  Teleoperate a Franka Panda robot using Haply haptic device with force feedback:
+
+   .. tab-set::
+      :sync-group: os
+
+      .. tab-item:: :icon:`fa-brands fa-linux` Linux
+         :sync: linux
+
+         .. code:: bash
+
+            ./isaaclab.sh -p scripts/demos/haply_teleoperation.py --websocket_uri ws://localhost:10001 --pos_sensitivity 1.65
+
+      .. tab-item:: :icon:`fa-brands fa-windows` Windows
+         :sync: windows
+
+         .. code:: batch
+
+            isaaclab.bat -p scripts\demos\haply_teleoperation.py --websocket_uri ws://localhost:10001 --pos_sensitivity 1.65
+
+   .. image:: ../_static/demos/haply_teleop_franka.jpg
+      :width: 100%
+      :alt: Haply teleoperation with force feedback
+
+   This demo requires Haply Inverse3 and VerseGrip devices.
+   The goal of this demo is to pick up the cube or touch it with the end-effector.
+   The Haply devices provide:
+
+   * 3 dimensional position tracking for end-effector control
+   * Directional force feedback for contact sensing
+   * Button inputs for gripper and end-effector rotation control
+
+   See :ref:`haply-teleoperation` for detailed setup instructions.
+
+
+
 -  Create and spawn procedurally generated terrains with different configurations:
 
    .. tab-set::

source/isaaclab/config/extension.toml

@@ -18,7 +18,8 @@ requirements = [
     "toml",
     "hidapi",
     "gymnasium==0.29.0",
-    "trimesh"
+    "trimesh",
+    "websockets"
 ]
 
 modules = [
@@ -27,7 +28,8 @@ modules = [
     "toml",
     "hid",
     "gymnasium",
-    "trimesh"
+    "trimesh",
+    "websockets"
 ]
 
 use_online_index=true

source/isaaclab/docs/CHANGELOG.rst

 Changelog
 ---------
 
+0.48.1 (2025-11-10)
+~~~~~~~~~~~~~~~~~~~
+
+Added
+^^^^^
+
+* Added :class:`~isaaclab.devices.haply.HaplyDevice` class for SE(3) teleoperation with dual Haply Inverse3 and Versegrip devices,
+  supporting robot manipulation with haptic feedback.
+* Added demo script ``scripts/demos/haply_teleoperation.py`` and documentation guide in
+  ``docs/source/how-to/haply_teleoperation.rst`` for Haply-based robot teleoperation.
+
+
 0.48.0 (2025-11-03)
 ~~~~~~~~~~~~~~~~~~~
 

source/isaaclab/isaaclab/devices/__init__.py

@@ -11,6 +11,7 @@ Currently, the following categories of devices are supported:
 * **Spacemouse**: 3D mouse with 6 degrees of freedom.
 * **Gamepad**: Gamepad with 2D two joysticks and buttons. Example: Xbox controller.
 * **OpenXR**: Uses hand tracking of index/thumb tip avg to drive the target pose. Gripping is done with pinching.
+* **Haply**: Haptic device (Inverse3 + VerseGrip) with position, orientation tracking and force feedback.
 
 All device interfaces inherit from the :class:`DeviceBase` class, which provides a
 common interface for all devices. The device interface reads the input data when
@@ -21,6 +22,7 @@ the peripheral device.
 
 from .device_base import DeviceBase, DeviceCfg, DevicesCfg
 from .gamepad import Se2Gamepad, Se2GamepadCfg, Se3Gamepad, Se3GamepadCfg
+from .haply import HaplyDevice, HaplyDeviceCfg
 from .keyboard import Se2Keyboard, Se2KeyboardCfg, Se3Keyboard, Se3KeyboardCfg
 from .openxr import ManusVive, ManusViveCfg, OpenXRDevice, OpenXRDeviceCfg
 from .retargeter_base import RetargeterBase, RetargeterCfg

source/isaaclab/isaaclab/devices/haply/__init__.py

+# Copyright (c) 2022-2025, The Isaac Lab Project Developers (https://github.com/isaac-sim/IsaacLab/blob/main/CONTRIBUTORS.md).
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+"""Haply device interface for teleoperation."""
+
+from .se3_haply import HaplyDevice, HaplyDeviceCfg
+
+__all__ = ["HaplyDevice", "HaplyDeviceCfg"]

source/isaaclab/isaaclab/devices/teleop_device_factory.py

@@ -12,6 +12,7 @@ from collections.abc import Callable
 
 from isaaclab.devices import DeviceBase, DeviceCfg
 from isaaclab.devices.gamepad import Se2Gamepad, Se2GamepadCfg, Se3Gamepad, Se3GamepadCfg
+from isaaclab.devices.haply import HaplyDevice, HaplyDeviceCfg
 from isaaclab.devices.keyboard import Se2Keyboard, Se2KeyboardCfg, Se3Keyboard, Se3KeyboardCfg
 from isaaclab.devices.openxr.retargeters import (
     G1LowerBodyStandingRetargeter,
@@ -47,6 +48,7 @@ DEVICE_MAP: dict[type[DeviceCfg], type[DeviceBase]] = {
     Se2KeyboardCfg: Se2Keyboard,
     Se2GamepadCfg: Se2Gamepad,
     Se2SpaceMouseCfg: Se2SpaceMouse,
+    HaplyDeviceCfg: HaplyDevice,
     OpenXRDeviceCfg: OpenXRDevice,
     ManusViveCfg: ManusVive,
 }

source/isaaclab/test/devices/test_device_constructors.py

@@ -13,12 +13,15 @@ simulation_app = AppLauncher(headless=True).app
 """Rest everything follows."""
 
 import importlib
+import json
 import torch
 
 import pytest
 
 # Import device classes to test
 from isaaclab.devices import (
+    HaplyDevice,
+    HaplyDeviceCfg,
     OpenXRDevice,
     OpenXRDeviceCfg,
     Se2Gamepad,
@@ -79,6 +82,11 @@ def mock_environment(mocker):
     omni_mock.kit.xr.core.XRPoseValidityFlags.POSITION_VALID = 1
     omni_mock.kit.xr.core.XRPoseValidityFlags.ORIENTATION_VALID = 2
 
+    # Mock Haply WebSocket
+    websockets_mock = mocker.MagicMock()
+    websocket_mock = mocker.MagicMock()
+    websockets_mock.connect.return_value.__aenter__.return_value = websocket_mock
+
     return {
         "carb": carb_mock,
         "omni": omni_mock,
@@ -89,6 +97,8 @@ def mock_environment(mocker):
         "settings": settings_mock,
         "hid": hid_mock,
         "device": device_mock,
+        "websockets": websockets_mock,
+        "websocket": websocket_mock,
     }
 
 
@@ -321,6 +331,141 @@ def test_openxr_constructors(mock_environment, mocker):
     device.reset()
 
 
+"""
+Test Haply devices.
+"""
+
+
+def test_haply_constructors(mock_environment, mocker):
+    """Test constructor for HaplyDevice."""
+    # Test config-based constructor
+    config = HaplyDeviceCfg(
+        websocket_uri="ws://localhost:10001",
+        pos_sensitivity=1.5,
+        data_rate=250.0,
+    )
+
+    # Mock the websockets module and asyncio
+    device_mod = importlib.import_module("isaaclab.devices.haply.se3_haply")
+    mocker.patch.dict("sys.modules", {"websockets": mock_environment["websockets"]})
+    mocker.patch.object(device_mod, "websockets", mock_environment["websockets"])
+
+    # Mock asyncio to prevent actual async operations
+    asyncio_mock = mocker.MagicMock()
+    mocker.patch.object(device_mod, "asyncio", asyncio_mock)
+
+    # Mock threading to prevent actual thread creation
+    threading_mock = mocker.MagicMock()
+    thread_instance = mocker.MagicMock()
+    threading_mock.Thread.return_value = thread_instance
+    thread_instance.is_alive.return_value = False
+    mocker.patch.object(device_mod, "threading", threading_mock)
+
+    # Mock time.time() for connection timeout simulation
+    time_mock = mocker.MagicMock()
+    time_mock.time.side_effect = [0.0, 0.1, 0.2, 0.3, 6.0]  # Will timeout
+    mocker.patch.object(device_mod, "time", time_mock)
+
+    # Create sample WebSocket response data
+    ws_response = {
+        "inverse3": [{
+            "device_id": "test_inverse3_123",
+            "state": {"cursor_position": {"x": 0.1, "y": 0.2, "z": 0.3}},
+        }],
+        "wireless_verse_grip": [{
+            "device_id": "test_versegrip_456",
+            "state": {
+                "orientation": {"x": 0.0, "y": 0.0, "z": 0.0, "w": 1.0},
+                "buttons": {"a": False, "b": False, "c": False},
+            },
+        }],
+    }
+
+    # Configure websocket mock to return JSON data
+    mock_environment["websocket"].recv = mocker.AsyncMock(return_value=json.dumps(ws_response))
+    mock_environment["websocket"].send = mocker.AsyncMock()
+
+    # The constructor will raise RuntimeError due to timeout, which is expected in test
+    with pytest.raises(RuntimeError, match="Failed to connect both Inverse3 and VerseGrip devices"):
+        haply = HaplyDevice(config)
+
+    # Now test successful connection by mocking time to not timeout
+    time_mock.time.side_effect = [0.0, 0.1, 0.2, 0.3, 0.4]  # Won't timeout
+
+    # Mock the connection status
+    mocker.patch.object(device_mod.HaplyDevice, "_start_websocket_thread")
+    haply = device_mod.HaplyDevice.__new__(device_mod.HaplyDevice)
+    haply._sim_device = config.sim_device
+    haply.websocket_uri = config.websocket_uri
+    haply.pos_sensitivity = config.pos_sensitivity
+    haply.data_rate = config.data_rate
+    haply.limit_force = config.limit_force
+    haply.connected = True
+    haply.inverse3_device_id = "test_inverse3_123"
+    haply.verse_grip_device_id = "test_versegrip_456"
+    haply.data_lock = threading_mock.Lock()
+    haply.force_lock = threading_mock.Lock()
+    haply._connected_lock = threading_mock.Lock()
+    haply._additional_callbacks = {}
+    haply._prev_buttons = {"a": False, "b": False, "c": False}
+    haply._websocket_thread = None  # Initialize to prevent AttributeError in __del__
+    haply.running = True
+    haply.cached_data = {
+        "position": torch.tensor([0.1, 0.2, 0.3], dtype=torch.float32).numpy(),
+        "quaternion": torch.tensor([0.0, 0.0, 0.0, 1.0], dtype=torch.float32).numpy(),
+        "buttons": {"a": False, "b": False, "c": False},
+        "inverse3_connected": True,
+        "versegrip_connected": True,
+    }
+    haply.feedback_force = {"x": 0.0, "y": 0.0, "z": 0.0}
+
+    # Verify configuration was applied correctly
+    assert haply.websocket_uri == "ws://localhost:10001"
+    assert haply.pos_sensitivity == 1.5
+    assert haply.data_rate == 250.0
+
+    # Test advance() returns expected type
+    result = haply.advance()
+    assert isinstance(result, torch.Tensor)
+    assert result.shape == (10,)  # (pos_x, pos_y, pos_z, qx, qy, qz, qw, btn_a, btn_b, btn_c)
+
+    # Test push_force with tensor (single force vector)
+    forces_within = torch.tensor([[1.0, 1.5, -0.5]], dtype=torch.float32)
+    position_zero = torch.tensor([0], dtype=torch.long)
+    haply.push_force(forces_within, position_zero)
+    assert haply.feedback_force["x"] == pytest.approx(1.0)
+    assert haply.feedback_force["y"] == pytest.approx(1.5)
+    assert haply.feedback_force["z"] == pytest.approx(-0.5)
+
+    # Test push_force with tensor (force limiting, default limit is 2.0 N)
+    forces_exceed = torch.tensor([[5.0, -10.0, 1.5]], dtype=torch.float32)
+    haply.push_force(forces_exceed, position_zero)
+    assert haply.feedback_force["x"] == pytest.approx(2.0)
+    assert haply.feedback_force["y"] == pytest.approx(-2.0)
+    assert haply.feedback_force["z"] == pytest.approx(1.5)
+
+    # Test push_force with position tensor (single index)
+    forces_multi = torch.tensor([[1.0, 2.0, 3.0], [0.5, 0.8, -0.3], [0.1, 0.2, 0.3]], dtype=torch.float32)
+    position_single = torch.tensor([1], dtype=torch.long)
+    haply.push_force(forces_multi, position=position_single)
+    assert haply.feedback_force["x"] == pytest.approx(0.5)
+    assert haply.feedback_force["y"] == pytest.approx(0.8)
+    assert haply.feedback_force["z"] == pytest.approx(-0.3)
+
+    # Test push_force with position tensor (multiple indices)
+    position_multi = torch.tensor([0, 2], dtype=torch.long)
+    haply.push_force(forces_multi, position=position_multi)
+    # Should sum forces[0] and forces[2]: [1.0+0.1, 2.0+0.2, 3.0+0.3] = [1.1, 2.2, 3.3]
+    # But clipped to [-2.0, 2.0]: [1.1, 2.0, 2.0]
+    assert haply.feedback_force["x"] == pytest.approx(1.1)
+    assert haply.feedback_force["y"] == pytest.approx(2.0)
+    assert haply.feedback_force["z"] == pytest.approx(2.0)
+
+    # Test reset functionality
+    haply.reset()
+    assert haply.feedback_force == {"x": 0.0, "y": 0.0, "z": 0.0}
+
+
 """
 Test teleop device factory.
 """