Adds live plots to managers (#893)

# Description

This adds a UI interface to the Managers in the `ManagerBasedEnv` and
The `MangerBasedRLEnv`. Additions include:
- UI widgets for `LiveLinePlot` and `ImagePlot`
- `ManagerLiveVisualizer/Cfg`: Given a `ManagerBase` (i.e.
action_manager, observation_manager, etc) and a config file this class
creates the the interface between managers and the UI.
- `EnvLiveVisualizer`: A 'manager' of `ManagerLiveVisualizer`. This is
added to the `ManagerBasedEnv` but is only called during the
initialization of the managers in `load_managers`
- Adds `get_active_iterable_terms` implementation methods to
ActionManager, ObservationManager, CommandsManager, CurriculumManager,
RewardManager, and TerminationManager. This method exports the active
term data and labels for each manager and is called by
ManagerLiveVisualizer.
- Additions to `BaseEnvWindow` and `RLEnvWindow` to register
`ManagerLiveVisualizer` UI interfaces for the chosen managers.

## Screenshots
[Screencast from 09-06-2024 01:20:18
PM.webm](https://github.com/user-attachments/assets/3ef0191d-5446-41bd-b274-43d886fb2d70)

## Implementation

![image](https://github.com/user-attachments/assets/49bd5493-3311-4c5c-a87c-6bbcd76a60fe)

## Type of change

- New feature (non-breaking change which adds functionality)

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

---------
Signed-off-by: jtigue-bdai <166445701+jtigue-bdai@users.noreply.github.com>
Signed-off-by: Pascal Roth <57946385+pascal-roth@users.noreply.github.com>
Signed-off-by: David Hoeller <dhoeller@nvidia.com>
Co-authored-by: zrene <zrene@ethz.ch>
Co-authored-by: James Tigue <jtigue@theaiinstitute.com>
Co-authored-by: jtigue-bdai <166445701+jtigue-bdai@users.noreply.github.com>
Co-authored-by: David Hoeller <dhoeller@nvidia.com>
Co-authored-by: Aravind EV <aravindev@live.in>
Co-authored-by: Kelly Guo <kellyg@nvidia.com>
Co-authored-by: Kelly Guo <kellyguo123@hotmail.com>

.pre-commit-config.yaml

@@ -38,7 +38,7 @@ repos:
       - id: pyupgrade
         args: ["--py310-plus"]
         # FIXME: This is a hack because Pytorch does not like: torch.Tensor | dict aliasing
-        exclude: "source/extensions/omni.isaac.lab/omni/isaac/lab/envs/common.py"
+        exclude: "source/extensions/omni.isaac.lab/omni/isaac/lab/envs/common.py|source/extensions/omni.isaac.lab/omni/isaac/lab/ui/widgets/image_plot.py"
   - repo: https://github.com/codespell-project/codespell
     rev: v2.2.6
     hooks:

source/extensions/omni.isaac.lab/config/extension.toml

 [package]
 
 # Note: Semantic Versioning is used: https://semver.org/
-version = "0.28.0"
+version = "0.29.0"
 
 # Description
 title = "Isaac Lab framework for Robot Learning"

source/extensions/omni.isaac.lab/docs/CHANGELOG.rst

 Changelog
 ---------
 
+0.29.0 (2024-12-15)
+~~~~~~~~~~~~~~~~~~~
+
+Added
+^^^^^
+
+* Added UI interface to the Managers in the ManagerBasedEnv and MangerBasedRLEnv classes.
+* Added UI widgets for :class:`LiveLinePlot` and :class:`ImagePlot`.
+* Added ``ManagerLiveVisualizer/Cfg``: Given a ManagerBase (i.e. action_manager, observation_manager, etc) and a config file this class creates the the interface between managers and the UI.
+* Added :class:`EnvLiveVisualizer`: A 'manager' of ManagerLiveVisualizer. This is added to the ManagerBasedEnv but is only called during the initialization of the managers in load_managers
+* Added ``get_active_iterable_terms`` implementation methods to ActionManager, ObservationManager, CommandsManager, CurriculumManager, RewardManager, and TerminationManager. This method exports the active term data and labels for each manager and is called by ManagerLiveVisualizer.
+* Additions to :class:`BaseEnvWindow` and :class:`RLEnvWindow` to register ManagerLiveVisualizer UI interfaces for the chosen managers.
+
+
 0.28.0 (2024-12-15)
 ~~~~~~~~~~~~~~~~~~~
 
@@ -93,7 +107,7 @@ Changed
 
 
 0.27.21 (2024-12-06)
-~~~~~~~~~~~~~~~~~~~
+~~~~~~~~~~~~~~~~~~~~
 
 Fixed
 ^^^^^

source/extensions/omni.isaac.lab/omni/isaac/lab/envs/manager_based_env.py

@@ -14,6 +14,7 @@ import omni.log
 from omni.isaac.lab.managers import ActionManager, EventManager, ObservationManager, RecorderManager
 from omni.isaac.lab.scene import InteractiveScene
 from omni.isaac.lab.sim import SimulationContext
+from omni.isaac.lab.ui.widgets import ManagerLiveVisualizer
 from omni.isaac.lab.utils.timer import Timer
 
 from .common import VecEnvObs
@@ -148,6 +149,8 @@ class ManagerBasedEnv:
         # we need to do this here after all the managers are initialized
         # this is because they dictate the sensors and commands right now
         if self.sim.has_gui() and self.cfg.ui_window_class_type is not None:
+            # setup live visualizers
+            self.setup_manager_visualizers()
             self._window = self.cfg.ui_window_class_type(self, window_name="IsaacLab")
         else:
             # if no window, then we don't need to store the window
@@ -233,6 +236,14 @@ class ManagerBasedEnv:
         if self.__class__ == ManagerBasedEnv and "startup" in self.event_manager.available_modes:
             self.event_manager.apply(mode="startup")
 
+    def setup_manager_visualizers(self):
+        """Creates live visualizers for manager terms."""
+
+        self.manager_visualizers = {
+            "action_manager": ManagerLiveVisualizer(manager=self.action_manager),
+            "observation_manager": ManagerLiveVisualizer(manager=self.observation_manager),
+        }
+
     """
     Operations - MDP.
     """

source/extensions/omni.isaac.lab/omni/isaac/lab/envs/manager_based_rl_env.py

@@ -16,6 +16,7 @@ from typing import Any, ClassVar
 from omni.isaac.version import get_version
 
 from omni.isaac.lab.managers import CommandManager, CurriculumManager, RewardManager, TerminationManager
+from omni.isaac.lab.ui.widgets import ManagerLiveVisualizer
 
 from .common import VecEnvStepReturn
 from .manager_based_env import ManagerBasedEnv
@@ -132,6 +133,18 @@ class ManagerBasedRLEnv(ManagerBasedEnv, gym.Env):
         if "startup" in self.event_manager.available_modes:
             self.event_manager.apply(mode="startup")
 
+    def setup_manager_visualizers(self):
+        """Creates live visualizers for manager terms."""
+
+        self.manager_visualizers = {
+            "action_manager": ManagerLiveVisualizer(manager=self.action_manager),
+            "observation_manager": ManagerLiveVisualizer(manager=self.observation_manager),
+            "command_manager": ManagerLiveVisualizer(manager=self.command_manager),
+            "termination_manager": ManagerLiveVisualizer(manager=self.termination_manager),
+            "reward_manager": ManagerLiveVisualizer(manager=self.reward_manager),
+            "curriculum_manager": ManagerLiveVisualizer(manager=self.curriculum_manager),
+        }
+
     """
     Operations - MDP
     """

source/extensions/omni.isaac.lab/omni/isaac/lab/envs/ui/base_env_window.py

@@ -16,6 +16,8 @@ import omni.kit.commands
 import omni.usd
 from pxr import PhysxSchema, Sdf, Usd, UsdGeom, UsdPhysics
 
+from omni.isaac.lab.ui.widgets import ManagerLiveVisualizer
+
 if TYPE_CHECKING:
     import omni.ui
 
@@ -57,6 +59,9 @@ class BaseEnvWindow:
             *self.env.scene.articulations.keys(),
         ]
 
+        # Listeners for environment selection changes
+        self._ui_listeners: list[ManagerLiveVisualizer] = []
+
         print("Creating window for environment.")
         # create window for UI
         self.ui_window = omni.ui.Window(
@@ -80,6 +85,10 @@ class BaseEnvWindow:
                 self._build_viewer_frame()
                 # create collapsable frame for debug visualization
                 self._build_debug_vis_frame()
+                with self.ui_window_elements["debug_frame"]:
+                    with self.ui_window_elements["debug_vstack"]:
+                        self._visualize_manager(title="Actions", class_name="action_manager")
+                        self._visualize_manager(title="Observations", class_name="observation_manager")
 
     def __del__(self):
         """Destructor for the window."""
@@ -200,9 +209,6 @@ class BaseEnvWindow:
         that has it implemented. If the element does not have a debug visualization implemented,
         a label is created instead.
         """
-        # import omni.isaac.ui.ui_utils as ui_utils
-        # import omni.ui
-
         # create collapsable frame for debug visualization
         self.ui_window_elements["debug_frame"] = omni.ui.CollapsableFrame(
             title="Scene Debug Visualization",
@@ -234,6 +240,26 @@ class BaseEnvWindow:
                     if elem is not None:
                         self._create_debug_vis_ui_element(name, elem)
 
+    def _visualize_manager(self, title: str, class_name: str) -> None:
+        """Checks if the attribute with the name 'class_name' can be visualized. If yes, create vis interface.
+
+        Args:
+            title: The title of the manager visualization frame.
+            class_name: The name of the manager to visualize.
+        """
+
+        if hasattr(self.env, class_name) and class_name in self.env.manager_visualizers:
+            manager = self.env.manager_visualizers[class_name]
+            if hasattr(manager, "has_debug_vis_implementation"):
+                self._create_debug_vis_ui_element(title, manager)
+            else:
+                print(
+                    f"ManagerLiveVisualizer cannot be created for manager: {class_name}, has_debug_vis_implementation"
+                    " does not exist"
+                )
+        else:
+            print(f"ManagerLiveVisualizer cannot be created for manager: {class_name}, Manager does not exist")
+
     """
     Custom callbacks for UI elements.
     """
@@ -357,6 +383,9 @@ class BaseEnvWindow:
             raise ValueError("Viewport camera controller is not initialized! Please check the rendering mode.")
         # store the desired env index, UI is 1-indexed
         vcc.set_view_env_index(model.as_int - 1)
+        # notify additional listeners
+        for listener in self._ui_listeners:
+            listener.set_env_selection(model.as_int - 1)
 
     """
     Helper functions - UI building.
@@ -379,14 +408,30 @@ class BaseEnvWindow:
                 alignment=omni.ui.Alignment.LEFT_CENTER,
                 tooltip=text,
             )
+            has_cfg = hasattr(elem, "cfg") and elem.cfg is not None
+            is_checked = False
+            if has_cfg:
+                is_checked = (hasattr(elem.cfg, "debug_vis") and elem.cfg.debug_vis) or (
+                    hasattr(elem, "debug_vis") and elem.debug_vis
+                )
             self.ui_window_elements[f"{name}_cb"] = SimpleCheckBox(
                 model=omni.ui.SimpleBoolModel(),
                 enabled=elem.has_debug_vis_implementation,
-                checked=elem.cfg.debug_vis if elem.cfg else False,
+                checked=is_checked,
                 on_checked_fn=lambda value, e=weakref.proxy(elem): e.set_debug_vis(value),
             )
             omni.isaac.ui.ui_utils.add_line_rect_flourish()
 
+        # Create a panel for the debug visualization
+        if isinstance(elem, ManagerLiveVisualizer):
+            self.ui_window_elements[f"{name}_panel"] = omni.ui.Frame(width=omni.ui.Fraction(1))
+            if not elem.set_vis_frame(self.ui_window_elements[f"{name}_panel"]):
+                print(f"Frame failed to set for ManagerLiveVisualizer: {name}")
+
+        # Add listener for environment selection changes
+        if isinstance(elem, ManagerLiveVisualizer):
+            self._ui_listeners.append(elem)
+
     async def _dock_window(self, window_title: str):
         """Docks the custom UI window to the property window."""
         # wait for the window to be created

source/extensions/omni.isaac.lab/omni/isaac/lab/envs/ui/manager_based_rl_env_window.py

@@ -34,5 +34,7 @@ class ManagerBasedRLEnvWindow(BaseEnvWindow):
         with self.ui_window_elements["main_vstack"]:
             with self.ui_window_elements["debug_frame"]:
                 with self.ui_window_elements["debug_vstack"]:
-                    self._create_debug_vis_ui_element("commands", self.env.command_manager)
-                    self._create_debug_vis_ui_element("actions", self.env.action_manager)
+                    self._visualize_manager(title="Commands", class_name="command_manager")
+                    self._visualize_manager(title="Rewards", class_name="reward_manager")
+                    self._visualize_manager(title="Curriculum", class_name="curriculum_manager")
+                    self._visualize_manager(title="Termination", class_name="termination_manager")

source/extensions/omni.isaac.lab/omni/isaac/lab/managers/action_manager.py

@@ -106,6 +106,7 @@ class ActionTerm(ManagerTermBase):
         # check if debug visualization is supported
         if not self.has_debug_vis_implementation:
             return False
+
         # toggle debug visualization objects
         self._set_debug_vis_impl(debug_vis)
         # toggle debug visualization handles
@@ -262,7 +263,26 @@ class ActionManager(ManagerBase):
     Operations.
     """
 
-    def set_debug_vis(self, debug_vis: bool) -> bool:
+    def get_active_iterable_terms(self, env_idx: int) -> Sequence[tuple[str, Sequence[float]]]:
+        """Returns the active terms as iterable sequence of tuples.
+
+        The first element of the tuple is the name of the term and the second element is the raw value(s) of the term.
+
+        Args:
+            env_idx: The specific environment to pull the active terms from.
+
+        Returns:
+            The active terms.
+        """
+        terms = []
+        idx = 0
+        for name, term in self._terms.items():
+            term_actions = self._action[env_idx, idx : idx + term.action_dim].cpu()
+            terms.append((name, term_actions.tolist()))
+            idx += term.action_dim
+        return terms
+
+    def set_debug_vis(self, debug_vis: bool):
         """Sets whether to visualize the action data.
         Args:
             debug_vis: Whether to visualize the action data.

source/extensions/omni.isaac.lab/omni/isaac/lab/managers/command_manager.py

@@ -296,7 +296,26 @@ class CommandManager(ManagerBase):
     Operations.
     """
 
-    def set_debug_vis(self, debug_vis: bool) -> bool:
+    def get_active_iterable_terms(self, env_idx: int) -> Sequence[tuple[str, Sequence[float]]]:
+        """Returns the active terms as iterable sequence of tuples.
+
+        The first element of the tuple is the name of the term and the second element is the raw value(s) of the term.
+
+        Args:
+            env_idx: The specific environment to pull the active terms from.
+
+        Returns:
+            The active terms.
+        """
+
+        terms = []
+        idx = 0
+        for name, term in self._terms.items():
+            terms.append((name, term.command[env_idx].cpu().tolist()))
+            idx += term.command.shape[1]
+        return terms
+
+    def set_debug_vis(self, debug_vis: bool):
         """Sets whether to visualize the command data.
 
         Args:

source/extensions/omni.isaac.lab/omni/isaac/lab/managers/curriculum_manager.py

@@ -138,6 +138,40 @@ class CurriculumManager(ManagerBase):
             state = term_cfg.func(self._env, env_ids, **term_cfg.params)
             self._curriculum_state[name] = state
 
+    def get_active_iterable_terms(self, env_idx: int) -> Sequence[tuple[str, Sequence[float]]]:
+        """Returns the active terms as iterable sequence of tuples.
+
+        The first element of the tuple is the name of the term and the second element is the raw value(s) of the term.
+
+        Args:
+            env_idx: The specific environment to pull the active terms from.
+
+        Returns:
+            The active terms.
+        """
+
+        terms = []
+
+        for term_name, term_state in self._curriculum_state.items():
+            if term_state is not None:
+                # deal with dict
+                data = []
+
+                if isinstance(term_state, dict):
+                    # each key is a separate state to log
+                    for key, value in term_state.items():
+                        if isinstance(value, torch.Tensor):
+                            value = value.item()
+                        terms[term_name].append(value)
+                else:
+                    # log directly if not a dict
+                    if isinstance(term_state, torch.Tensor):
+                        term_state = term_state.item()
+                    data.append(term_state)
+                terms.append((term_name, data))
+
+        return terms
+
     """
     Helper functions.
     """

source/extensions/omni.isaac.lab/omni/isaac/lab/managers/manager_base.py

@@ -193,6 +193,16 @@ class ManagerBase(ABC):
         # return the matching names
         return string_utils.resolve_matching_names(name_keys, list_of_strings)[1]
 
+    def get_active_iterable_terms(self, env_idx: int) -> Sequence[tuple[str, Sequence[float]]]:
+        """Returns the active terms as iterable sequence of tuples.
+
+        The first element of the tuple is the name of the term and the second element is the raw value(s) of the term.
+
+        Returns:
+            The active terms.
+        """
+        raise NotImplementedError
+
     """
     Implementation specific.
     """

source/extensions/omni.isaac.lab/omni/isaac/lab/managers/observation_manager.py

@@ -100,6 +100,9 @@ class ObservationManager(ManagerBase):
             else:
                 self._group_obs_dim[group_name] = group_term_dims
 
+        # Stores the latest observations.
+        self._obs_buffer: dict[str, torch.Tensor | dict[str, torch.Tensor]] | None = None
+
     def __str__(self) -> str:
         """Returns: A string representation for the observation manager."""
         msg = f"<ObservationManager> contains {len(self._group_obs_term_names)} groups.\n"
@@ -130,6 +133,43 @@ class ObservationManager(ManagerBase):
 
         return msg
 
+    def get_active_iterable_terms(self, env_idx: int) -> Sequence[tuple[str, Sequence[float]]]:
+        """Returns the active terms as iterable sequence of tuples.
+
+        The first element of the tuple is the name of the term and the second element is the raw value(s) of the term.
+
+        Args:
+            env_idx: The specific environment to pull the active terms from.
+
+        Returns:
+            The active terms.
+        """
+        terms = []
+
+        if self._obs_buffer is None:
+            self.compute()
+        obs_buffer: dict[str, torch.Tensor | dict[str, torch.Tensor]] = self._obs_buffer
+
+        for group_name, _ in self._group_obs_dim.items():
+            if not self.group_obs_concatenate[group_name]:
+                for name, term in obs_buffer[group_name].items():
+                    terms.append((group_name + "-" + name, term[env_idx].cpu().tolist()))
+                continue
+
+            idx = 0
+            # add info for each term
+            data = obs_buffer[group_name]
+            for name, shape in zip(
+                self._group_obs_term_names[group_name],
+                self._group_obs_term_dim[group_name],
+            ):
+                data_length = np.prod(shape)
+                term = data[env_idx, idx : idx + data_length]
+                terms.append((group_name + "-" + name, term.cpu().tolist()))
+                idx += data_length
+
+        return terms
+
     """
     Properties.
     """
@@ -212,6 +252,9 @@ class ObservationManager(ManagerBase):
         for group_name in self._group_obs_term_names:
             obs_buffer[group_name] = self.compute_group(group_name)
         # otherwise return a dict with observations of all groups
+
+        # Cache the observations.
+        self._obs_buffer = obs_buffer
         return obs_buffer
 
     def compute_group(self, group_name: str) -> torch.Tensor | dict[str, torch.Tensor]:

source/extensions/omni.isaac.lab/omni/isaac/lab/managers/reward_manager.py

@@ -61,6 +61,9 @@ class RewardManager(ManagerBase):
         # create buffer for managing reward per environment
         self._reward_buf = torch.zeros(self.num_envs, dtype=torch.float, device=self.device)
 
+        # Buffer which stores the current step reward for each term for each environment
+        self._step_reward = torch.zeros((self.num_envs, len(self._term_names)), dtype=torch.float, device=self.device)
+
     def __str__(self) -> str:
         """Returns: A string representation for reward manager."""
         msg = f"<RewardManager> contains {len(self._term_names)} active terms.\n"
@@ -148,6 +151,9 @@ class RewardManager(ManagerBase):
             # update episodic sum
             self._episode_sums[name] += value
 
+            # Update current reward for this step.
+            self._step_reward[:, self._term_names.index(name)] = value / dt
+
         return self._reward_buf
 
     """
@@ -186,6 +192,22 @@ class RewardManager(ManagerBase):
         # return the configuration
         return self._term_cfgs[self._term_names.index(term_name)]
 
+    def get_active_iterable_terms(self, env_idx: int) -> Sequence[tuple[str, Sequence[float]]]:
+        """Returns the active terms as iterable sequence of tuples.
+
+        The first element of the tuple is the name of the term and the second element is the raw value(s) of the term.
+
+        Args:
+            env_idx: The specific environment to pull the active terms from.
+
+        Returns:
+            The active terms.
+        """
+        terms = []
+        for idx, name in enumerate(self._term_names):
+            terms.append((name, [self._step_reward[env_idx, idx].cpu().item()]))
+        return terms
+
     """
     Helper functions.
     """

source/extensions/omni.isaac.lab/omni/isaac/lab/managers/termination_manager.py

@@ -184,6 +184,22 @@ class TerminationManager(ManagerBase):
         """
         return self._term_dones[name]
 
+    def get_active_iterable_terms(self, env_idx: int) -> Sequence[tuple[str, Sequence[float]]]:
+        """Returns the active terms as iterable sequence of tuples.
+
+        The first element of the tuple is the name of the term and the second element is the raw value(s) of the term.
+
+        Args:
+            env_idx: The specific environment to pull the active terms from.
+
+        Returns:
+            The active terms.
+        """
+        terms = []
+        for key in self._term_dones.keys():
+            terms.append((key, [self._term_dones[key][env_idx].float().cpu().item()]))
+        return terms
+
     """
     Operations - Term settings.
     """

source/extensions/omni.isaac.lab/omni/isaac/lab/ui/widgets/__init__.py

+# Copyright (c) 2022-2024, The Isaac Lab Project Developers.
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+from .image_plot import ImagePlot
+from .line_plot import LiveLinePlot
+from .manager_live_visualizer import ManagerLiveVisualizer
+from .ui_visualizer_base import UiVisualizerBase

source/extensions/omni.isaac.lab/omni/isaac/lab/ui/widgets/image_plot.py

+# Copyright (c) 2022-2024, The Isaac Lab Project Developers.
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+import numpy as np
+from matplotlib import cm
+from typing import TYPE_CHECKING, Optional
+
+import carb
+import omni
+import omni.log
+
+from .ui_widget_wrapper import UIWidgetWrapper
+
+if TYPE_CHECKING:
+    import omni.isaac.ui
+    import omni.ui
+
+
+class ImagePlot(UIWidgetWrapper):
+    """An image plot widget to display live data.
+
+    It has the following Layout where the mode frame is only useful for depth images:
+    +-------------------------------------------------------+
+    |                  containing_frame                     |
+    |+-----------------------------------------------------+|
+    |                   main_plot_frame                     |
+    ||+---------------------------------------------------+||
+    |||                    plot_frames                    |||
+    |||                                                   |||
+    |||                                                   |||
+    |||               (Image Plot Data)                   |||
+    |||                                                   |||
+    |||                                                   |||
+    |||+-------------------------------------------------+|||
+    |||                   mode_frame                      |||
+    |||                                                   |||
+    |||    [x][Absolute] [x][Grayscaled] [ ][Colorized]   |||
+    |+-----------------------------------------------------+|
+    +-------------------------------------------------------+
+
+    """
+
+    def __init__(
+        self,
+        image: Optional[np.ndarray] = None,
+        label: str = "",
+        widget_height: int = 200,
+        show_min_max: bool = True,
+        unit: tuple[float, str] = (1, ""),
+    ):
+        """Create an XY plot UI Widget with axis scaling, legends, and support for multiple plots.
+
+        Overlapping data is most accurately plotted when centered in the frame with reasonable axis scaling.
+        Pressing down the mouse gives the x and y values of each function at an x coordinate.
+
+        Args:
+            image: Image to display
+            label: Short descriptive text to the left of the plot
+            widget_height: Height of the plot in pixels
+            show_min_max: Whether to show the min and max values of the image
+            unit: Tuple of (scale, name) for the unit of the image
+        """
+        self._show_min_max = show_min_max
+        self._unit_scale = unit[0]
+        self._unit_name = unit[1]
+
+        self._curr_mode = "None"
+
+        self._has_built = False
+
+        self._enabled = True
+
+        self._byte_provider = omni.ui.ByteImageProvider()
+        if image is None:
+            carb.log_warn("image is NONE")
+            image = np.ones((480, 640, 3), dtype=np.uint8) * 255
+            image[:, :, 0] = 0
+            image[:, :240, 1] = 0
+
+        # if image is channel first, convert to channel last
+        if image.ndim == 3 and image.shape[0] in [1, 3, 4]:
+            image = np.moveaxis(image, 0, -1)
+
+        self._aspect_ratio = image.shape[1] / image.shape[0]
+        self._widget_height = widget_height
+        self._label = label
+        self.update_image(image)
+
+        plot_frame = self._create_ui_widget()
+
+        super().__init__(plot_frame)
+
+    def setEnabled(self, enabled: bool):
+        self._enabled = enabled
+
+    def update_image(self, image: np.ndarray):
+        if not self._enabled:
+            return
+
+        # if image is channel first, convert to channel last
+        if image.ndim == 3 and image.shape[0] in [1, 3, 4]:
+            image = np.moveaxis(image, 0, -1)
+
+        height, width = image.shape[:2]
+
+        if self._curr_mode == "Normalization":
+            image = (image - image.min()) / (image.max() - image.min())
+            image = (image * 255).astype(np.uint8)
+        elif self._curr_mode == "Colorization":
+            if image.ndim == 3 and image.shape[2] == 3:
+                omni.log.warn("Colorization mode is only available for single channel images")
+            else:
+                image = (image - image.min()) / (image.max() - image.min())
+                colormap = cm.get_cmap("jet")
+                if image.ndim == 3 and image.shape[2] == 1:
+                    image = (colormap(image).squeeze(2) * 255).astype(np.uint8)
+                else:
+                    image = (colormap(image) * 255).astype(np.uint8)
+
+        # convert image to 4-channel RGBA
+        if image.ndim == 2 or (image.ndim == 3 and image.shape[2] == 1):
+            image = np.dstack((image, image, image, np.full((height, width, 1), 255, dtype=np.uint8)))
+
+        elif image.ndim == 3 and image.shape[2] == 3:
+            image = np.dstack((image, np.full((height, width, 1), 255, dtype=np.uint8)))
+
+        self._byte_provider.set_bytes_data(image.flatten().data, [width, height])
+
+    def update_min_max(self, image: np.ndarray):
+        if self._show_min_max and hasattr(self, "_min_max_label"):
+            non_inf = image[np.isfinite(image)].flatten()
+            if len(non_inf) > 0:
+                self._min_max_label.text = self._get_unit_description(
+                    np.min(non_inf), np.max(non_inf), np.median(non_inf)
+                )
+            else:
+                self._min_max_label.text = self._get_unit_description(0, 0)
+
+    def _create_ui_widget(self):
+        containing_frame = omni.ui.Frame(build_fn=self._build_widget)
+        return containing_frame
+
+    def _get_unit_description(self, min_value: float, max_value: float, median_value: float = None):
+        return (
+            f"Min: {min_value * self._unit_scale:.2f} {self._unit_name} Max:"
+            f" {max_value * self._unit_scale:.2f} {self._unit_name}"
+            + (f" Median: {median_value * self._unit_scale:.2f} {self._unit_name}" if median_value is not None else "")
+        )
+
+    def _build_widget(self):
+
+        with omni.ui.VStack(spacing=3):
+            with omni.ui.HStack():
+                # Write the leftmost label for what this plot is
+                omni.ui.Label(
+                    self._label, width=omni.isaac.ui.ui_utils.LABEL_WIDTH, alignment=omni.ui.Alignment.LEFT_TOP
+                )
+                with omni.ui.Frame(width=self._aspect_ratio * self._widget_height, height=self._widget_height):
+                    self._base_plot = omni.ui.ImageWithProvider(self._byte_provider)
+
+            if self._show_min_max:
+                self._min_max_label = omni.ui.Label(self._get_unit_description(0, 0))
+
+            omni.ui.Spacer(height=8)
+            self._mode_frame = omni.ui.Frame(build_fn=self._build_mode_frame)
+
+            omni.ui.Spacer(width=5)
+        self._has_built = True
+
+    def _build_mode_frame(self):
+        """Build the frame containing the mode selection for the plots.
+
+        This is an internal function to build the frame containing the mode selection for the plots. This function
+        should only be called from within the build function of a frame.
+
+        The built widget has the following layout:
+        +-------------------------------------------------------+
+        |                   legends_frame                       |
+        ||+---------------------------------------------------+||
+        |||                                                   |||
+        |||    [x][Series 1] [x][Series 2] [ ][Series 3]      |||
+        |||                                                   |||
+        |||+-------------------------------------------------+|||
+        |+-----------------------------------------------------+|
+        +-------------------------------------------------------+
+        """
+        with omni.ui.HStack():
+            with omni.ui.HStack():
+
+                def _change_mode(value):
+                    self._curr_mode = value
+
+                omni.isaac.ui.ui_utils.dropdown_builder(
+                    label="Mode",
+                    type="dropdown",
+                    items=["Original", "Normalization", "Colorization"],
+                    tooltip="Select a mode",
+                    on_clicked_fn=_change_mode,
+                )

source/extensions/omni.isaac.lab/omni/isaac/lab/ui/widgets/line_plot.py

+# Copyright (c) 2022-2024, The Isaac Lab Project Developers.
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+from __future__ import annotations
+
+import colorsys
+import numpy as np
+from typing import TYPE_CHECKING
+
+import omni
+from omni.isaac.core.simulation_context import SimulationContext
+
+from .ui_widget_wrapper import UIWidgetWrapper
+
+if TYPE_CHECKING:
+    import omni.isaac.ui
+    import omni.ui
+
+
+class LiveLinePlot(UIWidgetWrapper):
+    """A 2D line plot widget to display live data.
+
+
+    This widget is used to display live data in a 2D line plot. It can be used to display multiple series
+    in the same plot.
+
+    It has the following Layout:
+    +-------------------------------------------------------+
+    |                  containing_frame                     |
+    |+-----------------------------------------------------+|
+    |                   main_plot_frame                     |
+    ||+---------------------------------------------------+||
+    |||         plot_frames + grid lines (Z_stacked)      |||
+    |||                                                   |||
+    |||                                                   |||
+    |||               (Live Plot Data)                    |||
+    |||                                                   |||
+    |||                                                   |||
+    |||+-------------------------------------------------+|||
+    |||                   legends_frame                   |||
+    |||                                                   |||
+    |||    [x][Series 1] [x][Series 2] [ ][Series 3]      |||
+    |||+-------------------------------------------------+|||
+    |||                   limits_frame                    |||
+    |||                                                   |||
+    |||        [Y-Limits] [min] [max] [Autoscale]         |||
+    |||+-------------------------------------------------+|||
+    |||                   filter_frame                    |||
+    |||                                                   |||
+    |||                                                   |||
+    |+-----------------------------------------------------+|
+    +-------------------------------------------------------+
+
+    """
+
+    def __init__(
+        self,
+        y_data: list[list[float]],
+        y_min: float = -10,
+        y_max: float = 10,
+        plot_height: int = 150,
+        show_legend: bool = True,
+        legends: list[str] | None = None,
+        max_datapoints: int = 200,
+    ):
+        """Create a new LiveLinePlot widget.
+
+        Args:
+            y_data: A list of lists of floats containing the data to plot. Each list of floats represents a series in the plot.
+            y_min: The minimum y value to display. Defaults to -10.
+            y_max: The maximum y value to display. Defaults to 10.
+            plot_height: The height of the plot in pixels. Defaults to 150.
+            show_legend: Whether to display the legend. Defaults to True.
+            legends: A list of strings containing the legend labels for each series. If None, the default labels are "Series_0", "Series_1", etc. Defaults to None.
+            max_datapoints: The maximum number of data points to display. If the number of data points exceeds this value, the oldest data points are removed. Defaults to 200.
+        """
+        super().__init__(self._create_ui_widget())
+        self.plot_height = plot_height
+        self.show_legend = show_legend
+        self._legends = legends if legends is not None else ["Series_" + str(i) for i in range(len(y_data))]
+        self._y_data = y_data
+        self._colors = self._get_distinct_hex_colors(len(y_data))
+        self._y_min = y_min if y_min is not None else -10
+        self._y_max = y_max if y_max is not None else 10
+        self._max_data_points = max_datapoints
+        self._show_legend = show_legend
+        self._series_visible = [True for _ in range(len(y_data))]
+        self._plot_frames = []
+        self._plots = []
+        self._plot_selected_values = []
+        self._is_built = False
+        self._filter_frame = None
+        self._filter_mode = None
+        self._last_values = None
+        self._is_paused = False
+
+        # Gets populated when widget is built
+        self._main_plot_frame = None
+
+        self._autoscale_model = omni.ui.SimpleBoolModel(True)
+
+    """Properties"""
+
+    @property
+    def autoscale_mode(self) -> bool:
+        return self._autoscale_model.as_bool
+
+    @property
+    def y_data(self) -> list[list[float]]:
+        """The current data in the plot."""
+        return self._y_data
+
+    @property
+    def y_min(self) -> float:
+        """The current minimum y value."""
+        return self._y_min
+
+    @property
+    def y_max(self) -> float:
+        """The current maximum y value."""
+        return self._y_max
+
+    @property
+    def legends(self) -> list[str]:
+        """The current legend labels."""
+        return self._legends
+
+    """ General Functions """
+
+    def clear(self):
+        """Clears the plot."""
+        self._y_data = [[] for _ in range(len(self._y_data))]
+        self._last_values = None
+
+        for plt in self._plots:
+            plt.set_data()
+
+        # self._container_frame.rebuild()
+
+    def add_datapoint(self, y_coords: list[float]):
+        """Add a data point to the plot.
+
+        The data point is added to the end of the plot. If the number of data points exceeds the maximum number
+        of data points, the oldest data point is removed.
+
+        ``y_coords`` is assumed to be a list of floats with the same length as the number of series in the plot.
+
+        Args:
+            y_coords: A list of floats containing the y coordinates of the new data points.
+        """
+
+        for idx, y_coord in enumerate(y_coords):
+
+            if len(self._y_data[idx]) > self._max_data_points:
+                self._y_data[idx] = self._y_data[idx][1:]
+
+            if self._filter_mode == "Lowpass":
+                if self._last_values is not None:
+                    alpha = 0.8
+                    y_coord = self._y_data[idx][-1] * alpha + y_coord * (1 - alpha)
+            elif self._filter_mode == "Integrate":
+                if self._last_values is not None:
+                    y_coord = self._y_data[idx][-1] + y_coord
+            elif self._filter_mode == "Derivative":
+                if self._last_values is not None:
+                    y_coord = (y_coord - self._last_values[idx]) / SimulationContext.instance().get_rendering_dt()
+
+            self._y_data[idx].append(float(y_coord))
+
+        if self._main_plot_frame is None:
+            # Widget not built, not visible
+            return
+
+        # Check if the widget has been built, i.e. the plot references have been created.
+        if not self._is_built or self._is_paused:
+            return
+
+        if len(self._y_data) != len(self._plots):
+            # Plots gotten out of sync, rebuild the widget
+            self._main_plot_frame.rebuild()
+            return
+
+        if self.autoscale_mode:
+            self._rescale_btn_pressed()
+
+        for idx, plt in enumerate(self._plots):
+            plt.set_data(*self._y_data[idx])
+
+        self._last_values = y_coords
+        # Autoscale the y-axis to the current data
+
+    """
+        Internal functions for building the UI.
+    """
+
+    def _build_stacked_plots(self, grid: bool = True):
+        """Builds multiple plots stacked on top of each other to display multiple series.
+
+        This is an internal function to build the plots. It should not be called from outside the class and only
+        from within the build function of a frame.
+
+        The built widget has the following layout:
+        +-------------------------------------------------------+
+        |                   main_plot_frame                     |
+        ||+---------------------------------------------------+||
+        |||                                                   |||
+        ||| y_max|*******-------------------*******|          |||
+        |||      |-------*****-----------**--------|          |||
+        |||     0|------------**-----***-----------|          |||
+        |||      |--------------***----------------|          |||
+        ||| y_min|---------------------------------|          |||
+        |||                                                   |||
+        |||+-------------------------------------------------+|||
+
+
+        Args:
+            grid: Whether to display grid lines. Defaults to True.
+        """
+
+        # Reset lists which are populated in the build function
+        self._plot_frames = []
+
+        # Define internal builder function
+        def _build_single_plot(y_data: list[float], color: int, plot_idx: int):
+            """Build a single plot.
+
+            This is an internal function to build a single plot with the given data and color. This function
+            should only be called from within the build function of a frame.
+
+            Args:
+                y_data: The data to plot.
+                color: The color of the plot.
+            """
+            plot = omni.ui.Plot(
+                omni.ui.Type.LINE,
+                self._y_min,
+                self._y_max,
+                *y_data,
+                height=self.plot_height,
+                style={"color": color, "background_color": 0x0},
+            )
+
+            if len(self._plots) <= plot_idx:
+                self._plots.append(plot)
+                self._plot_selected_values.append(omni.ui.SimpleStringModel(""))
+            else:
+                self._plots[plot_idx] = plot
+
+        # Begin building the widget
+        with omni.ui.HStack():
+            # Space to the left to add y-axis labels
+            omni.ui.Spacer(width=20)
+
+            # Built plots for each time series stacked on top of each other
+            with omni.ui.ZStack():
+                # Background rectangle
+                omni.ui.Rectangle(
+                    height=self.plot_height,
+                    style={
+                        "background_color": 0x0,
+                        "border_color": omni.ui.color.white,
+                        "border_width": 0.4,
+                        "margin": 0.0,
+                    },
+                )
+
+                # Draw grid lines and labels
+                if grid:
+                    # Calculate the number of grid lines to display
+                    # Absolute range of the plot
+                    plot_range = self._y_max - self._y_min
+                    grid_resolution = 10 ** np.floor(np.log10(0.5 * plot_range))
+
+                    plot_range /= grid_resolution
+
+                    # Fraction of the plot range occupied by the first and last grid line
+                    first_space = (self._y_max / grid_resolution) - np.floor(self._y_max / grid_resolution)
+                    last_space = np.ceil(self._y_min / grid_resolution) - self._y_min / grid_resolution
+
+                    # Number of grid lines to display
+                    n_lines = int(plot_range - first_space - last_space)
+
+                    plot_resolution = self.plot_height / plot_range
+
+                    with omni.ui.VStack():
+                        omni.ui.Spacer(height=plot_resolution * first_space)
+
+                        # Draw grid lines
+                        with omni.ui.VGrid(row_height=plot_resolution):
+                            for grid_line_idx in range(n_lines):
+                                # Create grid line
+                                with omni.ui.ZStack():
+                                    omni.ui.Line(
+                                        style={
+                                            "color": 0xAA8A8777,
+                                            "background_color": 0x0,
+                                            "border_width": 0.4,
+                                        },
+                                        alignment=omni.ui.Alignment.CENTER_TOP,
+                                        height=0,
+                                    )
+                                    with omni.ui.Placer(offset_x=-20):
+                                        omni.ui.Label(
+                                            f"{(self._y_max - first_space * grid_resolution - grid_line_idx * grid_resolution):.3f}",
+                                            width=8,
+                                            height=8,
+                                            alignment=omni.ui.Alignment.RIGHT_TOP,
+                                            style={
+                                                "color": 0xFFFFFFFF,
+                                                "font_size": 8,
+                                            },
+                                        )
+
+                # Create plots for each series
+                for idx, (data, color) in enumerate(zip(self._y_data, self._colors)):
+                    plot_frame = omni.ui.Frame(
+                        build_fn=lambda y_data=data, plot_idx=idx, color=color: _build_single_plot(
+                            y_data, color, plot_idx
+                        ),
+                    )
+                    plot_frame.visible = self._series_visible[idx]
+                    self._plot_frames.append(plot_frame)
+
+                # Create an invisible frame on top that will give a helpful tooltip
+                self._tooltip_frame = omni.ui.Plot(
+                    height=self.plot_height,
+                    style={"color": 0xFFFFFFFF, "background_color": 0x0},
+                )
+
+                self._tooltip_frame.set_mouse_pressed_fn(self._mouse_moved_on_plot)
+
+                # Create top label for the y-axis
+                with omni.ui.Placer(offset_x=-20, offset_y=-8):
+                    omni.ui.Label(
+                        f"{self._y_max:.3f}",
+                        width=8,
+                        height=2,
+                        alignment=omni.ui.Alignment.LEFT_TOP,
+                        style={"color": 0xFFFFFFFF, "font_size": 8},
+                    )
+
+                # Create bottom label for the y-axis
+                with omni.ui.Placer(offset_x=-20, offset_y=self.plot_height):
+                    omni.ui.Label(
+                        f"{self._y_min:.3f}",
+                        width=8,
+                        height=2,
+                        alignment=omni.ui.Alignment.LEFT_BOTTOM,
+                        style={"color": 0xFFFFFFFF, "font_size": 8},
+                    )
+
+    def _mouse_moved_on_plot(self, x, y, *args):
+        # Show a tooltip with x,y and function values
+        if len(self._y_data) == 0 or len(self._y_data[0]) == 0:
+            # There is no data in the plots, so do nothing
+            return
+
+        for idx, plot in enumerate(self._plots):
+            x_pos = plot.screen_position_x
+            width = plot.computed_width
+
+            location_x = (x - x_pos) / width
+
+            data = self._y_data[idx]
+            n_samples = len(data)
+            selected_sample = int(location_x * n_samples)
+            value = data[selected_sample]
+            # save the value in scientific notation
+            self._plot_selected_values[idx].set_value(f"{value:.3f}")
+
+    def _build_legends_frame(self):
+        """Build the frame containing the legend for the plots.
+
+        This is an internal function to build the frame containing the legend for the plots. This function
+        should only be called from within the build function of a frame.
+
+        The built widget has the following layout:
+        +-------------------------------------------------------+
+        |                   legends_frame                       |
+        ||+---------------------------------------------------+||
+        |||                                                   |||
+        |||    [x][Series 1] [x][Series 2] [ ][Series 3]      |||
+        |||                                                   |||
+        |||+-------------------------------------------------+|||
+        |+-----------------------------------------------------+|
+        +-------------------------------------------------------+
+        """
+        if not self._show_legend:
+            return
+
+        with omni.ui.HStack():
+            omni.ui.Spacer(width=32)
+
+            # Find the longest legend to determine the width of the frame
+            max_legend = max([len(legend) for legend in self._legends])
+            CHAR_WIDTH = 8
+            with omni.ui.VGrid(
+                row_height=omni.isaac.ui.ui_utils.LABEL_HEIGHT,
+                column_width=max_legend * CHAR_WIDTH + 6,
+            ):
+                for idx in range(len(self._y_data)):
+                    with omni.ui.HStack():
+                        model = omni.ui.SimpleBoolModel()
+                        model.set_value(self._series_visible[idx])
+                        omni.ui.CheckBox(model=model, tooltip="", width=4)
+                        model.add_value_changed_fn(lambda val, idx=idx: self._change_plot_visibility(idx, val.as_bool))
+                        omni.ui.Spacer(width=2)
+                        with omni.ui.VStack():
+                            omni.ui.Label(
+                                self._legends[idx],
+                                width=max_legend * CHAR_WIDTH,
+                                alignment=omni.ui.Alignment.LEFT,
+                                style={"color": self._colors[idx], "font_size": 12},
+                            )
+                            omni.ui.StringField(
+                                model=self._plot_selected_values[idx],
+                                width=max_legend * CHAR_WIDTH,
+                                alignment=omni.ui.Alignment.LEFT,
+                                style={"color": self._colors[idx], "font_size": 10},
+                                read_only=True,
+                            )
+
+    def _build_limits_frame(self):
+        """Build the frame containing the controls for the y-axis limits.
+
+        This is an internal function to build the frame containing the controls for the y-axis limits. This function
+        should only be called from within the build function of a frame.
+
+        The built widget has the following layout:
+        +-------------------------------------------------------+
+        |                   limits_frame                        |
+        ||+---------------------------------------------------+||
+        |||                                                   |||
+        |||         Limits    [min] [max] [Re-Sacle]          |||
+        |||         Autoscale[x]                              |||
+        |||    -------------------------------------------    |||
+        |||+-------------------------------------------------+|||
+        """
+        with omni.ui.VStack():
+            with omni.ui.HStack():
+                omni.ui.Label(
+                    "Limits",
+                    width=omni.isaac.ui.ui_utils.LABEL_WIDTH,
+                    alignment=omni.ui.Alignment.LEFT_CENTER,
+                )
+
+                self.lower_limit_drag = omni.ui.FloatDrag(name="min", enabled=True, alignment=omni.ui.Alignment.CENTER)
+                y_min_model = self.lower_limit_drag.model
+                y_min_model.set_value(self._y_min)
+                y_min_model.add_value_changed_fn(lambda x: self._set_y_min(x.as_float))
+                omni.ui.Spacer(width=2)
+
+                self.upper_limit_drag = omni.ui.FloatDrag(name="max", enabled=True, alignment=omni.ui.Alignment.CENTER)
+                y_max_model = self.upper_limit_drag.model
+                y_max_model.set_value(self._y_max)
+                y_max_model.add_value_changed_fn(lambda x: self._set_y_max(x.as_float))
+                omni.ui.Spacer(width=2)
+
+                omni.ui.Button(
+                    "Re-Scale",
+                    width=omni.isaac.ui.ui_utils.BUTTON_WIDTH,
+                    clicked_fn=self._rescale_btn_pressed,
+                    alignment=omni.ui.Alignment.LEFT_CENTER,
+                    style=omni.isaac.ui.ui_utils.get_style(),
+                )
+
+                omni.ui.CheckBox(model=self._autoscale_model, tooltip="", width=4)
+
+            omni.ui.Line(
+                style={"color": 0x338A8777},
+                width=omni.ui.Fraction(1),
+                alignment=omni.ui.Alignment.CENTER,
+            )
+
+    def _build_filter_frame(self):
+        """Build the frame containing the filter controls.
+
+        This is an internal function to build the frame containing the filter controls. This function
+        should only be called from within the build function of a frame.
+
+        The built widget has the following layout:
+        +-------------------------------------------------------+
+        |                   filter_frame                        |
+        ||+---------------------------------------------------+||
+        |||                                                   |||
+        |||                                                   |||
+        |||                                                   |||
+        |||+-------------------------------------------------+|||
+        |+-----------------------------------------------------+|
+        +-------------------------------------------------------+
+        """
+        with omni.ui.VStack():
+            with omni.ui.HStack():
+
+                def _filter_changed(value):
+                    self.clear()
+                    self._filter_mode = value
+
+                omni.isaac.ui.ui_utils.dropdown_builder(
+                    label="Filter",
+                    type="dropdown",
+                    items=["None", "Lowpass", "Integrate", "Derivative"],
+                    tooltip="Select a filter",
+                    on_clicked_fn=_filter_changed,
+                )
+
+                def _toggle_paused():
+                    self._is_paused = not self._is_paused
+
+                # Button
+                omni.ui.Button(
+                    "Play/Pause",
+                    width=omni.isaac.ui.ui_utils.BUTTON_WIDTH,
+                    clicked_fn=_toggle_paused,
+                    alignment=omni.ui.Alignment.LEFT_CENTER,
+                    style=omni.isaac.ui.ui_utils.get_style(),
+                )
+
+    def _create_ui_widget(self):
+        """Create the full UI widget."""
+
+        def _build_widget():
+            self._is_built = False
+            with omni.ui.VStack():
+                self._main_plot_frame = omni.ui.Frame(build_fn=self._build_stacked_plots)
+                omni.ui.Spacer(height=8)
+                self._legends_frame = omni.ui.Frame(build_fn=self._build_legends_frame)
+                omni.ui.Spacer(height=8)
+                self._limits_frame = omni.ui.Frame(build_fn=self._build_limits_frame)
+                omni.ui.Spacer(height=8)
+                self._filter_frame = omni.ui.Frame(build_fn=self._build_filter_frame)
+            self._is_built = True
+
+        containing_frame = omni.ui.Frame(build_fn=_build_widget)
+
+        return containing_frame
+
+    """ UI Actions Listener Functions """
+
+    def _change_plot_visibility(self, idx: int, visible: bool):
+        """Change the visibility of a plot at position idx."""
+        self._series_visible[idx] = visible
+        self._plot_frames[idx].visible = visible
+        # self._main_plot_frame.rebuild()
+
+    def _set_y_min(self, val: float):
+        """Update the y-axis minimum."""
+        self._y_min = val
+        self.lower_limit_drag.model.set_value(val)
+        self._main_plot_frame.rebuild()
+
+    def _set_y_max(self, val: float):
+        """Update the y-axis maximum."""
+        self._y_max = val
+        self.upper_limit_drag.model.set_value(val)
+        self._main_plot_frame.rebuild()
+
+    def _rescale_btn_pressed(self):
+        """Autoscale the y-axis to the current data."""
+        if any(self._series_visible):
+            y_min = np.round(
+                min([min(y) for idx, y in enumerate(self._y_data) if self._series_visible[idx]]),
+                4,
+            )
+            y_max = np.round(
+                max([max(y) for idx, y in enumerate(self._y_data) if self._series_visible[idx]]),
+                4,
+            )
+            if y_min == y_max:
+                y_max += 1e-4  # Make sure axes don't collapse
+
+            self._y_max = y_max
+            self._y_min = y_min
+
+        if hasattr(self, "lower_limit_drag") and hasattr(self, "upper_limit_drag"):
+            self.lower_limit_drag.model.set_value(self._y_min)
+            self.upper_limit_drag.model.set_value(self._y_max)
+
+        self._main_plot_frame.rebuild()
+
+    """ Helper Functions """
+
+    def _get_distinct_hex_colors(self, num_colors) -> list[int]:
+        """
+        This function returns a list of distinct colors for plotting.
+
+        Args:
+            num_colors (int): the number of colors to generate
+
+        Returns:
+            List[int]: a list of distinct colors in hexadecimal format 0xFFBBGGRR
+        """
+        # Generate equally spaced colors in HSV space
+        rgb_colors = [
+            colorsys.hsv_to_rgb(hue / num_colors, 0.75, 1) for hue in np.linspace(0, num_colors - 1, num_colors)
+        ]
+        # Convert to 0-255 RGB values
+        rgb_colors = [[int(c * 255) for c in rgb] for rgb in rgb_colors]
+        # Convert to 0xFFBBGGRR format
+        hex_colors = [0xFF * 16**6 + c[2] * 16**4 + c[1] * 16**2 + c[0] for c in rgb_colors]
+        return hex_colors

source/extensions/omni.isaac.lab/omni/isaac/lab/ui/widgets/manager_live_visualizer.py

+# Copyright (c) 2022-2024, The Isaac Lab Project Developers.
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+from __future__ import annotations
+
+import numpy
+import weakref
+from dataclasses import MISSING
+from typing import TYPE_CHECKING
+
+import carb
+import omni.kit.app
+from omni.isaac.core.simulation_context import SimulationContext
+
+from omni.isaac.lab.managers import ManagerBase
+from omni.isaac.lab.utils import configclass
+
+from .image_plot import ImagePlot
+from .line_plot import LiveLinePlot
+from .ui_visualizer_base import UiVisualizerBase
+
+if TYPE_CHECKING:
+    import omni.ui
+
+
+@configclass
+class ManagerLiveVisualizerCfg:
+    "Configuration for ManagerLiveVisualizer"
+
+    debug_vis: bool = False
+    """Flag used to set status of the live visualizers on startup. Defaults to closed."""
+    manager_name: str = MISSING
+    """Manager name that corresponds to the manager of interest in the ManagerBasedEnv and ManagerBasedRLEnv"""
+    term_names: list[str] | dict[str, list[str]] | None = None
+    """Specific term names specified in a Manager config that are chosen to be plotted. Defaults to None.
+
+    If None all terms will be plotted. For managers that utilize Groups (i.e. ObservationGroup) use a dictionary of
+    {group_names: [term_names]}.
+    """
+
+
+class ManagerLiveVisualizer(UiVisualizerBase):
+    """A interface object used to transfer data from a manager to a UI widget. This class handles the creation of UI
+    Widgets for selected terms given a ManagerLiveVisualizerCfg.
+    """
+
+    def __init__(self, manager: ManagerBase, cfg: ManagerLiveVisualizerCfg = ManagerLiveVisualizerCfg()):
+        """Initialize ManagerLiveVisualizer.
+
+        Args:
+            manager: The manager with terms to be plotted. The manager must have a get_active_iterable_terms method.
+            cfg: The configuration file used to select desired manager terms to be plotted.
+        """
+
+        self._manager = manager
+        self.debug_vis = cfg.debug_vis
+        self._env_idx: int = 0
+        self.cfg = cfg
+        self._viewer_env_idx = 0
+        self._vis_frame: omni.ui.Frame
+        self._vis_window: omni.ui.Window
+
+        # evaluate chosen terms if no terms provided use all available.
+        self.term_names = []
+
+        if self.cfg.term_names is not None:
+            # extract chosen terms
+            if isinstance(self.cfg.term_names, list):
+                for term_name in self.cfg.term_names:
+                    if term_name in self._manager.active_terms:
+                        self.term_names.append(term_name)
+                    else:
+                        carb.log_err(
+                            f"ManagerVisualizer Failure: ManagerTerm ({term_name}) does not exist in"
+                            f" Manager({self.cfg.manager_name})"
+                        )
+
+            # extract chosen group-terms
+            elif isinstance(self.cfg.term_names, dict):
+                # if manager is using groups and terms are saved as a dictionary
+                if isinstance(self._manager.active_terms, dict):
+                    for group, terms in self.cfg.term_names:
+                        if group in self._manager.active_terms.keys():
+                            for term_name in terms:
+                                if term_name in self._manager.active_terms[group]:
+                                    self.term_names.append(f"{group}-{term_name}")
+                                else:
+                                    carb.log_err(
+                                        f"ManagerVisualizer Failure: ManagerTerm ({term_name}) does not exist in"
+                                        f" Group({group})"
+                                    )
+                        else:
+                            carb.log_err(
+                                f"ManagerVisualizer Failure: Group ({group}) does not exist in"
+                                f" Manager({self.cfg.manager_name})"
+                            )
+                else:
+                    carb.log_err(
+                        f"ManagerVisualizer Failure: Manager({self.cfg.manager_name}) does not utilize grouping of"
+                        " terms."
+                    )
+
+    #
+    # Implementation checks
+    #
+
+    @property
+    def get_vis_frame(self) -> omni.ui.Frame:
+        """Getter for the UI Frame object tied to this visualizer."""
+        return self._vis_frame
+
+    @property
+    def get_vis_window(self) -> omni.ui.Window:
+        """Getter for the UI Window object tied to this visualizer."""
+        return self._vis_window
+
+    #
+    # Setters
+    #
+
+    def set_debug_vis(self, debug_vis: bool):
+        """Set the debug visualization external facing function.
+
+        Args:
+            debug_vis: Whether to enable or disable the debug visualization.
+        """
+        self._set_debug_vis_impl(debug_vis)
+
+    #
+    # Implementations
+    #
+
+    def _set_env_selection_impl(self, env_idx: int):
+        """Update the index of the selected environment to display.
+
+        Args:
+            env_idx: The index of the selected environment.
+        """
+        if env_idx > 0 and env_idx < self._manager.num_envs:
+            self._env_idx = env_idx
+        else:
+            carb.log_warn(f"Environment index is out of range (0,{self._manager.num_envs})")
+
+    def _set_vis_frame_impl(self, frame: omni.ui.Frame):
+        """Updates the assigned frame that can be used for visualizations.
+
+        Args:
+            frame: The debug visualization frame.
+        """
+        self._vis_frame = frame
+
+    def _debug_vis_callback(self, event):
+        """Callback for the debug visualization event."""
+
+        if not SimulationContext.instance().is_playing():
+            # Visualizers have not been created yet.
+            return
+
+        # get updated data and update visualization
+        for (_, term), vis in zip(
+            self._manager.get_active_iterable_terms(env_idx=self._env_idx), self._term_visualizers
+        ):
+            if isinstance(vis, LiveLinePlot):
+                vis.add_datapoint(term)
+            elif isinstance(vis, ImagePlot):
+                vis.update_image(numpy.array(term))
+
+    def _set_debug_vis_impl(self, debug_vis: bool):
+        """Set the debug visualization implementation.
+
+        Args:
+            debug_vis: Whether to enable or disable debug visualization.
+        """
+
+        if not hasattr(self, "_vis_frame"):
+            raise RuntimeError("No frame set for debug visualization.")
+
+        # Clear internal visualizers
+        self._term_visualizers = []
+        self._vis_frame.clear()
+
+        if debug_vis:
+            # if enabled create a subscriber for the post update event if it doesn't exist
+            if not hasattr(self, "_debug_vis_handle") or self._debug_vis_handle is None:
+                app_interface = omni.kit.app.get_app_interface()
+                self._debug_vis_handle = app_interface.get_post_update_event_stream().create_subscription_to_pop(
+                    lambda event, obj=weakref.proxy(self): obj._debug_vis_callback(event)
+                )
+        else:
+            # if disabled remove the subscriber if it exists
+            if self._debug_vis_handle is not None:
+                self._debug_vis_handle.unsubscribe()
+                self._debug_vis_handle = None
+
+            self._vis_frame.visible = False
+            return
+
+        self._vis_frame.visible = True
+
+        with self._vis_frame:
+            with omni.ui.VStack():
+                # Add a plot in a collapsible frame for each term available
+                for name, term in self._manager.get_active_iterable_terms(env_idx=self._env_idx):
+                    if name in self.term_names or len(self.term_names) == 0:
+                        frame = omni.ui.CollapsableFrame(
+                            name,
+                            collapsed=False,
+                            style={"border_color": 0xFF8A8777, "padding": 4},
+                        )
+                        with frame:
+                            # create line plot for single or multivariable signals
+                            len_term_shape = len(numpy.array(term).shape)
+                            if len_term_shape <= 2:
+                                plot = LiveLinePlot(
+                                    y_data=[[elem] for elem in term],
+                                    plot_height=150,
+                                    show_legend=True,
+                                )
+                                self._term_visualizers.append(plot)
+                            # create an image plot for 2d and greater data (i.e. mono and rgb images)
+                            elif len_term_shape == 3:
+                                image = ImagePlot(
+                                    image=numpy.array(term),
+                                    label=name,
+                                )
+                                self._term_visualizers.append(image)
+                            else:
+                                carb.log_warn(
+                                    f"ManagerLiveVisualizer: Term ({name}) is not a supported data type for"
+                                    " visualization."
+                                )
+                        frame.collapsed = True
+
+        self._debug_vis = debug_vis
+
+
+@configclass
+class DefaultManagerBasedEnvLiveVisCfg:
+    """Default configuration to use for the ManagerBasedEnv. Each chosen manager assumes all terms will be plotted."""
+
+    action_live_vis = ManagerLiveVisualizerCfg(manager_name="action_manager")
+    observation_live_vis = ManagerLiveVisualizerCfg(manager_name="observation_manager")
+
+
+@configclass
+class DefaultManagerBasedRLEnvLiveVisCfg(DefaultManagerBasedEnvLiveVisCfg):
+    """Default configuration to use for the ManagerBasedRLEnv. Each chosen manager assumes all terms will be plotted."""
+
+    curriculum_live_vis = ManagerLiveVisualizerCfg(manager_name="curriculum_manager")
+    command_live_vis = ManagerLiveVisualizerCfg(manager_name="command_manager")
+    reward_live_vis = ManagerLiveVisualizerCfg(manager_name="reward_manager")
+    termination_live_vis = ManagerLiveVisualizerCfg(manager_name="termination_manager")
+
+
+class EnvLiveVisualizer:
+    """A class to handle all ManagerLiveVisualizers used in an Environment."""
+
+    def __init__(self, cfg: object, managers: dict[str, ManagerBase]):
+        """Initialize the EnvLiveVisualizer.
+
+        Args:
+            cfg: The configuration file containing terms of ManagerLiveVisualizers.
+            managers: A dictionary of labeled managers. i.e. {"manager_name",manager}.
+        """
+        self.cfg = cfg
+        self.managers = managers
+        self._prepare_terms()
+
+    def _prepare_terms(self):
+        self._manager_visualizers: dict[str, ManagerLiveVisualizer] = dict()
+
+        # check if config is dict already
+        if isinstance(self.cfg, dict):
+            cfg_items = self.cfg.items()
+        else:
+            cfg_items = self.cfg.__dict__.items()
+
+        for term_name, term_cfg in cfg_items:
+            # check if term config is None
+            if term_cfg is None:
+                continue
+            # check if term config is viable
+            if isinstance(term_cfg, ManagerLiveVisualizerCfg):
+                # find appropriate manager name
+                manager = self.managers[term_cfg.manager_name]
+                self._manager_visualizers[term_cfg.manager_name] = ManagerLiveVisualizer(manager=manager, cfg=term_cfg)
+            else:
+                raise TypeError(
+                    f"Provided EnvLiveVisualizer term: '{term_name}' is not of type ManagerLiveVisualizerCfg"
+                )
+
+    @property
+    def manager_visualizers(self) -> dict[str, ManagerLiveVisualizer]:
+        """A dictionary of labeled ManagerLiveVisualizers associated manager name as key."""
+        return self._manager_visualizers

source/extensions/omni.isaac.lab/omni/isaac/lab/ui/widgets/ui_visualizer_base.py

+# Copyright (c) 2022-2024, The Isaac Lab Project Developers.
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+from __future__ import annotations
+
+import inspect
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    import omni.ui
+
+
+class UiVisualizerBase:
+    """Base Class for components that support debug visualizations that requires access to some UI elements.
+
+    This class provides a set of functions that can be used to assign ui interfaces.
+
+    The following functions are provided:
+
+    * :func:`set_debug_vis`: Assigns a debug visualization interface. This function is called by the main UI
+        when the checkbox for debug visualization is toggled.
+    * :func:`set_vis_frame`: Assigns a small frame within the isaac lab tab that can be used to visualize debug
+        information. Such as e.g. plots or images. It is called by the main UI on startup to create the frame.
+    * :func:`set_window`: Assigngs the main window that is used by the main UI. This allows the user
+        to have full controller over all UI elements. But be warned, with great power comes great responsibility.
+    """
+
+    """
+    Exposed Properties
+    """
+
+    @property
+    def has_debug_vis_implementation(self) -> bool:
+        """Whether the component has a debug visualization implemented."""
+        # check if function raises NotImplementedError
+        source_code = inspect.getsource(self._set_debug_vis_impl)
+        return "NotImplementedError" not in source_code
+
+    @property
+    def has_vis_frame_implementation(self) -> bool:
+        """Whether the component has a debug visualization implemented."""
+        # check if function raises NotImplementedError
+        source_code = inspect.getsource(self._set_vis_frame_impl)
+        return "NotImplementedError" not in source_code
+
+    @property
+    def has_window_implementation(self) -> bool:
+        """Whether the component has a debug visualization implemented."""
+        # check if function raises NotImplementedError
+        source_code = inspect.getsource(self._set_window_impl)
+        return "NotImplementedError" not in source_code
+
+    @property
+    def has_env_selection_implementation(self) -> bool:
+        """Whether the component has a debug visualization implemented."""
+        # check if function raises NotImplementedError
+        source_code = inspect.getsource(self._set_env_selection_impl)
+        return "NotImplementedError" not in source_code
+
+    """
+    Exposed Setters
+    """
+
+    def set_env_selection(self, env_selection: int) -> bool:
+        """Sets the selected environment id.
+
+        This function is called by the main UI when the user selects a different environment.
+
+        Args:
+            env_selection: The currently selected environment id.
+
+        Returns:
+            Whether the environment selection was successfully set. False if the component
+            does not support environment selection.
+        """
+        # check if environment selection is supported
+        if not self.has_env_selection_implementation:
+            return False
+        # set environment selection
+        self._set_env_selection_impl(env_selection)
+        return True
+
+    def set_window(self, window: omni.ui.Window) -> bool:
+        """Sets the current main ui window.
+
+        This function is called by the main UI when the window is created. It allows the component
+        to add custom UI elements to the window or to control the window and its elements.
+
+        Args:
+            window: The ui window.
+
+        Returns:
+            Whether the window was successfully set. False if the component
+            does not support this functionality.
+        """
+        # check if window is supported
+        if not self.has_window_implementation:
+            return False
+        # set window
+        self._set_window_impl(window)
+        return True
+
+    def set_vis_frame(self, vis_frame: omni.ui.Frame) -> bool:
+        """Sets the debug visualization frame.
+
+        This function is called by the main UI when the window is created. It allows the component
+        to modify a small frame within the orbit tab that can be used to visualize debug information.
+
+        Args:
+            vis_frame: The debug visualization frame.
+
+        Returns:
+            Whether the debug visualization frame was successfully set. False if the component
+            does not support debug visualization.
+        """
+        # check if debug visualization is supported
+        if not self.has_vis_frame_implementation:
+            return False
+        # set debug visualization frame
+        self._set_vis_frame_impl(vis_frame)
+        return True
+
+    """
+    Internal Implementation
+    """
+
+    def _set_env_selection_impl(self, env_idx: int):
+        """Set the environment selection."""
+        raise NotImplementedError(f"Environment selection is not implemented for {self.__class__.__name__}.")
+
+    def _set_window_impl(self, window: omni.ui.Window):
+        """Set the window."""
+        raise NotImplementedError(f"Window is not implemented for {self.__class__.__name__}.")
+
+    def _set_debug_vis_impl(self, debug_vis: bool):
+        """Set debug visualization state."""
+        raise NotImplementedError(f"Debug visualization is not implemented for {self.__class__.__name__}.")
+
+    def _set_vis_frame_impl(self, vis_frame: omni.ui.Frame):
+        """Set debug visualization into visualization objects.
+
+        This function is responsible for creating the visualization objects if they don't exist
+        and input ``debug_vis`` is True. If the visualization objects exist, the function should
+        set their visibility into the stage.
+        """
+        raise NotImplementedError(f"Debug visualization is not implemented for {self.__class__.__name__}.")

source/extensions/omni.isaac.lab/omni/isaac/lab/ui/widgets/ui_widget_wrapper.py

+# Copyright (c) 2022-2024, The Isaac Lab Project Developers.
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+# This file has been adapted from _isaac_sim/exts/omni.isaac.ui/omni/isaac/ui/element_wrappers/base_ui_element_wrappers.py
+
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+import omni
+
+if TYPE_CHECKING:
+    import omni.ui
+
+
+class UIWidgetWrapper:
+    """
+    Base class for creating wrappers around any subclass of omni.ui.Widget in order to provide an easy interface
+    for creating and managing specific types of widgets such as state buttons or file pickers.
+    """
+
+    def __init__(self, container_frame: omni.ui.Frame):
+        self._container_frame = container_frame
+
+    @property
+    def container_frame(self) -> omni.ui.Frame:
+        return self._container_frame
+
+    @property
+    def enabled(self) -> bool:
+        return self.container_frame.enabled
+
+    @enabled.setter
+    def enabled(self, value: bool):
+        self.container_frame.enabled = value
+
+    @property
+    def visible(self) -> bool:
+        return self.container_frame.visible
+
+    @visible.setter
+    def visible(self, value: bool):
+        self.container_frame.visible = value
+
+    def cleanup(self):
+        """
+        Perform any necessary cleanup
+        """
+        pass