Adds a render config to the simulation and tiledCamera limitations to the docs (#1246)

# Description

This change adds a render config to the simulation context and
highlights current limitations and workarounds for issues with the
TiledCamera class.

## Type of change

- This change requires a documentation update

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective or that my
feature works
- [x] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

.github/workflows/docs.yaml

@@ -2,6 +2,10 @@ name: Build & deploy docs
 
 on:
   push:
+    branches:
+      - main
+  pull_request:
+    types: [opened, synchronize, reopened]
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}

docs/index.rst

@@ -98,7 +98,6 @@ Table of Contents
    source/features/multi_gpu
    source/features/tiled_rendering
    source/features/reproducibility
-   .. source/features/motion_generators
 
 .. toctree::
    :maxdepth: 1

docs/source/api/lab/omni.isaac.lab.sim.rst

@@ -19,6 +19,7 @@
     SimulationContext
     SimulationCfg
     PhysxCfg
+    RenderCfg
 
   .. rubric:: Functions
 
@@ -46,6 +47,11 @@ Simulation Configuration
   :show-inheritance:
   :exclude-members: __init__
 
+.. autoclass:: RenderCfg
+  :members:
+  :show-inheritance:
+  :exclude-members: __init__
+
 Simulation Context Builder
 --------------------------
 

docs/source/features/hydra.rst

@@ -89,8 +89,8 @@ Elements in dictionaries are handled as a parameters in the hierarchy. For examp
 
 .. literalinclude:: ../../../source/extensions/omni.isaac.lab_tasks/omni/isaac/lab_tasks/manager_based/classic/cartpole/cartpole_env_cfg.py
     :language: python
-    :lines: 99-111
-    :emphasize-lines: 10
+    :lines: 90-114
+    :emphasize-lines: 11
 
 the ``position_range`` parameter can be modified with ``env.events.reset_cart_position.params.position_range="[-2.0, 2.0]"``.
 This example shows two noteworthy points:
@@ -112,7 +112,7 @@ For example, for the configuration of the Cartpole camera depth environment:
     :language: python
     :start-at: class CartpoleDepthCameraEnvCfg
     :end-at: tiled_camera.width
-    :emphasize-lines: 16
+    :emphasize-lines: 10, 15
 
 If the user were to modify the width of the camera, i.e. ``env.tiled_camera.width=128``, then the parameter
 ``env.observation_space=[80,128,1]`` must be updated and given as input as well.

docs/source/features/tiled_rendering.rst

@@ -8,7 +8,7 @@ Tiled-Camera Rendering
     This feature is only available from Isaac Sim version 4.2.0 onwards.
 
     Tiled rendering in combination with image processing networks require heavy memory resources, especially
-    at larger resolutions. We recommend running at 512 cameras in the scene on RTX 4090 GPUs or similar.
+    at larger resolutions. We recommend running 512 cameras in the scene on RTX 4090 GPUs or similar.
 
 
 Tiled rendering APIs provide a vectorized interface for collecting data from camera sensors.
@@ -129,3 +129,39 @@ Instance Segmentation
 - If ``colorize_instance_segmentation=True`` in the camera config, a 4-channel RGBA image will be returned with dimension (B, H, W, 4) and type ``torch.uint8``. The info ``idToLabels`` dictionary will be the mapping from color to USD prim path of that semantic entity. The info ``idToSemantics`` dictionary will be the mapping from color to semantic labels of that semantic entity.
 
 - If ``colorize_instance_segmentation=False``, a buffer of dimension (B, H, W, 1) of type ``torch.int32`` will be returned, containing the instance ID of each pixel. The info ``idToLabels`` dictionary will be the mapping from instance ID to USD prim path of that semantic entity. The info ``idToSemantics`` dictionary will be the mapping from instance ID to semantic labels of that semantic entity.
+
+
+Current Limitations
+-------------------
+
+Due to current limitations in the renderer, we can have only **one** :class:`~sensors.TiledCamera` instance in the scene.
+For use cases that require a setup with more than one camera, we can imitate the multi-camera behavior by moving the location
+of the camera in between render calls in a step.
+
+For example, in a stereo vision setup, the below snippet can be implemented:
+
+.. code-block:: python
+
+    # render image from "first" camera
+    camera_data_1 = self._tiled_camera.data.output["rgb"].clone() / 255.0
+    # update camera transform to the "second" camera location
+    self._tiled_camera.set_world_poses(
+        positions=pos,
+        orientations=rot,
+        convention="world"
+    )
+    # step the renderer
+    self.sim.render()
+    self._tiled_camera.update(0, force_recompute=True)
+    # render image from "second" camera
+    camera_data_2 = self._tiled_camera.data.output["rgb"].clone() / 255.0
+
+Note that this approach still limits the rendering resolution to be identical for all cameras. Currently, there is no workaround
+to achieve different resolution images using :class:`~sensors.TiledCamera`. The best approach is to use the largest resolution out of all of the
+desired resolutions and add additional scaling or cropping operations to the rendered output as a post-processing step.
+
+In addition, there may be visible quality differences when comparing render outputs of different numbers of environments.
+Currently, any combined resolution that has a width less than 265 pixels or height less than 265 will automatically switch
+to the DLAA anti-aliasing mode, which does not perform up-sampling during anti-aliasing. For resolutions larger than 265 in both
+width and height dimensions, we default to using the "performance" DLSS mode for anti-aliasing for performance benefits.
+Anti-aliasing modes and other rendering parameters can be specified in the :class:`~sim.RenderCfg`.

docs/source/overview/core-concepts/index.rst

@@ -9,3 +9,5 @@ This section we introduce core concepts in Isaac Lab.
 
   task_workflows
   actuators
+
+# motion_generators

source/extensions/omni.isaac.lab/config/extension.toml

 [package]
 
 # Note: Semantic Versioning is used: https://semver.org/
-version = "0.27.0"
+version = "0.27.1"
 
 # Description
 title = "Isaac Lab framework for Robot Learning"

source/extensions/omni.isaac.lab/docs/CHANGELOG.rst

 Changelog
 ---------
 
+0.27.1 (2024-10-20)
+~~~~~~~~~~~~~~~~~~~
+
+Added
+^^^^^
+
+* Added :class:`~omni.isaac.lab.sim.RenderCfg` and the attribute :attr:`~omni.isaac.lab.sim.SimulationCfg.render` for
+  specifying render related settings.
+
+
 0.27.0 (2024-10-14)
 ~~~~~~~~~~~~~~~~~~~
 

source/extensions/omni.isaac.lab/omni/isaac/lab/sim/__init__.py

@@ -28,7 +28,7 @@ To make it convenient to use the module, we recommend importing the module as fo
 
 from .converters import *  # noqa: F401, F403
 from .schemas import *  # noqa: F401, F403
-from .simulation_cfg import PhysxCfg, SimulationCfg  # noqa: F401, F403
+from .simulation_cfg import PhysxCfg, RenderCfg, SimulationCfg  # noqa: F401, F403
 from .simulation_context import SimulationContext, build_simulation_context  # noqa: F401, F403
 from .spawners import *  # noqa: F401, F403
 from .utils import *  # noqa: F401, F403

source/extensions/omni.isaac.lab/omni/isaac/lab/sim/simulation_cfg.py

@@ -152,6 +152,54 @@ class PhysxCfg:
     """Size of particle contacts stream buffer allocated in pinned host memory. Default is 2 ** 20."""
 
 
+@configclass
+class RenderCfg:
+    """Configuration for Omniverse RTX Renderer.
+
+    These parameters are used to configure the Omniverse RTX Renderer.
+    For more information, see the `Omniverse RTX Renderer documentation`_.
+
+    .. _Omniverse RTX Renderer documentation: https://docs.omniverse.nvidia.com/materials-and-rendering/latest/rtx-renderer.html
+    """
+
+    enable_translucency: bool = False
+    """Enables translucency for specular transmissive surfaces such as glass at the cost of some performance. Default is False."""
+
+    enable_reflections: bool = False
+    """Enables reflections at the cost of some performance. Default is False."""
+
+    enable_global_illumination: bool = False
+    """Enables Diffused Global Illumination at the cost of some performance. Default is False."""
+
+    antialiasing_mode: Literal["Off", "FXAA", "DLSS", "TAA", "DLAA"] = "DLSS"
+    """Selects the anti-aliasing mode to use. Defaults to DLSS."""
+
+    enable_dlssg: bool = False
+    """"Enables the use of DLSS-G.
+        DLSS Frame Generation boosts performance by using AI to generate more frames.
+        DLSS analyzes sequential frames and motion data to create additional high quality frames.
+        This feature requires an Ada Lovelace architecture GPU.
+        Enabling this feature also enables additional thread-related activities, which can hurt performance.
+        Default is False."""
+
+    dlss_mode: Literal[0, 1, 2, 3] = 0
+    """For DLSS anti-aliasing, selects the performance/quality tradeoff mode.
+       Valid values are 0 (Performance), 1 (Balanced), 2 (Quality), or 3 (Auto). Default is 0."""
+
+    enable_direct_lighting: bool = True
+    """Enable direct light contributions from lights."""
+
+    samples_per_pixel: int = 1
+    """Defines the Direct Lighting samples per pixel.
+       Higher values increase the direct lighting quality at the cost of performance. Default is 1."""
+
+    enable_shadows: bool = True
+    """Enables shadows at the cost of performance. When disabled, lights will not cast shadows. Defaults to True."""
+
+    enable_ambient_occlusion: bool = False
+    """Enables ambient occlusion at the cost of some performance. Default is False."""
+
+
 @configclass
 class SimulationCfg:
     """Configuration for simulation physics."""
@@ -234,3 +282,6 @@ class SimulationCfg:
 
     The material is created at the path: ``{physics_prim_path}/defaultMaterial``.
     """
+
+    render: RenderCfg = RenderCfg()
+    """Render settings. Default is RenderCfg()."""

source/extensions/omni.isaac.lab/omni/isaac/lab/sim/simulation_context.py

@@ -154,6 +154,26 @@ class SimulationContext(_SimulationContext):
         # flag for whether any GUI will be rendered (local, livestreamed or viewport)
         self._has_gui = self._local_gui or self._livestream_gui
 
+        # apply render settings from render config
+        carb_settings_iface.set_bool("/rtx/translucency/enabled", self.cfg.render.enable_translucency)
+        carb_settings_iface.set_bool("/rtx/reflections/enabled", self.cfg.render.enable_reflections)
+        carb_settings_iface.set_bool("/rtx/indirectDiffuse/enabled", self.cfg.render.enable_global_illumination)
+        carb_settings_iface.set_bool("/rtx/transient/dlssg/enabled", self.cfg.render.enable_dlssg)
+        carb_settings_iface.set_int("/rtx/post/dlss/execMode", self.cfg.render.dlss_mode)
+        carb_settings_iface.set_bool("/rtx/directLighting/enabled", self.cfg.render.enable_direct_lighting)
+        carb_settings_iface.set_int(
+            "/rtx/directLighting/sampledLighting/samplesPerPixel", self.cfg.render.samples_per_pixel
+        )
+        carb_settings_iface.set_bool("/rtx/shadows/enabled", self.cfg.render.enable_shadows)
+        carb_settings_iface.set_bool("/rtx/ambientOcclusion/enabled", self.cfg.render.enable_ambient_occlusion)
+        # set denoiser mode
+        try:
+            import omni.replicator.core as rep
+
+            rep.settings.set_render_rtx_realtime(antialiasing=self.cfg.render.antialiasing_mode)
+        except Exception:
+            pass
+
         # store the default render mode
         if not self._has_gui and not self._offscreen_render:
             # set default render mode

source/extensions/omni.isaac.lab/test/sim/test_simulation_render_config.py

+# Copyright (c) 2022-2024, The Isaac Lab Project Developers.
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+
+"""Launch Isaac Sim Simulator first."""
+
+from omni.isaac.lab.app import AppLauncher, run_tests
+
+# launch omniverse app
+app_launcher = AppLauncher(headless=True, enable_cameras=True)
+simulation_app = app_launcher.app
+
+"""Rest everything follows."""
+
+import unittest
+
+import carb
+
+from omni.isaac.lab.sim.simulation_cfg import RenderCfg, SimulationCfg
+from omni.isaac.lab.sim.simulation_context import SimulationContext
+
+
+class TestSimulationRenderConfig(unittest.TestCase):
+    """Tests for simulation context render config."""
+
+    """
+    Tests
+    """
+
+    def test_render_cfg(self):
+        """Test that the simulation context is created with the correct render cfg."""
+        enable_translucency = True
+        enable_reflections = True
+        enable_global_illumination = True
+        antialiasing_mode = "DLAA"
+        enable_dlssg = True
+        dlss_mode = 3
+        enable_direct_lighting = True
+        samples_per_pixel = 4
+        enable_shadows = True
+        enable_ambient_occlusion = True
+
+        render_cfg = RenderCfg(
+            enable_translucency=enable_translucency,
+            enable_reflections=enable_reflections,
+            enable_global_illumination=enable_global_illumination,
+            antialiasing_mode=antialiasing_mode,
+            enable_dlssg=enable_dlssg,
+            dlss_mode=dlss_mode,
+            enable_direct_lighting=enable_direct_lighting,
+            samples_per_pixel=samples_per_pixel,
+            enable_shadows=enable_shadows,
+            enable_ambient_occlusion=enable_ambient_occlusion,
+        )
+
+        cfg = SimulationCfg(render=render_cfg)
+
+        sim = SimulationContext(cfg)
+
+        self.assertEqual(sim.cfg.render.enable_translucency, enable_translucency)
+        self.assertEqual(sim.cfg.render.enable_reflections, enable_reflections)
+        self.assertEqual(sim.cfg.render.enable_global_illumination, enable_global_illumination)
+        self.assertEqual(sim.cfg.render.antialiasing_mode, antialiasing_mode)
+        self.assertEqual(sim.cfg.render.enable_dlssg, enable_dlssg)
+        self.assertEqual(sim.cfg.render.dlss_mode, dlss_mode)
+        self.assertEqual(sim.cfg.render.enable_direct_lighting, enable_direct_lighting)
+        self.assertEqual(sim.cfg.render.samples_per_pixel, samples_per_pixel)
+        self.assertEqual(sim.cfg.render.enable_shadows, enable_shadows)
+        self.assertEqual(sim.cfg.render.enable_ambient_occlusion, enable_ambient_occlusion)
+
+        carb_settings_iface = carb.settings.get_settings()
+        self.assertEqual(carb_settings_iface.get("/rtx/translucency/enabled"), sim.cfg.render.enable_translucency)
+        self.assertEqual(carb_settings_iface.get("/rtx/reflections/enabled"), sim.cfg.render.enable_reflections)
+        self.assertEqual(
+            carb_settings_iface.get("/rtx/indirectDiffuse/enabled"), sim.cfg.render.enable_global_illumination
+        )
+        self.assertEqual(carb_settings_iface.get("/rtx/transient/dlssg/enabled"), sim.cfg.render.enable_dlssg)
+        self.assertEqual(carb_settings_iface.get("/rtx/post/dlss/execMode"), sim.cfg.render.dlss_mode)
+        self.assertEqual(carb_settings_iface.get("/rtx/directLighting/enabled"), sim.cfg.render.enable_direct_lighting)
+        self.assertEqual(
+            carb_settings_iface.get("/rtx/directLighting/sampledLighting/samplesPerPixel"),
+            sim.cfg.render.samples_per_pixel,
+        )
+        self.assertEqual(carb_settings_iface.get("/rtx/shadows/enabled"), sim.cfg.render.enable_shadows)
+        self.assertEqual(
+            carb_settings_iface.get("/rtx/ambientOcclusion/enabled"), sim.cfg.render.enable_ambient_occlusion
+        )
+        self.assertEqual(carb_settings_iface.get("/rtx/post/aa/op"), 4)  # dlss = 3, dlaa=4
+
+    def test_render_cfg_defaults(self):
+        """Test that the simulation context is created with the correct render cfg."""
+        enable_translucency = False
+        enable_reflections = False
+        enable_global_illumination = False
+        antialiasing_mode = "DLSS"
+        enable_dlssg = False
+        dlss_mode = 0
+        enable_direct_lighting = False
+        samples_per_pixel = 1
+        enable_shadows = False
+        enable_ambient_occlusion = False
+
+        render_cfg = RenderCfg(
+            enable_translucency=enable_translucency,
+            enable_reflections=enable_reflections,
+            enable_global_illumination=enable_global_illumination,
+            antialiasing_mode=antialiasing_mode,
+            enable_dlssg=enable_dlssg,
+            dlss_mode=dlss_mode,
+            enable_direct_lighting=enable_direct_lighting,
+            samples_per_pixel=samples_per_pixel,
+            enable_shadows=enable_shadows,
+            enable_ambient_occlusion=enable_ambient_occlusion,
+        )
+
+        cfg = SimulationCfg(render=render_cfg)
+
+        sim = SimulationContext(cfg)
+
+        self.assertEqual(sim.cfg.render.enable_translucency, enable_translucency)
+        self.assertEqual(sim.cfg.render.enable_reflections, enable_reflections)
+        self.assertEqual(sim.cfg.render.enable_global_illumination, enable_global_illumination)
+        self.assertEqual(sim.cfg.render.antialiasing_mode, antialiasing_mode)
+        self.assertEqual(sim.cfg.render.enable_dlssg, enable_dlssg)
+        self.assertEqual(sim.cfg.render.dlss_mode, dlss_mode)
+        self.assertEqual(sim.cfg.render.enable_direct_lighting, enable_direct_lighting)
+        self.assertEqual(sim.cfg.render.samples_per_pixel, samples_per_pixel)
+        self.assertEqual(sim.cfg.render.enable_shadows, enable_shadows)
+        self.assertEqual(sim.cfg.render.enable_ambient_occlusion, enable_ambient_occlusion)
+
+        carb_settings_iface = carb.settings.get_settings()
+        self.assertEqual(carb_settings_iface.get("/rtx/translucency/enabled"), sim.cfg.render.enable_translucency)
+        self.assertEqual(carb_settings_iface.get("/rtx/reflections/enabled"), sim.cfg.render.enable_reflections)
+        self.assertEqual(
+            carb_settings_iface.get("/rtx/indirectDiffuse/enabled"), sim.cfg.render.enable_global_illumination
+        )
+        self.assertEqual(carb_settings_iface.get("/rtx/transient/dlssg/enabled"), sim.cfg.render.enable_dlssg)
+        self.assertEqual(carb_settings_iface.get("/rtx/post/dlss/execMode"), sim.cfg.render.dlss_mode)
+        self.assertEqual(carb_settings_iface.get("/rtx/directLighting/enabled"), sim.cfg.render.enable_direct_lighting)
+        self.assertEqual(
+            carb_settings_iface.get("/rtx/directLighting/sampledLighting/samplesPerPixel"),
+            sim.cfg.render.samples_per_pixel,
+        )
+        self.assertEqual(carb_settings_iface.get("/rtx/shadows/enabled"), sim.cfg.render.enable_shadows)
+        self.assertEqual(
+            carb_settings_iface.get("/rtx/ambientOcclusion/enabled"), sim.cfg.render.enable_ambient_occlusion
+        )
+        self.assertEqual(carb_settings_iface.get("/rtx/post/aa/op"), 3)  # dlss = 3, dlaa=4
+
+
+if __name__ == "__main__":
+    run_tests()