Commit a655ad95 authored by Kelly Guo's avatar Kelly Guo Committed by David Hoeller

Uses new replicator API for tiled rendering (#110)

# Description

This change switches to the newly added replicator API for tiled
rendering. All tiled rendering behaviors are the same as before. The API
provides a cleaner interface to initialize tiled rendering.

## Type of change
- New feature (non-breaking change which adds functionality)

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->
parent 91f80260
...@@ -11,7 +11,8 @@ Tiled Rendering ...@@ -11,7 +11,8 @@ Tiled Rendering
This feature is only available from Isaac Sim version 4.2.0 onwards. This feature is only available from Isaac Sim version 4.2.0 onwards.
Tiled rendering in combination with image processing networks require heavy memory resources, especially at larger resolutions. We recommend running at 256 cameras in the scene on RTX 4090 GPUs or similar. Tiled rendering in combination with image processing networks require heavy memory resources, especially at larger resolutions. We recommend running at 512 cameras in the scene on RTX 4090 GPUs or similar.
Tiled rendering APIs provide a vectorized interface for collecting data from camera sensors. Tiled rendering APIs provide a vectorized interface for collecting data from camera sensors.
This is useful for reinforcement learning environments requiring vision in the loop. This is useful for reinforcement learning environments requiring vision in the loop.
...@@ -59,6 +60,21 @@ environment. For example: ...@@ -59,6 +60,21 @@ environment. For example:
python source/standalone/workflows/rl_games/train.py --task=Isaac-Cartpole-RGB-Camera-Direct-v0 --headless --enable_cameras python source/standalone/workflows/rl_games/train.py --task=Isaac-Cartpole-RGB-Camera-Direct-v0 --headless --enable_cameras
.. warning::
There are currently a few limitations with tiled rendering:
* Number of cameras must be a perfect square
* Tile resolution must be a square
* Due to upsampling in the denoising process, image quality may appear different when running with different numbers of cameras.
To overcome this issue, we can use the DLAA denoiser at the cost of some performance.
.. code-block:: python
import omni.replicator.core as rep
rep.settings.set_render_rtx_realtime(antialiasing="DLAA")
Annotators and Data Types Annotators and Data Types
^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^
......
...@@ -56,10 +56,6 @@ exts."omni.renderer.core".present.enabled=false ...@@ -56,10 +56,6 @@ exts."omni.renderer.core".present.enabled=false
# Disabling these settings reduces renderer VRAM usage and improves rendering performance, but at some quality cost # Disabling these settings reduces renderer VRAM usage and improves rendering performance, but at some quality cost
rtx.raytracing.cached.enabled = false rtx.raytracing.cached.enabled = false
rtx.ambientOcclusion.enabled = false rtx.ambientOcclusion.enabled = false
rtx-transient.dlssg.enabled = false
rtx.sceneDb.ambientLightIntensity = 1.0
rtx.directLighting.sampledLighting.enabled = true
# Avoids unnecessary GPU context initialization # Avoids unnecessary GPU context initialization
renderer.multiGpu.maxGpuCount=1 renderer.multiGpu.maxGpuCount=1
......
...@@ -56,10 +56,6 @@ exts."omni.renderer.core".present.enabled=false ...@@ -56,10 +56,6 @@ exts."omni.renderer.core".present.enabled=false
# Disabling these settings reduces renderer VRAM usage and improves rendering performance, but at some quality cost # Disabling these settings reduces renderer VRAM usage and improves rendering performance, but at some quality cost
rtx.raytracing.cached.enabled = false rtx.raytracing.cached.enabled = false
rtx.ambientOcclusion.enabled = false rtx.ambientOcclusion.enabled = false
rtx-transient.dlssg.enabled = false
rtx.sceneDb.ambientLightIntensity = 1.0
rtx.directLighting.sampledLighting.enabled = true
# Avoids unnecessary GPU context initialization # Avoids unnecessary GPU context initialization
renderer.multiGpu.maxGpuCount=1 renderer.multiGpu.maxGpuCount=1
......
...@@ -12,12 +12,11 @@ from collections.abc import Sequence ...@@ -12,12 +12,11 @@ from collections.abc import Sequence
from tensordict import TensorDict from tensordict import TensorDict
from typing import TYPE_CHECKING, Any from typing import TYPE_CHECKING, Any
import carb
import omni.usd import omni.usd
import warp as wp import warp as wp
from omni.isaac.core.prims import XFormPrimView from omni.isaac.core.prims import XFormPrimView
from omni.isaac.version import get_version from omni.isaac.version import get_version
from pxr import Usd, UsdGeom from pxr import UsdGeom
from omni.isaac.lab.utils.warp.kernels import reshape_tiled_image from omni.isaac.lab.utils.warp.kernels import reshape_tiled_image
...@@ -184,22 +183,10 @@ class TiledCamera(Camera): ...@@ -184,22 +183,10 @@ class TiledCamera(Camera):
sensor_prim = UsdGeom.Camera(cam_prim) sensor_prim = UsdGeom.Camera(cam_prim)
self._sensor_prims.append(sensor_prim) self._sensor_prims.append(sensor_prim)
# get full resolution for all tiles # Create replicator tiled render product
full_resolution = self._tiled_image_shape() rp = rep.create.render_product_tiled(
cameras=self._view.prim_paths, tile_resolution=(self.cfg.width, self.cfg.height)
# Set carb settings for tiled rendering )
carb_settings = carb.settings.get_settings()
carb_settings.set("/rtx/viewTile/height", self.cfg.height)
carb_settings.set("/rtx/viewTile/width", self.cfg.width)
carb_settings.set("/rtx/viewTile/count", self._view.count)
# Create render product
rp = rep.create.render_product(self._view.prim_paths[0], full_resolution)
# Attach all cameras to render product
rp_prim = stage.GetPrimAtPath(rp.path)
with Usd.EditContext(stage, stage.GetSessionLayer()):
rp_prim.GetRelationship("camera").SetTargets(self._view.prim_paths)
self._render_product_paths = [rp.path] self._render_product_paths = [rp.path]
# Define the annotators based on requested data types # Define the annotators based on requested data types
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment