Skip to content

K
KincoActuatorIsaacLab
Commit a655ad95 authored by Kelly Guo's avatar Kelly Guo Committed by David Hoeller
Browse files
Options

Uses new replicator API for tiled rendering (#110) 

# Description

This change switches to the newly added replicator API for tiled
rendering. All tiled rendering behaviors are the same as before. The API
provides a cleaner interface to initialize tiled rendering.

## Type of change
- New feature (non-breaking change which adds functionality)

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [x] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

<!--
As you go through the checklist above, you can mark something as done by
putting an x character in it

For example,
- [x] I have done this task
- [ ] I have not done this task
-->
parent 91f80260
Hide whitespace changes
Inline Side-by-side
Showing with 212 additions and 202 deletions
docs/source/features/tiled_rendering.rst
View file @ a655ad95
...@@ -11,7 +11,8 @@ Tiled Rendering ...@@ -11,7 +11,8 @@ Tiled Rendering
This feature is only available from Isaac Sim version 4.2.0 onwards. This feature is only available from Isaac Sim version 4.2.0 onwards.
Tiled rendering in combination with image processing networks require heavy memory resources, especially at larger resolutions. We recommend running at 256 cameras in the scene on RTX 4090 GPUs or similar. Tiled rendering in combination with image processing networks require heavy memory resources, especially at larger resolutions. We recommend running at 512 cameras in the scene on RTX 4090 GPUs or similar.
Tiled rendering APIs provide a vectorized interface for collecting data from camera sensors. Tiled rendering APIs provide a vectorized interface for collecting data from camera sensors.
This is useful for reinforcement learning environments requiring vision in the loop. This is useful for reinforcement learning environments requiring vision in the loop.
...@@ -59,6 +60,21 @@ environment. For example: ...@@ -59,6 +60,21 @@ environment. For example:
python source/standalone/workflows/rl_games/train.py --task=Isaac-Cartpole-RGB-Camera-Direct-v0 --headless --enable_cameras python source/standalone/workflows/rl_games/train.py --task=Isaac-Cartpole-RGB-Camera-Direct-v0 --headless --enable_cameras
.. warning::
There are currently a few limitations with tiled rendering:
* Number of cameras must be a perfect square
* Tile resolution must be a square
* Due to upsampling in the denoising process, image quality may appear different when running with different numbers of cameras.
To overcome this issue, we can use the DLAA denoiser at the cost of some performance.
.. code-block:: python
import omni.replicator.core as rep
rep.settings.set_render_rtx_realtime(antialiasing="DLAA")
Annotators and Data Types Annotators and Data Types
^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^
......
source/apps/isaaclab.python.headless.rendering.kit
View file @ a655ad95
...@@ -56,10 +56,6 @@ exts."omni.renderer.core".present.enabled=false ...@@ -56,10 +56,6 @@ exts."omni.renderer.core".present.enabled=false
# Disabling these settings reduces renderer VRAM usage and improves rendering performance, but at some quality cost # Disabling these settings reduces renderer VRAM usage and improves rendering performance, but at some quality cost
rtx.raytracing.cached.enabled = false rtx.raytracing.cached.enabled = false
rtx.ambientOcclusion.enabled = false rtx.ambientOcclusion.enabled = false
rtx-transient.dlssg.enabled = false
rtx.sceneDb.ambientLightIntensity = 1.0
rtx.directLighting.sampledLighting.enabled = true
# Avoids unnecessary GPU context initialization # Avoids unnecessary GPU context initialization
renderer.multiGpu.maxGpuCount=1 renderer.multiGpu.maxGpuCount=1
......
source/apps/isaaclab.python.rendering.kit
View file @ a655ad95
...@@ -56,10 +56,6 @@ exts."omni.renderer.core".present.enabled=false ...@@ -56,10 +56,6 @@ exts."omni.renderer.core".present.enabled=false
# Disabling these settings reduces renderer VRAM usage and improves rendering performance, but at some quality cost # Disabling these settings reduces renderer VRAM usage and improves rendering performance, but at some quality cost
rtx.raytracing.cached.enabled = false rtx.raytracing.cached.enabled = false
rtx.ambientOcclusion.enabled = false rtx.ambientOcclusion.enabled = false
rtx-transient.dlssg.enabled = false
rtx.sceneDb.ambientLightIntensity = 1.0
rtx.directLighting.sampledLighting.enabled = true
# Avoids unnecessary GPU context initialization # Avoids unnecessary GPU context initialization
renderer.multiGpu.maxGpuCount=1 renderer.multiGpu.maxGpuCount=1
......
source/extensions/omni.isaac.lab/omni/isaac/lab/sensors/camera/tiled_camera.py
View file @ a655ad95
...@@ -12,12 +12,11 @@ from collections.abc import Sequence ...@@ -12,12 +12,11 @@ from collections.abc import Sequence
from tensordict import TensorDict from tensordict import TensorDict
from typing import TYPE_CHECKING, Any from typing import TYPE_CHECKING, Any
import carb
import omni.usd import omni.usd
import warp as wp import warp as wp
from omni.isaac.core.prims import XFormPrimView from omni.isaac.core.prims import XFormPrimView
from omni.isaac.version import get_version from omni.isaac.version import get_version
from pxr import Usd, UsdGeom from pxr import UsdGeom
from omni.isaac.lab.utils.warp.kernels import reshape_tiled_image from omni.isaac.lab.utils.warp.kernels import reshape_tiled_image
...@@ -184,22 +183,10 @@ class TiledCamera(Camera): ...@@ -184,22 +183,10 @@ class TiledCamera(Camera):
sensor_prim = UsdGeom.Camera(cam_prim) sensor_prim = UsdGeom.Camera(cam_prim)
self._sensor_prims.append(sensor_prim) self._sensor_prims.append(sensor_prim)
# get full resolution for all tiles # Create replicator tiled render product
full_resolution = self._tiled_image_shape() rp = rep.create.render_product_tiled(
cameras=self._view.prim_paths, tile_resolution=(self.cfg.width, self.cfg.height)
# Set carb settings for tiled rendering )
carb_settings = carb.settings.get_settings()
carb_settings.set("/rtx/viewTile/height", self.cfg.height)
carb_settings.set("/rtx/viewTile/width", self.cfg.width)
carb_settings.set("/rtx/viewTile/count", self._view.count)
# Create render product
rp = rep.create.render_product(self._view.prim_paths[0], full_resolution)
# Attach all cameras to render product
rp_prim = stage.GetPrimAtPath(rp.path)
with Usd.EditContext(stage, stage.GetSessionLayer()):
rp_prim.GetRelationship("camera").SetTargets(self._view.prim_paths)
self._render_product_paths = [rp.path] self._render_product_paths = [rp.path]
# Define the annotators based on requested data types # Define the annotators based on requested data types
......
source/extensions/omni.isaac.lab/test/sensors/test_tiled_camera.py
View file @ a655ad95
...@@ -123,8 +123,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -123,8 +123,9 @@ class TestTiledCamera(unittest.TestCase):
def test_multi_camera_init(self): def test_multi_camera_init(self):
"""Test multi-camera initialization.""" """Test multi-camera initialization."""
prim_utils.create_prim("/World/Origin_00", "Xform") num_cameras = 4
prim_utils.create_prim("/World/Origin_01", "Xform") for i in range(num_cameras):
prim_utils.create_prim(f"/World/Origin_{i}", "Xform")
# Create camera # Create camera
camera_cfg = copy.deepcopy(self.camera_cfg) camera_cfg = copy.deepcopy(self.camera_cfg)
...@@ -137,7 +138,7 @@ class TestTiledCamera(unittest.TestCase): ...@@ -137,7 +138,7 @@ class TestTiledCamera(unittest.TestCase):
# Check if camera is initialized # Check if camera is initialized
self.assertTrue(camera.is_initialized) self.assertTrue(camera.is_initialized)
# Check if camera prim is set correctly and that it is a camera prim # Check if camera prim is set correctly and that it is a camera prim
self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_01/CameraSensor") self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_1/CameraSensor")
self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera) self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera)
# Simulate for a few steps # Simulate for a few steps
...@@ -147,11 +148,11 @@ class TestTiledCamera(unittest.TestCase): ...@@ -147,11 +148,11 @@ class TestTiledCamera(unittest.TestCase):
self.sim.step() self.sim.step()
# Check buffers that exists and have correct shapes # Check buffers that exists and have correct shapes
self.assertEqual(camera.data.pos_w.shape, (2, 3)) self.assertEqual(camera.data.pos_w.shape, (num_cameras, 3))
self.assertEqual(camera.data.quat_w_ros.shape, (2, 4)) self.assertEqual(camera.data.quat_w_ros.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_world.shape, (2, 4)) self.assertEqual(camera.data.quat_w_world.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_opengl.shape, (2, 4)) self.assertEqual(camera.data.quat_w_opengl.shape, (num_cameras, 4))
self.assertEqual(camera.data.intrinsic_matrices.shape, (2, 3, 3)) self.assertEqual(camera.data.intrinsic_matrices.shape, (num_cameras, 3, 3))
self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width)) self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width))
# Simulate physics # Simulate physics
...@@ -163,20 +164,21 @@ class TestTiledCamera(unittest.TestCase): ...@@ -163,20 +164,21 @@ class TestTiledCamera(unittest.TestCase):
# check image data # check image data
for im_type, im_data in camera.data.output.to_dict().items(): for im_type, im_data in camera.data.output.to_dict().items():
if im_type == "rgb": if im_type == "rgb":
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 3)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 3))
self.assertGreater((im_data[0] / 255.0).mean().item(), 0.0) for i in range(4):
self.assertGreater((im_data[1] / 255.0).mean().item(), 0.0) self.assertGreater((im_data[i] / 255.0).mean().item(), 0.0)
elif im_type == "depth": elif im_type == "depth":
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 1)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 1))
self.assertGreater(im_data[0].mean().item(), 0.0) for i in range(4):
self.assertGreater(im_data[1].mean().item(), 0.0) self.assertGreater(im_data[i].mean().item(), 0.0)
del camera del camera
def test_rgb_only_camera(self): def test_rgb_only_camera(self):
"""Test initialization with only RGB.""" """Test initialization with only RGB."""
prim_utils.create_prim("/World/Origin_00", "Xform") num_cameras = 4
prim_utils.create_prim("/World/Origin_01", "Xform") for i in range(num_cameras):
prim_utils.create_prim(f"/World/Origin_{i}", "Xform")
# Create camera # Create camera
camera_cfg = copy.deepcopy(self.camera_cfg) camera_cfg = copy.deepcopy(self.camera_cfg)
...@@ -190,7 +192,7 @@ class TestTiledCamera(unittest.TestCase): ...@@ -190,7 +192,7 @@ class TestTiledCamera(unittest.TestCase):
# Check if camera is initialized # Check if camera is initialized
self.assertTrue(camera.is_initialized) self.assertTrue(camera.is_initialized)
# Check if camera prim is set correctly and that it is a camera prim # Check if camera prim is set correctly and that it is a camera prim
self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_01/CameraSensor") self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_1/CameraSensor")
self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera) self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera)
self.assertListEqual(sorted(camera.data.output.keys()), sorted(["rgb", "rgba"])) self.assertListEqual(sorted(camera.data.output.keys()), sorted(["rgb", "rgba"]))
# Simulate for a few steps # Simulate for a few steps
...@@ -200,11 +202,11 @@ class TestTiledCamera(unittest.TestCase): ...@@ -200,11 +202,11 @@ class TestTiledCamera(unittest.TestCase):
self.sim.step() self.sim.step()
# Check buffers that exists and have correct shapes # Check buffers that exists and have correct shapes
self.assertEqual(camera.data.pos_w.shape, (2, 3)) self.assertEqual(camera.data.pos_w.shape, (num_cameras, 3))
self.assertEqual(camera.data.quat_w_ros.shape, (2, 4)) self.assertEqual(camera.data.quat_w_ros.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_world.shape, (2, 4)) self.assertEqual(camera.data.quat_w_world.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_opengl.shape, (2, 4)) self.assertEqual(camera.data.quat_w_opengl.shape, (num_cameras, 4))
self.assertEqual(camera.data.intrinsic_matrices.shape, (2, 3, 3)) self.assertEqual(camera.data.intrinsic_matrices.shape, (num_cameras, 3, 3))
self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width)) self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width))
# Simulate physics # Simulate physics
...@@ -215,9 +217,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -215,9 +217,9 @@ class TestTiledCamera(unittest.TestCase):
camera.update(self.dt) camera.update(self.dt)
# check image data # check image data
im_data = camera.data.output["rgb"] im_data = camera.data.output["rgb"]
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 3)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 3))
self.assertGreater((im_data[0] / 255.0).mean().item(), 0.0) for i in range(4):
self.assertGreater((im_data[1] / 255.0).mean().item(), 0.0) self.assertGreater((im_data[i] / 255.0).mean().item(), 0.0)
# Check data type of image # Check data type of image
self.assertEqual(camera.data.output["rgb"].dtype, torch.uint8) self.assertEqual(camera.data.output["rgb"].dtype, torch.uint8)
del camera del camera
...@@ -252,8 +254,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -252,8 +254,9 @@ class TestTiledCamera(unittest.TestCase):
def test_depth_only_camera(self): def test_depth_only_camera(self):
"""Test initialization with only depth.""" """Test initialization with only depth."""
prim_utils.create_prim("/World/Origin_00", "Xform") num_cameras = 4
prim_utils.create_prim("/World/Origin_01", "Xform") for i in range(num_cameras):
prim_utils.create_prim(f"/World/Origin_{i}", "Xform")
# Create camera # Create camera
camera_cfg = copy.deepcopy(self.camera_cfg) camera_cfg = copy.deepcopy(self.camera_cfg)
...@@ -267,7 +270,7 @@ class TestTiledCamera(unittest.TestCase): ...@@ -267,7 +270,7 @@ class TestTiledCamera(unittest.TestCase):
# Check if camera is initialized # Check if camera is initialized
self.assertTrue(camera.is_initialized) self.assertTrue(camera.is_initialized)
# Check if camera prim is set correctly and that it is a camera prim # Check if camera prim is set correctly and that it is a camera prim
self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_01/CameraSensor") self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_1/CameraSensor")
self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera) self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera)
self.assertListEqual(sorted(camera.data.output.keys()), sorted(["depth", "distance_to_image_plane"])) self.assertListEqual(sorted(camera.data.output.keys()), sorted(["depth", "distance_to_image_plane"]))
...@@ -278,11 +281,11 @@ class TestTiledCamera(unittest.TestCase): ...@@ -278,11 +281,11 @@ class TestTiledCamera(unittest.TestCase):
self.sim.step() self.sim.step()
# Check buffers that exists and have correct shapes # Check buffers that exists and have correct shapes
self.assertEqual(camera.data.pos_w.shape, (2, 3)) self.assertEqual(camera.data.pos_w.shape, (num_cameras, 3))
self.assertEqual(camera.data.quat_w_ros.shape, (2, 4)) self.assertEqual(camera.data.quat_w_ros.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_world.shape, (2, 4)) self.assertEqual(camera.data.quat_w_world.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_opengl.shape, (2, 4)) self.assertEqual(camera.data.quat_w_opengl.shape, (num_cameras, 4))
self.assertEqual(camera.data.intrinsic_matrices.shape, (2, 3, 3)) self.assertEqual(camera.data.intrinsic_matrices.shape, (num_cameras, 3, 3))
self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width)) self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width))
# Simulate physics # Simulate physics
...@@ -293,9 +296,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -293,9 +296,9 @@ class TestTiledCamera(unittest.TestCase):
camera.update(self.dt) camera.update(self.dt)
# check image data # check image data
im_data = camera.data.output["depth"] im_data = camera.data.output["depth"]
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 1)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 1))
self.assertGreater(im_data[0].mean().item(), 0.0) for i in range(4):
self.assertGreater(im_data[1].mean().item(), 0.0) self.assertGreater((im_data[i]).mean().item(), 0.0)
# Check data type of image # Check data type of image
self.assertEqual(camera.data.output["depth"].dtype, torch.float) self.assertEqual(camera.data.output["depth"].dtype, torch.float)
del camera del camera
...@@ -303,8 +306,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -303,8 +306,9 @@ class TestTiledCamera(unittest.TestCase):
def test_rgba_only_camera(self): def test_rgba_only_camera(self):
"""Test initialization with only RGBA.""" """Test initialization with only RGBA."""
prim_utils.create_prim("/World/Origin_00", "Xform") num_cameras = 4
prim_utils.create_prim("/World/Origin_01", "Xform") for i in range(num_cameras):
prim_utils.create_prim(f"/World/Origin_{i}", "Xform")
# Create camera # Create camera
camera_cfg = copy.deepcopy(self.camera_cfg) camera_cfg = copy.deepcopy(self.camera_cfg)
...@@ -318,7 +322,7 @@ class TestTiledCamera(unittest.TestCase): ...@@ -318,7 +322,7 @@ class TestTiledCamera(unittest.TestCase):
# Check if camera is initialized # Check if camera is initialized
self.assertTrue(camera.is_initialized) self.assertTrue(camera.is_initialized)
# Check if camera prim is set correctly and that it is a camera prim # Check if camera prim is set correctly and that it is a camera prim
self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_01/CameraSensor") self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_1/CameraSensor")
self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera) self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera)
self.assertListEqual(list(camera.data.output.keys()), ["rgba"]) self.assertListEqual(list(camera.data.output.keys()), ["rgba"])
...@@ -329,11 +333,11 @@ class TestTiledCamera(unittest.TestCase): ...@@ -329,11 +333,11 @@ class TestTiledCamera(unittest.TestCase):
self.sim.step() self.sim.step()
# Check buffers that exists and have correct shapes # Check buffers that exists and have correct shapes
self.assertEqual(camera.data.pos_w.shape, (2, 3)) self.assertEqual(camera.data.pos_w.shape, (num_cameras, 3))
self.assertEqual(camera.data.quat_w_ros.shape, (2, 4)) self.assertEqual(camera.data.quat_w_ros.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_world.shape, (2, 4)) self.assertEqual(camera.data.quat_w_world.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_opengl.shape, (2, 4)) self.assertEqual(camera.data.quat_w_opengl.shape, (num_cameras, 4))
self.assertEqual(camera.data.intrinsic_matrices.shape, (2, 3, 3)) self.assertEqual(camera.data.intrinsic_matrices.shape, (num_cameras, 3, 3))
self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width)) self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width))
# Simulate physics # Simulate physics
...@@ -344,9 +348,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -344,9 +348,9 @@ class TestTiledCamera(unittest.TestCase):
camera.update(self.dt) camera.update(self.dt)
# check image data # check image data
for _, im_data in camera.data.output.to_dict().items(): for _, im_data in camera.data.output.to_dict().items():
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 4)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 4))
self.assertGreater((im_data[0] / 255.0).mean().item(), 0.0) for i in range(4):
self.assertGreater((im_data[1] / 255.0).mean().item(), 0.0) self.assertGreater((im_data[i] / 255.0).mean().item(), 0.0)
# Check data type of image # Check data type of image
self.assertEqual(camera.data.output["rgba"].dtype, torch.uint8) self.assertEqual(camera.data.output["rgba"].dtype, torch.uint8)
del camera del camera
...@@ -354,8 +358,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -354,8 +358,9 @@ class TestTiledCamera(unittest.TestCase):
def test_distance_to_camera_only_camera(self): def test_distance_to_camera_only_camera(self):
"""Test initialization with only distance_to_camera.""" """Test initialization with only distance_to_camera."""
prim_utils.create_prim("/World/Origin_00", "Xform") num_cameras = 4
prim_utils.create_prim("/World/Origin_01", "Xform") for i in range(num_cameras):
prim_utils.create_prim(f"/World/Origin_{i}", "Xform")
# Create camera # Create camera
camera_cfg = copy.deepcopy(self.camera_cfg) camera_cfg = copy.deepcopy(self.camera_cfg)
...@@ -369,7 +374,7 @@ class TestTiledCamera(unittest.TestCase): ...@@ -369,7 +374,7 @@ class TestTiledCamera(unittest.TestCase):
# Check if camera is initialized # Check if camera is initialized
self.assertTrue(camera.is_initialized) self.assertTrue(camera.is_initialized)
# Check if camera prim is set correctly and that it is a camera prim # Check if camera prim is set correctly and that it is a camera prim
self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_01/CameraSensor") self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_1/CameraSensor")
self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera) self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera)
self.assertListEqual(list(camera.data.output.keys()), ["distance_to_camera"]) self.assertListEqual(list(camera.data.output.keys()), ["distance_to_camera"])
...@@ -380,11 +385,11 @@ class TestTiledCamera(unittest.TestCase): ...@@ -380,11 +385,11 @@ class TestTiledCamera(unittest.TestCase):
self.sim.step() self.sim.step()
# Check buffers that exists and have correct shapes # Check buffers that exists and have correct shapes
self.assertEqual(camera.data.pos_w.shape, (2, 3)) self.assertEqual(camera.data.pos_w.shape, (num_cameras, 3))
self.assertEqual(camera.data.quat_w_ros.shape, (2, 4)) self.assertEqual(camera.data.quat_w_ros.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_world.shape, (2, 4)) self.assertEqual(camera.data.quat_w_world.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_opengl.shape, (2, 4)) self.assertEqual(camera.data.quat_w_opengl.shape, (num_cameras, 4))
self.assertEqual(camera.data.intrinsic_matrices.shape, (2, 3, 3)) self.assertEqual(camera.data.intrinsic_matrices.shape, (num_cameras, 3, 3))
self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width)) self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width))
# Simulate physics # Simulate physics
...@@ -395,9 +400,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -395,9 +400,9 @@ class TestTiledCamera(unittest.TestCase):
camera.update(self.dt) camera.update(self.dt)
# check image data # check image data
for _, im_data in camera.data.output.to_dict().items(): for _, im_data in camera.data.output.to_dict().items():
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 1)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 1))
self.assertGreater(im_data[0].mean().item(), 0.0) for i in range(4):
self.assertGreater(im_data[1].mean().item(), 0.0) self.assertGreater((im_data[i]).mean().item(), 0.0)
# Check data type of image # Check data type of image
self.assertEqual(camera.data.output["distance_to_camera"].dtype, torch.float) self.assertEqual(camera.data.output["distance_to_camera"].dtype, torch.float)
del camera del camera
...@@ -405,8 +410,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -405,8 +410,9 @@ class TestTiledCamera(unittest.TestCase):
def test_distance_to_image_plane_only_camera(self): def test_distance_to_image_plane_only_camera(self):
"""Test initialization with only distance_to_image_plane.""" """Test initialization with only distance_to_image_plane."""
prim_utils.create_prim("/World/Origin_00", "Xform") num_cameras = 4
prim_utils.create_prim("/World/Origin_01", "Xform") for i in range(num_cameras):
prim_utils.create_prim(f"/World/Origin_{i}", "Xform")
# Create camera # Create camera
camera_cfg = copy.deepcopy(self.camera_cfg) camera_cfg = copy.deepcopy(self.camera_cfg)
...@@ -420,7 +426,7 @@ class TestTiledCamera(unittest.TestCase): ...@@ -420,7 +426,7 @@ class TestTiledCamera(unittest.TestCase):
# Check if camera is initialized # Check if camera is initialized
self.assertTrue(camera.is_initialized) self.assertTrue(camera.is_initialized)
# Check if camera prim is set correctly and that it is a camera prim # Check if camera prim is set correctly and that it is a camera prim
self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_01/CameraSensor") self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_1/CameraSensor")
self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera) self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera)
self.assertListEqual(list(camera.data.output.keys()), ["distance_to_image_plane"]) self.assertListEqual(list(camera.data.output.keys()), ["distance_to_image_plane"])
...@@ -431,11 +437,11 @@ class TestTiledCamera(unittest.TestCase): ...@@ -431,11 +437,11 @@ class TestTiledCamera(unittest.TestCase):
self.sim.step() self.sim.step()
# Check buffers that exists and have correct shapes # Check buffers that exists and have correct shapes
self.assertEqual(camera.data.pos_w.shape, (2, 3)) self.assertEqual(camera.data.pos_w.shape, (num_cameras, 3))
self.assertEqual(camera.data.quat_w_ros.shape, (2, 4)) self.assertEqual(camera.data.quat_w_ros.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_world.shape, (2, 4)) self.assertEqual(camera.data.quat_w_world.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_opengl.shape, (2, 4)) self.assertEqual(camera.data.quat_w_opengl.shape, (num_cameras, 4))
self.assertEqual(camera.data.intrinsic_matrices.shape, (2, 3, 3)) self.assertEqual(camera.data.intrinsic_matrices.shape, (num_cameras, 3, 3))
self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width)) self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width))
# Simulate physics # Simulate physics
...@@ -446,9 +452,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -446,9 +452,9 @@ class TestTiledCamera(unittest.TestCase):
camera.update(self.dt) camera.update(self.dt)
# check image data # check image data
for _, im_data in camera.data.output.to_dict().items(): for _, im_data in camera.data.output.to_dict().items():
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 1)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 1))
self.assertGreater(im_data[0].mean().item(), 0.0) for i in range(4):
self.assertGreater(im_data[1].mean().item(), 0.0) self.assertGreater((im_data[i]).mean().item(), 0.0)
# Check data type of image # Check data type of image
self.assertEqual(camera.data.output["distance_to_image_plane"].dtype, torch.float) self.assertEqual(camera.data.output["distance_to_image_plane"].dtype, torch.float)
del camera del camera
...@@ -456,8 +462,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -456,8 +462,9 @@ class TestTiledCamera(unittest.TestCase):
def test_normals_only_camera(self): def test_normals_only_camera(self):
"""Test initialization with only normals.""" """Test initialization with only normals."""
prim_utils.create_prim("/World/Origin_00", "Xform") num_cameras = 4
prim_utils.create_prim("/World/Origin_01", "Xform") for i in range(num_cameras):
prim_utils.create_prim(f"/World/Origin_{i}", "Xform")
# Create camera # Create camera
camera_cfg = copy.deepcopy(self.camera_cfg) camera_cfg = copy.deepcopy(self.camera_cfg)
...@@ -471,7 +478,7 @@ class TestTiledCamera(unittest.TestCase): ...@@ -471,7 +478,7 @@ class TestTiledCamera(unittest.TestCase):
# Check if camera is initialized # Check if camera is initialized
self.assertTrue(camera.is_initialized) self.assertTrue(camera.is_initialized)
# Check if camera prim is set correctly and that it is a camera prim # Check if camera prim is set correctly and that it is a camera prim
self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_01/CameraSensor") self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_1/CameraSensor")
self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera) self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera)
self.assertListEqual(list(camera.data.output.keys()), ["normals"]) self.assertListEqual(list(camera.data.output.keys()), ["normals"])
...@@ -482,11 +489,11 @@ class TestTiledCamera(unittest.TestCase): ...@@ -482,11 +489,11 @@ class TestTiledCamera(unittest.TestCase):
self.sim.step() self.sim.step()
# Check buffers that exists and have correct shapes # Check buffers that exists and have correct shapes
self.assertEqual(camera.data.pos_w.shape, (2, 3)) self.assertEqual(camera.data.pos_w.shape, (num_cameras, 3))
self.assertEqual(camera.data.quat_w_ros.shape, (2, 4)) self.assertEqual(camera.data.quat_w_ros.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_world.shape, (2, 4)) self.assertEqual(camera.data.quat_w_world.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_opengl.shape, (2, 4)) self.assertEqual(camera.data.quat_w_opengl.shape, (num_cameras, 4))
self.assertEqual(camera.data.intrinsic_matrices.shape, (2, 3, 3)) self.assertEqual(camera.data.intrinsic_matrices.shape, (num_cameras, 3, 3))
self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width)) self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width))
# Simulate physics # Simulate physics
...@@ -497,9 +504,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -497,9 +504,9 @@ class TestTiledCamera(unittest.TestCase):
camera.update(self.dt) camera.update(self.dt)
# check image data # check image data
for _, im_data in camera.data.output.to_dict().items(): for _, im_data in camera.data.output.to_dict().items():
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 3)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 3))
self.assertGreater(im_data[0].mean().item(), 0.0) for i in range(4):
self.assertGreater(im_data[1].mean().item(), 0.0) self.assertGreater((im_data[i]).mean().item(), 0.0)
# Check data type of image # Check data type of image
self.assertEqual(camera.data.output["normals"].dtype, torch.float) self.assertEqual(camera.data.output["normals"].dtype, torch.float)
del camera del camera
...@@ -507,8 +514,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -507,8 +514,9 @@ class TestTiledCamera(unittest.TestCase):
def test_motion_vectors_only_camera(self): def test_motion_vectors_only_camera(self):
"""Test initialization with only motion_vectors.""" """Test initialization with only motion_vectors."""
prim_utils.create_prim("/World/Origin_00", "Xform") num_cameras = 4
prim_utils.create_prim("/World/Origin_01", "Xform") for i in range(num_cameras):
prim_utils.create_prim(f"/World/Origin_{i}", "Xform")
# Create camera # Create camera
camera_cfg = copy.deepcopy(self.camera_cfg) camera_cfg = copy.deepcopy(self.camera_cfg)
...@@ -522,7 +530,7 @@ class TestTiledCamera(unittest.TestCase): ...@@ -522,7 +530,7 @@ class TestTiledCamera(unittest.TestCase):
# Check if camera is initialized # Check if camera is initialized
self.assertTrue(camera.is_initialized) self.assertTrue(camera.is_initialized)
# Check if camera prim is set correctly and that it is a camera prim # Check if camera prim is set correctly and that it is a camera prim
self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_01/CameraSensor") self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_1/CameraSensor")
self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera) self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera)
self.assertListEqual(list(camera.data.output.keys()), ["motion_vectors"]) self.assertListEqual(list(camera.data.output.keys()), ["motion_vectors"])
...@@ -533,11 +541,11 @@ class TestTiledCamera(unittest.TestCase): ...@@ -533,11 +541,11 @@ class TestTiledCamera(unittest.TestCase):
self.sim.step() self.sim.step()
# Check buffers that exists and have correct shapes # Check buffers that exists and have correct shapes
self.assertEqual(camera.data.pos_w.shape, (2, 3)) self.assertEqual(camera.data.pos_w.shape, (num_cameras, 3))
self.assertEqual(camera.data.quat_w_ros.shape, (2, 4)) self.assertEqual(camera.data.quat_w_ros.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_world.shape, (2, 4)) self.assertEqual(camera.data.quat_w_world.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_opengl.shape, (2, 4)) self.assertEqual(camera.data.quat_w_opengl.shape, (num_cameras, 4))
self.assertEqual(camera.data.intrinsic_matrices.shape, (2, 3, 3)) self.assertEqual(camera.data.intrinsic_matrices.shape, (num_cameras, 3, 3))
self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width)) self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width))
# Simulate physics # Simulate physics
...@@ -548,9 +556,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -548,9 +556,9 @@ class TestTiledCamera(unittest.TestCase):
camera.update(self.dt) camera.update(self.dt)
# check image data # check image data
for _, im_data in camera.data.output.to_dict().items(): for _, im_data in camera.data.output.to_dict().items():
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 2)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 2))
self.assertGreater(im_data[0].mean().item(), 0.0) for i in range(4):
self.assertGreater(im_data[1].mean().item(), 0.0) self.assertGreater((im_data[i]).mean().item(), 0.0)
# Check data type of image # Check data type of image
self.assertEqual(camera.data.output["motion_vectors"].dtype, torch.float) self.assertEqual(camera.data.output["motion_vectors"].dtype, torch.float)
del camera del camera
...@@ -558,8 +566,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -558,8 +566,9 @@ class TestTiledCamera(unittest.TestCase):
def test_semantic_segmentation_colorize_only_camera(self): def test_semantic_segmentation_colorize_only_camera(self):
"""Test initialization with only semantic_segmentation.""" """Test initialization with only semantic_segmentation."""
prim_utils.create_prim("/World/Origin_00", "Xform") num_cameras = 4
prim_utils.create_prim("/World/Origin_01", "Xform") for i in range(num_cameras):
prim_utils.create_prim(f"/World/Origin_{i}", "Xform")
# Create camera # Create camera
camera_cfg = copy.deepcopy(self.camera_cfg) camera_cfg = copy.deepcopy(self.camera_cfg)
...@@ -573,7 +582,7 @@ class TestTiledCamera(unittest.TestCase): ...@@ -573,7 +582,7 @@ class TestTiledCamera(unittest.TestCase):
# Check if camera is initialized # Check if camera is initialized
self.assertTrue(camera.is_initialized) self.assertTrue(camera.is_initialized)
# Check if camera prim is set correctly and that it is a camera prim # Check if camera prim is set correctly and that it is a camera prim
self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_01/CameraSensor") self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_1/CameraSensor")
self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera) self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera)
self.assertListEqual(list(camera.data.output.keys()), ["semantic_segmentation"]) self.assertListEqual(list(camera.data.output.keys()), ["semantic_segmentation"])
...@@ -584,11 +593,11 @@ class TestTiledCamera(unittest.TestCase): ...@@ -584,11 +593,11 @@ class TestTiledCamera(unittest.TestCase):
self.sim.step() self.sim.step()
# Check buffers that exists and have correct shapes # Check buffers that exists and have correct shapes
self.assertEqual(camera.data.pos_w.shape, (2, 3)) self.assertEqual(camera.data.pos_w.shape, (num_cameras, 3))
self.assertEqual(camera.data.quat_w_ros.shape, (2, 4)) self.assertEqual(camera.data.quat_w_ros.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_world.shape, (2, 4)) self.assertEqual(camera.data.quat_w_world.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_opengl.shape, (2, 4)) self.assertEqual(camera.data.quat_w_opengl.shape, (num_cameras, 4))
self.assertEqual(camera.data.intrinsic_matrices.shape, (2, 3, 3)) self.assertEqual(camera.data.intrinsic_matrices.shape, (num_cameras, 3, 3))
self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width)) self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width))
# Simulate physics # Simulate physics
...@@ -599,9 +608,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -599,9 +608,9 @@ class TestTiledCamera(unittest.TestCase):
camera.update(self.dt) camera.update(self.dt)
# check image data # check image data
for _, im_data in camera.data.output.to_dict().items(): for _, im_data in camera.data.output.to_dict().items():
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 4)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 4))
self.assertGreater((im_data[0] / 255.0).mean().item(), 0.0) for i in range(4):
self.assertGreater((im_data[1] / 255.0).mean().item(), 0.0) self.assertGreater((im_data[i] / 255.0).mean().item(), 0.0)
# Check data type of image and info # Check data type of image and info
self.assertEqual(camera.data.output["semantic_segmentation"].dtype, torch.uint8) self.assertEqual(camera.data.output["semantic_segmentation"].dtype, torch.uint8)
self.assertEqual(type(camera.data.info["semantic_segmentation"]), dict) self.assertEqual(type(camera.data.info["semantic_segmentation"]), dict)
...@@ -610,8 +619,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -610,8 +619,9 @@ class TestTiledCamera(unittest.TestCase):
def test_instance_segmentation_fast_colorize_only_camera(self): def test_instance_segmentation_fast_colorize_only_camera(self):
"""Test initialization with only instance_segmentation_fast.""" """Test initialization with only instance_segmentation_fast."""
prim_utils.create_prim("/World/Origin_00", "Xform") num_cameras = 4
prim_utils.create_prim("/World/Origin_01", "Xform") for i in range(num_cameras):
prim_utils.create_prim(f"/World/Origin_{i}", "Xform")
# Create camera # Create camera
camera_cfg = copy.deepcopy(self.camera_cfg) camera_cfg = copy.deepcopy(self.camera_cfg)
...@@ -625,7 +635,7 @@ class TestTiledCamera(unittest.TestCase): ...@@ -625,7 +635,7 @@ class TestTiledCamera(unittest.TestCase):
# Check if camera is initialized # Check if camera is initialized
self.assertTrue(camera.is_initialized) self.assertTrue(camera.is_initialized)
# Check if camera prim is set correctly and that it is a camera prim # Check if camera prim is set correctly and that it is a camera prim
self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_01/CameraSensor") self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_1/CameraSensor")
self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera) self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera)
self.assertListEqual(list(camera.data.output.keys()), ["instance_segmentation_fast"]) self.assertListEqual(list(camera.data.output.keys()), ["instance_segmentation_fast"])
...@@ -636,11 +646,11 @@ class TestTiledCamera(unittest.TestCase): ...@@ -636,11 +646,11 @@ class TestTiledCamera(unittest.TestCase):
self.sim.step() self.sim.step()
# Check buffers that exists and have correct shapes # Check buffers that exists and have correct shapes
self.assertEqual(camera.data.pos_w.shape, (2, 3)) self.assertEqual(camera.data.pos_w.shape, (num_cameras, 3))
self.assertEqual(camera.data.quat_w_ros.shape, (2, 4)) self.assertEqual(camera.data.quat_w_ros.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_world.shape, (2, 4)) self.assertEqual(camera.data.quat_w_world.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_opengl.shape, (2, 4)) self.assertEqual(camera.data.quat_w_opengl.shape, (num_cameras, 4))
self.assertEqual(camera.data.intrinsic_matrices.shape, (2, 3, 3)) self.assertEqual(camera.data.intrinsic_matrices.shape, (num_cameras, 3, 3))
self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width)) self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width))
# Simulate physics # Simulate physics
...@@ -651,9 +661,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -651,9 +661,9 @@ class TestTiledCamera(unittest.TestCase):
camera.update(self.dt) camera.update(self.dt)
# check image data # check image data
for _, im_data in camera.data.output.to_dict().items(): for _, im_data in camera.data.output.to_dict().items():
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 4)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 4))
self.assertGreater((im_data[0] / 255.0).mean().item(), 0.0) for i in range(num_cameras):
self.assertGreater((im_data[1] / 255.0).mean().item(), 0.0) self.assertGreater((im_data[i] / 255.0).mean().item(), 0.0)
# Check data type of image and info # Check data type of image and info
self.assertEqual(camera.data.output["instance_segmentation_fast"].dtype, torch.uint8) self.assertEqual(camera.data.output["instance_segmentation_fast"].dtype, torch.uint8)
self.assertEqual(type(camera.data.info["instance_segmentation_fast"]), dict) self.assertEqual(type(camera.data.info["instance_segmentation_fast"]), dict)
...@@ -662,8 +672,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -662,8 +672,9 @@ class TestTiledCamera(unittest.TestCase):
def test_instance_id_segmentation_fast_colorize_only_camera(self): def test_instance_id_segmentation_fast_colorize_only_camera(self):
"""Test initialization with only instance_id_segmentation_fast.""" """Test initialization with only instance_id_segmentation_fast."""
prim_utils.create_prim("/World/Origin_00", "Xform") num_cameras = 4
prim_utils.create_prim("/World/Origin_01", "Xform") for i in range(num_cameras):
prim_utils.create_prim(f"/World/Origin_{i}", "Xform")
# Create camera # Create camera
camera_cfg = copy.deepcopy(self.camera_cfg) camera_cfg = copy.deepcopy(self.camera_cfg)
...@@ -677,7 +688,7 @@ class TestTiledCamera(unittest.TestCase): ...@@ -677,7 +688,7 @@ class TestTiledCamera(unittest.TestCase):
# Check if camera is initialized # Check if camera is initialized
self.assertTrue(camera.is_initialized) self.assertTrue(camera.is_initialized)
# Check if camera prim is set correctly and that it is a camera prim # Check if camera prim is set correctly and that it is a camera prim
self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_01/CameraSensor") self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_1/CameraSensor")
self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera) self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera)
self.assertListEqual(list(camera.data.output.keys()), ["instance_id_segmentation_fast"]) self.assertListEqual(list(camera.data.output.keys()), ["instance_id_segmentation_fast"])
...@@ -688,11 +699,11 @@ class TestTiledCamera(unittest.TestCase): ...@@ -688,11 +699,11 @@ class TestTiledCamera(unittest.TestCase):
self.sim.step() self.sim.step()
# Check buffers that exists and have correct shapes # Check buffers that exists and have correct shapes
self.assertEqual(camera.data.pos_w.shape, (2, 3)) self.assertEqual(camera.data.pos_w.shape, (num_cameras, 3))
self.assertEqual(camera.data.quat_w_ros.shape, (2, 4)) self.assertEqual(camera.data.quat_w_ros.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_world.shape, (2, 4)) self.assertEqual(camera.data.quat_w_world.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_opengl.shape, (2, 4)) self.assertEqual(camera.data.quat_w_opengl.shape, (num_cameras, 4))
self.assertEqual(camera.data.intrinsic_matrices.shape, (2, 3, 3)) self.assertEqual(camera.data.intrinsic_matrices.shape, (num_cameras, 3, 3))
self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width)) self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width))
# Simulate physics # Simulate physics
...@@ -703,9 +714,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -703,9 +714,9 @@ class TestTiledCamera(unittest.TestCase):
camera.update(self.dt) camera.update(self.dt)
# check image data # check image data
for _, im_data in camera.data.output.to_dict().items(): for _, im_data in camera.data.output.to_dict().items():
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 4)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 4))
self.assertGreater((im_data[0] / 255.0).mean().item(), 0.0) for i in range(num_cameras):
self.assertGreater((im_data[1] / 255.0).mean().item(), 0.0) self.assertGreater((im_data[i] / 255.0).mean().item(), 0.0)
# Check data type of image and info # Check data type of image and info
self.assertEqual(camera.data.output["instance_id_segmentation_fast"].dtype, torch.uint8) self.assertEqual(camera.data.output["instance_id_segmentation_fast"].dtype, torch.uint8)
self.assertEqual(type(camera.data.info["instance_id_segmentation_fast"]), dict) self.assertEqual(type(camera.data.info["instance_id_segmentation_fast"]), dict)
...@@ -714,8 +725,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -714,8 +725,9 @@ class TestTiledCamera(unittest.TestCase):
def test_semantic_segmentation_non_colorize_only_camera(self): def test_semantic_segmentation_non_colorize_only_camera(self):
"""Test initialization with only semantic_segmentation.""" """Test initialization with only semantic_segmentation."""
prim_utils.create_prim("/World/Origin_00", "Xform") num_cameras = 4
prim_utils.create_prim("/World/Origin_01", "Xform") for i in range(num_cameras):
prim_utils.create_prim(f"/World/Origin_{i}", "Xform")
# Create camera # Create camera
camera_cfg = copy.deepcopy(self.camera_cfg) camera_cfg = copy.deepcopy(self.camera_cfg)
...@@ -730,7 +742,7 @@ class TestTiledCamera(unittest.TestCase): ...@@ -730,7 +742,7 @@ class TestTiledCamera(unittest.TestCase):
# Check if camera is initialized # Check if camera is initialized
self.assertTrue(camera.is_initialized) self.assertTrue(camera.is_initialized)
# Check if camera prim is set correctly and that it is a camera prim # Check if camera prim is set correctly and that it is a camera prim
self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_01/CameraSensor") self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_1/CameraSensor")
self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera) self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera)
self.assertListEqual(list(camera.data.output.keys()), ["semantic_segmentation"]) self.assertListEqual(list(camera.data.output.keys()), ["semantic_segmentation"])
...@@ -741,11 +753,11 @@ class TestTiledCamera(unittest.TestCase): ...@@ -741,11 +753,11 @@ class TestTiledCamera(unittest.TestCase):
self.sim.step() self.sim.step()
# Check buffers that exists and have correct shapes # Check buffers that exists and have correct shapes
self.assertEqual(camera.data.pos_w.shape, (2, 3)) self.assertEqual(camera.data.pos_w.shape, (num_cameras, 3))
self.assertEqual(camera.data.quat_w_ros.shape, (2, 4)) self.assertEqual(camera.data.quat_w_ros.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_world.shape, (2, 4)) self.assertEqual(camera.data.quat_w_world.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_opengl.shape, (2, 4)) self.assertEqual(camera.data.quat_w_opengl.shape, (num_cameras, 4))
self.assertEqual(camera.data.intrinsic_matrices.shape, (2, 3, 3)) self.assertEqual(camera.data.intrinsic_matrices.shape, (num_cameras, 3, 3))
self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width)) self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width))
# Simulate physics # Simulate physics
...@@ -756,9 +768,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -756,9 +768,9 @@ class TestTiledCamera(unittest.TestCase):
camera.update(self.dt) camera.update(self.dt)
# check image data # check image data
for _, im_data in camera.data.output.to_dict().items(): for _, im_data in camera.data.output.to_dict().items():
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 1)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 1))
self.assertGreater((im_data[0] / 255.0).mean().item(), 0.0) for i in range(num_cameras):
self.assertGreater((im_data[1] / 255.0).mean().item(), 0.0) self.assertGreater(im_data[i].to(dtype=float).mean().item(), 0.0)
# Check data type of image and info # Check data type of image and info
self.assertEqual(camera.data.output["semantic_segmentation"].dtype, torch.int32) self.assertEqual(camera.data.output["semantic_segmentation"].dtype, torch.int32)
self.assertEqual(type(camera.data.info["semantic_segmentation"]), dict) self.assertEqual(type(camera.data.info["semantic_segmentation"]), dict)
...@@ -768,8 +780,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -768,8 +780,9 @@ class TestTiledCamera(unittest.TestCase):
def test_instance_segmentation_fast_non_colorize_only_camera(self): def test_instance_segmentation_fast_non_colorize_only_camera(self):
"""Test initialization with only instance_segmentation_fast.""" """Test initialization with only instance_segmentation_fast."""
prim_utils.create_prim("/World/Origin_00", "Xform") num_cameras = 4
prim_utils.create_prim("/World/Origin_01", "Xform") for i in range(num_cameras):
prim_utils.create_prim(f"/World/Origin_{i}", "Xform")
# Create camera # Create camera
camera_cfg = copy.deepcopy(self.camera_cfg) camera_cfg = copy.deepcopy(self.camera_cfg)
...@@ -784,7 +797,7 @@ class TestTiledCamera(unittest.TestCase): ...@@ -784,7 +797,7 @@ class TestTiledCamera(unittest.TestCase):
# Check if camera is initialized # Check if camera is initialized
self.assertTrue(camera.is_initialized) self.assertTrue(camera.is_initialized)
# Check if camera prim is set correctly and that it is a camera prim # Check if camera prim is set correctly and that it is a camera prim
self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_01/CameraSensor") self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_1/CameraSensor")
self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera) self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera)
self.assertListEqual(list(camera.data.output.keys()), ["instance_segmentation_fast"]) self.assertListEqual(list(camera.data.output.keys()), ["instance_segmentation_fast"])
...@@ -795,11 +808,11 @@ class TestTiledCamera(unittest.TestCase): ...@@ -795,11 +808,11 @@ class TestTiledCamera(unittest.TestCase):
self.sim.step() self.sim.step()
# Check buffers that exists and have correct shapes # Check buffers that exists and have correct shapes
self.assertEqual(camera.data.pos_w.shape, (2, 3)) self.assertEqual(camera.data.pos_w.shape, (num_cameras, 3))
self.assertEqual(camera.data.quat_w_ros.shape, (2, 4)) self.assertEqual(camera.data.quat_w_ros.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_world.shape, (2, 4)) self.assertEqual(camera.data.quat_w_world.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_opengl.shape, (2, 4)) self.assertEqual(camera.data.quat_w_opengl.shape, (num_cameras, 4))
self.assertEqual(camera.data.intrinsic_matrices.shape, (2, 3, 3)) self.assertEqual(camera.data.intrinsic_matrices.shape, (num_cameras, 3, 3))
self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width)) self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width))
# Simulate physics # Simulate physics
...@@ -810,9 +823,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -810,9 +823,9 @@ class TestTiledCamera(unittest.TestCase):
camera.update(self.dt) camera.update(self.dt)
# check image data # check image data
for _, im_data in camera.data.output.to_dict().items(): for _, im_data in camera.data.output.to_dict().items():
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 1)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 1))
self.assertGreater((im_data[0] / 255.0).mean().item(), 0.0) for i in range(num_cameras):
self.assertGreater((im_data[1] / 255.0).mean().item(), 0.0) self.assertGreater(im_data[i].to(dtype=float).mean().item(), 0.0)
# Check data type of image and info # Check data type of image and info
self.assertEqual(camera.data.output["instance_segmentation_fast"].dtype, torch.int32) self.assertEqual(camera.data.output["instance_segmentation_fast"].dtype, torch.int32)
self.assertEqual(type(camera.data.info["instance_segmentation_fast"]), dict) self.assertEqual(type(camera.data.info["instance_segmentation_fast"]), dict)
...@@ -821,8 +834,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -821,8 +834,9 @@ class TestTiledCamera(unittest.TestCase):
def test_instance_id_segmentation_fast_non_colorize_only_camera(self): def test_instance_id_segmentation_fast_non_colorize_only_camera(self):
"""Test initialization with only instance_id_segmentation_fast.""" """Test initialization with only instance_id_segmentation_fast."""
prim_utils.create_prim("/World/Origin_00", "Xform") num_cameras = 4
prim_utils.create_prim("/World/Origin_01", "Xform") for i in range(num_cameras):
prim_utils.create_prim(f"/World/Origin_{i}", "Xform")
# Create camera # Create camera
camera_cfg = copy.deepcopy(self.camera_cfg) camera_cfg = copy.deepcopy(self.camera_cfg)
...@@ -837,7 +851,7 @@ class TestTiledCamera(unittest.TestCase): ...@@ -837,7 +851,7 @@ class TestTiledCamera(unittest.TestCase):
# Check if camera is initialized # Check if camera is initialized
self.assertTrue(camera.is_initialized) self.assertTrue(camera.is_initialized)
# Check if camera prim is set correctly and that it is a camera prim # Check if camera prim is set correctly and that it is a camera prim
self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_01/CameraSensor") self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_1/CameraSensor")
self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera) self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera)
self.assertListEqual(list(camera.data.output.keys()), ["instance_id_segmentation_fast"]) self.assertListEqual(list(camera.data.output.keys()), ["instance_id_segmentation_fast"])
...@@ -848,11 +862,11 @@ class TestTiledCamera(unittest.TestCase): ...@@ -848,11 +862,11 @@ class TestTiledCamera(unittest.TestCase):
self.sim.step() self.sim.step()
# Check buffers that exists and have correct shapes # Check buffers that exists and have correct shapes
self.assertEqual(camera.data.pos_w.shape, (2, 3)) self.assertEqual(camera.data.pos_w.shape, (num_cameras, 3))
self.assertEqual(camera.data.quat_w_ros.shape, (2, 4)) self.assertEqual(camera.data.quat_w_ros.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_world.shape, (2, 4)) self.assertEqual(camera.data.quat_w_world.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_opengl.shape, (2, 4)) self.assertEqual(camera.data.quat_w_opengl.shape, (num_cameras, 4))
self.assertEqual(camera.data.intrinsic_matrices.shape, (2, 3, 3)) self.assertEqual(camera.data.intrinsic_matrices.shape, (num_cameras, 3, 3))
self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width)) self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width))
# Simulate physics # Simulate physics
...@@ -863,9 +877,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -863,9 +877,9 @@ class TestTiledCamera(unittest.TestCase):
camera.update(self.dt) camera.update(self.dt)
# check image data # check image data
for _, im_data in camera.data.output.to_dict().items(): for _, im_data in camera.data.output.to_dict().items():
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 1)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 1))
self.assertGreater((im_data[0] / 255.0).mean().item(), 0.0) for i in range(num_cameras):
self.assertGreater((im_data[1] / 255.0).mean().item(), 0.0) self.assertGreater(im_data[i].to(dtype=float).mean().item(), 0.0)
# Check data type of image and info # Check data type of image and info
self.assertEqual(camera.data.output["instance_id_segmentation_fast"].dtype, torch.int32) self.assertEqual(camera.data.output["instance_id_segmentation_fast"].dtype, torch.int32)
self.assertEqual(type(camera.data.info["instance_id_segmentation_fast"]), dict) self.assertEqual(type(camera.data.info["instance_id_segmentation_fast"]), dict)
...@@ -886,8 +900,9 @@ class TestTiledCamera(unittest.TestCase): ...@@ -886,8 +900,9 @@ class TestTiledCamera(unittest.TestCase):
"instance_id_segmentation_fast", "instance_id_segmentation_fast",
] ]
prim_utils.create_prim("/World/Origin_00", "Xform") num_cameras = 4
prim_utils.create_prim("/World/Origin_01", "Xform") for i in range(num_cameras):
prim_utils.create_prim(f"/World/Origin_{i}", "Xform")
# Create camera # Create camera
camera_cfg = copy.deepcopy(self.camera_cfg) camera_cfg = copy.deepcopy(self.camera_cfg)
...@@ -901,7 +916,7 @@ class TestTiledCamera(unittest.TestCase): ...@@ -901,7 +916,7 @@ class TestTiledCamera(unittest.TestCase):
# Check if camera is initialized # Check if camera is initialized
self.assertTrue(camera.is_initialized) self.assertTrue(camera.is_initialized)
# Check if camera prim is set correctly and that it is a camera prim # Check if camera prim is set correctly and that it is a camera prim
self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_01/CameraSensor") self.assertEqual(camera._sensor_prims[1].GetPath().pathString, "/World/Origin_1/CameraSensor")
self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera) self.assertIsInstance(camera._sensor_prims[0], UsdGeom.Camera)
self.assertListEqual(sorted(camera.data.output.keys()), sorted(all_annotator_types)) self.assertListEqual(sorted(camera.data.output.keys()), sorted(all_annotator_types))
...@@ -912,11 +927,11 @@ class TestTiledCamera(unittest.TestCase): ...@@ -912,11 +927,11 @@ class TestTiledCamera(unittest.TestCase):
self.sim.step() self.sim.step()
# Check buffers that exists and have correct shapes # Check buffers that exists and have correct shapes
self.assertEqual(camera.data.pos_w.shape, (2, 3)) self.assertEqual(camera.data.pos_w.shape, (num_cameras, 3))
self.assertEqual(camera.data.quat_w_ros.shape, (2, 4)) self.assertEqual(camera.data.quat_w_ros.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_world.shape, (2, 4)) self.assertEqual(camera.data.quat_w_world.shape, (num_cameras, 4))
self.assertEqual(camera.data.quat_w_opengl.shape, (2, 4)) self.assertEqual(camera.data.quat_w_opengl.shape, (num_cameras, 4))
self.assertEqual(camera.data.intrinsic_matrices.shape, (2, 3, 3)) self.assertEqual(camera.data.intrinsic_matrices.shape, (num_cameras, 3, 3))
self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width)) self.assertEqual(camera.data.image_shape, (self.camera_cfg.height, self.camera_cfg.width))
# Simulate physics # Simulate physics
...@@ -928,24 +943,24 @@ class TestTiledCamera(unittest.TestCase): ...@@ -928,24 +943,24 @@ class TestTiledCamera(unittest.TestCase):
# check image data # check image data
for data_type, im_data in camera.data.output.to_dict().items(): for data_type, im_data in camera.data.output.to_dict().items():
if data_type in ["rgb", "normals"]: if data_type in ["rgb", "normals"]:
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 3)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 3))
elif data_type in [ elif data_type in [
"rgba", "rgba",
"semantic_segmentation", "semantic_segmentation",
"instance_segmentation_fast", "instance_segmentation_fast",
"instance_id_segmentation_fast", "instance_id_segmentation_fast",
]: ]:
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 4)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 4))
self.assertGreater((im_data[0] / 255.0).mean().item(), 0.0) for i in range(num_cameras):
self.assertGreater((im_data[1] / 255.0).mean().item(), 0.0) self.assertGreater((im_data[i] / 255.0).mean().item(), 0.0)
elif data_type in ["motion_vectors"]: elif data_type in ["motion_vectors"]:
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 2)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 2))
self.assertGreater(im_data[0].mean().item(), 0.0) for i in range(num_cameras):
self.assertGreater(im_data[1].mean().item(), 0.0) self.assertGreater(im_data[i].mean().item(), 0.0)
elif data_type in ["depth", "distance_to_camera", "distance_to_image_plane"]: elif data_type in ["depth", "distance_to_camera", "distance_to_image_plane"]:
self.assertEqual(im_data.shape, (2, self.camera_cfg.height, self.camera_cfg.width, 1)) self.assertEqual(im_data.shape, (num_cameras, self.camera_cfg.height, self.camera_cfg.width, 1))
self.assertGreater(im_data[0].mean().item(), 0.0) for i in range(num_cameras):
self.assertGreater(im_data[1].mean().item(), 0.0) self.assertGreater(im_data[i].mean().item(), 0.0)
# access image data and compare dtype # access image data and compare dtype
output = camera.data.output output = camera.data.output
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment