Commit 66665928 authored by shauryadNv's avatar shauryadNv Committed by Kelly Guo

Fixes documentation inconsistencies for Mimic-Cosmos (#502)

# Description

Fixed some inconsistencies in the Mimic-Cosmos doc. Also fixed a minor
bug in the evaluation script.

## Type of change

- Bug fix (non-breaking change which fixes an issue)
- This change requires a documentation update

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

---------
Signed-off-by: 's avatarrwiltz <165190220+rwiltz@users.noreply.github.com>
Signed-off-by: 's avatarKelly Guo <kellyguo123@hotmail.com>
Signed-off-by: 's avatarAshwin Varghese Kuruttukulam <123109010+ashwinvkNV@users.noreply.github.com>
Signed-off-by: 's avatarKelly Guo <kellyg@nvidia.com>
Signed-off-by: 's avatarMichael Gussert <michael@gussert.com>
Signed-off-by: 's avatarsamibouziri <79418773+samibouziri@users.noreply.github.com>
Signed-off-by: 's avatarMayank Mittal <12863862+Mayankm96@users.noreply.github.com>
Signed-off-by: 's avatarKyle Morgenstein <34984693+KyleM73@users.noreply.github.com>
Signed-off-by: 's avatarHongyu Li <lihongyu0807@icloud.com>
Signed-off-by: 's avatarToni-SM <toni.semu@gmail.com>
Signed-off-by: 's avatarJames Tigue <166445701+jtigue-bdai@users.noreply.github.com>
Signed-off-by: 's avatarPascal Roth <57946385+pascal-roth@users.noreply.github.com>
Signed-off-by: 's avatarVictor Khaustov <3192677+vi3itor@users.noreply.github.com>
Signed-off-by: 's avatarAlvinC <alvincny529@gmail.com>
Signed-off-by: 's avatarTyler Lum <tylergwlum@gmail.com>
Signed-off-by: 's avatarMiguel Alonso Jr. <76960110+miguelalonsojr@users.noreply.github.com>
Signed-off-by: 's avatarrenaudponcelet <renaud.poncelet@gmail.com>
Co-authored-by: 's avatarlotusl-code <lotusl@nvidia.com>
Co-authored-by: 's avatarKelly Guo <kellyg@nvidia.com>
Co-authored-by: 's avatarjaczhangnv <jaczhang@nvidia.com>
Co-authored-by: 's avatarrwiltz <165190220+rwiltz@users.noreply.github.com>
Co-authored-by: 's avatarYanzi Zhu <yanziz@nvidia.com>
Co-authored-by: 's avatarnv-mhaselton <mhaselton@nvidia.com>
Co-authored-by: 's avatarcosmith-nvidia <141183495+cosmith-nvidia@users.noreply.github.com>
Co-authored-by: 's avatarMichael Gussert <michael@gussert.com>
Co-authored-by: 's avatarCY Chen <cyc@nvidia.com>
Co-authored-by: 's avataroahmednv <oahmed@Nvidia.com>
Co-authored-by: 's avatarAshwin Varghese Kuruttukulam <123109010+ashwinvkNV@users.noreply.github.com>
Co-authored-by: 's avatarRafael Wiltz <rwiltz@nvidia.com>
Co-authored-by: 's avatarPeter Du <peterd@nvidia.com>
Co-authored-by: 's avatarmatthewtrepte <mtrepte@nvidia.com>
Co-authored-by: 's avatarchengronglai <chengrongl@nvidia.com>
Co-authored-by: 's avatarpulkitg01 <pulkitg@nvidia.com>
Co-authored-by: 's avatarConnor Smith <cosmith@nvidia.com>
Co-authored-by: 's avatarAshwin Varghese Kuruttukulam <ashwinvk@nvidia.com>
Co-authored-by: 's avatarKelly Guo <kellyguo123@hotmail.com>
Co-authored-by: 's avatarMayank Mittal <12863862+Mayankm96@users.noreply.github.com>
Co-authored-by: 's avatarsamibouziri <79418773+samibouziri@users.noreply.github.com>
Co-authored-by: 's avatarJames Smith <142246516+jsmith-bdai@users.noreply.github.com>
Co-authored-by: 's avatarShundo Kishi <syundo0730@gmail.com>
Co-authored-by: 's avatarSheikh Dawood <sabdulajees@nvidia.com>
Co-authored-by: 's avatarToni-SM <aserranomuno@nvidia.com>
Co-authored-by: 's avatarGonglitian <70052908+Gonglitian@users.noreply.github.com>
Co-authored-by: 's avatarJames Tigue <166445701+jtigue-bdai@users.noreply.github.com>
Co-authored-by: 's avatarMayank Mittal <mittalma@leggedrobotics.com>
Co-authored-by: 's avatarKyle Morgenstein <34984693+KyleM73@users.noreply.github.com>
Co-authored-by: 's avatarJohnson Sun <20457146+j3soon@users.noreply.github.com>
Co-authored-by: 's avatarPascal Roth <57946385+pascal-roth@users.noreply.github.com>
Co-authored-by: 's avatarHongyu Li <lihongyu0807@icloud.com>
Co-authored-by: 's avatarJean-Francois-Lafleche <57650687+Jean-Francois-Lafleche@users.noreply.github.com>
Co-authored-by: 's avatarWei Jinqi <changshanshi@outlook.com>
Co-authored-by: 's avatarLouis LE LAY <le.lay.louis@gmail.com>
Co-authored-by: 's avatarHarsh Patel <hapatel@theaiinstitute.com>
Co-authored-by: 's avatarKousheek Chakraborty <kousheekc@gmail.com>
Co-authored-by: 's avatarVictor Khaustov <3192677+vi3itor@users.noreply.github.com>
Co-authored-by: 's avatarAlvinC <alvincny529@gmail.com>
Co-authored-by: 's avatarFelipe Mohr <50018670+felipemohr@users.noreply.github.com>
Co-authored-by: 's avatarAdAstra7 <87345760+likecanyon@users.noreply.github.com>
Co-authored-by: 's avatargao <ziqi.gao@iff-extern.fraunhofer.de>
Co-authored-by: 's avatarTyler Lum <tylergwlum@gmail.com>
Co-authored-by: 's avatar-T.K.- <t_k_233@outlook.com>
Co-authored-by: 's avatarClemens Schwarke <96480707+ClemensSchwarke@users.noreply.github.com>
Co-authored-by: 's avatarMiguel Alonso Jr. <76960110+miguelalonsojr@users.noreply.github.com>
Co-authored-by: 's avatarMiguel Alonso Jr. <miguel.alonso@nfinite.app>
Co-authored-by: 's avatarrenaudponcelet <renaud.poncelet@gmail.com>
Co-authored-by: 's avatarAles Borovicka <aborovicka@nvidia.com>
Co-authored-by: 's avatarnv-mm <mmagdics@nvidia.com>
Co-authored-by: 's avatarAntoine RICHARD <antoiner@nvidia.com>
parent f01733c3
...@@ -29,7 +29,7 @@ In the following example, we will show you how to use Isaac Lab Mimic to generat ...@@ -29,7 +29,7 @@ In the following example, we will show you how to use Isaac Lab Mimic to generat
The number of demonstrations can be increased or decreased, 1000 demonstrations have been shown to provide good training results for this task. The number of demonstrations can be increased or decreased, 1000 demonstrations have been shown to provide good training results for this task.
Additionally, the number of environments in the ``--num_envs`` parameter can be adjusted to speed up data generation. Additionally, the number of environments in the ``--num_envs`` parameter can be adjusted to speed up data generation.
The suggested number of 10 can be executed on a moderate laptop GPU. The suggested number of 10 can be executed on a moderate laptop CPU.
On a more powerful desktop machine, use a larger number of environments for a significant speedup of this step. On a more powerful desktop machine, use a larger number of environments for a significant speedup of this step.
Cosmos Augmentation Cosmos Augmentation
...@@ -77,8 +77,8 @@ Example usage for the cube stacking task: ...@@ -77,8 +77,8 @@ Example usage for the cube stacking task:
.. code:: bash .. code:: bash
python scripts/tools/hdf5_to_mp4.py \ python scripts/tools/hdf5_to_mp4.py \
--input_file datasets/mimic_generated_dataset.hdf5 \ --input_file datasets/mimic_dataset_1k.hdf5 \
--output_dir datasets/mimic_generated_dataset_mp4 --output_dir datasets/mimic_dataset_1k_mp4
Running Cosmos for Visual Augmentation Running Cosmos for Visual Augmentation
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
...@@ -163,7 +163,7 @@ Example command to use the Cosmos Transfer1 model for this usecase: ...@@ -163,7 +163,7 @@ Example command to use the Cosmos Transfer1 model for this usecase:
export NUM_GPU="${NUM_GPU:=1}" export NUM_GPU="${NUM_GPU:=1}"
PYTHONPATH=$(pwd) torchrun --nproc_per_node=$NUM_GPU --nnodes=1 --node_rank=0 cosmos_transfer1/diffusion/inference/transfer.py \ PYTHONPATH=$(pwd) torchrun --nproc_per_node=$NUM_GPU --nnodes=1 --node_rank=0 cosmos_transfer1/diffusion/inference/transfer.py \
--checkpoint_dir $CHECKPOINT_DIR \ --checkpoint_dir $CHECKPOINT_DIR \
--video_save_folder outputs/mimic_cosmos \ --video_save_folder outputs/cosmos_dataset_1k_mp4 \
--controlnet_specs ./controlnet_specs/demo_0.json \ --controlnet_specs ./controlnet_specs/demo_0.json \
--offload_text_encoder_model \ --offload_text_encoder_model \
--offload_guardrail_models \ --offload_guardrail_models \
...@@ -176,10 +176,10 @@ Example ``./controlnet_specs/demo_0.json`` json file to use with the above comma ...@@ -176,10 +176,10 @@ Example ``./controlnet_specs/demo_0.json`` json file to use with the above comma
{ {
"prompt": "A robotic arm is picking up and stacking cubes inside a foggy industrial scrapyard at dawn, surrounded by piles of old robotic parts and twisted metal. The background includes large magnetic cranes, rusted conveyor belts, and flickering yellow floodlights struggling to penetrate the fog. The robot arm is bright teal with a glossy surface and silver stripes on the outer edges; the joints rotate smoothly and the pistons reflect a pale cyan hue. The robot arm is mounted on a table that is light oak wood with a natural grain pattern and a glossy varnish that reflects overhead lights softly; small burn marks dot one corner. The arm is connected to the base mounted on the table. The bottom cube is deep blue, the second cube is bright red, and the top cube is vivid green, maintaining their correct order after stacking. Sunlight pouring in from a large, open window bathes the table and robotic arm in a warm golden light. The shadows are soft, and the scene feels natural and inviting with a slight contrast between light and shadow.", "prompt": "A robotic arm is picking up and stacking cubes inside a foggy industrial scrapyard at dawn, surrounded by piles of old robotic parts and twisted metal. The background includes large magnetic cranes, rusted conveyor belts, and flickering yellow floodlights struggling to penetrate the fog. The robot arm is bright teal with a glossy surface and silver stripes on the outer edges; the joints rotate smoothly and the pistons reflect a pale cyan hue. The robot arm is mounted on a table that is light oak wood with a natural grain pattern and a glossy varnish that reflects overhead lights softly; small burn marks dot one corner. The arm is connected to the base mounted on the table. The bottom cube is deep blue, the second cube is bright red, and the top cube is vivid green, maintaining their correct order after stacking. Sunlight pouring in from a large, open window bathes the table and robotic arm in a warm golden light. The shadows are soft, and the scene feels natural and inviting with a slight contrast between light and shadow.",
"negative_prompt": "The video captures a game playing, with bad crappy graphics and cartoonish frames. It represents a recording of old outdated games. The images are very pixelated and of poor CG quality. There are many subtitles in the footage. Overall, the video is unrealistic and appears cg. Plane background.", "negative_prompt": "The video captures a game playing, with bad crappy graphics and cartoonish frames. It represents a recording of old outdated games. The images are very pixelated and of poor CG quality. There are many subtitles in the footage. Overall, the video is unrealistic and appears cg. Plane background.",
"input_video_path" : "mimic_generated_dataset_mp4/demo_0_table_cam.mp4", "input_video_path" : "mimic_dataset_1k_mp4/demo_0_table_cam.mp4",
"sigma_max": 50, "sigma_max": 50,
"vis": { "vis": {
"input_control": "mimic_generated_dataset_mp4/demo_0_table_cam.mp4", "input_control": "mimic_dataset_1k_mp4/demo_0_table_cam.mp4",
"control_weight": 0.3 "control_weight": 0.3
}, },
"edge": { "edge": {
...@@ -230,8 +230,8 @@ Example usage for the cube stacking task: ...@@ -230,8 +230,8 @@ Example usage for the cube stacking task:
.. code:: bash .. code:: bash
python scripts/tools/mp4_to_hdf5.py \ python scripts/tools/mp4_to_hdf5.py \
--input_file datasets/mimic_generated_dataset.hdf5 \ --input_file datasets/mimic_dataset_1k.hdf5 \
--videos_dir datasets/cosmos_dataset_mp4 \ --videos_dir datasets/cosmos_dataset_1k_mp4 \
--output_file datasets/cosmos_dataset_1k.hdf5 --output_file datasets/cosmos_dataset_1k.hdf5
Pre-generated Dataset Pre-generated Dataset
...@@ -270,7 +270,7 @@ Example usage for the cube stacking task: ...@@ -270,7 +270,7 @@ Example usage for the cube stacking task:
.. code:: bash .. code:: bash
python scripts/tools/merge_hdf5_datasets.py \ python scripts/tools/merge_hdf5_datasets.py \
--input_files datasets/mimic_generated_dataset.hdf5 datasets/cosmos_dataset.hdf5 \ --input_files datasets/mimic_dataset_1k.hdf5 datasets/cosmos_dataset_1k.hdf5 \
--output_file datasets/mimic_cosmos_dataset.hdf5 --output_file datasets/mimic_cosmos_dataset.hdf5
Model Training and Evaluation Model Training and Evaluation
...@@ -299,7 +299,8 @@ Using the generated data, we can now train a visuomotor BC agent for ``Isaac-Sta ...@@ -299,7 +299,8 @@ Using the generated data, we can now train a visuomotor BC agent for ``Isaac-Sta
./isaaclab.sh -p scripts/imitation_learning/robomimic/train.py \ ./isaaclab.sh -p scripts/imitation_learning/robomimic/train.py \
--task Isaac-Stack-Cube-Franka-IK-Rel-Visuomotor-v0 --algo bc \ --task Isaac-Stack-Cube-Franka-IK-Rel-Visuomotor-v0 --algo bc \
--dataset ./datasets/mimic_cosmos_dataset.hdf5 --dataset ./datasets/mimic_cosmos_dataset.hdf5 \
--name bc_rnn_image_franka_stack_mimic_cosmos
.. note:: .. note::
By default the trained models and logs will be saved to ``IssacLab/logs/robomimic``. By default the trained models and logs will be saved to ``IssacLab/logs/robomimic``.
...@@ -377,8 +378,8 @@ Example usage for the cube stacking task: ...@@ -377,8 +378,8 @@ Example usage for the cube stacking task:
./isaaclab.sh -p scripts/imitation_learning/robomimic/robust_eval.py \ ./isaaclab.sh -p scripts/imitation_learning/robomimic/robust_eval.py \
--task Isaac-Stack-Cube-Franka-IK-Rel-Visuomotor-v0 \ --task Isaac-Stack-Cube-Franka-IK-Rel-Visuomotor-v0 \
--input_dir logs/robomimic/Isaac-Stack-Cube-Franka-IK-Rel-Visuomotor-v0/bc_rnn_image_franka_stack_mimic_cosmos_table_only/*/models \ --input_dir logs/robomimic/Isaac-Stack-Cube-Franka-IK-Rel-Visuomotor-v0/bc_rnn_image_franka_stack_mimic_cosmos/*/models \
--log_dir robust_results/bc_rnn_image_franka_stack_mimic_cosmos_table_only \ --log_dir robust_results/bc_rnn_image_franka_stack_mimic_cosmos \
--log_file result \ --log_file result \
--enable_cameras \ --enable_cameras \
--seeds 0 \ --seeds 0 \
......
...@@ -235,7 +235,7 @@ def main() -> None: ...@@ -235,7 +235,7 @@ def main() -> None:
env_cfg.eval_mode = True env_cfg.eval_mode = True
# Create environment # Create environment
env = gym.make(args_cli.task, cfg=env_cfg) env = gym.make(args_cli.task, cfg=env_cfg).unwrapped
# Acquire device # Acquire device
device = TorchUtils.get_torch_device(try_to_use_cuda=True) device = TorchUtils.get_torch_device(try_to_use_cuda=True)
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment