Adds custom dependency installation from extension.toml inside Docker (#552)

This PR adds back in the functionality that was added with
https://github.com/isaac-orbit/IsaacLab/pull/542

- New feature (non-breaking change which adds functionality)

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [ ] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

---------
Signed-off-by: James Smith <142246516+jsmith-bdai@users.noreply.github.com>
Co-authored-by: Mayank Mittal <12863862+Mayankm96@users.noreply.github.com>

docker/Dockerfile.base

@@ -52,6 +52,12 @@ COPY ../ ${ISAACLAB_PATH}
 # Set up a symbolic link between the installed Isaac Sim root folder and _isaac_sim in the Isaac Lab directory
 RUN ln -sf ${ISAACSIM_ROOT_PATH} ${ISAACLAB_PATH}/_isaac_sim
 
+# Install apt dependencies for extensions that declare them in their extension.toml
+RUN --mount=type=cache,target=/var/cache/apt \
+    ${ISAACLAB_PATH}/isaaclab.sh -p ${ISAACLAB_PATH}/tools/install_deps.py apt ${ISAACLAB_PATH}/source/extensions && \
+    apt -y autoremove && apt clean autoclean && \
+    rm -rf /var/lib/apt/lists/*
+
 # for singularity usage, have to create the directories that will binded
 RUN mkdir -p ${ISAACSIM_ROOT_PATH}/kit/cache && \
     mkdir -p ${DOCKER_USER_HOME}/.cache/ov && \

docker/Dockerfile.ros2

@@ -22,6 +22,9 @@ RUN --mount=type=cache,target=/var/cache/apt \
     ros-humble-rmw-fastrtps-cpp \
     # This includes various dev tools including colcon
     ros-dev-tools && \
+    # Install rosdeps for extensions that declare a ros_ws in
+    # their extension.toml
+    ${ISAACLAB_PATH}/isaaclab.sh -p ${ISAACLAB_PATH}/tools/install_deps.py rosdep ${ISAACLAB_PATH}/source/extensions && \
     apt -y autoremove && apt clean autoclean && \
     rm -rf /var/lib/apt/lists/* && \
     # Add sourcing of setup.bash to .bashrc

docs/source/setup/developer.rst

@@ -190,6 +190,34 @@ important to note that Omniverse also provides a similar
 However, it requires going through the build process and does not support testing of the python module in
 standalone applications.
 
+Extension Dependency Management
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Certain extensions may have dependencies which need to be installed before the extension can be run.
+While Python dependencies can be expressed via the ``INSTALL_REQUIRES`` array in ``setup.py``, we need
+a separate installation pipeline to handle non-Python dependencies. We have therefore created
+an additional setup procedure, ``python tools/install_deps.py {dep_type} {extensions_dir}``, which scans the ``extension.toml``
+file of the directories under the ``{extensions_dir}`` (such as ``${ISAACLAB_PATH}/source/extensions``) for ``apt`` and ``rosdep`` dependencies.
+
+This example ``extension.toml`` has both ``apt_deps`` and ``ros_ws`` specified, so both
+``apt`` and ``rosdep`` packages will be installed if ``python tools/install_deps.py all ${ISAACLAB_PATH}/source/extensions``
+is passed:
+
+.. code-block:: toml
+
+   [isaaclab_settings]
+   apt_deps = ["example_package"]
+   ros_ws = "path/from/extension_root/to/ros_ws"
+
+From the ``apt_deps`` in the above example, the package ``example_package`` would be installed via ``apt``.
+From the ``ros_ws``, a ``rosdep install --from-paths {ros_ws}/src --ignore-src`` command will be called.
+This will install all the `ROS package.xml dependencies <https://docs.ros.org/en/humble/Tutorials/Intermediate/Rosdep.html>`__
+in the directory structure below. Currently the ROS distro is assumed to be ``humble``.
+
+``apt`` deps are automatically installed this way during the build process of the ``Dockerfile.base``,
+and ``rosdep`` deps during the build process of ``Dockerfile.ros2``.
+
+
 Standalone applications
 ~~~~~~~~~~~~~~~~~~~~~~~
 

tools/install_deps.py

+# Copyright (c) 2022-2024, The Isaac Lab Project Developers.
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+"""
+A script with various methods of installing dependencies
+defined in an extension.toml
+"""
+
+import argparse
+import os
+import shutil
+import toml
+from subprocess import run
+
+# add argparse arguments
+parser = argparse.ArgumentParser(description="Utility to install dependencies based on an extension.toml")
+parser.add_argument("type", type=str, choices=["all", "apt", "rosdep"], help="The type of packages to install")
+parser.add_argument("extensions_dir", type=str, help="The path to the directory beneath which we search for extensions")
+
+
+def install_apt_packages(paths: list[str]):
+    """Attempts to install apt packages for Isaac Lab extensions.
+    For each path in arg paths, it looks in {extension_root}/config/extension.toml for [isaac_lab_settings][apt_deps]
+    and then attempts to install them. Exits on failure to stop the build process
+    from continuing despite missing dependencies.
+
+    Args:
+        paths: A list of paths to the extension root
+
+    Raises:
+        RuntimeError: If 'apt' is not a known command
+    """
+    for path in paths:
+        if shutil.which("apt"):
+            if not os.path.exists(f"{path}/config/extension.toml"):
+                raise RuntimeError(
+                    "During the installation of an IsaacSim extension's dependencies, an extension.toml was unable to"
+                    " be found. All IsaacSim extensions must have a configuring .toml at"
+                    " (extension_root)/config/extension.toml"
+                )
+            with open(f"{path}/config/extension.toml") as fd:
+                ext_toml = toml.load(fd)
+                if "isaac_lab_settings" in ext_toml and "apt_deps" in ext_toml["isaac_lab_settings"]:
+                    deps = ext_toml["isaac_lab_settings"]["apt_deps"]
+                    print(f"[INFO] Installing the following apt packages: {deps}")
+                    run_and_print(["apt-get", "update"])
+                    run_and_print(["apt-get", "install", "-y"] + deps)
+                else:
+                    print("[INFO] No apt packages to install")
+        else:
+            raise RuntimeError("Exiting because 'apt' is not a known command")
+
+
+def install_rosdep_packages(paths: list[str]):
+    """Attempts to install rosdep packages for Isaac Lab extensions.
+    For each path in arg paths, it looks in {extension_root}/config/extension.toml for [isaac_lab_settings][ros_ws]
+    and then attempts to install all rosdeps under that workspace.
+    Exits on failure to stop the build process from continuing despite missing dependencies.
+
+    Args:
+        path: A list of paths to the extension roots
+
+    Raises:
+        RuntimeError: If 'rosdep' is not a known command
+    """
+    for path in paths:
+        if shutil.which("rosdep"):
+            if not os.path.exists(f"{path}/config/extension.toml"):
+                raise RuntimeError(
+                    "During the installation of an IsaacSim extension's dependencies, an extension.toml was unable to"
+                    " be found. All IsaacSim extensions must have a configuring .toml at"
+                    " (extension_root)/config/extension.toml"
+                )
+            with open(f"{path}/config/extension.toml") as fd:
+                ext_toml = toml.load(fd)
+                if "isaac_lab_settings" in ext_toml and "ros_ws" in ext_toml["isaac_lab_settings"]:
+                    ws_path = ext_toml["isaac_lab_settings"]["ros_ws"]
+                    if not os.path.exists("/etc/ros/rosdep/sources.list.d/20-default.list"):
+                        run_and_print(["rosdep", "init"])
+                        run_and_print(["rosdep", "update", "--rosdistro=humble"])
+                    run_and_print([
+                        "rosdep",
+                        "install",
+                        "--from-paths",
+                        f"{path}/{ws_path}/src",
+                        "--ignore-src",
+                        "-y",
+                        "--rosdistro=humble",
+                    ])
+                else:
+                    print("[INFO] No rosdep packages to install")
+        else:
+            raise RuntimeError("Exiting because 'rosdep' is not a known command")
+
+
+def run_and_print(args: list[str]):
+    """Runs a subprocess.run(args=args, capture_output=True, check=True),
+    and prints the output
+
+    Args:
+        args: a list of arguments to be passed to subprocess.run()
+    """
+    completed_process = run(args=args, capture_output=True, check=True)
+    print(f"{str(completed_process.stdout, encoding='utf-8')}")
+
+
+def main():
+    args = parser.parse_args()
+    # Get immediate children of args.extensions_dir
+    extension_paths = [os.path.join(args.extensions_dir, x) for x in next(os.walk(args.extensions_dir))[1]]
+    if args.type == "all":
+        install_apt_packages(extension_paths)
+        install_rosdep_packages(extension_paths)
+    elif args.type == "apt":
+        install_apt_packages(extension_paths)
+    elif args.type == "rosdep":
+        install_rosdep_packages(extension_paths)
+    else:
+        raise ValueError(f"'Invalid type dependency: '{args.type}'. Available options: ['all', 'apt', 'rosdep'].")
+
+
+if __name__ == "__main__":
+    main()