Adds dependency installation procedure to `./orbit.sh` (#514)

# Description

~Adds a new package `orbit_hooks` to `/orbit/docker/orbit_hooks`. This
is a small library of `setuptools.Command` child classes, built at the
beginning of `Dockerfile.base`. They can be referenced in the `setup.py`
of extensions which want hooks, such as ROS or apt packages installed at
Dockerfile build time.~

~The currently existing hooks are `InstallAptDeps` and `InstallRosDeps`.
These will automatically perform certain installation processes:~

~`InstallAptDeps` looks in the extension's `extension.toml` for an
`orbit_hooks` `apt_deps` list, which will be installed during the
`Dockerfile.base` build procedure.~

~`InstallRosDeps` looks in the extension's `extension.toml` for an
`orbit_hooks` `ros_ws` path. In building `Dockerfile.ros2`, we will
perform a `rosdep` installation of any dependencies in ros packages
beneath `ros_ws`.~

~Additionally, I have added the script `check_and_install_deps.py`,
which scans all subdirs in `orbit/source/extensions` for the existence
of certain setup.py cmdclasses (`install_apt_deps`,`install_ros_deps`)
and calls them if they exist.~

~For an extension to make use of this optional install procedure, they
will need to `import orbit_hooks` in their `setup.py` the relevant
command class dict (`INSTALL_ALL_DEPS`, `INSTALL_APT_DEPS`,
`INSTALL_ROS_DEPS`) and to have the corresponding values in their
`extension.toml`. For an example, [this
PR](https://github.com/bdaiinstitute/orbit.eval_sim/pull/36) has the
related modifications to its `extension.toml` and its `setup.py`.~

**Update**:
Adds a new script `/tools/install_deps.py` which has commands for
installing ros and apt dependencies in extensions via options set in the
`extension.toml`.

`install_apt_packages` looks in the extension's `extension.toml` for an
`orbit_settings` `apt_deps` list, which will be installed during the
`Dockerfile.base` build procedure.

`install_rosdep_packages` looks in the extension's `extension.toml` for
an `orbit_settings` `ros_ws` path. In building `Dockerfile.ros2`, we
will perform a `rosdep` installation of any dependencies in ros packages
beneath `ros_ws`.

Added section about this in `developers.rst`, as well as reference to it
in the this [draft
PR](https://github.com/isaac-orbit/orbit.ext_template/pull/18) for
`orbit.ext_template`

## Type of change

- New feature (non-breaking change which adds functionality)

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./orbit.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have run all the tests with `./orbit.sh --test` and they pass
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there

---------
Signed-off-by: Hunter Hansen <50837800+hhansen-bdai@users.noreply.github.com>

docker/Dockerfile.base

@@ -7,8 +7,9 @@
 # Please check above link for license information.
 
 # Base image
-ARG ISAACSIM_VERSION
-FROM nvcr.io/nvidia/isaac-sim:${ISAACSIM_VERSION} AS base
+ARG ISAACSIM_VERSION_ARG
+FROM nvcr.io/nvidia/isaac-sim:${ISAACSIM_VERSION_ARG} AS base
+ENV ISAACSIM_VERSION=${ISAACSIM_VERSION_ARG}
 
 # Set default RUN shell to bash
 SHELL ["/bin/bash", "-c"]
@@ -19,11 +20,14 @@ LABEL description="Dockerfile for building and running the Orbit framework insid
 
 # Arguments
 # Path to Isaac Sim root folder
-ARG ISAACSIM_ROOT_PATH
+ARG ISAACSIM_ROOT_PATH_ARG
+ENV ISAACSIM_ROOT_PATH=${ISAACSIM_ROOT_PATH_ARG}
 # Path to orbit directory
-ARG ORBIT_PATH
+ARG ORBIT_PATH_ARG
+ENV ORBIT_PATH=${ORBIT_PATH_ARG}
 # Home dir of docker user, typically '/root'
-ARG DOCKER_USER_HOME
+ARG DOCKER_USER_HOME_ARG
+ENV DOCKER_USER_HOME=${DOCKER_USER_HOME_ARG}
 
 # Set environment variables
 ENV LANG=C.UTF-8
@@ -46,8 +50,14 @@ COPY ../ ${ORBIT_PATH}
 # Set up a symbolic link between the installed Isaac Sim root folder and _isaac_sim in the orbit directory
 RUN ln -sf ${ISAACSIM_ROOT_PATH} ${ORBIT_PATH}/_isaac_sim
 
+# Install apt dependencies for extensions that declare them in their extension.toml
+RUN --mount=type=cache,target=/var/cache/apt \
+    ${ORBIT_PATH}/orbit.sh --install-deps apt && \
+    apt -y autoremove && apt clean autoclean && \
+    rm -rf /var/lib/apt/lists/*
+
 # for singularity usage, have to create the directories that will binded
-RUN mkdir -p ${ISAACSIM_ROOT_PATH}/kit/cache && \
+RUN mkdir -p ${ISAACSIM_ROOT_PATH}/kit/cachecd && \
     mkdir -p ${DOCKER_USER_HOME}/.cache/ov && \
     mkdir -p ${DOCKER_USER_HOME}/.cache/pip && \
     mkdir -p ${DOCKER_USER_HOME}/.cache/nvidia/GLCache &&  \

docker/Dockerfile.ros2

@@ -4,8 +4,6 @@ FROM orbit-base AS ros2
 
 # Which ROS2 apt package to install
 ARG ROS2_APT_PACKAGE
-# Home of the docker user, generally /root
-ARG DOCKER_USER_HOME
 
 # ROS2 Humble Apt installations
 RUN --mount=type=cache,target=/var/cache/apt \
@@ -24,6 +22,9 @@ RUN --mount=type=cache,target=/var/cache/apt \
     ros-humble-rmw-fastrtps-cpp \
     # This includes various dev tools including colcon
     ros-dev-tools && \
+    # Install rosdeps for extensions that declare a ros_ws in
+    # their extension.toml
+    ${ORBIT_PATH}/orbit.sh --install-deps rosdep && \
     apt -y autoremove && apt clean autoclean && \
     rm -rf /var/lib/apt/lists/* && \
     # Add sourcing of setup.bash to .bashrc

docker/docker-compose.yaml

@@ -57,7 +57,6 @@ x-default-orbit-volumes: &default-orbit-volumes
 
 x-default-orbit-environment: &default-orbit-environment
   - ISAACSIM_PATH=${DOCKER_ORBIT_PATH}/_isaac_sim
-  - ORBIT_PATH=${DOCKER_ORBIT_PATH}
   - OMNI_KIT_ALLOW_ROOT=1
 
 x-default-orbit-deploy: &default-orbit-deploy
@@ -77,10 +76,10 @@ services:
       context: ../
       dockerfile: docker/Dockerfile.base
       args:
-        - ISAACSIM_VERSION=${ISAACSIM_VERSION}
-        - ISAACSIM_ROOT_PATH=${DOCKER_ISAACSIM_ROOT_PATH}
-        - ORBIT_PATH=${DOCKER_ORBIT_PATH}
-        - DOCKER_USER_HOME=${DOCKER_USER_HOME}
+        - ISAACSIM_VERSION_ARG=${ISAACSIM_VERSION}
+        - ISAACSIM_ROOT_PATH_ARG=${DOCKER_ISAACSIM_ROOT_PATH}
+        - ORBIT_PATH_ARG=${DOCKER_ORBIT_PATH}
+        - DOCKER_USER_HOME_ARG=${DOCKER_USER_HOME}
     image: orbit-base
     container_name: orbit-base
     environment: *default-orbit-environment
@@ -107,7 +106,6 @@ services:
         # avoid a warning message when building only the base profile
         # with the .env.base file
         - ROS2_APT_PACKAGE=${ROS2_APT_PACKAGE:-NONE}
-        - DOCKER_USER_HOME=${DOCKER_USER_HOME}
     image: orbit-ros2
     container_name: orbit-ros2
     environment: *default-orbit-environment

docs/source/setup/developer.rst

@@ -188,6 +188,34 @@ important to note that Omniverse also provides a similar
 However, it requires going through the build process and does not support testing of the python module in
 standalone applications.
 
+Extension Dependency Management
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Certain extensions may have dependencies which need to be installed before the extension can be run.
+While Python dependencies can be expressed via the ``INSTALL_REQUIRES`` array in ``setup.py``, we need
+a separate installation pipeline to handle non-Python dependencies. We have therefore created
+an additional setup procedure, ``./orbit.sh --install-deps {dep_type}``, which scans the ``extension.toml``
+file of the directories under ``/orbit/source/extensions`` for ``apt`` and ``rosdep`` dependencies.
+
+This example ``extension.toml`` has both ``apt_deps`` and ``ros_ws`` specified, so both
+``apt`` and ``rosdep`` packages will be installed if ``./orbit.sh --install-deps all``
+is passed:
+
+.. code-block:: toml
+
+   [orbit_settings]
+   apt_deps = ["example_package"]
+   ros_ws = "path/from/extension_root/to/ros_ws"
+
+From the ``apt_deps`` in the above example, the package ``example_package`` would be installed via ``apt``.
+From the ``ros_ws``, a ``rosdep install --from-paths {ros_ws}/src --ignore-src`` command will be called.
+This will install all the `ROS package.xml dependencies <https://docs.ros.org/en/humble/Tutorials/Intermediate/Rosdep.html>`__
+in the directory structure below. Currently the ROS distro is assumed to be ``humble``.
+
+``apt`` deps are automatically installed this way during the build process of the ``Dockerfile.base``,
+and ``rosdep`` deps during the build process of ``Dockerfile.ros2``.
+
+
 Standalone applications
 ~~~~~~~~~~~~~~~~~~~~~~~
 

orbit.sh

@@ -79,13 +79,22 @@ install_orbit_extension() {
     # retrieve the python executable
     python_exe=$(extract_python_exe)
     # if the directory contains setup.py then install the python module
-    if [ -f "$1/setup.py" ];
-    then
+    if [ -f "$1/setup.py" ]; then
         echo -e "\t module: $1"
         ${python_exe} -m pip install --editable $1
     fi
 }
 
+install_extension_deps() {
+    # retrieve the python executable
+    set -e
+    path="$1"
+    cmd="$2"
+    python_exe=$(extract_python_exe)
+    echo -e "\t Installing deps for module: $1"
+    ${python_exe} ${ORBIT_PATH}/tools/install_deps.py ${cmd} $1
+}
+
 # setup anaconda environment for orbit
 setup_conda_env() {
     # get environment name from input
@@ -195,6 +204,7 @@ print_help () {
     echo -e "\t-h, --help           Display the help content."
     echo -e "\t-i, --install        Install the extensions inside Orbit."
     echo -e "\t-e, --extra [LIB]    Install learning frameworks (rl_games, rsl_rl, sb3) as extra dependencies. Default is 'all'."
+    echo -e "\t--install-deps [dep_type] Install dependencies for extensions (apt, rosdep, all) from each extension.toml. Default is 'all'."
     echo -e "\t-f, --format         Run pre-commit to format the code and check lints."
     echo -e "\t-p, --python         Run the python executable provided by Isaac Sim or virtual environment (if active)."
     echo -e "\t-s, --sim            Run the simulator executable (isaac-sim.sh) provided by Isaac Sim."
@@ -254,6 +264,30 @@ while [[ $# -gt 0 ]]; do
             ${python_exe} -m pip install -e ${ORBIT_PATH}/source/extensions/omni.isaac.orbit_tasks["${framework_name}"]
             shift # past argument
             ;;
+        --install-deps)
+            # install the deps for extensions in source/extensions directory
+            if [ -z "$2" ]; then
+                dep_type="all"
+            else
+                dep_type=$2
+                shift # past argument
+            fi
+            echo "[INFO] Installing ${dep_type} dependencies for extensions inside orbit repository..."
+            # recursively look into directories and install
+            # all extension dependencies
+            export -f extract_python_exe
+            export -f install_extension_deps
+            # check if dep_type is installed, if not "all"
+            if [ "$dep_type" = "all" ] || command -v "$dep_type" &>/dev/null; then
+                find -L "${ORBIT_PATH}/source/extensions" -mindepth 1 -maxdepth 1 -type d -print0 | xargs -0 -I {} bash -c 'install_extension_deps "$1" "$2"' _ {} "${dep_type}"
+            else
+                echo "[ERROR] Not installing ${dep_type} deps, ${dep_type} not a known command"
+                exit 1
+            fi
+            # unset local variables
+            unset install_extension_deps
+            shift # past argument
+            ;;
         -c|--conda)
             # use default name if not provided
             if [ -z "$2" ]; then

tools/install_deps.py

+# Copyright (c) 2022-2024, The ORBIT Project Developers.
+# All rights reserved.
+#
+# SPDX-License-Identifier: BSD-3-Clause
+
+"""
+A script with various methods of installing dependencies
+defined in an extension.toml
+"""
+
+import argparse
+import os
+import shutil
+import sys
+import toml
+from subprocess import SubprocessError, run
+
+# add argparse arguments
+parser = argparse.ArgumentParser(description="Utility to install dependencies based on an extension.toml")
+parser.add_argument("type", type=str, choices=["all", "apt", "rosdep"], help="The type of packages to install")
+parser.add_argument("path", type=str, help="The path to the extension which will have its deps installed")
+
+
+def install_apt_packages(path):
+    """
+    A function which attempts to install apt packages for Orbit extensions.
+    It looks in {extension_root}/config/extension.toml for [orbit_settings][apt_deps]
+    and then attempts to install them. Exits on failure to stop the build process
+    from continuing despite missing dependencies.
+
+    Args:
+        path: A path to the extension root
+    """
+    try:
+        if shutil.which("apt"):
+            with open(f"{path}/config/extension.toml") as fd:
+                ext_toml = toml.load(fd)
+                if "orbit_settings" in ext_toml and "apt_deps" in ext_toml["orbit_settings"]:
+                    deps = ext_toml["orbit_settings"]["apt_deps"]
+                    print(f"[INFO] Installing the following apt packages: {deps}")
+                    run_and_print(["apt-get", "update"])
+                    run_and_print(["apt-get", "install", "-y"] + deps)
+                else:
+                    print("[INFO] No apt packages to install")
+        else:
+            raise RuntimeError("Exiting because 'apt' is not a known command")
+    except SubprocessError as e:
+        print(f"[ERROR]: {str(e.stderr, encoding='utf-8')}")
+        sys.exit(1)
+    except Exception as e:
+        print(f"[ERROR]: {e}")
+        sys.exit(1)
+
+
+def install_rosdep_packages(path):
+    """
+    A function which attempts to install rosdep packages for Orbit extensions.
+    It looks in {extension_root}/config/extension.toml for [orbit_settings][ros_ws]
+    and then attempts to install all rosdeps under that workspace.
+    Exits on failure to stop the build process from continuing despite missing dependencies.
+
+    Args:
+        path: A path to the extension root
+    """
+    try:
+        if shutil.which("rosdep"):
+            with open(f"{path}/config/extension.toml") as fd:
+                ext_toml = toml.load(fd)
+                if "orbit_settings" in ext_toml and "ros_ws" in ext_toml["orbit_settings"]:
+                    ws_path = ext_toml["orbit_settings"]["ros_ws"]
+                    if not os.path.exists("/etc/ros/rosdep/sources.list.d/20-default.list"):
+                        run_and_print(["rosdep", "init"])
+                        run_and_print(["rosdep", "update", "--rosdistro=humble"])
+                    run_and_print([
+                        "rosdep",
+                        "install",
+                        "--from-paths",
+                        f"{path}/{ws_path}/src",
+                        "--ignore-src",
+                        "-y",
+                        "--rosdistro=humble",
+                    ])
+                else:
+                    print("[INFO] No rosdep packages to install")
+        else:
+            raise RuntimeError("Exiting because 'rosdep' is not a known command")
+    except SubprocessError as e:
+        print(f"[ERROR]: {str(e.stderr, encoding='utf-8')}")
+        sys.exit(1)
+    except Exception as e:
+        print(f"[ERROR]: {e}")
+        sys.exit(1)
+
+
+def run_and_print(args):
+    """
+    Runs a subprocess.run(args=args, capture_output=True, check=True),
+    and prints the output
+    """
+    completed_process = run(args=args, capture_output=True, check=True)
+    print(f"{str(completed_process.stdout, encoding='utf-8')}")
+
+
+def main():
+    args = parser.parse_args()
+    if args.type == "all":
+        install_apt_packages(args.path)
+        install_rosdep_packages(args.path)
+    elif args.type == "apt":
+        install_apt_packages(args.path)
+    elif args.type == "rosdep":
+        install_rosdep_packages(args.path)
+    else:
+        print(f"[ERROR] '{args.type}' type dependencies not installable")
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()