Unverified Commit 5ffff16f authored by Mayank Mittal's avatar Mayank Mittal Committed by GitHub

Cleans up instructions for custom dependency installation (#621)

# Description

This MR mainly fixes the documentation to make instructions clearer. The
script was unclear about where the workspace path needs to be and how it
can be resolved. The MR makes sure these cases are dealt and reported
correctly.

## Type of change

- This change requires a documentation update

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there
parent ff6c9657
...@@ -191,29 +191,46 @@ standalone applications. ...@@ -191,29 +191,46 @@ standalone applications.
Extension Dependency Management Extension Dependency Management
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Certain extensions may have dependencies which need to be installed before the extension can be run. Certain extensions may have dependencies which require installation of additional packages before the extension
While Python dependencies can be expressed via the ``INSTALL_REQUIRES`` array in ``setup.py``, we need can be used. While Python dependencies are handled by the `setuptools <https://setuptools.readthedocs.io/en/latest/>`__
a separate installation pipeline to handle non-Python dependencies. We have therefore created package and specified in the ``setup.py`` file, non-Python dependencies such as `ROS <https://www.ros.org/>`__
an additional setup procedure, ``python tools/install_deps.py {dep_type} {extensions_dir}``, which scans the ``extension.toml`` packages or `apt <https://en.wikipedia.org/wiki/APT_(software)>`__ packages are not handled by setuptools.
file of the directories under the ``{extensions_dir}`` (such as ``${ISAACLAB_PATH}/source/extensions``) for ``apt`` and ``rosdep`` dependencies. To handle these dependencies, we have created an additional setup procedure described in the next section.
This example ``extension.toml`` has both ``apt_deps`` and ``ros_ws`` specified, so both There are two types of dependencies that can be specified in the ``extension.toml`` file
``apt`` and ``rosdep`` packages will be installed if ``python tools/install_deps.py all ${ISAACLAB_PATH}/source/extensions`` under the ``isaac_lab_settings`` section:
is passed:
1. **apt_deps**: A list of apt packages that need to be installed. These are installed using the
`apt <https://ubuntu.com/server/docs/package-management>`__ package manager.
2. **ros_ws**: The path to the ROS workspace that contains the ROS packages. These are installed using
the `rosdep <https://docs.ros.org/en/humble/Tutorials/Intermediate/Rosdep.html>`__ dependency manager.
As an example, the following ``extension.toml`` file specifies the dependencies for the extension:
.. code-block:: toml .. code-block:: toml
[isaaclab_settings] [isaac_lab_settings]
apt_deps = ["example_package"] # apt dependencies
ros_ws = "path/from/extension_root/to/ros_ws" apt_deps = ["libboost-all-dev"]
From the ``apt_deps`` in the above example, the package ``example_package`` would be installed via ``apt``. # ROS workspace
From the ``ros_ws``, a ``rosdep install --from-paths {ros_ws}/src --ignore-src`` command will be called. # note: if this path is relative, it is relative to the extension directory's root
This will install all the `ROS package.xml dependencies <https://docs.ros.org/en/humble/Tutorials/Intermediate/Rosdep.html>`__ ros_ws = "/home/user/catkin_ws"
in the directory structure below. Currently the ROS distro is assumed to be ``humble``.
``apt`` deps are automatically installed this way during the build process of the ``Dockerfile.base``, These dependencies are installed using the ``install_deps.py`` script provided in the ``tools`` directory.
and ``rosdep`` deps during the build process of ``Dockerfile.ros2``. To install all dependencies for all extensions, run the following command:
.. code-block:: bash
# execute from the root of the repository
# the script expects the type of dependencies to install and the path to the extensions directory
# available types are: 'apt', 'rosdep' and 'all'
python tools/install_deps.py all ${ISAACLAB_PATH}/source/extensions
.. note::
Currently, this script is automatically executed during the build process of the ``Dockerfile.base``
and ``Dockerfile.ros2``. This ensures that all the 'apt' and 'rosdep' dependencies are installed
before building the extensions respectively.
Standalone applications Standalone applications
......
...@@ -4,8 +4,27 @@ ...@@ -4,8 +4,27 @@
# SPDX-License-Identifier: BSD-3-Clause # SPDX-License-Identifier: BSD-3-Clause
""" """
A script with various methods of installing dependencies This script is a utility to install dependencies mentioned in an extension.toml file of an extension.
defined in an extension.toml
The script takes in two arguments:
1. type: The type of dependencies to install. It can be one of the following: ['all', 'apt', 'rosdep'].
2. extensions_dir: The path to the directory beneath which we search for extensions.
The script will search for all extensions in the extensions_dir and then look for an extension.toml file in each
extension's config directory. If the extension.toml file exists, the script will look for the following keys in the
[isaac_lab_settings] section:
* **apt_deps**: A list of apt packages to install.
* **ros_ws**: The path to the ROS workspace in the extension. If the path is not absolute, the script assumes that
the path is relative to the extension root and resolves it accordingly.
If the type is 'all', the script will install both apt and rosdep packages. If the type is 'apt', the script will only
install apt packages. If the type is 'rosdep', the script will only install rosdep packages.
For more information, please check the `documentation`_.
.. _documentation: https://isaac-sim.github.io/IsaacLab/source/setup/developer.html#extension-dependency-management
""" """
import argparse import argparse
...@@ -15,31 +34,36 @@ import toml ...@@ -15,31 +34,36 @@ import toml
from subprocess import run from subprocess import run
# add argparse arguments # add argparse arguments
parser = argparse.ArgumentParser(description="Utility to install dependencies based on an extension.toml") parser = argparse.ArgumentParser(description="A utility to install dependencies based on extension.toml files.")
parser.add_argument("type", type=str, choices=["all", "apt", "rosdep"], help="The type of packages to install") parser.add_argument("type", type=str, choices=["all", "apt", "rosdep"], help="The type of packages to install.")
parser.add_argument("extensions_dir", type=str, help="The path to the directory beneath which we search for extensions") parser.add_argument("extensions_dir", type=str, help="The path to the directory containing extensions.")
parser.add_argument("--ros_distro", type=str, default="humble", help="The ROS distribution to use for rosdep.")
def install_apt_packages(paths: list[str]): def install_apt_packages(paths: list[str]):
"""Attempts to install apt packages for Isaac Lab extensions. """Installs apt packages listed in the extension.toml file for Isaac Lab extensions.
For each path in arg paths, it looks in {extension_root}/config/extension.toml for [isaac_lab_settings][apt_deps]
and then attempts to install them. Exits on failure to stop the build process For each path in the input list of paths, the function looks in ``{path}/config/extension.toml`` for
from continuing despite missing dependencies. the ``[isaac_lab_settings][apt_deps]`` key. It then attempts to install the packages listed in the
value of the key. The function exits on failure to stop the build process from continuing despite missing
dependencies.
Args: Args:
paths: A list of paths to the extension root paths: A list of paths to the extension's root.
Raises: Raises:
RuntimeError: If 'apt' is not a known command FileNotFoundError: If the extension.toml file is not found.
SystemError: If 'apt' is not a known command. This is a system error.
""" """
for path in paths: for path in paths:
if shutil.which("apt"): if shutil.which("apt"):
# Check if the extension.toml file exists
if not os.path.exists(f"{path}/config/extension.toml"): if not os.path.exists(f"{path}/config/extension.toml"):
raise RuntimeError( raise FileNotFoundError(
"During the installation of an IsaacSim extension's dependencies, an extension.toml was unable to" "During the installation of 'apt' dependencies, unable to find a"
" be found. All IsaacSim extensions must have a configuring .toml at" f" valid file at: {path}/config/extension.toml."
" (extension_root)/config/extension.toml"
) )
# Load the extension.toml file and check for apt_deps
with open(f"{path}/config/extension.toml") as fd: with open(f"{path}/config/extension.toml") as fd:
ext_toml = toml.load(fd) ext_toml = toml.load(fd)
if "isaac_lab_settings" in ext_toml and "apt_deps" in ext_toml["isaac_lab_settings"]: if "isaac_lab_settings" in ext_toml and "apt_deps" in ext_toml["isaac_lab_settings"]:
...@@ -48,77 +72,105 @@ def install_apt_packages(paths: list[str]): ...@@ -48,77 +72,105 @@ def install_apt_packages(paths: list[str]):
run_and_print(["apt-get", "update"]) run_and_print(["apt-get", "update"])
run_and_print(["apt-get", "install", "-y"] + deps) run_and_print(["apt-get", "install", "-y"] + deps)
else: else:
print("[INFO] No apt packages to install") print(f"[INFO] No apt packages specified for the extension at: {path}")
else: else:
raise RuntimeError("Exiting because 'apt' is not a known command") raise SystemError("Unable to find 'apt' command. Please ensure that 'apt' is installed on your system.")
def install_rosdep_packages(paths: list[str], ros_distro: str = "humble"):
"""Installs ROS dependencies listed in the extension.toml file for Isaac Lab extensions.
def install_rosdep_packages(paths: list[str]): For each path in the input list of paths, the function looks in ``{path}/config/extension.toml`` for
"""Attempts to install rosdep packages for Isaac Lab extensions. the ``[isaac_lab_settings][ros_ws]`` key. It then attempts to install the ROS dependencies under the workspace
For each path in arg paths, it looks in {extension_root}/config/extension.toml for [isaac_lab_settings][ros_ws] listed in the value of the key. The function exits on failure to stop the build process from continuing despite
and then attempts to install all rosdeps under that workspace. missing dependencies.
Exits on failure to stop the build process from continuing despite missing dependencies.
If the path to the ROS workspace is not absolute, the function assumes that the path is relative to the extension
root and resolves it accordingly. The function also checks if the ROS workspace exists before proceeding with
the installation of ROS dependencies. If the ROS workspace does not exist, the function raises an error.
Args: Args:
path: A list of paths to the extension roots path: A list of paths to the extension roots.
ros_distro: The ROS distribution to use for rosdep. Default is 'humble'.
Raises: Raises:
RuntimeError: If 'rosdep' is not a known command FileNotFoundError: If the extension.toml file is not found under the path.
FileNotFoundError: If a valid ROS workspace is not found while installing ROS dependencies.
SystemError: If 'rosdep' is not a known command. This is raised if 'rosdep' is not installed on the system.
""" """
for path in paths: for path in paths:
if shutil.which("rosdep"): if shutil.which("rosdep"):
# Check if the extension.toml file exists
if not os.path.exists(f"{path}/config/extension.toml"): if not os.path.exists(f"{path}/config/extension.toml"):
raise RuntimeError( raise FileNotFoundError(
"During the installation of an IsaacSim extension's dependencies, an extension.toml was unable to" "During the installation of 'rosdep' dependencies, unable to find a"
" be found. All IsaacSim extensions must have a configuring .toml at" f" valid file at: {path}/config/extension.toml."
" (extension_root)/config/extension.toml"
) )
# Load the extension.toml file and check for ros_ws
with open(f"{path}/config/extension.toml") as fd: with open(f"{path}/config/extension.toml") as fd:
ext_toml = toml.load(fd) ext_toml = toml.load(fd)
if "isaac_lab_settings" in ext_toml and "ros_ws" in ext_toml["isaac_lab_settings"]: if "isaac_lab_settings" in ext_toml and "ros_ws" in ext_toml["isaac_lab_settings"]:
# resolve the path to the ROS workspace
ws_path = ext_toml["isaac_lab_settings"]["ros_ws"] ws_path = ext_toml["isaac_lab_settings"]["ros_ws"]
if not os.path.abspath(ws_path):
ws_path = os.path.join(path, ws_path)
# check if the workspace exists
if not os.path.exists(f"{ws_path}/src"):
raise FileNotFoundError(
"During the installation of 'rosdep' dependencies, unable to find a"
f" valid ROS workspace at: {path}/{ws_path}."
)
# install rosdep if not already installed
if not os.path.exists("/etc/ros/rosdep/sources.list.d/20-default.list"): if not os.path.exists("/etc/ros/rosdep/sources.list.d/20-default.list"):
run_and_print(["rosdep", "init"]) run_and_print(["rosdep", "init"])
run_and_print(["rosdep", "update", "--rosdistro=humble"]) run_and_print(["rosdep", "update", f"--rosdistro={ros_distro}"])
# install rosdep packages
run_and_print([ run_and_print([
"rosdep", "rosdep",
"install", "install",
"--from-paths", "--from-paths",
f"{path}/{ws_path}/src", f"{ws_path}/src",
"--ignore-src", "--ignore-src",
"-y", "-y",
"--rosdistro=humble", f"--rosdistro={ros_distro}",
]) ])
else: else:
print("[INFO] No rosdep packages to install") print(f"[INFO] No rosdep packages specified for the extension at: {path}")
else: else:
raise RuntimeError("Exiting because 'rosdep' is not a known command") raise SystemError(
"Unable to find 'rosdep' command. Please ensure that 'rosdep' is installed on your system."
"You can install it by running:\n\t sudo apt-get install python3-rosdep"
)
def run_and_print(args: list[str]): def run_and_print(args: list[str]):
"""Runs a subprocess.run(args=args, capture_output=True, check=True), """Runs a subprocess and prints the output to stdout.
and prints the output
This function wraps subprocess.run() and prints the output to stdout.
Args: Args:
args: a list of arguments to be passed to subprocess.run() args: A list of arguments to pass to subprocess.run().
""" """
completed_process = run(args=args, capture_output=True, check=True) completed_process = run(args=args, capture_output=True, check=True)
print(f"{str(completed_process.stdout, encoding='utf-8')}") print(f"{str(completed_process.stdout, encoding='utf-8')}")
def main(): def main():
# Parse the command line arguments
args = parser.parse_args() args = parser.parse_args()
# Get immediate children of args.extensions_dir # Get immediate children of args.extensions_dir
extension_paths = [os.path.join(args.extensions_dir, x) for x in next(os.walk(args.extensions_dir))[1]] extension_paths = [os.path.join(args.extensions_dir, x) for x in next(os.walk(args.extensions_dir))[1]]
# Install dependencies based on the type
if args.type == "all": if args.type == "all":
install_apt_packages(extension_paths) install_apt_packages(extension_paths)
install_rosdep_packages(extension_paths) install_rosdep_packages(extension_paths, args.ros_distro)
elif args.type == "apt": elif args.type == "apt":
install_apt_packages(extension_paths) install_apt_packages(extension_paths)
elif args.type == "rosdep": elif args.type == "rosdep":
install_rosdep_packages(extension_paths) install_rosdep_packages(extension_paths, args.ros_distro)
else: else:
raise ValueError(f"'Invalid type dependency: '{args.type}'. Available options: ['all', 'apt', 'rosdep'].") raise ValueError(f"'Invalid dependency type: '{args.type}'. Available options: ['all', 'apt', 'rosdep'].")
if __name__ == "__main__": if __name__ == "__main__":
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment