Unverified Commit 5ffff16f authored by Mayank Mittal's avatar Mayank Mittal Committed by GitHub

Cleans up instructions for custom dependency installation (#621)

# Description

This MR mainly fixes the documentation to make instructions clearer. The
script was unclear about where the workspace path needs to be and how it
can be resolved. The MR makes sure these cases are dealt and reported
correctly.

## Type of change

- This change requires a documentation update

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there
parent ff6c9657
......@@ -191,29 +191,46 @@ standalone applications.
Extension Dependency Management
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Certain extensions may have dependencies which need to be installed before the extension can be run.
While Python dependencies can be expressed via the ``INSTALL_REQUIRES`` array in ``setup.py``, we need
a separate installation pipeline to handle non-Python dependencies. We have therefore created
an additional setup procedure, ``python tools/install_deps.py {dep_type} {extensions_dir}``, which scans the ``extension.toml``
file of the directories under the ``{extensions_dir}`` (such as ``${ISAACLAB_PATH}/source/extensions``) for ``apt`` and ``rosdep`` dependencies.
Certain extensions may have dependencies which require installation of additional packages before the extension
can be used. While Python dependencies are handled by the `setuptools <https://setuptools.readthedocs.io/en/latest/>`__
package and specified in the ``setup.py`` file, non-Python dependencies such as `ROS <https://www.ros.org/>`__
packages or `apt <https://en.wikipedia.org/wiki/APT_(software)>`__ packages are not handled by setuptools.
To handle these dependencies, we have created an additional setup procedure described in the next section.
This example ``extension.toml`` has both ``apt_deps`` and ``ros_ws`` specified, so both
``apt`` and ``rosdep`` packages will be installed if ``python tools/install_deps.py all ${ISAACLAB_PATH}/source/extensions``
is passed:
There are two types of dependencies that can be specified in the ``extension.toml`` file
under the ``isaac_lab_settings`` section:
1. **apt_deps**: A list of apt packages that need to be installed. These are installed using the
`apt <https://ubuntu.com/server/docs/package-management>`__ package manager.
2. **ros_ws**: The path to the ROS workspace that contains the ROS packages. These are installed using
the `rosdep <https://docs.ros.org/en/humble/Tutorials/Intermediate/Rosdep.html>`__ dependency manager.
As an example, the following ``extension.toml`` file specifies the dependencies for the extension:
.. code-block:: toml
[isaaclab_settings]
apt_deps = ["example_package"]
ros_ws = "path/from/extension_root/to/ros_ws"
[isaac_lab_settings]
# apt dependencies
apt_deps = ["libboost-all-dev"]
From the ``apt_deps`` in the above example, the package ``example_package`` would be installed via ``apt``.
From the ``ros_ws``, a ``rosdep install --from-paths {ros_ws}/src --ignore-src`` command will be called.
This will install all the `ROS package.xml dependencies <https://docs.ros.org/en/humble/Tutorials/Intermediate/Rosdep.html>`__
in the directory structure below. Currently the ROS distro is assumed to be ``humble``.
# ROS workspace
# note: if this path is relative, it is relative to the extension directory's root
ros_ws = "/home/user/catkin_ws"
``apt`` deps are automatically installed this way during the build process of the ``Dockerfile.base``,
and ``rosdep`` deps during the build process of ``Dockerfile.ros2``.
These dependencies are installed using the ``install_deps.py`` script provided in the ``tools`` directory.
To install all dependencies for all extensions, run the following command:
.. code-block:: bash
# execute from the root of the repository
# the script expects the type of dependencies to install and the path to the extensions directory
# available types are: 'apt', 'rosdep' and 'all'
python tools/install_deps.py all ${ISAACLAB_PATH}/source/extensions
.. note::
Currently, this script is automatically executed during the build process of the ``Dockerfile.base``
and ``Dockerfile.ros2``. This ensures that all the 'apt' and 'rosdep' dependencies are installed
before building the extensions respectively.
Standalone applications
......
......@@ -4,8 +4,27 @@
# SPDX-License-Identifier: BSD-3-Clause
"""
A script with various methods of installing dependencies
defined in an extension.toml
This script is a utility to install dependencies mentioned in an extension.toml file of an extension.
The script takes in two arguments:
1. type: The type of dependencies to install. It can be one of the following: ['all', 'apt', 'rosdep'].
2. extensions_dir: The path to the directory beneath which we search for extensions.
The script will search for all extensions in the extensions_dir and then look for an extension.toml file in each
extension's config directory. If the extension.toml file exists, the script will look for the following keys in the
[isaac_lab_settings] section:
* **apt_deps**: A list of apt packages to install.
* **ros_ws**: The path to the ROS workspace in the extension. If the path is not absolute, the script assumes that
the path is relative to the extension root and resolves it accordingly.
If the type is 'all', the script will install both apt and rosdep packages. If the type is 'apt', the script will only
install apt packages. If the type is 'rosdep', the script will only install rosdep packages.
For more information, please check the `documentation`_.
.. _documentation: https://isaac-sim.github.io/IsaacLab/source/setup/developer.html#extension-dependency-management
"""
import argparse
......@@ -15,31 +34,36 @@ import toml
from subprocess import run
# add argparse arguments
parser = argparse.ArgumentParser(description="Utility to install dependencies based on an extension.toml")
parser.add_argument("type", type=str, choices=["all", "apt", "rosdep"], help="The type of packages to install")
parser.add_argument("extensions_dir", type=str, help="The path to the directory beneath which we search for extensions")
parser = argparse.ArgumentParser(description="A utility to install dependencies based on extension.toml files.")
parser.add_argument("type", type=str, choices=["all", "apt", "rosdep"], help="The type of packages to install.")
parser.add_argument("extensions_dir", type=str, help="The path to the directory containing extensions.")
parser.add_argument("--ros_distro", type=str, default="humble", help="The ROS distribution to use for rosdep.")
def install_apt_packages(paths: list[str]):
"""Attempts to install apt packages for Isaac Lab extensions.
For each path in arg paths, it looks in {extension_root}/config/extension.toml for [isaac_lab_settings][apt_deps]
and then attempts to install them. Exits on failure to stop the build process
from continuing despite missing dependencies.
"""Installs apt packages listed in the extension.toml file for Isaac Lab extensions.
For each path in the input list of paths, the function looks in ``{path}/config/extension.toml`` for
the ``[isaac_lab_settings][apt_deps]`` key. It then attempts to install the packages listed in the
value of the key. The function exits on failure to stop the build process from continuing despite missing
dependencies.
Args:
paths: A list of paths to the extension root
paths: A list of paths to the extension's root.
Raises:
RuntimeError: If 'apt' is not a known command
FileNotFoundError: If the extension.toml file is not found.
SystemError: If 'apt' is not a known command. This is a system error.
"""
for path in paths:
if shutil.which("apt"):
# Check if the extension.toml file exists
if not os.path.exists(f"{path}/config/extension.toml"):
raise RuntimeError(
"During the installation of an IsaacSim extension's dependencies, an extension.toml was unable to"
" be found. All IsaacSim extensions must have a configuring .toml at"
" (extension_root)/config/extension.toml"
raise FileNotFoundError(
"During the installation of 'apt' dependencies, unable to find a"
f" valid file at: {path}/config/extension.toml."
)
# Load the extension.toml file and check for apt_deps
with open(f"{path}/config/extension.toml") as fd:
ext_toml = toml.load(fd)
if "isaac_lab_settings" in ext_toml and "apt_deps" in ext_toml["isaac_lab_settings"]:
......@@ -48,77 +72,105 @@ def install_apt_packages(paths: list[str]):
run_and_print(["apt-get", "update"])
run_and_print(["apt-get", "install", "-y"] + deps)
else:
print("[INFO] No apt packages to install")
print(f"[INFO] No apt packages specified for the extension at: {path}")
else:
raise RuntimeError("Exiting because 'apt' is not a known command")
raise SystemError("Unable to find 'apt' command. Please ensure that 'apt' is installed on your system.")
def install_rosdep_packages(paths: list[str], ros_distro: str = "humble"):
"""Installs ROS dependencies listed in the extension.toml file for Isaac Lab extensions.
def install_rosdep_packages(paths: list[str]):
"""Attempts to install rosdep packages for Isaac Lab extensions.
For each path in arg paths, it looks in {extension_root}/config/extension.toml for [isaac_lab_settings][ros_ws]
and then attempts to install all rosdeps under that workspace.
Exits on failure to stop the build process from continuing despite missing dependencies.
For each path in the input list of paths, the function looks in ``{path}/config/extension.toml`` for
the ``[isaac_lab_settings][ros_ws]`` key. It then attempts to install the ROS dependencies under the workspace
listed in the value of the key. The function exits on failure to stop the build process from continuing despite
missing dependencies.
If the path to the ROS workspace is not absolute, the function assumes that the path is relative to the extension
root and resolves it accordingly. The function also checks if the ROS workspace exists before proceeding with
the installation of ROS dependencies. If the ROS workspace does not exist, the function raises an error.
Args:
path: A list of paths to the extension roots
path: A list of paths to the extension roots.
ros_distro: The ROS distribution to use for rosdep. Default is 'humble'.
Raises:
RuntimeError: If 'rosdep' is not a known command
FileNotFoundError: If the extension.toml file is not found under the path.
FileNotFoundError: If a valid ROS workspace is not found while installing ROS dependencies.
SystemError: If 'rosdep' is not a known command. This is raised if 'rosdep' is not installed on the system.
"""
for path in paths:
if shutil.which("rosdep"):
# Check if the extension.toml file exists
if not os.path.exists(f"{path}/config/extension.toml"):
raise RuntimeError(
"During the installation of an IsaacSim extension's dependencies, an extension.toml was unable to"
" be found. All IsaacSim extensions must have a configuring .toml at"
" (extension_root)/config/extension.toml"
raise FileNotFoundError(
"During the installation of 'rosdep' dependencies, unable to find a"
f" valid file at: {path}/config/extension.toml."
)
# Load the extension.toml file and check for ros_ws
with open(f"{path}/config/extension.toml") as fd:
ext_toml = toml.load(fd)
if "isaac_lab_settings" in ext_toml and "ros_ws" in ext_toml["isaac_lab_settings"]:
# resolve the path to the ROS workspace
ws_path = ext_toml["isaac_lab_settings"]["ros_ws"]
if not os.path.abspath(ws_path):
ws_path = os.path.join(path, ws_path)
# check if the workspace exists
if not os.path.exists(f"{ws_path}/src"):
raise FileNotFoundError(
"During the installation of 'rosdep' dependencies, unable to find a"
f" valid ROS workspace at: {path}/{ws_path}."
)
# install rosdep if not already installed
if not os.path.exists("/etc/ros/rosdep/sources.list.d/20-default.list"):
run_and_print(["rosdep", "init"])
run_and_print(["rosdep", "update", "--rosdistro=humble"])
run_and_print(["rosdep", "update", f"--rosdistro={ros_distro}"])
# install rosdep packages
run_and_print([
"rosdep",
"install",
"--from-paths",
f"{path}/{ws_path}/src",
f"{ws_path}/src",
"--ignore-src",
"-y",
"--rosdistro=humble",
f"--rosdistro={ros_distro}",
])
else:
print("[INFO] No rosdep packages to install")
print(f"[INFO] No rosdep packages specified for the extension at: {path}")
else:
raise RuntimeError("Exiting because 'rosdep' is not a known command")
raise SystemError(
"Unable to find 'rosdep' command. Please ensure that 'rosdep' is installed on your system."
"You can install it by running:\n\t sudo apt-get install python3-rosdep"
)
def run_and_print(args: list[str]):
"""Runs a subprocess.run(args=args, capture_output=True, check=True),
and prints the output
"""Runs a subprocess and prints the output to stdout.
This function wraps subprocess.run() and prints the output to stdout.
Args:
args: a list of arguments to be passed to subprocess.run()
args: A list of arguments to pass to subprocess.run().
"""
completed_process = run(args=args, capture_output=True, check=True)
print(f"{str(completed_process.stdout, encoding='utf-8')}")
def main():
# Parse the command line arguments
args = parser.parse_args()
# Get immediate children of args.extensions_dir
extension_paths = [os.path.join(args.extensions_dir, x) for x in next(os.walk(args.extensions_dir))[1]]
# Install dependencies based on the type
if args.type == "all":
install_apt_packages(extension_paths)
install_rosdep_packages(extension_paths)
install_rosdep_packages(extension_paths, args.ros_distro)
elif args.type == "apt":
install_apt_packages(extension_paths)
elif args.type == "rosdep":
install_rosdep_packages(extension_paths)
install_rosdep_packages(extension_paths, args.ros_distro)
else:
raise ValueError(f"'Invalid type dependency: '{args.type}'. Available options: ['all', 'apt', 'rosdep'].")
raise ValueError(f"'Invalid dependency type: '{args.type}'. Available options: ['all', 'apt', 'rosdep'].")
if __name__ == "__main__":
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment