Skip to content

K
KincoActuatorIsaacLab
Unverified Commit 173b5f3e authored by Mayank Mittal's avatar Mayank Mittal Committed by GitHub
Browse files
Options

Cleans up the docker container interface and utilities (#823) 

# Description

Mainly documentation fixes and ensuring the code quality remains
consistent. Also adds the `container.sh` script which calls the python
script for compatibility reasons. We can remove it in later releases.

## Type of change

- Bug fix (non-breaking change which fixes an issue)
- This change requires a documentation update

## Checklist

- [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
`./isaaclab.sh --format`
- [x] I have made corresponding changes to the documentation
- [x] My changes generate no new warnings
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have updated the changelog and the corresponding version in the
extension's `config/extension.toml` file
- [x] I have added my name to the `CONTRIBUTORS.md` or my name already
exists there
parent 7379dcee
Show whitespace changes
Inline Side-by-side
Showing with 536 additions and 253 deletions
docker/.env.ros2
View file @ 173b5f3e
......@@ -3,7 +3,7 @@
###
# Set the version of the ROS2 apt package to install (ros-base, desktop, desktop-full)
ROS2_APT_PACKAGE=ros-base
# Se t ROS2 middleware implementation to use (e.g. rmw_fastrtps_cpp, rmw_cyclonedds_cpp)
# Set ROS2 middleware implementation to use (e.g. rmw_fastrtps_cpp, rmw_cyclonedds_cpp)
RMW_IMPLEMENTATION=rmw_fastrtps_cpp
# Path to fastdds.xml file to use (only needed when using fastdds)
FASTRTPS_DEFAULT_PROFILES_FILE=${DOCKER_USER_HOME}/.ros/fastdds.xml
......
docker/container.py
View file @ 173b5f3e
......@@ -9,24 +9,32 @@ import argparse
import shutil
from pathlib import Path
from utils import x11_utils
from utils.isaaclab_container_interface import IsaacLabContainerInterface
from utils import ContainerInterface, x11_utils
def main():
def parse_cli_args() -> argparse.Namespace:
"""Parse command line arguments.
This function creates a parser object and adds subparsers for each command. The function then parses the
command line arguments and returns the parsed arguments.
Returns:
The parsed command line arguments.
"""
parser = argparse.ArgumentParser(description="Utility for using Docker with Isaac Lab.")
subparsers = parser.add_subparsers(dest="command", required=True)
# We have to create separate parent parsers for common options to our subparsers
parent_parser = argparse.ArgumentParser(add_help=False)
parent_parser.add_argument("profile", nargs="?", default="base", help="Optional container profile specification.")
parent_parser.add_argument(
"profile", nargs="?", default="base", help="Optional container profile specification. Example: 'base' or 'ros'."
)
parent_parser.add_argument(
"--files",
nargs="*",
default=None,
help=(
"Allows additional .yaml files to be passed to the docker compose command. Files will be merged with"
" docker-compose.yaml in the order in which they are provided."
"Allows additional '.yaml' files to be passed to the docker compose command. These files will be merged"
" with 'docker-compose.yaml' in their provided order."
),
)
parent_parser.add_argument(
......@@ -34,12 +42,13 @@ def main():
nargs="*",
default=None,
help=(
"Allows additional .env files to be passed to the docker compose command. Files will be merged with"
" .env.base in the order in which they are provided."
"Allows additional '.env' files to be passed to the docker compose command. These files will be merged with"
" '.env.base' in their provided order."
),
)
# Actual command definition begins here
subparsers = parser.add_subparsers(dest="command", required=True)
subparsers.add_parser(
"start",
help="Build the docker image and create the container in detached mode.",
......@@ -64,37 +73,55 @@ def main():
)
subparsers.add_parser("stop", help="Stop the docker container and remove it.", parents=[parent_parser])
# parse the arguments to determine the command
args = parser.parse_args()
return args
def main(args: argparse.Namespace):
"""Main function for the Docker utility."""
# check if docker is installed
if not shutil.which("docker"):
raise RuntimeError("Docker is not installed! Please check the 'Docker Guide' for instruction.")
raise RuntimeError(
"Docker is not installed! Please check the 'Docker Guide' for instruction: "
"https://isaac-sim.github.io/IsaacLab/source/deployment/docker.html"
)
# Creating container interface
ci = IsaacLabContainerInterface(
# creating container interface
ci = ContainerInterface(
context_dir=Path(__file__).resolve().parent, profile=args.profile, yamls=args.files, envs=args.env_files
)
print(f"[INFO] Using container profile: {ci.profile}")
if args.command == "start":
# check if x11 forwarding is enabled
x11_outputs = x11_utils.x11_check(ci.statefile)
# if x11 forwarding is enabled, add the x11 yaml and environment variables
if x11_outputs is not None:
(x11_yaml, x11_envar) = x11_outputs
ci.add_yamls += x11_yaml
ci.environ.update(x11_envar)
# start the container
ci.start()
elif args.command == "enter":
# refresh the x11 forwarding
x11_utils.x11_refresh(ci.statefile)
# enter the container
ci.enter()
elif args.command == "config":
ci.config(args.output_yaml)
elif args.command == "copy":
ci.copy()
elif args.command == "stop":
# stop the container
ci.stop()
# cleanup the x11 forwarding
x11_utils.x11_cleanup(ci.statefile)
else:
raise RuntimeError(f"Invalid command provided: {args.command}")
raise RuntimeError(f"Invalid command provided: {args.command}. Please check the help message.")
if __name__ == "__main__":
main()
args_cli = parse_cli_args()
main(args_cli)
docker/container.sh 0 → 100755
View file @ 173b5f3e
#!/usr/bin/env bash
# Copyright (c) 2022-2024, The Isaac Lab Project Developers.
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
# print warning of deprecated script in yellow
echo -e "\e[33m------------------------------------------------------------"
echo -e "WARNING: This script is deprecated and will be removed in the future. Please use 'docker/container.py' instead."
echo -e "------------------------------------------------------------\e[0m\n"
# obtain current directory
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
# call the python script
python3 "${SCRIPT_DIR}/container.py" "${@:1}"
docker/utils/__init__.py
View file @ 173b5f3e
......@@ -2,3 +2,7 @@
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
from .container_interface import ContainerInterface
__all__ = ["ContainerInterface"]
docker/utils/isaaclab_container_interface.py → docker/utils/container_interface.py
View file @ 173b5f3e
......@@ -2,6 +2,7 @@
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
from __future__ import annotations
import os
......@@ -10,96 +11,70 @@ import subprocess
from pathlib import Path
from typing import Any
from utils.statefile import Statefile
from .state_file import StateFile
class IsaacLabContainerInterface:
"""
Interface for managing Isaac Lab containers.
"""
class ContainerInterface:
"""A helper class for managing Isaac Lab containers."""
def __init__(
self,
context_dir: Path,
profile: str = "base",
statefile: None | Statefile = None,
yamls: list[str] | None = None,
envs: list[str] | None = None,
statefile: StateFile | None = None,
):
"""
Initialize the IsaacLabContainerInterface with the given parameters.
"""Initialize the container interface with the given parameters.
Args:
context_dir : The context directory for Docker operations.
statefile : An instance of the Statefile class to manage state variables. If not provided, initializes a Statefile(path=self.context_dir/.container.cfg).
profile : The profile name for the container. Defaults to "base".
yamls : A list of yamls to extend docker-compose.yaml. They will be extended in the order they are provided.
envs : A list of envs to extend .env.base. They will be extended in the order they are provided.
context_dir: The context directory for Docker operations.
profile: The profile name for the container. Defaults to "base".
yamls: A list of yaml files to extend ``docker-compose.yaml`` settings. These are extended in the order
they are provided.
envs: A list of environment variable files to extend the ``.env.base`` file. These are extended in the order
they are provided.
statefile: An instance of the :class:`Statefile` class to manage state variables. Defaults to None, in
which case a new configuration object is created by reading the configuration file at the path
``context_dir/.container.cfg``.
"""
# set the context directory
self.context_dir = context_dir
# create a state-file if not provided
# the state file is a manager of run-time state variables that are saved to a file
if statefile is None:
self.statefile = Statefile(path=self.context_dir / ".container.cfg")
self.statefile = StateFile(path=self.context_dir / ".container.cfg")
else:
self.statefile = statefile
# set the profile and container name
self.profile = profile
if self.profile == "isaaclab":
# Silently correct from isaaclab to base,
# because isaaclab is a commonly passed arg
# Silently correct from isaaclab to base, because isaaclab is a commonly passed arg
# but not a real profile
self.profile = "base"
self.container_name = f"isaac-lab-{self.profile}"
self.image_name = f"isaac-lab-{self.profile}:latest"
self.environ = os.environ
self.resolve_image_extension(yamls, envs)
self.load_dot_vars()
def resolve_image_extension(self, yamls: list[str] | None = None, envs: list[str] | None = None):
"""
Resolve the image extension by setting up YAML files, profiles, and environment files for the Docker compose command.
Args:
yamls (List[str], optional): A list of yamls to extend docker-compose.yaml. They will be extended in the order they are provided.
envs (List[str], optional): A list of envs to extend .env.base. They will be extended in the order they are provided.
"""
self.add_yamls = ["--file", "docker-compose.yaml"]
self.add_profiles = ["--profile", f"{self.profile}"]
self.add_env_files = ["--env-file", ".env.base"]
if self.profile != "base":
self.add_env_files += ["--env-file", f".env.{self.profile}"]
if yamls is not None:
for yaml in yamls:
self.add_yamls += ["--file", yaml]
# keep the environment variables from the current environment
self.environ = os.environ
if envs is not None:
for env in envs:
self.add_env_files += ["--env-file", env]
# resolve the image extension through the passed yamls and envs
self._resolve_image_extension(yamls, envs)
# load the environment variables from the .env files
self._parse_dot_vars()
def load_dot_vars(self):
"""
Load environment variables from .env files into a dictionary.
The environment variables are read in order and overwritten if there are name conflicts,
mimicking the behavior of Docker compose.
Operations.
"""
self.dot_vars: dict[str, Any] = {}
if len(self.add_env_files) % 2 != 0:
raise RuntimeError(
"The parameter self.add_env_files is configured incorrectly. There should be an even number of"
" arguments."
)
for i in range(1, len(self.add_env_files), 2):
with open(self.context_dir / self.add_env_files[i]) as f:
self.dot_vars.update(dict(line.strip().split("=", 1) for line in f if "=" in line))
def is_container_running(self) -> bool:
"""
Check if the container is running.
If the container is not running, return False.
"""Check if the container is running.
Returns:
bool: True if the container is running, False otherwise.
True if the container is running, otherwise False.
"""
status = subprocess.run(
["docker", "container", "inspect", "-f", "{{.State.Status}}", self.container_name],
......@@ -110,22 +85,22 @@ class IsaacLabContainerInterface:
return status == "running"
def does_image_exist(self) -> bool:
"""
Check if the Docker image exists.
If the image does not exist, return False.
"""Check if the Docker image exists.
Returns:
bool: True if the image exists, False otherwise.
True if the image exists, otherwise False.
"""
result = subprocess.run(["docker", "image", "inspect", self.image_name], capture_output=True, text=True)
return result.returncode == 0
def start(self):
"""
Build and start the Docker container using the Docker compose command.
"""
print(f"[INFO] Building the docker image and starting the container {self.container_name} in the background...")
"""Build and start the Docker container using the Docker compose command."""
print(
f"[INFO] Building the docker image and starting the container '{self.container_name}' in the"
" background...\n"
)
# build the image for the base profile
subprocess.run(
[
"docker",
......@@ -141,6 +116,8 @@ class IsaacLabContainerInterface:
cwd=self.context_dir,
env=self.environ,
)
# build the image for the profile
subprocess.run(
["docker", "compose"]
+ self.add_yamls
......@@ -153,14 +130,13 @@ class IsaacLabContainerInterface:
)
def enter(self):
"""
Enter the running container by executing a bash shell.
"""Enter the running container by executing a bash shell.
Raises:
RuntimeError: If the container is not running.
"""
if self.is_container_running():
print(f"[INFO] Entering the existing {self.container_name} container in a bash session...")
print(f"[INFO] Entering the existing '{self.container_name}' container in a bash session...\n")
subprocess.run([
"docker",
"exec",
......@@ -172,17 +148,16 @@ class IsaacLabContainerInterface:
"bash",
])
else:
raise RuntimeError(f"The container '{self.container_name}' is not running")
raise RuntimeError(f"The container '{self.container_name}' is not running.")
def stop(self):
"""
Stop the running container using the Docker compose command.
"""Stop the running container using the Docker compose command.
Raises:
RuntimeError: If the container is not running.
"""
if self.is_container_running():
print(f"[INFO] Stopping the launched docker container {self.container_name}...")
print(f"[INFO] Stopping the launched docker container '{self.container_name}'...\n")
subprocess.run(
["docker", "compose"] + self.add_yamls + self.add_profiles + self.add_env_files + ["down"],
check=False,
......@@ -193,33 +168,40 @@ class IsaacLabContainerInterface:
raise RuntimeError(f"Can't stop container '{self.container_name}' as it is not running.")
def copy(self, output_dir: Path | None = None):
"""
Copy artifacts from the running container to the host machine.
"""Copy artifacts from the running container to the host machine.
Args:
output_dir: The directory to copy the artifacts to. Defaults to self.context_dir.
output_dir: The directory to copy the artifacts to. Defaults to None, in which case
the context directory is used.
Raises:
RuntimeError: If the container is not running.
"""
if self.is_container_running():
print(f"[INFO] Copying artifacts from the 'isaac-lab-{self.container_name}' container...")
print(f"[INFO] Copying artifacts from the '{self.container_name}' container...\n")
if output_dir is None:
output_dir = self.context_dir
# create a directory to store the artifacts
output_dir = output_dir.joinpath("artifacts")
if not output_dir.is_dir():
output_dir.mkdir()
# define dictionary of mapping from docker container path to host machine path
docker_isaac_lab_path = Path(self.dot_vars["DOCKER_ISAACLAB_PATH"])
artifacts = {
Path(self.dot_vars["DOCKER_ISAACLAB_PATH"]).joinpath("logs"): output_dir.joinpath("logs"),
Path(self.dot_vars["DOCKER_ISAACLAB_PATH"]).joinpath("docs/_build"): output_dir.joinpath("docs"),
Path(self.dot_vars["DOCKER_ISAACLAB_PATH"]).joinpath("data_storage"): output_dir.joinpath(
"data_storage"
),
docker_isaac_lab_path.joinpath("logs"): output_dir.joinpath("logs"),
docker_isaac_lab_path.joinpath("docs/_build"): output_dir.joinpath("docs"),
docker_isaac_lab_path.joinpath("data_storage"): output_dir.joinpath("data_storage"),
}
# print the artifacts to be copied
for container_path, host_path in artifacts.items():
print(f"\t -{container_path} -> {host_path}")
# remove the existing artifacts
for path in artifacts.values():
shutil.rmtree(path, ignore_errors=True)
# copy the artifacts
for container_path, host_path in artifacts.items():
subprocess.run(
[
......@@ -232,25 +214,83 @@ class IsaacLabContainerInterface:
)
print("\n[INFO] Finished copying the artifacts from the container.")
else:
raise RuntimeError(f"The container '{self.container_name}' is not running")
raise RuntimeError(f"The container '{self.container_name}' is not running.")
def config(self, output_yaml: Path | None = None) -> None:
"""
Generate a docker-compose.yaml from the passed yamls, .envs, and either print to the
terminal or create a yaml at output_yaml
def config(self, output_yaml: Path | None = None):
"""Process the Docker compose configuration based on the passed yamls and environment files.
If the :attr:`output_yaml` is not None, the configuration is written to the file. Otherwise, it is printed to
the terminal.
Args:
output_yaml (Path, optional): The absolute path of the yaml file to write the output to, if any. Defaults
to None, and simply prints to the terminal
output_yaml: The path to the yaml file where the configuration is written to. Defaults
to None, in which case the configuration is printed to the terminal.
"""
print("[INFO] Configuring the passed options into a yaml...")
print("[INFO] Configuring the passed options into a yaml...\n")
# resolve the output argument
if output_yaml is not None:
output = ["--output", output_yaml]
else:
output = []
# run the docker compose config command to generate the configuration
subprocess.run(
["docker", "compose"] + self.add_yamls + self.add_profiles + self.add_env_files + ["config"] + output,
check=False,
cwd=self.context_dir,
env=self.environ,
)
"""
Helper functions.
"""
def _resolve_image_extension(self, yamls: list[str] | None = None, envs: list[str] | None = None):
"""
Resolve the image extension by setting up YAML files, profiles, and environment files for the Docker compose command.
Args:
yamls: A list of yaml files to extend ``docker-compose.yaml`` settings. These are extended in the order
they are provided.
envs: A list of environment variable files to extend the ``.env.base`` file. These are extended in the order
they are provided.
"""
self.add_yamls = ["--file", "docker-compose.yaml"]
self.add_profiles = ["--profile", f"{self.profile}"]
self.add_env_files = ["--env-file", ".env.base"]
# extend env file based on profile
if self.profile != "base":
self.add_env_files += ["--env-file", f".env.{self.profile}"]
# extend the env file based on the passed envs
if envs is not None:
for env in envs:
self.add_env_files += ["--env-file", env]
# extend the docker-compose.yaml based on the passed yamls
if yamls is not None:
for yaml in yamls:
self.add_yamls += ["--file", yaml]
def _parse_dot_vars(self):
"""Parse the environment variables from the .env files.
Based on the passed ".env" files, this function reads the environment variables and stores them in a dictionary.
The environment variables are read in order and overwritten if there are name conflicts, mimicking the behavior
of Docker compose.
"""
self.dot_vars: dict[str, Any] = {}
# check if the number of arguments is even for the env files
if len(self.add_env_files) % 2 != 0:
raise RuntimeError(
"The parameters for env files are configured incorrectly. There should be an even number of arguments."
f" Received: {self.add_env_files}."
)
# read the environment variables from the .env files
for i in range(1, len(self.add_env_files), 2):
with open(self.context_dir / self.add_env_files[i]) as f:
self.dot_vars.update(dict(line.strip().split("=", 1) for line in f if "=" in line))
docker/utils/statefile.py → docker/utils/state_file.py
View file @ 173b5f3e
......@@ -2,6 +2,7 @@
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
from __future__ import annotations
import configparser
......@@ -10,124 +11,141 @@ from pathlib import Path
from typing import Any
def load_cfg_file(path: Path) -> ConfigParser:
"""
Load the contents of a config file.
If the file exists, read its contents and return them as a dictionary.
If the file does not exist, create an empty file and return an empty dictionary.
Args:
statefile: The path to the config file.
Returns:
dict: The contents of the config file as a dictionary.
"""
cfg = ConfigParser()
cfg.read(path)
return cfg
def save_cfg_file(path: Path, cfg: ConfigParser):
"""
Save a dictionary to a config file.
Args:
path: The path to the config file.
data: The data to be saved to the config file.
"""
with open(path, "w+") as file:
cfg.write(file)
class StateFile:
"""A class to manage state variables parsed from a configuration file.
This class provides a simple interface to set, get, and delete variables from a configuration
object. It also provides the ability to save the configuration object to a file.
class Statefile:
"""
A class to manage state variables stored in a cfg file.
It thinly wraps around the ConfigParser class from the configparser module.
"""
def __init__(self, path: Path, namespace: str | None = None):
"""
Initialize the Statefile object with the path to the cfg file.
"""Initialize the class instance and load the configuration file.
Args:
path: The path to the cfg file.
namespace: Namespace a section of the cfg.
Defaults to None, and all member functions will have
to specify section or else set Statefile.namespace directly.
path: The path to the configuration file.
namespace: The default namespace to use when setting and getting variables.
Namespace corresponds to a section in the configuration file. Defaults to None,
meaning all member functions will have to specify the section explicitly,
or :attr:`StateFile.namespace` must be set manually.
"""
self.path = path
self.namespace = namespace
self.load_cfg()
# load the configuration file
self.load()
def __del__(self):
"""
Save self.loaded_cfg to self.path upon deconstruction
Save the loaded configuration to the initial file path upon deconstruction. This helps
ensure that the configuration file is always up to date.
"""
self.save_cfg()
# save the configuration file
self.save()
def set_variable(self, key: str, value: Any, section: str | None = None):
"""
Set a variable in the cfg file.
Operations.
"""
def set_variable(self, key: str, value: Any, section: str | None = None):
"""Set a variable into the configuration object.
Note:
Since we use the ConfigParser class, the section names are case-sensitive but the keys are not.
Args:
key: The key of the variable to be set.
value: The value of the variable to be set.
section: section of the cfg. Defaults to the self.namespace
section: The section of the configuration object to set the variable in.
Defaults to None, in which case the default section is used.
Raises:
configparser.Error: If no section is specified and the default section is None.
"""
# resolve the section
if section is None:
if self.namespace is None:
raise configparser.Error("No section specified")
raise configparser.Error("No section specified. Please specify a section or set StateFile.namespace.")
section = self.namespace
# create section if it does not exist
if section not in self.loaded_cfg.sections():
self.loaded_cfg.add_section(section)
# set the variable
self.loaded_cfg.set(section, key, value)
def load_variable(self, key: str, section: str | None = None) -> Any:
"""
Load a variable from the cfg file.
def get_variable(self, key: str, section: str | None = None) -> Any:
"""Get a variable from the configuration object.
Note:
Since we use the ConfigParser class, the section names are case-sensitive but the keys are not.
Args:
key: The key of the variable to be loaded.
section: section of the cfg. Defaults to the self.namespace
section: The section of the configuration object to read the variable from.
Defaults to None, in which case the default section is used.
Returns:
any: The value of the variable, or None if the key does not exist.
The value of the variable. It is None if the key does not exist.
Raises:
configparser.Error: If no section is specified and the default section is None.
"""
# resolve the section
if section is None:
if self.namespace is None:
raise configparser.Error("No section specified")
raise configparser.Error("No section specified. Please specify a section or set StateFile.namespace.")
section = self.namespace
return self.loaded_cfg.get(section, key, fallback=None)
def delete_variable(self, key: str, section: str | None = None):
"""
Delete a variable from the cfg file.
"""Delete a variable from the configuration object.
Note:
Since we use the ConfigParser class, the section names are case-sensitive but the keys are not.
Args:
key: The key of the variable to be deleted.
section: section of the cfg. Defaults to self.namespace
section: The section of the configuration object to remove the variable from.
Defaults to None, in which case the default section is used.
Raises:
configparser.Error: If no section is specified and the default section is None.
configparser.NoSectionError: If the section does not exist in the configuration object.
configparser.NoOptionError: If the key does not exist in the section.
"""
# resolve the section
if section is None:
if self.namespace is None:
raise configparser.Error("No section specified")
raise configparser.Error("No section specified. Please specify a section or set StateFile.namespace.")
section = self.namespace
# check if the section exists
if section not in self.loaded_cfg.sections():
raise configparser.NoSectionError(f"Section {section} does not exist in {self.path}")
raise configparser.NoSectionError(f"Section '{section}' does not exist in the file: {self.path}")
# check if the key exists
if self.loaded_cfg.has_option(section, key):
self.loaded_cfg.remove_option(section, key)
else:
raise configparser.NoOptionError(option=key, section=section)
def load_cfg(self):
"""
Calls load_cfg_file() to populate self.loaded_cfg with the
data stored at self.path
Operations - File I/O.
"""
self.loaded_cfg = load_cfg_file(self.path)
def save_cfg(self):
"""
Calls save_cfg_file() to write the contents of self.loaded_cfg
to the file at self.path
def load(self):
"""Load the configuration file into memory.
This function reads the contents of the configuration file into memory.
If the file does not exist, it creates an empty file.
"""
save_cfg_file(self.path, self.loaded_cfg)
self.loaded_cfg = ConfigParser()
self.loaded_cfg.read(self.path)
def save(self):
"""Save the configuration file to disk."""
with open(self.path, "w+") as f:
self.loaded_cfg.write(f)
docker/utils/x11_utils.py
View file @ 173b5f3e
......@@ -2,121 +2,156 @@
# All rights reserved.
#
# SPDX-License-Identifier: BSD-3-Clause
"""Utility functions for managing X11 forwarding in the docker container."""
from __future__ import annotations
import os
import shutil
import subprocess
import sys
from pathlib import Path
from utils.statefile import Statefile
from .state_file import StateFile
# This method of x11 enabling forwarding was inspired by osrf/rocker
# https://github.com/osrf/rocker
def configure_x11(statefile: Statefile) -> dict[str, str]:
"""
Configure X11 forwarding by creating and managing a temporary .xauth file.
def configure_x11(statefile: StateFile) -> dict[str, str]:
"""Configure X11 forwarding by creating and managing a temporary .xauth file.
If xauth is not installed, the function prints an error message and exits. The message
instructs the user to install xauth with 'apt install xauth'.
If xauth is not installed, prompt the user to install it. If the .xauth file
does not exist, create it and configure it with the necessary xauth cookie.
If the .xauth file does not exist, the function creates it and configures it with the necessary
xauth cookie.
Args:
statefile: An instance of the Statefile class to manage state variables.
statefile: An instance of the configuration file class.
Returns:
dict: A dictionary where the key is __ISAACLAB_TMP_XAUTH (referenced in x11.yaml)
and the value is the corresponding tmp file which has been created.
A dictionary with two key-value pairs:
- "__ISAACLAB_TMP_XAUTH": The path to the temporary .xauth file.
- "__ISAACLAB_TMP_DIR": The path to the directory where the temporary .xauth file is stored.
"""
# check if xauth is installed
if not shutil.which("xauth"):
print("[INFO] xauth is not installed.")
print("[INFO] Please install it with 'apt install xauth'")
exit(1)
# set the namespace to X11 for the statefile
statefile.namespace = "X11"
__ISAACLAB_TMP_XAUTH = statefile.load_variable("__ISAACLAB_TMP_XAUTH")
if __ISAACLAB_TMP_XAUTH is None or not Path(__ISAACLAB_TMP_XAUTH).exists():
__ISAACLAB_TMP_DIR = subprocess.run(["mktemp", "-d"], capture_output=True, text=True, check=True).stdout.strip()
__ISAACLAB_TMP_XAUTH = create_x11_tmpfile(tmpdir=Path(__ISAACLAB_TMP_DIR))
statefile.set_variable("__ISAACLAB_TMP_XAUTH", str(__ISAACLAB_TMP_XAUTH))
# load the value of the temporary xauth file
tmp_xauth_value = statefile.get_variable("__ISAACLAB_TMP_XAUTH")
if tmp_xauth_value is None or not Path(tmp_xauth_value).exists():
# create a temporary directory to store the .xauth file
tmp_dir = subprocess.run(["mktemp", "-d"], capture_output=True, text=True, check=True).stdout.strip()
# create the .xauth file
tmp_xauth_value = create_x11_tmpfile(tmpdir=Path(tmp_dir))
# set the statefile variable
statefile.set_variable("__ISAACLAB_TMP_XAUTH", str(tmp_xauth_value))
else:
__ISAACLAB_TMP_DIR = Path(__ISAACLAB_TMP_XAUTH).parent
return {"__ISAACLAB_TMP_XAUTH": str(__ISAACLAB_TMP_XAUTH), "__ISAACLAB_TMP_DIR": str(__ISAACLAB_TMP_DIR)}
tmp_dir = Path(tmp_xauth_value).parent
return {"__ISAACLAB_TMP_XAUTH": str(tmp_xauth_value), "__ISAACLAB_TMP_DIR": str(tmp_dir)}
def x11_check(statefile: Statefile) -> tuple[list[str], dict[str, str]] | None:
"""
Check and configure X11 forwarding based on user input and existing state.
Prompt the user to enable or disable X11 forwarding if not already configured.
Configure X11 forwarding if enabled.
def x11_check(statefile: StateFile) -> tuple[list[str], dict[str, str]] | None:
"""Check and configure X11 forwarding based on user input and existing state.
This function checks if X11 forwarding is enabled in the configuration file. If it is not configured,
the function prompts the user to enable or disable X11 forwarding. If X11 forwarding is enabled, the function
configures X11 forwarding by creating a temporary .xauth file.
Args:
statefile: An instance of the Statefile class to manage state variables.
statefile: An instance of the configuration file class.
Returns:
list or str: A list containing the x11.yaml file configuration option if X11 forwarding is enabled,
otherwise None
If X11 forwarding is enabled, the function returns a tuple containing the following:
- A list containing the x11.yaml file configuration option for docker-compose.
- A dictionary containing the environment variables for the container.
If X11 forwarding is disabled, the function returns None.
"""
# set the namespace to X11 for the statefile
statefile.namespace = "X11"
__ISAACLAB_X11_FORWARDING_ENABLED = statefile.load_variable("__ISAACLAB_X11_FORWARDING_ENABLED")
if __ISAACLAB_X11_FORWARDING_ENABLED is None:
print("[INFO] X11 forwarding from the Isaac Lab container is off by default.")
# check if X11 forwarding is enabled
is_x11_forwarding_enabled = statefile.get_variable("X11_FORWARDING_ENABLED")
if is_x11_forwarding_enabled is None:
print("[INFO] X11 forwarding from the Isaac Lab container is disabled by default.")
print(
"[INFO] It will fail if there is no display, or this script is being run via ssh without proper"
" configuration."
)
x11_answer = input("Would you like to enable it? (y/N) ")
# parse the user's input
if x11_answer.lower() == "y":
__ISAACLAB_X11_FORWARDING_ENABLED = "1"
statefile.set_variable("__ISAACLAB_X11_FORWARDING_ENABLED", "1")
is_x11_forwarding_enabled = "1"
print("[INFO] X11 forwarding is enabled from the container.")
else:
__ISAACLAB_X11_FORWARDING_ENABLED = "0"
statefile.set_variable("__ISAACLAB_X11_FORWARDING_ENABLED", "0")
is_x11_forwarding_enabled = "0"
print("[INFO] X11 forwarding is disabled from the container.")
# remember the user's choice and set the statefile variable
statefile.set_variable("X11_FORWARDING_ENABLED", is_x11_forwarding_enabled)
else:
print(f"[INFO] X11 Forwarding is configured as {__ISAACLAB_X11_FORWARDING_ENABLED} in .container.cfg")
if __ISAACLAB_X11_FORWARDING_ENABLED == "1":
print("[INFO] To disable X11 forwarding, set __ISAACLAB_X11_FORWARDING_ENABLED=0 in .container.cfg")
# print the current configuration
print(f"[INFO] X11 Forwarding is configured as '{is_x11_forwarding_enabled}' in '.container.cfg'.")
# print help message to enable/disable X11 forwarding
if is_x11_forwarding_enabled == "1":
print("\tTo disable X11 forwarding, set 'X11_FORWARDING_ENABLED=0' in '.container.cfg'.")
else:
print("[INFO] To enable X11 forwarding, set __ISAACLAB_X11_FORWARDING_ENABLED=1 in .container.cfg")
print("\tTo enable X11 forwarding, set 'X11_FORWARDING_ENABLED=1' in '.container.cfg'.")
if __ISAACLAB_X11_FORWARDING_ENABLED == "1":
x11_envar = configure_x11(statefile)
if is_x11_forwarding_enabled == "1":
x11_envars = configure_x11(statefile)
# If X11 forwarding is enabled, return the proper args to
# compose the x11.yaml file. Else, return an empty string.
return (["--file", "x11.yaml"], x11_envar)
return ["--file", "x11.yaml"], x11_envars
return None
def x11_cleanup(statefile: Statefile):
"""
Clean up the temporary .xauth file used for X11 forwarding.
def x11_cleanup(statefile: StateFile):
"""Clean up the temporary .xauth file used for X11 forwarding.
If the .xauth file exists, delete it and remove the corresponding state variable.
If the .xauth file exists, this function deletes it and remove the corresponding state variable.
Args:
statefile: An instance of the Statefile class to manage state variables.
statefile: An instance of the configuration file class.
"""
# set the namespace to X11 for the statefile
statefile.namespace = "X11"
__ISAACLAB_TMP_XAUTH = statefile.load_variable("__ISAACLAB_TMP_XAUTH")
if __ISAACLAB_TMP_XAUTH is not None and Path(__ISAACLAB_TMP_XAUTH).exists():
print(f"[INFO] Removing temporary Isaac Lab .xauth file {__ISAACLAB_TMP_XAUTH}.")
Path(__ISAACLAB_TMP_XAUTH).unlink()
# load the value of the temporary xauth file
tmp_xauth_value = statefile.get_variable("__ISAACLAB_TMP_XAUTH")
# if the file exists, delete it and remove the state variable
if tmp_xauth_value is not None and Path(tmp_xauth_value).exists():
print(f"[INFO] Removing temporary Isaac Lab '.xauth' file: {tmp_xauth_value}.")
Path(tmp_xauth_value).unlink()
statefile.delete_variable("__ISAACLAB_TMP_XAUTH")
def create_x11_tmpfile(tmpfile: Path | None = None, tmpdir: Path | None = None) -> Path:
"""
Creates an .xauth file with an MIT-MAGIC-COOKIE derived from the current DISPLAY,
returns its location as a Path.
"""Creates an .xauth file with an MIT-MAGIC-COOKIE derived from the current ``DISPLAY`` environment variable.
Args:
tmpfile: A Path to a file which will be filled with the correct .xauth info
tmpdir: A Path to the directory where a random tmp file will be made,
used as an --tmpdir arg to mktemp
tmpfile: A Path to a file which will be filled with the correct .xauth info.
tmpdir: A Path to the directory where a random tmp file will be made.
This is used as an ``--tmpdir arg`` to ``mktemp`` bash command.
Returns:
The Path to the .xauth file.
"""
if tmpfile is None:
if tmpdir is None:
......@@ -132,28 +167,61 @@ def create_x11_tmpfile(tmpfile: Path | None = None, tmpdir: Path | None = None)
else:
tmpfile.touch()
tmp_xauth = tmpfile
# Derive current MIT-MAGIC-COOKIE and make it universally addressable
xauth_cookie = subprocess.run(
["xauth", "nlist", os.environ["DISPLAY"]], capture_output=True, text=True, check=True
).stdout.replace("ffff", "")
# Merge the new cookie into the create .tmp file
subprocess.run(["xauth", "-f", tmp_xauth, "nmerge", "-"], input=xauth_cookie, text=True, check=True)
return tmp_xauth
def x11_refresh(statefile: Statefile):
"""
If x11 is enabled, generates a new .xauth file with the current MIT-MAGIC-COOKIE-1,
using the same filename so that the bind-mount and
XAUTHORITY var from build-time still work. DISPLAY will also
need to be updated in the container environment command.
def x11_refresh(statefile: StateFile):
"""Refresh the temporary .xauth file used for X11 forwarding.
If x11 is enabled, this function generates a new .xauth file with the current MIT-MAGIC-COOKIE-1.
The new file uses the same filename so that the bind-mount and ``XAUTHORITY`` var from build-time
still work.
As the envar ``DISPLAY` informs the contents of the MIT-MAGIC-COOKIE-1, that value within the container
will also need to be updated to the current value on the host. Currently, this done automatically in
:meth:`ContainerInterface.enter` method.
The function exits if X11 forwarding is enabled but the temporary .xauth file does not exist. In this case,
the user must rebuild the container.
Args:
statefile: An instance of the Statefile class to manage state variables.
statefile: An instance of the configuration file class.
"""
# set the namespace to X11 for the statefile
statefile.namespace = "X11"
__ISAACLAB_TMP_XAUTH = Path(statefile.load_variable("__ISAACLAB_TMP_XAUTH"))
if __ISAACLAB_TMP_XAUTH is not None and __ISAACLAB_TMP_XAUTH.exists():
__ISAACLAB_TMP_XAUTH.unlink()
create_x11_tmpfile(tmpfile=__ISAACLAB_TMP_XAUTH)
statefile.set_variable("__ISAACLAB_TMP_XAUTH", str(__ISAACLAB_TMP_XAUTH))
# check if X11 forwarding is enabled
is_x11_forwarding_enabled = statefile.get_variable("X11_FORWARDING_ENABLED")
# load the value of the temporary xauth file
tmp_xauth_value = statefile.get_variable("__ISAACLAB_TMP_XAUTH")
# print the current configuration
if is_x11_forwarding_enabled is not None:
status = "enabled" if is_x11_forwarding_enabled == "1" else "disabled"
print(f"[INFO] X11 Forwarding is {status} from the settings in '.container.cfg'")
# if the file exists, delete it and create a new one
if tmp_xauth_value is not None and Path(tmp_xauth_value).exists():
# remove the file and create a new one
Path(tmp_xauth_value).unlink()
create_x11_tmpfile(tmpfile=Path(tmp_xauth_value))
# update the statefile with the new path
statefile.set_variable("__ISAACLAB_TMP_XAUTH", str(tmp_xauth_value))
elif tmp_xauth_value is None:
if is_x11_forwarding_enabled is not None and is_x11_forwarding_enabled == "1":
print(
"[ERROR] X11 forwarding is enabled but the temporary .xauth file does not exist."
" Please rebuild the container by running: './docker/container.py start'"
)
sys.exit(1)
else:
print("[INFO] X11 forwarding is disabled. No action taken.")
docs/source/deployment/docker.rst
View file @ 173b5f3e
......@@ -73,14 +73,14 @@ Directory Organization
The root of the Isaac Lab repository contains the ``docker`` directory that has various files and scripts
needed to run Isaac Lab inside a Docker container. A subset of these are summarized below:
* ``Dockerfile.base``: Defines the isaaclab image by overlaying Isaac Lab dependencies onto the Isaac Sim Docker image.
``Dockerfiles`` which end with something else, (i.e. ``Dockerfile.ros2``) build an `image_extension <#isaac-lab-image-extensions>`_.
* ``docker-compose.yaml``: Creates mounts to allow direct editing of Isaac Lab code from the host machine that runs
* **Dockerfile.base**: Defines the base Isaac Lab image by overlaying its dependencies onto the Isaac Sim Docker image.
Dockerfiles which end with something else, (i.e. ``Dockerfile.ros2``) build an `image extension <#isaac-lab-image-extensions>`_.
* **docker-compose.yaml**: Creates mounts to allow direct editing of Isaac Lab code from the host machine that runs
the container. It also creates several named volumes such as ``isaac-cache-kit`` to
store frequently re-used resources compiled by Isaac Sim, such as shaders, and to retain logs, data, and documents.
* ``base.env``: Stores environment variables required for the ``base`` build process and the container itself. ``.env``
files which end with something else (i.e. ``.env.ros2``) define these for `image_extension <#isaac-lab-image-extensions>`_.
* ``container.py``: A script that interfaces with tools in ``utils`` to configure and build the image,
* **.env.base**: Stores environment variables required for the ``base`` build process and the container itself. ``.env``
files which end with something else (i.e. ``.env.ros2``) define these for `image extension <#isaac-lab-image-extensions>`_.
* **container.py**: A utility script that interfaces with tools in ``utils`` to configure and build the image,
and run and interact with the container.
Running the Container
......@@ -95,22 +95,22 @@ Running the Container
For a faster development cycle, we mount the following directories in the Isaac Lab repository into the container
so that you can edit their files from the host machine:
* ``source``: This is the directory that contains the Isaac Lab source code.
* ``docs``: This is the directory that contains the source code for Isaac Lab documentation. This is overlaid except
* **IsaacLab/source**: This is the directory that contains the Isaac Lab source code.
* **IsaacLab/docs**: This is the directory that contains the source code for Isaac Lab documentation. This is overlaid except
for the ``_build`` subdirectory where build artifacts are stored.
The script ``container.py`` parallels three basic ``docker compose`` commands. Each can accept an `image_extension argument <#isaac-lab-image-extensions>`_,
or else they will default to image_extension ``base``:
The script ``container.py`` parallels basic ``docker compose`` commands. Each can accept an `image extension argument <#isaac-lab-image-extensions>`_,
or else they will default to the ``base`` image extension. These commands are:
1. ``start``: This builds the image and brings up the container in detached mode (i.e. in the background).
2. ``enter``: This begins a new bash process in an existing isaaclab container, and which can be exited
* **start**: This builds the image and brings up the container in detached mode (i.e. in the background).
* **enter**: This begins a new bash process in an existing Isaac Lab container, and which can be exited
without bringing down the container.
3. ``config``: This outputs the compose.yaml which would be result from the inputs given to ``container.py start``. This command is useful
* **config**: This outputs the compose.yaml which would be result from the inputs given to ``container.py start``. This command is useful
for debugging a compose configuration.
4. ``copy``: This copies the ``logs``, ``data_storage`` and ``docs/_build`` artifacts, from the ``isaac-lab-logs``, ``isaac-lab-data`` and ``isaac-lab-docs``
* **copy**: This copies the ``logs``, ``data_storage`` and ``docs/_build`` artifacts, from the ``isaac-lab-logs``, ``isaac-lab-data`` and ``isaac-lab-docs``
volumes respectively, to the ``docker/artifacts`` directory. These artifacts persist between docker container instances and are shared between image extensions.
5. ``stop``: This brings down the container and removes it.
* **stop**: This brings down the container and removes it.
The following shows how to launch the container in a detached state and enter it:
......@@ -118,15 +118,15 @@ The following shows how to launch the container in a detached state and enter it
# Launch the container in detached mode
# We don't pass an image extension arg, so it defaults to 'base'
python docker/container.py start
./docker/container.py start
# If we want to add .env or .yaml files to customize our compose config,
# we can simply specify them in the same manner as the compose cli
# python docker/container.py start --file my-compose.yaml --env-file .env.my-vars
# ./docker/container.py start --file my-compose.yaml --env-file .env.my-vars
# Enter the container
# We pass 'base' explicitly, but if we hadn't it would default to 'base'
python docker/container.py enter base
./docker/container.py enter base
To copy files from the base container to the host machine, you can use the following command:
......@@ -141,18 +141,26 @@ directories to the ``docker/artifacts`` directory. This is useful for copying th
.. code:: bash
# stop the container
python docker/container.py stop
./docker/container.py stop
X11 forwarding
~~~~~~~~~~~~~~
The container supports X11 forwarding, which allows the user to run GUI applications from the container and display them
on the host machine.
The container supports X11 forwarding, which allows the user to run GUI applications from the container
and display them on the host machine.
The first time a container is started with ``python docker/container.py start``, the script prompts
the user whether to activate X11 forwarding. This will create a file ``docker/.container.cfg`` to store the user's choice.
Subsequently, X11 forwarding can be toggled by changing ``__ISAACLAB_X11_FORWARDING_ENABLED`` to 0 or 1 in ``docker/.container.cfg``.
The first time a container is started with ``./docker/container.py start``, the script prompts
the user whether to activate X11 forwarding. This will create a file at ``docker/.container.cfg``
to store the user's choice for future runs.
If you want to change the choice, you can set the parameter ``X11_FORWARDING_ENABLED`` to '0' or '1'
in the ``docker/.container.cfg`` file to disable or enable X11 forwarding, respectively. After that, you need to
re-build the container by running ``./docker/container.py start``. The rebuilding process ensures that the changes
are applied to the container. Otherwise, the changes will not take effect.
After the container is started, you can enter the container and run GUI applications from it with X11 forwarding enabled.
The display will be forwarded to the host machine.
Python Interpreter
......@@ -233,22 +241,23 @@ Isaac Lab Image Extensions
--------------------------
The produced image depends upon the arguments passed to ``container.py start`` and ``container.py stop``. These
commands accept an ``image_extension`` as an additional argument. If no argument is passed, then these
commands default to ``base``. Currently, the only valid ``image_extension`` arguments are (``base``, ``ros2``).
Only one ``image_extension`` can be passed at a time, and the produced container will be named ``isaac-lab-${profile}``.
commands accept an image extension parameter as an additional argument. If no argument is passed, then this
parameter defaults to ``base``. Currently, the only valid values are (``base``, ``ros2``).
Only one image extension can be passed at a time. The produced container will be named ``isaac-lab-${profile}``,
where ``${profile}`` is the image extension name.
.. code:: bash
# start base by default
python docker/container.py start
./docker/container.py start
# stop base explicitly
python docker/container.py stop base
./docker/container.py stop base
# start ros2 container
python docker/container.py start ros2
./docker/container.py start ros2
# stop ros2 container
python docker/container.py stop ros2
./docker/container.py stop ros2
The passed ``image_extension`` argument will build the image defined in ``Dockerfile.${image_extension}``,
The passed image extension argument will build the image defined in ``Dockerfile.${image_extension}``,
with the corresponding `profile`_ in the ``docker-compose.yaml`` and the envars from ``.env.${image_extension}``
in addition to the ``.env.base``, if any.
......@@ -258,54 +267,21 @@ ROS2 Image Extension
In ``Dockerfile.ros2``, the container installs ROS2 Humble via an `apt package`_, and it is sourced in the ``.bashrc``.
The exact version is specified by the variable ``ROS_APT_PACKAGE`` in the ``.env.ros2`` file,
defaulting to ``ros-base``. Other relevant ROS2 variables are also specified in the ``.env.ros2`` file,
including variables defining the `various middleware`_ options. The container defaults to ``FastRTPS``, but ``CylconeDDS``
is also supported. Each of these middlewares can be `tuned`_ using their corresponding ``.xml`` files under ``docker/.ros``.
Known Issues
------------
Invalid mount config for type "bind"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you see the following error when building the container:
including variables defining the `various middleware`_ options.
.. code:: text
The container defaults to ``FastRTPS``, but ``CylconeDDS`` is also supported. Each of these middlewares can be
`tuned`_ using their corresponding ``.xml`` files under ``docker/.ros``.
⠋ Container isaaclab Creating 0.0s
Error response from daemon: invalid mount config for type "bind": bind source path does not exist: ${HOME}/.Xauthority
This means that the ``.Xauthority`` file is not present in the home directory of the host machine.
The portion of the docker-compose.yaml that enables this is commented out by default, so this shouldn't
happen unless it has been altered. This file is required for X11 forwarding to work. To fix this, you can
create an empty ``.Xauthority`` file in your home directory.
.. code:: bash
.. dropdown:: Parameters for ROS2 Image Extension
:icon: code
touch ${HOME}/.Xauthority
.. literalinclude:: ../../../docker/.env.ros2
:language: bash
A similar error but requires a different fix:
.. code:: text
⠋ Container isaaclab Creating 0.0s
Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /tmp/.X11-unix
This means that the folder/files are either not present or not accessible on the host machine.
The portion of the docker-compose.yaml that enables this is commented out by default, so this
shouldn't happen unless it has been altered. This usually happens when you have multiple docker
versions installed on your machine. To fix this, you can try the following:
* Remove all docker versions from your machine.
.. code:: bash
sudo apt remove docker*
sudo apt remove docker docker-engine docker.io containerd runc docker-desktop docker-compose-plugin
sudo snap remove docker
sudo apt clean autoclean && sudo apt autoremove --yes
* Install the latest version of docker based on the instructions in the setup section.
Known Issues
------------
WebRTC Streaming
~~~~~~~~~~~~~~~~
......
docs/source/deployment/run_docker_example.rst
View file @ 173b5f3e
......@@ -105,7 +105,7 @@ the following command to retrieve our logs from the Docker container and put the
.. code-block:: console
python container.py copy
./container.py copy
We will see a terminal readout reporting the artifacts we have retrieved from the container. If we navigate to
......@@ -120,12 +120,12 @@ but we have retrieved our logs and wish to go inspect them. We can stop the Isaa
.. code-block:: console
python container.py stop
./container.py stop
This will bring down the Docker Isaac Lab container. The image will persist and remain available for further use, as will
the contents of any `volumes`_. If we wish to free up the disk space taken by the image, (~20.1GB), and do not mind repeating
the build process when we next run ``python container.py start``, we may enter the following command to delete the **isaac-lab-base** image:
the build process when we next run ``./container.py start``, we may enter the following command to delete the **isaac-lab-base** image:
.. code-block:: console
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment