• matthewtrepte's avatar
    Adds training benchmark unit tests with input config (#2503) · 4737968a
    matthewtrepte authored
    # Description
    
    <!--
    Thank you for your interest in sending a pull request. Please make sure
    to check the contribution guidelines.
    
    Link:
    https://isaac-sim.github.io/IsaacLab/main/source/refs/contributing.html
    -->
    
    Add unit test for training and evaluating environments across all
    workflows using an input config.
    
    The config determines which environments to train, how long to train,
    and training KPI thresholds.
    
    Every training + evaluation is one pytest, which fails if any of the
    thresholds aren't met.
    
    A KPI json can be created to summarize the trainings. This json also
    functions as a KPI payload which will be uploaded to a Grafana Dashboard
    every week using OSMO CI, using the full mode in the config. Can be used
    to track improvements and regressions.
    
    Dashboard -
    [LINK](https://grafana.nvidia.com/d/cejwy2w46gtmob/24b94e54-6b8b-50a4-969a-121dc573a34c?orgId=105)
    
    <!-- As a practice, it is recommended to open an issue to have
    discussions on the proposed pull request.
    This makes it easier for the community to keep track of what is being
    developed or added, and if a given feature
    is demanded by more than one party. -->
    
    ## Type of change
    
    <!-- As you go through the list, delete the ones that are not
    applicable. -->
    
    - Bug fix (non-breaking change which fixes an issue)
    - New feature (non-breaking change which adds functionality)
    - Breaking change (fix or feature that would cause existing
    functionality to not work as expected)
    - This change requires a documentation update
    
    ## Screenshots
    
    Please attach before and after screenshots of the change if applicable.
    
    <!--
    Example:
    
    | Before | After |
    | ------ | ----- |
    | _gif/png before_ | _gif/png after_ |
    
    To upload images to a PR -- simply drag and drop an image while in edit
    mode and it should upload the image directly. You can then paste that
    source into the above before/after sections.
    -->
    
    ## Checklist
    
    - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
    `./isaaclab.sh --format`
    - [ ] I have made corresponding changes to the documentation
    - [ ] My changes generate no new warnings
    - [ ] I have added tests that prove my fix is effective or that my
    feature works
    - [ ] I have updated the changelog and the corresponding version in the
    extension's `config/extension.toml` file
    - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
    exists there
    
    <!--
    As you go through the checklist above, you can mark something as done by
    putting an x character in it
    
    For example,
    - [x] I have done this task
    - [ ] I have not done this task
    -->
    
    ---------
    Co-authored-by: 's avatarKelly Guo <kellyg@nvidia.com>
    Co-authored-by: 's avatarKelly Guo <kellyguo123@hotmail.com>
    4737968a
Name
Last commit
Last update
.aws Loading commit data...
.github Loading commit data...
.vscode Loading commit data...
apps Loading commit data...
docker Loading commit data...
docs Loading commit data...
scripts Loading commit data...
source Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.flake8 Loading commit data...
.gitattributes Loading commit data...
.gitignore Loading commit data...
.pre-commit-config.yaml Loading commit data...
CONTRIBUTING.md Loading commit data...
CONTRIBUTORS.md Loading commit data...
LICENSE Loading commit data...
LICENSE-mimic Loading commit data...
README.md Loading commit data...
SECURITY.md Loading commit data...
VERSION Loading commit data...
environment.yml Loading commit data...
isaaclab.bat Loading commit data...
isaaclab.sh Loading commit data...
pyproject.toml Loading commit data...