-
jsmith-bdai authored
# Description This PR adds a script to run all the unit tests (any Python script named `test_*.py`) and report on the success rate and timing. It also enables the skipping of known breaking tests via tests_to_skip.py. The tests can be run via `orbit -t`, and the test output will be printed to both a log (`test_results.log`) and to the console. This can be used for CI/CD but also can be temporarily used to run each test without invoking them manually one by one until we have a better solution. A few tests are currently broken; for now, these are added to tests_to_skip - I will put follow-up issues + PRs to fix these as soon as possible. I recommend we add running tests as a requirement before merging PRs until the CI/CD project is complete, too. One way to help facilitate this is to have PR authors copy/paste the results at the end of the test run to the PR description. Fixes #309 ## Type of change - New feature (non-breaking change which adds functionality) ## Screenshots An example test result summary with timeout set to 60 seconds running on my workstation: ``` =================== Test Result Summary =================== Total: 39 Passing: 29 Failing: 0 Skipped: 8 Timing Out: 2 Passing Percentage: 94.87% Total Time Elapsed: 0.0h31.0m52.38s ``` ## Checklist - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with `./orbit.sh --format` - [ ] I have made corresponding changes to the documentation - [x] My changes generate no new warnings - [x] I have added tests that prove my fix is effective or that my feature works - [ ] I have updated the changelog and the corresponding version in the extension's `config/extension.toml` file - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already exists there