• jsmith-bdai's avatar
    Adds a runner script to execute all tests in the `source` directory (#327) · 51ccd99f
    jsmith-bdai authored
    # Description
    
    This PR adds a script to run all the unit tests (any Python script named
    `test_*.py`) and report on the success rate and timing. It also enables
    the skipping of known breaking tests via tests_to_skip.py. The tests can
    be run via `orbit -t`, and the test output will be printed to both a log
    (`test_results.log`) and to the console.
    
    This can be used for CI/CD but also can be temporarily used to run each
    test without invoking them manually one by one until we have a better
    solution.
    
    A few tests are currently broken; for now, these are added to
    tests_to_skip - I will put follow-up issues + PRs to fix these as soon
    as possible. I recommend we add running tests as a requirement before
    merging PRs until the CI/CD project is complete, too. One way to help
    facilitate this is to have PR authors copy/paste the results at the end
    of the test run to the PR description.
    
    Fixes #309 
    
    ## Type of change
    
    - New feature (non-breaking change which adds functionality)
    
    ## Screenshots
    
    An example test result summary with timeout set to 60 seconds running on
    my workstation:
    
    ```
    ===================
    Test Result Summary
    ===================
    Total: 39
    Passing: 29
    Failing: 0
    Skipped: 8
    Timing Out: 2
    Passing Percentage: 94.87%
    Total Time Elapsed: 0.0h31.0m52.38s
    ```
    
    ## Checklist
    
    - [x] I have run the [`pre-commit` checks](https://pre-commit.com/) with
    `./orbit.sh --format`
    - [ ] I have made corresponding changes to the documentation
    - [x] My changes generate no new warnings
    - [x] I have added tests that prove my fix is effective or that my
    feature works
    - [ ] I have updated the changelog and the corresponding version in the
    extension's `config/extension.toml` file
    - [ ] I have added my name to the `CONTRIBUTORS.md` or my name already
    exists there
    51ccd99f
tests_to_skip.py 716 Bytes