Open
Description
Support more runner strategies
Currently, the runner is very straightforward, it executes every test case sequentially, there are different strategies it can support:
- Maybe the runner can execute multiple test cases in parallel, and you can possibly configure how many in parallel at a time. This is linked to the issue about leaking test cases from other tests you are not executing against, I mentioned a possible approach to handle this there (Test Cases may be leaked through RCE #15 (comment))
- The runner has the option of executing a certain group of test cases first, and only executing the others if the first group passes. This is a very common use case, as judging platforms want to often not run against private test cases if the sample test cases didn't pass in the first place. There are many solutions for this, maybe each test case can have a "depend_on" attribute which tells it depends on other test case to succeed, and only runs if that test case passed successfully.