[FEATURE]: Validate prompts referenced in treatments.yaml, surface errors in yaml #122
Description
User Story
As an experiment builder, I want a more proactive warning if the markdown prompts that I've referenced are valid, so that I don't have to find bugs when I try and start up the game.
Feature description
Currently, you won't get a warning that a markdown file is invalid if it doesn't parse as a prompt file - because (for good reason) we don't always validate every markdown.
But, this means that it's possible not to have an error raised for a file when the conditions for parsing it as a prompt are not met - ie, the name and type metadata aren't specified.
We do have another source of information telling us that a particular file shoud be parsed as a prompt file, which is when the experiment designer references it from the treatments.yaml. So, when we validate the treatments.yaml, it would be cool if we could go and check if each of the referenced prompt files has been validated using our parser, and if not (or that validation is out of date) trigger the validation. Then, if the validation fails, surface a warning in the treatment file that the referenced prompt is not valid.
We may be able to tap into vscode's memory of what has been tested with what tests, so that we don't have to re-test every single prompt file every single time the treatments file changes. That would be ideal.
What would this feature enable?
An additional source of information that helps researchers build experiments faster.