Exiting the entire pipeline early if a condition is met [help]

[bakaburg1]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Hi! Is there a way to conclude a pipeline if a condition is met?
I have a target that compiles a set of combinations of parameters and then filters out the already processed combinations based on a set of files, each with the combination results.
The leftover combinations are then mapped to branches to be processed.

I decided to use the files as ground truth since I needed something easy to inspect; also, the mere existence of the file is not enough to skip the reprocessing since its content is also relevant to this decision.

But if all combinations have already been processed, the pipeline does error: map() cannot be run on an empty set.

I'd like to have a more graceful stopping if this condition is met, but I could not find a graceful way of stopping the pipelines without marking all downstream targets as errored.
The best I could come up with is a conditional stop() and error = "abridge" in the combination target which produces this:

✔ skipping targets (1 so far)...
▶ dispatched target combinations
▶ recorded workspace combinations
✖ errored target combinations
Error: All combinations were already processed.

✖ errored pipeline [2.114 seconds]

It's better than a full-fledged error or a bunch of errored branches, but still, it's reported as an errored pipeline, which is not truthful.

Is there any more elegant solution that marks the pipeline as successfully completed?

[wlandau]

targets isn't designed for this, so there isn't a perfect solution. The error sounds like it gets the job done. Another thing you could try is to have each target check the condition and then run tar_cancel() if it is already met.

[koefoeden]

I had the same issue, and ended up with a solution that I think works quite nicely (although a bit hacky). You simply wrap your target, or a list of any number of targets, inside an if else block containing your condition, for example by using the following tar_hook_outer-command: tar_hook_outer(conditional_targets_list, hook = if (skip_condition) fs::file_create("NULL.txt") else .x)
)

See example below:

not_a_conditional_target <- tar_target(...)

conditional_targets_list <- list(
    tar_target(conditional_target , foo())
    tar_target(conditional_downstream_target, conditional_target)
)

# Pipeline definition at the bottom of the script
list(
    not_a_conditional_target, 
    conditional_targets_list %>% tar_hook_outer(hook = if (your_condition) fs::file_create("NULL.txt") else .x)
)

Basically, all the targets that are wrapped by this command just return a dummy character (and creates a dummy file as well) which is thus a valid return value (i.e. no erroring) for both tar_target and tarchetypes::tar_file-targets (i.e. tar_target() with format = "file"). And since these targets don't care about the upstream targets either during the skip_condition, you can chain many of them.

You can even use this method to dynamically end the pipeline early, or even (effectively) select a specific path through the target DAG during run-time, while ending up at the same downstream targets - which I think is very powerful! For this, you'd just check for which target did not return NULL.txt in the "joining target" in a similar if-else block, and then going forward with the correct dependecy.