Workflow Designer Enhancement - Parameters & Context

[vivekorch]

ST2 WORKFLOW DESIGNER
USER EXPERIENCE ENHANCEMENTS

1. Problem Statement
ST2 Workflow Composer Graphical User Interface requires parameter names in a non-intuitive syntax. For example, standard way of adding parameters to an action requires a strict notation

<% ctx().myParameter %>

ctx() which represents "Context" need not be exposed to the end user. Further, strict mandate of "<% " is error prone.
Same issue exists in <% failed() %> and <% succeeded() %> constructs in TRANSITIONS.
Secondly, users need to remember the parameter names and type them correctly. A mistyped parameter name gets revealed much later resulting in inefficient workflow creation process.

2. Expected Behavior
Just like modern cars don?t need users to change gears, the details of ctx() etc need to be taken care of by GUI logic. Users are only concerned with parameters, and should be allowed to enter them in most intuitive manner.

Users need to be given an optional drop down menu to select parameters from. This way they can chose right parameter.

In case users choose to type the parameter name, automatic auto-completion should take place as users type out the parameter.

The list of parameters should be taken from workflow parameters, as defined by users. This list can change, meaning users can add/delete/modify parameter. This should be handled intelligently. Meaning backend logic should keep track of parameters as defined by users and which one of those are being used in different actions. If a parameter is used in one or more actions, users should not be allowed to delete it from workflow; they should first edit the corresponding actions.

In TRANSITIONS tab, the "When" clause should be a drop down that has "Succeeded", "Failed".

3. Use Cases
3.1. Workflow Creation
Recommended workflow creation steps are as follows
* Create new workflow by clicking on "+" button
* Select your Pack from drop down menu
* Name your workflow
* Add description
* Save the workflow
At this point workflow is created but not useful yet

3.2. Adding Parameters/Vars to your workflow
* Select PARAMETER or VARS tab
* Add parameters/vars by selecting "ADD PAREMETER"/"ADD VARIABLE". Name your parameters/vars clearly so that they can be used unambiguously in your workflow
* Orquesta forces some rules. For example no dashes are allowed. GUI must check for these rules and reject names that don't follow them, and suggest appropriate changes via useful help messages

3.3. Adding Actions to your Workflow
* Drag and drop required action(s) in the workflow
* Join the actions using arrows (see St2 documentation on how to do this) and create your workflow
* Review your workflow to make sure it does what you need it to do. Goal is to verify logic and not details

3.4. Adding Conditional Branching to your Workflow
* The 'When' clause will have a drop down for "Failed" and "Succeded". Pick what you want and create rest of the logic as before

3.5. Add Parameters and Variables to your Workflow
Most of the suggested changes will appear in this step
* For each action, fill in all the required parameters
* There are three ways you can pass parameter to an action. Easiest one is to have a hard coded constant/number. This is not recommended, but sometimes you may end up doing this.

Second is that you may want to pass results from earlier action as parameter.

Third, and most common, is that your parameter is passed from workflow parameter definition. For this purpose there should be a dropdown that lists currently defined workflow parameters. You can chose right parameter. GUI must take care of making sure right datatype is maintained when displaying the list. In this context, users should also be allowed to type the parameter, GUI providing auto-completed list as user types.

3.6. Deleting/Editing a parameter from Workflow
* Users can delete a parameter from GUI by selecting PARAMETERS tab and "Trash Can" icon next to given parameter
* GUI should check if this parameter is currently being used by any actions in the workflow
* If parameter is being used then delete action should not be allowed, with proper error message. Error message should list all the actions that are using the parameter
* Same logic for Editing a parameter. Specifically, changing the data type should not be permitted if parameter is being used. Changing description is OK

3.7. Backward Compatibility
Since all the additions are GUI based, there should be 100% backward compatibility. In other words all the workflows created using St2 3.2 or 3.3 version should be viewable/editable in new format.

Note about YAML and other files
This is a GUI specific enhancement. All the YAML and other data files should remain orthogonal to this development. Specifically, there is no change in them. Further, YAML files from older releases such as 3.2 and 3.3 should immediately enjoy the new GUI features without any changes. The saved files from the enhanced GUI should be exactly same as release 3.2 and 3.3.

Note about drop down menu
There should be some way to override the dropdown menu option and type whatever you want to type. If GUI supports this natively then it is well and good. If not then there should be an option to bypass dropdown.

This is important because additional options may be added in the supporting file structure. The GUI may be late in catching up to these features. Users should still be able to enjoy the new features without waiting for "next release of GUI which supports these feature"

[amanda11]

I think this makes a really good discussion and review on the current flow of the existing workflow designer, and I think it would be great to get some feedback from other users to see how they are using it, and what are their common cases, and their feedback on the proposal. Thanks for raising the proposal, I think there are some good points in here.

I like the workflow creation suggestion. I think the workflow creation would be a great addition compared to the way the EWC designer supported workflows.

I like the add parameter to workflows comments and the extra validation, and I like the delete parameter suggestions and checks on removing a parameter in use (the only problem I can see is if a parameter name is the same as something the workflow is publishing, then you might realise that you don't need it as a parameter - but the UI wouldn't know that... unless it also checks before deletion if its ever been in a publish - but this might be an edge case that could be ignored, as I don't think it is that common).

In transitions then I can see succeeded and failed are the most common cases, so might be good to have an easy way to choose them and then add your and/or etc to it. It would need to cater for the case where transition when doesn't have either of succeeded or failed as part of the condition, as that isn't always the case, and the logic for the transition might just be on parameter values etc. For me personally, then succeeded and failed probably accounts for the majority of transitions, but the number that are more complex isn't an insignificant number.

My concern would be changing the way parameters to actions are edited. If it was changed to pull-down, and/or auto-completion, then how to ensure it's still easy for users to put in complex parameters etc. e.g. I can use Jinja or YAQL in my evaluations, those parameter values could have complex YAQL or Jinja expressions being applied to a parameter or result from a previous task. Similarly I might be getting values other than the ctx e.g. from the keyvalue store.

I personally find that the number of tasks in a workflow that just require pure workflow parameters wouldn't be the common case. For me personally the major benefit on adding/editing parameters would be syntax checking of the statements rather than a pull-down.

I can see that this might benefit people new to StackStorm, and maybe would help them bring up some simple workflows quickly. I wonder if it would be possible to have the pull-downs so they were unobtrusive, so you could use them if you wanted, but ignore if you just wanted to have free-form edit (or could switch modes easily between the two alternatively).

[mickmcgrath13]

Some great points in here!

I agree with the abstraction of the syntax (i.e. removal of ctx() or providing dropdown/autocomplete for succeeded() etc) so long as the underlying yaml is still available and configurable (to @amanda11 's point).

This could be accomplished by enhancements to the abstraction (i.e. gui-heavy) ui while still making available the full yaml-based config via the UI.

I agree that, at scale, the more complex transition logic is likely to be more common, so we should account for that, and I don't see why we can't have both.

Additionally, as @amanda11 alluded, ctx isn't the only 'prefix' available for use in a workflow (result(), task, and st2kv, for example)

So, again, to @amanda11 's point

I wonder if it would be possible to have the pull-downs so they were unobtrusive, so you could use them if you wanted, but ignore if you just wanted to have free-form edit (or could switch modes easily between the two alternatively).

I think that if we want an abstraction layer like this in the GUI, this is the only approach that makes sense, and from an implementation perspective, it's definitely doable.

[vivekorch]

Thanks all. I have updated the proposal based on comments.

[m4dcoder]

Workflow parameters, variables that are defined at the top of the workflow, and variables that are published on task transition have different scope. For example, workflow parameters and vars defined at top of the workflow are global for the workflow execution. The vars that are defined on task transition are only available for the branch after it has been published. Ideally, the workflow designer should be able to track the context per branch and the drop down should only show the variables that are with in scope.

The opinions above already cover this to some extent. Any place where variables in the context can be assigned can also be complex YAQL and Jinja expressions. The UX enhancement should be able to handle simple cases as well as complex expressions. It's good idea to have a drop down to allow users handle the simple use cases or for new users to learn the basics. But once selected, I think the workflow designer should print the actual form in the free form edit text box (i.e. select succeeded but print succeeded(), select var1 but print ctx().var1).

We can also allow an option for users to write & test more complex YAQL or Jinja expressions before populating the text box. We have a separate community tool called the yaqluator at http://yaqluator.com. If we can develop a simplified version as an extended part of the workflow designer and with context data specific to the workflow, I think it can help reduce the barrier of entry.

[nzlosh]

On the point of yaqluator, there is a tool built by a community member that is the same sort things as yaqluator, but it handles jinja and yaql expressions and allows users to fetch executions to populate data sets. The user can then write expression for the fetched data set using StackStorm Orquesta syntax. Perhaps we could consider integrating it?
https://github.com/Xorso/orquestaevaluator

There is one issue with orquestaevalutator which is there is no licence associated with the project. lorddaren/orquestaevaluator#2, so the developer needs to state which licence the project uses before we could consider integrating it.

[lorddaren]

There is one issue with orquestaevalutator which is there is no licence associated with the project. Xorso/orquestaevaluator#2, so the developer needs to state which licence the project uses before we could consider integrating it.

I have updated the license to MIT so that it can be used in the project. Anyway we can help contribute to the core just let me know.