Async Execution Functions #3
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds support for node execution functions defined as async. When a node's execution function is defined as async, we can continue executing other nodes while it is processing. Standard uses of `await` should "just work", but people will still have to be careful if they spawn actual threads. Because torch doesn't really have async/await versions of functions, this won't particularly help with most locally-executing nodes, but it does work for e.g. web requests to other machines. Remaining work: 1. The UI doesn't properly display multiple concurrent node executions. 2. I probably need some work to handle the case where one node hits an out-of-memory error (or other exception) while another node is concurrently executing. 3. If people are doing node expansion within an async function and using the `GraphBuilder`, they'll need to provide a manual prefix. The default one won't necessarily work properly in this case. This looks easy to fix in Python 3.12+ with contextvars, but I have to figure out how to do it in 3.11 and earlier.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit adds support for node execution functions defined as async. When
a node's execution function is defined as async, we can continue
executing other nodes while it is processing.
Standard uses of
awaitshould "just work", but people will still haveto be careful if they spawn actual threads. Because torch doesn't really
have async/await versions of functions, this won't particularly help
with most locally-executing nodes, but it does work for e.g. web
requests to other machines.
Remaining work:
out-of-memory error (or other exception) while another node is
concurrently executing.
the
GraphBuilder, they'll need to provide a manual prefix. Thedefault one won't necessarily work properly in this case. This looks
easy to fix in Python 3.12+ with contextvars, but I have to figure
out how to do it in 3.11 and earlier.