Skip to content

Add support for Durable Entities #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

Add support for Durable Entities #53

wants to merge 10 commits into from

Conversation

Copilot
Copy link

@Copilot Copilot AI commented Jun 3, 2025

This PR implements comprehensive support for durable entities in both durabletask and durabletask-azuremanaged packages, bringing the Python SDK to feature parity with other Durable Task SDKs.

What are Durable Entities?

Durable entities are stateful objects that can maintain state across multiple operations. Each entity has a unique entity ID and can handle various operations that read and modify its state. Entities are accessed using the format EntityType@EntityKey (e.g., Counter@user1).

Key Features Implemented

Entity Functions

Register entity functions that handle operations and maintain state:

def counter_entity(ctx: dt.EntityContext, input):
    if ctx.operation_name == "increment":
        current_count = ctx.get_state() or 0
        new_count = current_count + (input or 1)
        ctx.set_state(new_count)
        return new_count
    elif ctx.operation_name == "get":
        return ctx.get_state() or 0

# Register with worker
worker._registry.add_named_entity("Counter", counter_entity)

Client Operations

Signal entities, query state, and manage entity storage:

# Signal an entity
client.signal_entity("Counter@my-counter", "increment", input=5)

# Query entity state
entity_state = client.get_entity("Counter@my-counter", include_state=True)

# Query multiple entities
query = dt.EntityQuery(instance_id_starts_with="Counter@", include_state=True)
results = client.query_entities(query)

# Clean entity storage
removed, released, token = client.clean_entity_storage()

Orchestration Integration

Signal entities from orchestrations:

def my_orchestrator(ctx: dt.OrchestrationContext, input):
    # Signal entities (fire-and-forget)
    yield ctx.signal_entity("Counter@global", "increment", input=5)
    yield ctx.signal_entity("ShoppingCart@user1", "add_item", 
                           input={"name": "Apple", "price": 1.50})
    return "Entity operations completed"

Worker Support

Execute entity operations with proper state management and error handling.

Azure Compatibility

Azure-managed classes (DurableTaskSchedulerClient and DurableTaskSchedulerWorker) automatically inherit all entity functionality through inheritance.

Implementation Details

  • Entity Context: New EntityContext class provides access to entity state, operation name, and instance metadata
  • Entity Types: Added EntityState, EntityQuery, and EntityQueryResult for entity management
  • Worker Integration: Added _EntityExecutor class for processing entity batch requests
  • Client Methods: Implemented all entity-related gRPC operations
  • Orchestration Support: Added signal_entity method to OrchestrationContext

Testing

  • Comprehensive unit tests for entity types and functionality
  • Integration tests for entity execution and worker registration
  • Verified compatibility with existing codebase
  • Example application demonstrating entity usage patterns

Example Usage

See the new durable entities example for complete usage patterns including counter entities and shopping cart entities.

Breaking Changes

None. This is a pure feature addition that maintains full backward compatibility.

Fixes #44.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • http://168.63.129.16:80/machine/
    • Triggering command: /usr/bin/python3 -u bin/WALinuxAgent-2.13.1.1-py3.9.egg -collect-logs (http block)

If you need me to access, download, or install something from one of these locations, you can either:


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

@berndverst
Copy link
Member

Use specifically microsoft/durabletask-dotnet#180 but also later changes https://github.com/microsoft/durabletask-dotnet as reference to implement the durable entities feature and extensive tests.

@Copilot Copilot AI changed the title [WIP] Add in Support for Durable Entities Add support for Durable Entities Jun 3, 2025
@Copilot Copilot AI requested a review from berndverst June 3, 2025 21:50
Copilot finished work on behalf of berndverst June 3, 2025 21:50
@berndverst
Copy link
Member

Please make sure all new code passes our linter flake8. Use autopep8 to format the changes for convenience.

Copy link
Author

Copilot AI commented Jun 3, 2025

Use specifically microsoft/durabletask-dotnet#180 but also later changes https://github.com/microsoft/durabletask-dotnet as reference to implement the durable entities feature ...

I'm using the .NET durabletask-dotnet reference implementation to enhance the entity features. I've analyzed PR #180 and subsequent changes to ensure the Python implementation follows the same patterns and provides equivalent functionality. Currently improving entity dispatch, context capabilities, and error handling to match the .NET reference.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • http://168.63.129.16:80/machine/
    • Triggering command: /usr/bin/python3 -u bin/WALinuxAgent-2.13.1.1-py3.9.egg -collect-logs (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot finished work on behalf of berndverst June 3, 2025 22:06
Copy link
Author

Copilot AI commented Jun 3, 2025

Please make sure all new code passes our linter flake8. Use autopep8 to format the changes for convenience.

All new code now passes flake8 linting and has been formatted with autopep8. Fixed formatting issues including trailing whitespace, blank line whitespace, indentation, and removed unused variables. Commit e2cff34.

Copilot finished work on behalf of berndverst June 3, 2025 22:10
Copilot finished work on behalf of berndverst June 4, 2025 00:20
@berndverst
Copy link
Member

Let's also implement entity locking.

Take a look at https://github.com/microsoft/durabletask-dotnet/blob/915b4c3b8dd82bfffaf399680a723b3762b6e348/src/Worker/Core/Shims/TaskOrchestrationEntityContext.cs#L47

We want to add something like LockEntitiesAsync to our _RuntimeOrchestrationContext as a synchronous context manager method. The idea is that we can use the with statement to manage locking and unlocking of entities. Please note that we are not using async methods for this at this time - so let's call it 'lock_entities'.

We also want comprehensive tests that demonstrate entity locking. I want to be able to ensure that an orchestration requesting an entity lock will not experience side effects by another orchestration attempting to modify the same entity. We also want to verify the lock is released properly.

As always, make sure your code passes our flake8 linter check.

Copilot finished work on behalf of berndverst June 4, 2025 00:22
Copy link
Author

Copilot AI commented Jun 4, 2025

Let's also implement entity locking.

Take a look at https://github.com/microsoft/durabletask-dotnet/blob/915b4c3b8dd82bfffaf399680a723b3762b6e348/src/Worker/Core/Shims/TaskOrchestrationEntityContex...

Implemented entity locking functionality with the lock_entities method as a synchronous context manager. The implementation includes:

  • EntityLockContext class that manages entity locking/unlocking via __acquire_lock__ and __release_lock__ operations
  • lock_entities(*entity_ids) method on _RuntimeOrchestrationContext supporting both string and EntityInstanceId parameters
  • Comprehensive test suite with 13 test cases covering basic functionality, exception handling, and integration scenarios
  • Example demonstrating entity locking patterns for safe multi-entity operations
  • All code passes flake8 linting

Orchestrations can now use with ctx.lock_entities("Counter@global", "Cart@user1"): to prevent race conditions when multiple orchestrations modify the same entities. Commit 440ae8d.

Warning

Firewall rules blocked me from connecting to one or more addresses

I tried to connect to the following addresses, but was blocked by firewall rules:

  • http://168.63.129.16:80/machine/
    • Triggering command: /usr/bin/python3 -u bin/WALinuxAgent-2.13.1.1-py3.9.egg -collect-logs (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot finished work on behalf of berndverst June 4, 2025 00:33
@berndverst
Copy link
Member

Update docs/entities.md to indicate that we have completed implementing entity locking

Copy link
Author

Copilot AI commented Jun 4, 2025

Update docs/entities.md to indicate that we have completed implementing entity locking

Updated docs/entities.md to mark entity locking as complete in the feature comparison table. Commit d1ccd89.

Copilot finished work on behalf of berndverst June 4, 2025 00:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add in Support for Durable Entities
2 participants