Implement ConfigeratorPlanLoader with robust error handling #3356
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
meta-cla
bot
added
the
CLA Signed
|
This pull request was exported from Phabricator. Differential Revision: D81573577 |
added 3 commits
September 8, 2025 07:54
…#3327) Summary: Pull Request resolved: pytorch#3327 internal Context: This change is part of the effort in improving planners overall UX and reliability. This Diff: 1. Add ConfigeratorStats to upload sharding plan to config store. **How is a sharding plan stored in Configerator?** The Thrift definition of a sharding plan includes two fields: Topology and Dict[int, ShardingOption]. 1. Topology: The Topology field contains the information mentioned in this diff D79142495. 2. Dict[int, ShardingOption]: This field represents a dictionary where each key is a 64-bit hash of a sharding option, and the value is the corresponding Thrift-converted sharding option. The hash is calculated using the storage_hash function within the ShardingOption object, which takes into account factors such as the fqn, sharding type, and compute kernel. **How can a loaded plan be merged with an enumerated search space?** **Background:** When a plan is preserved during the logging stage, a hash is generated to ensure that the same plan can be loaded and validated later. The [hash is calculated](https://www.internalfb.com/code/fbsource/[fdf90ff2be9041f867bc6c9e4aec6ee94862fa11]/fbcode/torchrec/distributed/planner/types.py?lines=1010-1026) using input fields such as topology, batch size, constraints, storage reservation, and storage reservation policy, as well as fields from the sharding options like fqn, sharding type, kernel type, shards, and cache parameters. Once the plan is loaded and validated, we can safely assume that all loaded sharding options are a 1:1 map of enumerated sharded options. During the loading process, we traverse the enumerated search space, calculate the storage hash for each sharding option, look up the corresponding sharding option from the loaded plan, and replace the Shards of the enumerated sharding option with those of the loaded sharding option. This approach enables us to generate precise sharding options that can be seamlessly converted into a sharing plan as done by the planner and this also ensures consistent logging while also facilitating plan replay. Differential Revision: D81185992
Summary: Pull Request resolved: pytorch#3331 internal Context: Planner stats db is introduced in this diff to track metadata and perf metrics associated with sharding plan. This Diff: 1. Added methods to insert, select and delete planner stats db row. 2. UTs for planner planner stats db 3. Integration of planner stats db with ConfigeratorStats Differential Revision: D81216987
Differential Revision: D81571293
aliafzal
force-pushed
the
export-D81573577
branch
from
2fc97ae to
d30c654
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 8, 2025
Summary: Add ConfigeratorPlanLoader an implementation of the PlanLoader interface to enable: **Key Features:** 1. Plan Retrieval: Loads compressed sharding plans from Configerator using plan_id 2. Database Integration: Queries PlannerStatsDB to get storage location and context hash 3. Decompression: Uses zstd to decompress stored plan data 4. Thrift Conversion: Deserializes Thrift structures and converts back to Python ShardingOption objects 5. Error Handling: Failure scenarios with configurable fallback behavior **Error Handling & Fallback Scenarios:** The implementation supports two distinct error handling modes controlled by `enable_fallback`: **Normal Mode (enable_fallback=False - Default):** - Raises `PlannerError` with `PLAN_LOADING_FAILED` type for any failure - Error scenarios include: - Network connectivity issues (Configerator service unavailable) - Invalid plan id or config path - Data decompression failures - Thrift deserialization errors - Thrift-to-Python conversion failures **Fallback Mode (enable_fallback=True):** - Returns `None` instead of raising exceptions on loading failures - Logs detailed warning messages with plan_id, config_path, and error details - Enables graceful degradation where system can fall back to alternative planning strategies - Suitable for development, experimentation, or scenarios prioritizing availability over strict error handling - Warning logs include full context for debugging: plan ID, Configerator path, and original error Differential Revision: D81573577
aliafzal
force-pushed
the
export-D81573577
branch
from
d30c654 to
95fe9f7
Compare
aliafzal
added a commit
to aliafzal/torchrec
that referenced
this pull request
Sep 8, 2025
Summary: Add ConfigeratorPlanLoader an implementation of the PlanLoader interface to enable: **Key Features:** 1. Plan Retrieval: Loads compressed sharding plans from Configerator using plan_id 2. Database Integration: Queries PlannerStatsDB to get storage location and context hash 3. Decompression: Uses zstd to decompress stored plan data 4. Thrift Conversion: Deserializes Thrift structures and converts back to Python ShardingOption objects 5. Error Handling: Failure scenarios with configurable fallback behavior **Error Handling & Fallback Scenarios:** The implementation supports two distinct error handling modes controlled by `enable_fallback`: **Normal Mode (enable_fallback=False - Default):** - Raises `PlannerError` with `PLAN_LOADING_FAILED` type for any failure - Error scenarios include: - Network connectivity issues (Configerator service unavailable) - Invalid plan id or config path - Data decompression failures - Thrift deserialization errors - Thrift-to-Python conversion failures **Fallback Mode (enable_fallback=True):** - Returns `None` instead of raising exceptions on loading failures - Logs detailed warning messages with plan_id, config_path, and error details - Enables graceful degradation where system can fall back to alternative planning strategies - Suitable for development, experimentation, or scenarios prioritizing availability over strict error handling - Warning logs include full context for debugging: plan ID, Configerator path, and original error Differential Revision: D81573577
|
This pull request was exported from Phabricator. Differential Revision: D81573577 |
Summary: Pull Request resolved: pytorch#3356 Add ConfigeratorPlanLoader an implementation of the PlanLoader interface to enable: **Key Features:** 1. Plan Retrieval: Loads compressed sharding plans from Configerator using plan_id 2. Database Integration: Queries PlannerStatsDB to get storage location and context hash 3. Decompression: Uses zstd to decompress stored plan data 4. Thrift Conversion: Deserializes Thrift structures and converts back to Python ShardingOption objects 5. Error Handling: Failure scenarios with configurable fallback behavior **Error Handling & Fallback Scenarios:** The implementation supports two distinct error handling modes controlled by `enable_fallback`: **Normal Mode (enable_fallback=False - Default):** - Raises `PlannerError` with `PLAN_LOADING_FAILED` type for any failure - Error scenarios include: - Network connectivity issues (Configerator service unavailable) - Invalid plan id or config path - Data decompression failures - Thrift deserialization errors - Thrift-to-Python conversion failures **Fallback Mode (enable_fallback=True):** - Returns `None` instead of raising exceptions on loading failures - Logs detailed warning messages with plan_id, config_path, and error details - Enables graceful degradation where system can fall back to alternative planning strategies - Suitable for development, experimentation, or scenarios prioritizing availability over strict error handling - Warning logs include full context for debugging: plan ID, Configerator path, and original error Differential Revision: D81573577
|
This pull request was exported from Phabricator. Differential Revision: D81573577 |
aliafzal
force-pushed
the
export-D81573577
branch
from
95fe9f7 to
9b377f8
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Add ConfigeratorPlanLoader an implementation of the PlanLoader interface to enable:
Key Features:
Error Handling & Fallback Scenarios:
The implementation supports two distinct error handling modes controlled by
enable_fallback:Normal Mode (enable_fallback=False - Default):
PlannerErrorwithPLAN_LOADING_FAILEDtype for any failureFallback Mode (enable_fallback=True):
Noneinstead of raising exceptions on loading failuresDifferential Revision: D81573577