support usage tracking with response api to _merge_usage_entries #8728
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
With response API you get additional usage token information in the form of litellm's InputTokenDetails and OutputTokenDetails.
Because these are objects the current _merge_usage_entries cannot handle them and thus cause the entire request to fail.
This PR fixes that.
This pull request improves the aggregation logic for usage tracking and adds comprehensive tests to ensure correct merging of token usage details, including handling of custom objects and
None
values.Improvements to usage aggregation logic
_merge_usage_entries
method indspy/utils/usage_tracker.py
to correctly merge usage entries when values are custom objects (with a__dict__
), ensuring that details from types likeInputTokensDetails
andOutputTokensDetails
are properly aggregated._merge_usage_entries
for better readability and consistency.Expanded test coverage
tests/utils/test_usage_tracker.py
to verify that input and output token details are accurately aggregated across multiple usage entries, including cases with missing orNone
values.InputTokensDetails
andOutputTokensDetails
fromlitellm.types.llms.openai
to support new test scenarios.