Skip to content

Conversation

sobychacko
Copy link
Contributor

…citly cache tools

This commit fixes a bug where the SYSTEM_ONLY cache strategy was incorrectly placing cache_control on tool definitions, making it functionally identical to SYSTEM_AND_TOOLS.

Problem:

  • SYSTEM_ONLY and SYSTEM_AND_TOOLS both placed cache breakpoints on tools AND system messages, using 2 breakpoints
  • This violated the documented behavior and wasted breakpoints
  • The strategies were functionally identical

Root Cause:

  • CacheEligibilityResolver.resolveToolCacheControl() checked if eligible message types included SYSTEM, which both strategies have
  • It should have checked the strategy directly instead

Solution:

  • Changed resolveToolCacheControl() to use allowlist approach
  • Only SYSTEM_AND_TOOLS and CONVERSATION_HISTORY explicitly cache tools
  • SYSTEM_ONLY now places cache_control only on system message
  • Tools still get cached implicitly via Anthropic's cache hierarchy (tools → system → messages)

Impact:

  • SYSTEM_ONLY: Uses 1 breakpoint (system only) instead of 2
  • SYSTEM_AND_TOOLS: Uses 2 breakpoints (last tool + system) as before
  • Users can now optimize breakpoint usage more effectively

Changes:

  • Fix CacheEligibilityResolver.resolveToolCacheControl() to check strategy directly instead of eligible message types
  • Enhance testSystemOnlyCacheStrategy() to include tools and verify they don't have cache_control
  • Update toolCacheControlRespectsStrategy() test to verify all strategies behave correctly

Thank you for taking time to contribute this pull request!
You might have already read the contributor guide, but as a reminder, please make sure to:

  • Add a Signed-off-by line to each commit (git commit -s) per the DCO
  • Rebase your changes on the latest main branch and squash your commits
  • Add/Update unit tests as needed
  • Run a build and make sure all tests pass prior to submission

For more details, please check the contributor guide.
Thank you upfront!

@sobychacko
Copy link
Contributor Author

@adase11 I made some slight semantic changes here. Could you take a look and see what you think?

@adase11
Copy link
Contributor

adase11 commented Oct 2, 2025

@adase11 I made some slight semantic changes here. Could you take a look and see what you think?

Can do, sorry I am traveling until next week but I'll try to get to it in a timely manner. I also saw your other question and am giving it some thought. Thanks!

…citly cache tools

This commit fixes a bug where the SYSTEM_ONLY cache strategy was
incorrectly placing cache_control on tool definitions, making it
functionally identical to SYSTEM_AND_TOOLS.

Problem:
- SYSTEM_ONLY and SYSTEM_AND_TOOLS both placed cache breakpoints on
  tools AND system messages, using 2 breakpoints
- This violated the documented behavior and wasted breakpoints
- The strategies were functionally identical

Root Cause:
- CacheEligibilityResolver.resolveToolCacheControl() checked if
  eligible message types included SYSTEM, which both strategies have
- It should have checked the strategy directly instead

Solution:
- Changed resolveToolCacheControl() to use allowlist approach
- Only SYSTEM_AND_TOOLS and CONVERSATION_HISTORY explicitly cache tools
- SYSTEM_ONLY now places cache_control only on system message
- Tools still get cached implicitly via Anthropic's cache hierarchy
  (tools → system → messages)

Impact:
- SYSTEM_ONLY: Uses 1 breakpoint (system only) instead of 2
- SYSTEM_AND_TOOLS: Uses 2 breakpoints (last tool + system) as before
- Users can now optimize breakpoint usage more effectively

Changes:
- Fix CacheEligibilityResolver.resolveToolCacheControl() to check
  strategy directly instead of eligible message types
- Enhance testSystemOnlyCacheStrategy() to include tools and verify
  they don't have cache_control
- Update toolCacheControlRespectsStrategy() test to verify all
  strategies behave correctly

Signed-off-by: Soby Chacko <[email protected]>
@sobychacko sobychacko force-pushed the fix/anthropic-system-only-cache-strategy branch from 82d63b2 to 56e2e0a Compare October 3, 2025 17:38
  This commit introduces the TOOLS_ONLY caching strategy, enabling scenarios
  where large tool definitions should be cached while system prompts remain
  dynamic and uncached.

  Key Changes:
  - Add TOOLS_ONLY enum to AnthropicCacheStrategy with comprehensive javadocs
  - Update CacheEligibilityResolver to support TOOLS_ONLY strategy
  - Enhance all strategy javadocs with detailed use cases and token guidance
  - Add comprehensive unit tests covering all caching scenarios
  - Update documentation with TOOLS_ONLY examples and cascade invalidation

  Use Cases:
  - Multi-tenant SaaS applications with shared tools but per-tenant system prompts
  - A/B testing scenarios with stable tools but variable system instructions
  - Applications with large tool sets (5000+ tokens) and dynamic contexts

  Technical Details:
  - TOOLS_ONLY uses 1 cache breakpoint on the last tool definition
  - System messages are NOT cached, processed fresh on each request
  - Supports Anthropic's cache hierarchy (tools → system → messages)
  - Compatible with cascade invalidation behavior

  Documentation:
  - Added strategy comparison table with breakpoint usage
  - Added multi-tenant SaaS use case example
  - Updated best practices with cascade invalidation explanation
  - Clarified SYSTEM_AND_TOOLS independence behavior

Signed-off-by: Soby Chacko <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants