WIP - Support thinking mode for Anthropic models #170

rhys117 · 2025-05-14T12:47:09Z

Resolves #154

Description

Add thinking support for Anthropic models.

Implementation Details

Set thinking on the supported model in models.json
Expose thinking response on Message via .thinking

TODO:

Bedrock tests
Ensure support for acts_as persistent models

…guration

rhys117 · 2025-06-06T10:42:22Z

lib/ruby_llm/providers/anthropic/chat.rb

@@ -39,36 +39,51 @@ def build_base_payload(chat_messages, temperature, model, stream)
          {
            model: model,
            messages: chat_messages.map { |msg| format_message(msg) },
-            temperature: temperature,


rhys117 · 2025-06-06T13:01:00Z

Hey @crmne, can you please let me know if this looks to be heading in a direction that you're happy with architecturally?

Cheers!

crmne

Hey @rhys117!

Thanks for tackling this feature!

Issue: Inconsistent Naming

We've got with_reasoning(), reasoning_content, default_reasoning_budget, but Anthropic's API calls it "thinking" - and we're mixing both terms in capabilities. This breaks our clean API pattern.

Solution: Standardize on "thinking"

Let's match Anthropic's terminology throughout:

# Method
def with_thinking(budget: nil)
  # budget parameter instead of separate config call
end

# Response  
response.thinking     # not reasoning_content
response.thinking?    # boolean check

# Config
default_thinking_budget  # not reasoning

# Error
UnsupportedThinkingError

Usage:

chat = RubyLLM.chat(model: 'claude-3-7-sonnet')
  .with_thinking(budget: 2048)
  .with_temperature(1.0)

response = chat.ask("Short story about a dog")
puts response.thinking
puts response.content

This keeps our API consistent and beautiful. Thoughts?

rhys117 · 2025-06-12T12:59:59Z

That all sounds good to me, I'll take a look at this again in the coming days

pedrofs · 2025-07-14T13:22:58Z

lib/ruby_llm/chat.rb

-      @temperature = 0.7
+      @thinking = @config.default_thinking
+      @thinking_budget = @config.default_thinking_budget
+      @temperature = @config.default_temperature


This default_temperature doesn't exist:

undefined method 'default_temperature' for an instance of RubyLLM::Configuration

This has been corrected, thanks to @hiemanshu

pedrofs

I'm waiting on this feature, and it's kinda working for me. I had to remove the default_temperature and also implement a thinking? to the Model::Info in a dumb way but could be more robust:

class RubyLLM::Model::Info
  ...
  def thinking?
    id.include?("opus-4") || id.include?("sonnet-4") || id.include?("3-7-sonnet")
  end
  ...
end

Any chance to move on with this PR?

pedrofs · 2025-07-14T16:21:45Z

@rhys117, I would be happy to help push this PR through the finish line. Let me know if you want.

crmne · 2025-07-16T17:56:25Z

@pedrofs the best would be to implement it for all providers. Do you want to take on the challenge?

pedrofs · 2025-07-17T01:21:19Z

Hey, @crmne. I can put some work by the end of July. And will be happy to help!

Great work in the gem btw!

Get thinking working with bedrock

rhys117 · 2025-07-17T03:01:05Z

Hey @crmne and @pedrofs, sorry I started this and haven't had time to finish it off. I've merged some changes from @hiemanshu here as well around adding bedrock support.

I will work on the 'acts as' support tomorrow and clean some of this up. I'll be pretty time poor again afterwards, so it would be great if you want to pick this up @pedrofs!

pedrofs · 2025-07-17T10:06:03Z

Hey @crmne and @pedrofs, sorry I started this and haven't had time to finish it off. I've merged some changes from @hiemanshu here as well around adding bedrock support.

I will work on the 'acts as' support tomorrow and clean some of this up. I'll be pretty time poor again afterwards, so it would be great if you want to pick this up @pedrofs!

Sounds good! Do you want me to keep adding things on this branch or patch mine and move from there?

rhys117 · 2025-07-18T02:19:13Z

Hey @crmne and @pedrofs, sorry I started this and haven't had time to finish it off. I've merged some changes from @hiemanshu here as well around adding bedrock support.
I will work on the 'acts as' support tomorrow and clean some of this up. I'll be pretty time poor again afterwards, so it would be great if you want to pick this up @pedrofs!

Sounds good! Do you want me to keep adding things on this branch or patch mine and move from there?

Whatever is easiest, I've added you and Hiemanshu as collaborators to my fork so you can work directly off this branch if you like

- Add 'thinking' to anthropic capabilities - Add 'thinking' to bedrock capabilities (for anthropic supported models) - Update models.json file to reflect changes

- incorrectly using 'supports_thinking?' instead of 'thinking?'

- Adds chat_thinking_spec.rb - Adds THINKING_MODELS and includes anthropic provider models - Adds NON_THINKING_MODELS and includes anthropic provider models - Adds cassette files

- use ** instead of using including unused parameters

rhys117

There was a little more here to clean up than I remembered. I've gone through and corrected the pieces that I can see needed attention to get this working within the RubyLLM::Message class.

Can someone with bedrock keys please add to the spec appropriately?

Unfortunately, I did not have time to look into the 'acts as' support. @crmne, out of curiosity, what do you think about merging this without it and then working towards iterating on this in another pull request to avoid this one getting any larger?

Manual testing -

3.2.2 :007 > chat = RubyLLM::Chat.new(model: 'claude-sonnet-4-20250514').with_thinking
 =>
#<RubyLLM::Chat:0x0000000100d73e98
...
3.2.2 :008 > chat.ask("Give me a short story about a cat.").thinking
 => "I'll write a short story about a cat. Let me think of an interesting premise - maybe something heartwarming or with a little adventure. I'll keep it concise but engaging."

# Streaming 
3.2.2 :010 > chat.ask("Give me a short story about a cat.") { |chunk| p chunk.thinking }
nil
nil
nil
"I"
"'ll write a short story about a cat."
" Let me think of an interesting premise - maybe something heartwarming or"
" with a little adventure. I'll keep it concise but"
" engaging."
nil
...

rhys117 · 2025-07-18T06:15:44Z

lib/ruby_llm/chat.rb

-      @temperature = 0.7
+      @thinking = @config.default_thinking
+      @thinking_budget = @config.default_thinking_budget
+      @temperature = @config.default_temperature


This has been corrected, thanks to @hiemanshu

rhys117 · 2025-07-18T06:34:39Z

lib/ruby_llm/providers/gemini/chat.rb

@@ -11,7 +11,7 @@ def completion_url
          "models/#{@model}:generateContent"
        end

-        def render_payload(messages, tools:, temperature:, model:, stream: false) # rubocop:disable Lint/UnusedMethodArgument


Discarding unused params using '**' is my preference here, but would be keen to hear others' opinions here please

rhys117 added 7 commits May 14, 2025 22:40

feat: wip - add thinking content to messages

81ed712

Merge branch 'main' into 154-thinking

a47870a

chore: add thinking to capabilities

b6e1bb0

chore: pass thinking through from chat initialisation

ecb69c9

chore: add very basic config for thinking budget through global confi…

a014b77

…guration

bug: fix config missing comma

ddb0ae1

chore: add streaming content

6d66491

rhys117 commented Jun 6, 2025

View reviewed changes

chore: rename to use existing reasoning capability

7da672e

crmne requested changes Jun 11, 2025

View reviewed changes

rhys117 and others added 3 commits June 22, 2025 20:37

Merge branch 'main' into 154-thinking

c948b0e

chore: rename to thinking

6b4fb83

Get thinking working with bedrock

7ec6733

pedrofs reviewed Jul 14, 2025

View reviewed changes

Merge branch 'main' into 154-thinking

8709018

crmne added the enhancement New feature or request label Jul 16, 2025

Merge pull request #1 from recitalsoftware/154-thinking

b8fb932

Get thinking working with bedrock

rhys117 added 4 commits July 18, 2025 14:48

chore: update anthropic capabilities with thinking

5577bae

- Add 'thinking' to anthropic capabilities - Add 'thinking' to bedrock capabilities (for anthropic supported models) - Update models.json file to reflect changes

chore: move temperature setting to param

5c02af2

chore: use 'thinking' capability instead of reasoning in Model::Info

153440c

chore: allow thinking capabilties on assumed models

627ffe0

rhys117 added 4 commits July 18, 2025 15:25

bug: fix call to check if thinking supported in 'with_thinking'

8a6453d

- incorrectly using 'supports_thinking?' instead of 'thinking?'

test: add basic spec for anthropic models

cc1ce5f

- Adds chat_thinking_spec.rb - Adds THINKING_MODELS and includes anthropic provider models - Adds NON_THINKING_MODELS and includes anthropic provider models - Adds cassette files

Merge branch 'main' into 154-thinking

87fa6a5

bug: ensure render_payload args compatibility across all providers

06daa1c

- use ** instead of using including unused parameters

rhys117 commented Jul 18, 2025

View reviewed changes

Uh oh!

WIP - Support thinking mode for Anthropic models #170

Are you sure you want to change the base?

WIP - Support thinking mode for Anthropic models #170

Uh oh!

Conversation

rhys117 commented May 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Implementation Details

TODO:

Uh oh!

rhys117 Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

rhys117 commented Jun 6, 2025

Uh oh!

crmne left a comment

Choose a reason for hiding this comment

Issue: Inconsistent Naming

Solution: Standardize on "thinking"

Uh oh!

rhys117 commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pedrofs Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

rhys117 Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

pedrofs left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pedrofs commented Jul 14, 2025

Uh oh!

crmne commented Jul 16, 2025

Uh oh!

pedrofs commented Jul 17, 2025

Uh oh!

rhys117 commented Jul 17, 2025

Uh oh!

pedrofs commented Jul 17, 2025

Uh oh!

rhys117 commented Jul 18, 2025

Uh oh!

rhys117 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rhys117 Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

rhys117 Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rhys117 commented May 14, 2025 •

edited

Loading

rhys117 commented Jun 12, 2025 •

edited

Loading

pedrofs left a comment •

edited

Loading

rhys117 left a comment •

edited

Loading