Support options on completion calls #130
Conversation
parruda
force-pushed
the
parruda/support-options-on-completion-calls
branch
from
7a73a65 to
d3b8091
Compare
|
Needed this too 🙏 |
|
Does that make sense @crmne? If it does I can write some tests and move the PR to Ready |
|
@adenta I saw your comment on the other PR, so do you think this method is good? Just waiting for confirmation from folks more familiar with the gem before I put any more work on it (tests, etc) |
|
@crmne |
|
@crmne What do you think of this? Would love to see it merged (after conflict resolution ofc), since it would allow for any option with the underlying API. |
There was a problem hiding this comment.
Thanks for the PR! I can see the real need here - providers keep adding unique features and we can't wrap every single one.
I like the direction, but let's fix the merge order:
render_payload(...).merge(options) is dangerous because user options could override critical RubyLLM parameters like model, messages, stream, etc.
Simple fix: Change it to options.merge(render_payload(...)) instead. This way user options go in first, then RubyLLM's required parameters take precedence.
For tests: I want to see this tested for each provider - verify that custom options work without breaking normal chat functionality across OpenAI, Anthropic, Gemini, etc.
Make that change and add provider tests, then this is good to merge! 🚀
|
@parruda do you feel like implementing the feedback above? Would love to see this PR merged 🙏 |
|
@danielfriis I've got my hands full with https://github.com/parruda/claude-swarm so feel free to take it over. I also really want this feature! |
|
We also need this to pass metadata to LLM proxies like LiteLLM for Langfuse. I would be happy and willing to contribute. |
|
I implemented @crmne's comments from #130 (review) above, added tests, and created a new PR for |
|
I would be have to aid in reviewing to push this feature forward |
…lying API payload (#265) ## What this does Implements `with_request_options` (renamed from `with_options` due to ActiveRecord conflict -- see conversation) with @crmne's suggestions from comment #130 (review) and tested against all providers. This allows users to set arbitrary options on the payload before it's sent to the provider's API endpoint. The render_payload takes precedence. Demo: ```ruby chat = RubyLLM .chat(model: "qwen3", provider: :ollama) .with_request_options(response_format: {type: "json_object"}) .with_instructions("Answer with a JSON object with the key `result` and a numerical value.") response = chat.ask("What is the square root of 64?") response.content => "{\n \"result\": 8\n}" ``` This is a power-user feature, and is specific to each provider (and model, to a lesser extent). I added a brief section to the docs. For tests: different providers supported different options, so tests are divided by provider. (Note that `deep_merge` is required for Gemini in particular because it relies on a top-level `generationConfig` object.) ## Type of change - [ ] Bug fix - [X] New feature - [ ] Breaking change - [ ] Documentation - [ ] Performance improvement ## Scope check - [X] I read the [Contributing Guide](https://github.com/crmne/ruby_llm/blob/main/CONTRIBUTING.md) - [X] This aligns with RubyLLM's focus on **LLM communication** - [X] This isn't application-specific logic that belongs in user code - [X] This benefits most users, not just my specific use case ## Quality check - [ ] I ran `overcommit --install` and all hooks pass - [X] I tested my changes thoroughly - [X] I updated documentation if needed - [X] I didn't modify auto-generated files manually (`models.json`, `aliases.json`) ## API changes - [ ] Breaking change - [X] New public methods/classes - [ ] Changed method signatures - [ ] No API changes ## Related issues - #130 - #131 - #221 --------- Co-authored-by: Carmine Paolino <[email protected]>
This is a draft for allowing passing API parameters in chat completions.