Skip to content

Support options on completion calls #130

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

parruda
Copy link

@parruda parruda commented Apr 21, 2025

This is a draft for allowing passing API parameters in chat completions.

@parruda parruda force-pushed the parruda/support-options-on-completion-calls branch from 7a73a65 to d3b8091 Compare April 21, 2025 18:29
@danielfriis
Copy link

Needed this too 🙏

@parruda
Copy link
Author

parruda commented Apr 22, 2025

Does that make sense @crmne? If it does I can write some tests and move the PR to Ready

@adenta adenta mentioned this pull request Apr 23, 2025
@parruda
Copy link
Author

parruda commented Apr 24, 2025

@adenta I saw your comment on the other PR, so do you think this method is good? Just waiting for confirmation from folks more familiar with the gem before I put any more work on it (tests, etc)

@MadBomber
Copy link

@crmne
Need this to address a building trend in providers attempt to distinguish their products without major impact to endpoint API compatibility. For example: https://docs.x.ai/docs/guides/live-search#data-sources-and-parameters

@danielfriis
Copy link

@crmne What do you think of this?

Would love to see it merged (after conflict resolution ofc), since it would allow for any option with the underlying API.

Copy link
Owner

@crmne crmne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! I can see the real need here - providers keep adding unique features and we can't wrap every single one.

I like the direction, but let's fix the merge order:

render_payload(...).merge(options) is dangerous because user options could override critical RubyLLM parameters like model, messages, stream, etc.

Simple fix: Change it to options.merge(render_payload(...)) instead. This way user options go in first, then RubyLLM's required parameters take precedence.

For tests: I want to see this tested for each provider - verify that custom options work without breaking normal chat functionality across OpenAI, Anthropic, Gemini, etc.

Make that change and add provider tests, then this is good to merge! 🚀

@danielfriis
Copy link

@parruda do you feel like implementing the feedback above? Would love to see this PR merged 🙏

@parruda
Copy link
Author

parruda commented Jun 19, 2025

@danielfriis I've got my hands full with https://github.com/parruda/claude-swarm so feel free to take it over. I also really want this feature!

@howdymatt
Copy link

We also need this to pass metadata to LLM proxies like LiteLLM for Langfuse. I would be happy and willing to contribute.

@compumike
Copy link
Contributor

I implemented @crmne's comments from #130 (review) above, added tests, and created a new PR for with_options: #265

@howdymatt
Copy link

I would be have to aid in reviewing to push this feature forward

crmne added a commit that referenced this pull request Jul 21, 2025
…lying API payload (#265)

## What this does

Implements `with_request_options` (renamed from `with_options` due to
ActiveRecord conflict -- see conversation) with @crmne's suggestions
from comment
#130 (review)
and tested against all providers.

This allows users to set arbitrary options on the payload before it's
sent to the provider's API endpoint. The render_payload takes
precedence.

Demo:

```ruby
chat = RubyLLM
  .chat(model: "qwen3", provider: :ollama)
  .with_request_options(response_format: {type: "json_object"})
  .with_instructions("Answer with a JSON object with the key `result` and a numerical value.")
response = chat.ask("What is the square root of 64?")
response.content
=> "{\n  \"result\": 8\n}"
```

This is a power-user feature, and is specific to each provider (and
model, to a lesser extent). I added a brief section to the docs.

For tests: different providers supported different options, so tests are
divided by provider.

(Note that `deep_merge` is required for Gemini in particular because it
relies on a top-level `generationConfig` object.)

## Type of change

- [ ] Bug fix
- [X] New feature
- [ ] Breaking change
- [ ] Documentation
- [ ] Performance improvement

## Scope check

- [X] I read the [Contributing
Guide](https://github.com/crmne/ruby_llm/blob/main/CONTRIBUTING.md)
- [X] This aligns with RubyLLM's focus on **LLM communication**
- [X] This isn't application-specific logic that belongs in user code
- [X] This benefits most users, not just my specific use case

## Quality check

- [ ] I ran `overcommit --install` and all hooks pass
- [X] I tested my changes thoroughly
- [X] I updated documentation if needed
- [X] I didn't modify auto-generated files manually (`models.json`,
`aliases.json`)

## API changes

- [ ] Breaking change
- [X] New public methods/classes
- [ ] Changed method signatures
- [ ] No API changes

## Related issues

- #130
- #131
- #221

---------

Co-authored-by: Carmine Paolino <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Passing 'custom' options Allow passing other API options like max_completion_tokens
6 participants