Skip to content

Prevent chat model from streaming after first full _valid_ response object #387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: development
Choose a base branch
from

Conversation

ehlowr0ld
Copy link
Contributor

@ehlowr0ld ehlowr0ld commented May 15, 2025

Some models do not stop responding after the json response is finished. They continue with plaintext or try to continue the conversation and output more response objects. This PR prevents reading response past the first FULL VALID json response.

@ehlowr0ld ehlowr0ld changed the title Prevent chat model from streaming after first full response object Prevent chat model from streaming after first full _valid_ response object May 15, 2025
@frdel
Copy link
Owner

frdel commented May 19, 2025

@ehlowr0ld shouldn't we do this by tracking closing brackets instead? When the dirtyjson class traverses the text, it reaches points of "self.stack.pop()" where branches of the object are being closed. Once the stack is 0 length, the json is complete.
I don't really like the idea if specifying required properties, as they are prompt based and thoughts for example can be optional.

@ehlowr0ld
Copy link
Contributor Author

@frdel This was the only robust way I could make it work - without the required properties sometimes the message was cutting off for example tool_args. There is the method parse_string_checked which needs the required properties - and the normal parse string which works like before but with a little more checks under the hood which still can be used in other places.

@ehlowr0ld ehlowr0ld force-pushed the feat/local_pr_response_checking branch from a33eadd to 27d41f5 Compare June 1, 2025 14:07
@ehlowr0ld ehlowr0ld force-pushed the feat/local_pr_response_checking branch from 27d41f5 to 2b8e389 Compare June 1, 2025 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants