Skip to content

Implements handling of partial frames by the streaming decompressor #88

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

gucki
Copy link

@gucki gucki commented May 16, 2024

Fixes #87

Example use case:

read_buffer = "".force_encoding(Encoding::ASCII_8BIT)
loop do
  logger.debug("read partial")
  buf = reader.readpartial(64 * 1024)
  read_buffer << buf
  logger.debug("read #{buf.bytesize} bytes (#{read_buffer.bytesize} bytes buffered)")

  while !read_buffer.empty?
    result, data, read_buffer_pos = decompressor.decompress2(read_buffer)
    read_buffer.slice!(0, read_buffer_pos)
    logger.debug("decompressed #{data.bytesize} bytes (result #{result}, consumed #{read_buffer_pos} bytes, #{read_buffer.bytesize} left in read buffer)")
    writer.write(data)
    break if read_buffer_pos == 0
  end
end

It works according to the zstd streaming decompression documentation:

Use ZSTD_decompressStream() repetitively to consume your input.
The function will update both `pos` fields.
If `input.pos < input.size`, some input has not been consumed.
It's up to the caller to present again remaining data.
The function tries to flush all data decoded immediately, respecting output buffer size.
If `output.pos < output.size`, decoder has flushed everything it could.
But if `output.pos == output.size`, there might be some data left within internal buffers.,
In which case, call ZSTD_decompressStream() again to flush whatever remains in the buffer.
Note : with no additional input provided, amount of data flushed is necessarily <= ZSTD_BLOCKSIZE_MAX.
@return : 0 when a frame is completely decoded and fully flushed,
      or an error code, which can be tested using ZSTD_isError(),
      or any other value > 0, which means there is still some decoding or flushing to do to complete current frame :
                              the return value is a suggested next input size (just a hint for better latency)
                              that will never request more than the remaining frame size.

@SpringMT
Copy link
Owner

Thank you for the PR.
I'll check it later.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Streaming decompression questions
2 participants