Skip to content

op-batcher: make shadow compressor more memory efficient #16570

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 8 commits into
base: develop
Choose a base branch
from

Conversation

geoknee
Copy link
Contributor

@geoknee geoknee commented Jun 25, 2025

WIP

#16537

geoknee added 5 commits June 25, 2025 10:20
keeping two compressors around can use a lot of memory. Here, we get the same benefits by just caching the input and overwriting it to the compressor when the wrapper compressor is flushed or closed.
we always write and flush. iff we go over the limit, we roll back.

when we close we recompress the input data for better results.
Copy link

codecov bot commented Jun 25, 2025

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.24%. Comparing base (65117fa) to head (fc8e1f2).
Report is 21 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff              @@
##           develop   #16570       +/-   ##
============================================
+ Coverage    82.39%   96.24%   +13.85%     
============================================
  Files          158      106       -52     
  Lines         8817     4584     -4233     
============================================
- Hits          7265     4412     -2853     
+ Misses        1412      172     -1240     
+ Partials       140        0      -140     
Flag Coverage Δ
cannon-go-tests-64 ?
contracts-bedrock-tests 96.24% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

see 52 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@geoknee
Copy link
Contributor Author

geoknee commented Jun 25, 2025

Through local profiling (see #16572) I am seeing a 2x-3x improvement in memory usage per pending channel with this change.

@geoknee
Copy link
Contributor Author

geoknee commented Jun 25, 2025

However, as is this change is causing safe head progression tests to fail, so that needs to be addressed!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant