Compression research

[bhazzard]

Sample the EOS block log to determine what portion of transactions are compressed in the block log.

Determine how much smaller blocks would be compressed with several top compression algorithms.

• Also, what is the cost in time for each compression algorithm?

[heifner]

Also explore P2P compression of complete stream or only parts such as packed_transaction and signed_block.

[heifner]

info  2025-06-25T13:59:47.811 spring-ut block_log.cpp:742             open                 ] Log has 6372256 blocks
info  2025-06-25T13:59:47.811 spring-ut blocklog.cpp:338              read_log             ] existing block log contains block num 431297468 through block num 437669723
info  2025-06-25T14:08:36.385 spring-ut blocklog.cpp:355              read_log             ] opening fork_db
info  2025-06-25T14:08:36.681 spring-ut blocklog.cpp:368              read_log             ] existing reversible fork_db block num 437669724 through block num 437671115
info  2025-06-25T14:08:36.683 spring-ut fork_database.cpp:170         close_impl           ] Writing fork_database 1392 blocks with root 437669723:1a164f5bd45c114d853695f22e5b83b320f1ab10118562ba73f977109cd203be and head 437671115:1a1654cb6d698d717db06b904a61bae97a2b7f24d6098643d6a8d90f38e402aa
info  2025-06-25T14:08:36.741 spring-ut blocklog.cpp:38               report               ] spring-util - reading both block log and fork_db took 528929 msec
info  2025-06-25T14:08:36.741 spring-ut blocklog.cpp:393              read_log             ] compressed 22128448017 vs 29107816968
info  2025-06-25T14:08:36.741 spring-ut blocklog.cpp:394              read_log             ] total compressed 6473423 vs 44043850
info  2025-06-25T14:37:47.643 spring-ut blocklog.cpp:399              read_log             ] total compressed size 22128448017 vs 8105268600

The number of compressed trx is ~14.7%
Compression ratio of those compressed is: ~24%
Total compressed size is 273% of uncompressed size.

[greg7mdp]

Total compressed size is 273% of uncompressed size.

Would suggest that compression is not helping 😉

[heifner]

Total compressed size is 273% of uncompressed size.

Would suggest that compression is not helping 😉

That is suppose to mean that the size of all the compressed data is almost 3x the size of all the uncompressed data. So most of the data is compressed even though the number of compressed trxs is only about 24% of the total.

[greg7mdp]

I think it is clearer to say that:

• considering only compressed transactions (14% of total trxs), compression reduced the size by 24%
• considering all transactions, compression reduced the overall size by 18%

[greg7mdp]

Out of curiosity, what time period do these numbers cover? Do they take the exsat data into account?