Replace C bindings for CRC32 fusion calculation #9
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The C bindings approach has proven to be problematic across a variety of CI/CD implementations, particularly when cross-compiling to other targets. Moving to a pure Rust implementation should eliminate those issues.
By fusing CLMUL and native CRC32 instructions, we can achieve more throughput on platforms which have native CRC32 instruction support. These were generated using the ‘fast-crc32’ project, then converted from C to Rust with the help of Claude.ai. https://github.com/corsix/fast-crc32/ https://www.corsix.org/content/fast-crc32c-4k https://www.corsix.org/content/alternative-exposition-crc32_4k_pclmulqdq https://dougallj.wordpress.com/2022/05/22/faster-crc32-on-the-apple-m1/
AVX512 support has landed in 1.89.0, so we can deprecate the feature flag and instead rely on the Rust version for AVX512 support, including VPCLMULQDQ. https://releases.rs/docs/1.89.0/ rust-lang/rust#138940
Updates the README to include latest benchmarking numbers and information about acceleration targets.
There was a problem hiding this comment.
Pull Request Overview
This PR removes fragile C bindings for CRC-32/ISCSI and CRC-32/ISO-HDLC, replacing them with pure Rust implementations gated by Rust 1.89+ feature detection, and updates benchmark code accordingly.
- Deleted all generated C source and header files under
include/ - Simplified
build.rsby removing C compilation paths - Adjusted
benches/benchmark.rsto refine the set of algorithms, add measurement times, and swap benchmark order
Reviewed Changes
Copilot reviewed 37 out of 37 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| include/* | All C-based CRC implementation files and headers deleted |
| build.rs | Removed C-binding build logic and feature-flag overrides |
| benches/benchmark.rs | Updated CRC algorithm list, enabled Duration measurement, reordered |
I’m going to put this in another PR instead, waiting for 1.89 to hit stable first. I’ll keep this gated on the “vpclmulqdq” feature flag until then.
Elminate redundant features that are already implictly enabled. https://doc.rust-lang.org/reference/attributes/codegen.html#r-attributes.codegen.target_feature.x86 https://doc.rust-lang.org/reference/attributes/codegen.html#r-attributes.codegen.target_feature.aarch64
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The Problems
The C bindings used to accelerate CRC-32/ISCSI and CRC-32/ISO-HDLC using a fusion approach were fragile and problematic for some environments and build processes.
The Solution
Remove the C bindings and replace them with pure Rust implementations. This also resulted in a material performance boost for smaller inputs (1 KiB).
Changes
Planned version bump
MINORLinks
Notes
This makes this library 100% pure Rust. 👍