Add script to generate fuzz coverage #3718

Prabhat1308 · 2025-04-08T19:41:18Z

This script generates the fuzz coverage in a html file when run .

Considerations

The script assumes a corpus directory in fuzz directory.
Also assumes the corpus for a particular target to be under the name of the target itself. for eg - corpus for base32_target.rs would be under corpus/base32_target.
The corpus I worked with was generated using libFuzzer it may/may not work with corpus generated from other fuzzers.

How to run

# from the root directory
chmod +x contrib/generate_fuzz_coverage.sh
contrib/generate_fuzz_coverage.sh

ldk-reviews-bot · 2025-04-08T19:41:21Z

👋 I see @jkczyz was un-assigned.
If you'd like another reviewer assignemnt, please click here.

Prabhat1308 · 2025-04-08T19:47:19Z

If needed I can also push my script to generate corpus from libFuzzer that create the corpus according to the considerations if the reviewers deem it worthy .

TheBlueMatt

I wonder if we should include this coverage data in our codecov output in PRs...On the one hand its not "reliable" coverage data in that we aren't actually testing the behavior, but on the other hand we are at least hitting the code lines and presumable would find crashes. WDYT?

contrib/generate_fuzz_coverage.sh

TheBlueMatt · 2025-04-09T14:17:25Z

contrib/generate_fuzz_coverage.sh

+
+    if [ -d "$TARGET_CORPUS_DIR" ]; then
+        echo "Running cargo llvm-cov for target: $TARGET_NAME with corpus $TARGET_CORPUS_DIR"
+        if ! CARGO_TARGET_DIR=./target cargo +${TOOLCHAIN} llvm-cov run --no-report --manifest-path "$FUZZ_DIR/Cargo.toml" --target "$HOST_TRIPLE" --features "$COMBINED_FEATURES" --bin "$TARGET_NAME" -- "$TARGET_CORPUS_DIR" -runs=1; then


Rather than running the fuzz target specifically, can we just use cargo llvm-cov to run the tests? For each binary in the fuzz tests we already support running cargo test and it'll pull all the files from fuzz/test_cases/X and run it through the fuzz processor. Seems easier than relying on the libfuzzer logic.

Not sure about that. Running on test_cases with cargo test would remove the fuzziness since now it will be run test_cases and wont run the fuzzer but since its coverage report we are basically doing the same thing. I would have to check on this.

Someone doing fuzzing can presumably copy their corpus from wherever it is to test_cases. In the comment you note that we currently expect a corpus in corpus, but it could well be any folder name?

I have changed the script to use cargo test instance now. Surprisingly reduced the code by 99%. Although I don't see the test_cases in any branch . Maybe you can try running the script and provide some feedback since I dont know the format on how test_cases directory has the inputs.

Right, test_cases isn't automatically created anywhere, its just read by the tests if it exists. Not sure what to do about that wrt the script here, maybe we should have it look for the llvm-libfuzzer, afl, and honggfuzz dirs, copy them to test (if its empty) and then run cargo test?

Did you want to go ahead and do this?

Hi . sorry for the late response . Have been busy recently with my academics . Would implement this by the end of this month hopefully.

I updated the PR according to the comment , finding the corpus directories is not feasible since there is no clear indication/factor that can catch that this is a corpus directory apart from the object format which might also be different depending on the fuzzer. Also the location of corpus can be out of the project and some people tend to keep their project directory clean . Instead what I have done is to check for test_cases and if its not present/empty it will throw a message and instruction to copy paste (but not halt the fuzz run).

ldk-reviews-bot · 2025-04-09T14:20:29Z

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

Prabhat1308 · 2025-04-09T15:17:18Z

I wonder if we should include this coverage data in our codecov output in PRs...On the one hand its not "reliable" coverage data in that we aren't actually testing the behavior, but on the other hand we are at least hitting the code lines and presumable would find crashes. WDYT?

I would prefer to not add it in codecov until we have a good public corpora of fuzz inputs to run the report on.

TheBlueMatt · 2025-05-14T15:50:29Z

I would prefer to not add it in codecov until we have a good public corpora of fuzz inputs to run the report on.

One reason to do this would be to only include "trivially-reachable" things in the codecov output. We don't really want to include stuff as coverage if the fuzzer happened to reach it only once, but if the fuzzer found its way into it in a minute of CI time, its probably decent coverage :)

Prabhat1308 · 2025-05-18T21:19:47Z

I would prefer to not add it in codecov until we have a good public corpora of fuzz inputs to run the report on.

One reason to do this would be to only include "trivially-reachable" things in the codecov output. We don't really want to include stuff as coverage if the fuzzer happened to reach it only once, but if the fuzzer found its way into it in a minute of CI time, its probably decent coverage :)

Makes sense if thats the case.

TheBlueMatt · 2025-06-12T18:26:56Z

Do you intend to go ahead and run this in CI? Or would you prefer not to?

Prabhat1308 · 2025-06-13T09:29:07Z

added this script to run in CI .

Prabhat1308 · 2025-06-15T19:42:55Z

1.63 versions are failing because llvm-cov requires 1.65 , other 2 failures are because of storage issue on the CI instance.

TheBlueMatt

Oops, so sorry I missed your comments here.

TheBlueMatt · 2025-06-30T15:11:23Z

ci/ci-tests.sh

@@ -137,3 +137,28 @@ RUSTFLAGS="--cfg=splicing" cargo test --verbose --color always -p lightning
 RUSTFLAGS="--cfg=async_payments" cargo test --verbose --color always -p lightning
 [ "$CI_MINIMIZE_DISK_USAGE" != "" ] && cargo clean
 RUSTFLAGS="--cfg=lsps1_service" cargo test --verbose --color always -p lightning-liquidity
+
+# Generate fuzz coverage report
+echo -e "\n\nGenerating fuzz coverage report"


Ah, rather than doing this in ci-tests.sh let's do this in a new script which is called from the coverage job in CI.

contrib/generate_fuzz_coverage.sh

ci/ci-fuzz.sh

graphite-app · 2025-06-30T16:15:25Z

contrib/generate_fuzz_coverage.sh

+export RUSTFLAGS="--cfg=fuzzing --cfg=secp256k1_fuzz --cfg=hashes_fuzz"
+# ignore anything in fuzz directory since we don't want coverage of targets 
+cargo llvm-cov --html --ignore-filename-regex "fuzz/" --output-dir "$OUTPUT_DIR"


The script currently checks for the existence of test corpus files but doesn't appear to use them when generating coverage with cargo llvm-cov. For accurate fuzz coverage reporting, the command should incorporate these corpus files as test inputs.

Consider modifying the cargo llvm-cov command to explicitly use the corpus files, perhaps with something like:

# For each target with a corpus for target_dir in test_cases/*; do if [ -d "$target_dir" ]; then target_name=$(basename "$target_dir") cargo llvm-cov --html --test-dir "$target_dir" --target "$target_name" ... fi done

This would ensure the coverage report reflects code paths exercised by the fuzzing corpus rather than just standard test execution.

Suggested change

export RUSTFLAGS="--cfg=fuzzing --cfg=secp256k1_fuzz --cfg=hashes_fuzz"

# ignore anything in fuzz directory since we don't want coverage of targets

cargo llvm-cov --html --ignore-filename-regex "fuzz/" --output-dir "$OUTPUT_DIR"

export RUSTFLAGS="--cfg=fuzzing --cfg=secp256k1_fuzz --cfg=hashes_fuzz"

# Process each fuzz target with its corpus

mkdir -p "$OUTPUT_DIR"

for target_dir in fuzz/corpus/*; do

if [ -d "$target_dir" ]; then

target_name=$(basename "$target_dir")

echo "Generating coverage for target: $target_name"

# Run coverage for this specific target with its corpus

cargo llvm-cov --html --ignore-filename-regex "fuzz/" \

--fuzz-target "$target_name" \

--corpus-dir "$target_dir" \

--output-path "$OUTPUT_DIR/$target_name"

fi

done

# Generate a combined report

cargo llvm-cov --html --ignore-filename-regex "fuzz/" --output-dir "$OUTPUT_DIR/combined"

Spotted by Diamond

Is this helpful? React 👍 or 👎 to let us know.

Hmm I dont think this is the correct way ? Cargo-llvm will use all the tests_cases (if present) and all targets in this scenario. Individually calling them seems redundant work

contrib/generate_fuzz_coverage.sh

Prabhat1308 · 2025-07-01T15:26:07Z

Failure caused because of no storage left on the ci infra

TheBlueMatt

Man I hate github CI. I think if we add a deletion of some of the files in target after the first cargo llvm-cov in the CI build.yml it should pass.

contrib/generate_fuzz_coverage.sh

Prabhat1308 · 2025-07-08T16:22:42Z

Working locally when run with the CI workflow.

ci/ci-fuzz.sh

contrib/generate_fuzz_coverage.sh

Prabhat1308 · 2025-07-08T17:22:51Z

CI passed . had to nuke the entire target directory

tnull · 2025-07-10T09:41:23Z

Might be worth giving CI a kick now that we fixed the No space left on device issue (for now) with #3916.

adds the coverage report generation in CI . Also make the coverage script more CI friendly.

ldk-reviews-bot requested a review from jkczyz April 8, 2025 19:51

jkczyz requested review from TheBlueMatt and removed request for jkczyz April 8, 2025 21:50

TheBlueMatt reviewed Apr 9, 2025

View reviewed changes

Prabhat1308 force-pushed the pv/add_cov_script branch from c45d613 to 8f005a0 Compare April 9, 2025 16:21

Prabhat1308 force-pushed the pv/add_cov_script branch from 8f005a0 to 10679bb Compare June 2, 2025 18:55

Prabhat1308 force-pushed the pv/add_cov_script branch 3 times, most recently from a554c14 to a460b7a Compare June 15, 2025 16:54

TheBlueMatt reviewed Jun 30, 2025

View reviewed changes

graphite-app bot reviewed Jun 30, 2025

View reviewed changes

contrib/generate_fuzz_coverage.sh Show resolved Hide resolved

ci/ci-fuzz.sh Outdated Show resolved Hide resolved

ci/ci-fuzz.sh Outdated Show resolved Hide resolved

graphite-app bot reviewed Jun 30, 2025

View reviewed changes

Prabhat1308 force-pushed the pv/add_cov_script branch from c81c70f to 4baa925 Compare June 30, 2025 17:38

graphite-app bot reviewed Jun 30, 2025

View reviewed changes

contrib/generate_fuzz_coverage.sh Outdated Show resolved Hide resolved

TheBlueMatt reviewed Jul 2, 2025

View reviewed changes

contrib/generate_fuzz_coverage.sh Outdated Show resolved Hide resolved

graphite-app bot reviewed Jul 8, 2025

View reviewed changes

contrib/generate_fuzz_coverage.sh Show resolved Hide resolved

Prabhat1308 force-pushed the pv/add_cov_script branch from 934753f to e028e19 Compare July 8, 2025 16:20

graphite-app bot reviewed Jul 8, 2025

View reviewed changes

contrib/generate_fuzz_coverage.sh Outdated Show resolved Hide resolved

Prabhat1308 force-pushed the pv/add_cov_script branch from e028e19 to f3ec93c Compare July 8, 2025 16:24

graphite-app bot reviewed Jul 8, 2025

View reviewed changes

ci/ci-fuzz.sh Show resolved Hide resolved

Prabhat1308 force-pushed the pv/add_cov_script branch from f3ec93c to 029d266 Compare July 8, 2025 16:49

graphite-app bot reviewed Jul 8, 2025

View reviewed changes

ci/ci-fuzz.sh Show resolved Hide resolved

contrib/generate_fuzz_coverage.sh Show resolved Hide resolved

Prabhat1308 force-pushed the pv/add_cov_script branch from 029d266 to d9b6814 Compare July 8, 2025 17:02

graphite-app bot reviewed Jul 8, 2025

View reviewed changes

contrib/generate_fuzz_coverage.sh Show resolved Hide resolved

Prabhat1308 added 4 commits July 10, 2025 15:31

add script to generate fuzz coverage

b0196f4

add coverage report generation in CI

62dfd35

adds the coverage report generation in CI . Also make the coverage script more CI friendly.

create new fuzz script and add to coverage job

cf039d7

space management and use codecov

248ba29

Prabhat1308 force-pushed the pv/add_cov_script branch from d9b6814 to 248ba29 Compare July 10, 2025 10:01

-export RUSTFLAGS="--cfg=fuzzing --cfg=secp256k1_fuzz --cfg=hashes_fuzz"
-# ignore anything in fuzz directory since we don't want coverage of targets
-cargo llvm-cov --html --ignore-filename-regex "fuzz/" --output-dir "$OUTPUT_DIR"
+export RUSTFLAGS="--cfg=fuzzing --cfg=secp256k1_fuzz --cfg=hashes_fuzz"
+# Process each fuzz target with its corpus
+mkdir -p "$OUTPUT_DIR"
+for target_dir in fuzz/corpus/*; do
+  if [ -d "$target_dir" ]; then
+    target_name=$(basename "$target_dir")
+    echo "Generating coverage for target: $target_name"
+    # Run coverage for this specific target with its corpus
+    cargo llvm-cov --html --ignore-filename-regex "fuzz/" \
+      --fuzz-target "$target_name" \
+      --corpus-dir "$target_dir" \
+      --output-path "$OUTPUT_DIR/$target_name"
+  fi
+done
+# Generate a combined report
+cargo llvm-cov --html --ignore-filename-regex "fuzz/" --output-dir "$OUTPUT_DIR/combined"

Add script to generate fuzz coverage #3718

Are you sure you want to change the base?

Add script to generate fuzz coverage #3718

Conversation

Prabhat1308 commented Apr 8, 2025

Uh oh!

ldk-reviews-bot commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Prabhat1308 commented Apr 8, 2025

Uh oh!

TheBlueMatt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Prabhat1308 Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ldk-reviews-bot commented Apr 9, 2025

Uh oh!

Prabhat1308 commented Apr 9, 2025

Uh oh!

TheBlueMatt commented May 14, 2025

Uh oh!

Prabhat1308 commented May 18, 2025

Uh oh!

TheBlueMatt commented Jun 12, 2025

Uh oh!

Prabhat1308 commented Jun 13, 2025

Uh oh!

Prabhat1308 commented Jun 15, 2025

Uh oh!

TheBlueMatt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

graphite-app bot Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

Prabhat1308 Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Prabhat1308 commented Jul 1, 2025

Uh oh!

TheBlueMatt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Prabhat1308 commented Jul 8, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Prabhat1308 commented Jul 8, 2025

Uh oh!

ldk-reviews-bot commented Apr 8, 2025 •

edited

Loading

Prabhat1308 Apr 9, 2025 •

edited

Loading

Prabhat1308 Jun 30, 2025 •

edited

Loading