difftest: nextest support and speedups #334

Firestar99 · 2025-07-14T17:10:37Z

Some difftest refactors of things I just want to have:

switch to mimic-libtest from tester
add profile-based test filtering via nextest, as before:
- cargo nextest run will ignore difftests
- cargo difftest will only run difftests
output is now in tests/difftests/tests/target/difftest so you can easily inspect the output instead of having to search your tmp dir
moved runner to new runner crate, bin just wraps it as a "test without harness"
- split was needed to have both tests within the runner and exposing it as a test runner
moved lib into tests/difftests/tests workspace
- prevents "Updating crates.io index" on every single test, causing massive slowdowns
- created types crate with shared mod config between lib and runner
- we no longer compile rustc_codegen_spirv, wgpu, ash etc. in both workspaces, now only in difftest workspace

Speedups

master reference
rerunning difftest without changes:
- Linux: 45s -> 6s
- Windows: 195s -> 80s (whatever is going on there)
CI difftest times:
- Linux: 12m -> 8m
- Windows: 22m -> 16m
total CI time: 23m -> 17m

Firestar99 · 2025-07-15T11:27:52Z

Even if you specify nextest filters that exclude a certain package, nextest will still build it, just not run it to discover any potential tests. I'll have to add back in the manual package excludes in CI to get back to previous CI speeds.

feature request: nextest-rs/nextest#154

Firestar99 · 2025-07-15T22:03:06Z

@LegNeato this one is ready to review, apart from the macOS difftest failing. Could you do me a favor and debug that one on your mac? It's only difftests::arch::workgroup_memory failing with I/O error: No such file or directory (os error 2) when running the ash variant.
(especially since github is taking forever to launch CI again...)

LegNeato · 2025-07-16T11:47:19Z

Pushed a fix. The problem was you were making the output directory and coming up with temp_output_path, but you weren't creating temp_output_path. Preivously the contract was the harness creates the file and passes the path to the test. The tests expected the file to be created by the harness so they could just blindly open and write and the output Differs expected the file to exist so they could just read.

LegNeato · 2025-07-16T11:48:28Z

.github/workflows/ci.yaml

      - name: cargo fetch --locked
        run: cargo fetch --locked --target $TARGET
      - name: cargo fetch --locked difftests
        run: cargo fetch --locked --manifest-path=tests/difftests/tests/Cargo.toml --target $TARGET
-      - name: test difftest
-        run: cargo test -p "difftest*" --release --no-default-features --features "use-installed-tools"
+      - name: test difftest-runner


My main concern here is we are making CI smarter when best practice is to make it as dumb as possible.

I'm still a little unsure what to do about precompiling the tests, which is not just an CI issue but also when building from your own machine. Currently, it'll look like the tests are just stuck eating CPU for minutes, apart from the occasional slow warning from nextest, as all tests are blocked on the first test holding the target folder lock and compiling all dependencies. So at least for CI, instead of windows taking 7-8min of "nothing happening", we get cargo's compilation progress reporting.

We could move precompiling tests to the listing stage, but that would make nextest list take forever on first execution. Not even sure if we can even print compilation progress, as when listing tests we must only write a list of tests to stdout and nothing else, otherwise nextest will error out. Though I haven't tested if piping through stderr would work.

Maybe we could alias cargo difftest to be proceeded by a test compile before running nextest?

Isn't that basically what we had? It ran cargo build --release in the tests workspace regardless of what was running. The thinking was that most of the deps would be shared, but I guess I soon as I added ash vs wgpu that went out the window.

Crates in a workspace will compile all their dependencies into the target dir of their workspace, regardless of where they're from. So on master, the difftest lib crate would be compiled once in the difftest workspace's target and a second time in the root workspace's target. Now only types is shared like that, and a lot lighter since it's just plain types and no wgpu or ash.

tests/difftests/bin/src/main.rs

tests/difftests/runner/src/lib.rs

tests/difftests/tests/rust-toolchain.toml

LegNeato · 2025-07-16T11:55:02Z

nextest list compiles things like wgpu, is that expected?

LegNeato · 2025-07-16T11:56:30Z

nextest list also doesn't show any compiletests or difftests and nextest test doesn't run them...I would expect it to run all of them by default so we can tell people "install nextest, run nextest test before putting up a PR". But I guess this PR is just replacing the impl for cargo difftest and that can wait.

BTW, running cargo test from the root now appears to run difftests...

Firestar99 · 2025-07-16T21:27:56Z

nextest has a -P profile option to select a profile from the folllowing ones I configured:

rust-gpu/.config/nextest.toml

Lines 1 to 10 in 506469f

    
           [profile.default] 
        
           default-filter = '!package(difftest*) & !package(compiletest*) & !package(example-runner-*)' 
        
           fail-fast = false 
        
           [profile.difftests] 
        
           default-filter = 'package(difftests)' 
        
           slow-timeout = "2m" 
        
           [profile.difftest-runner] 
        
           default-filter = 'package(difftest-runner) | package(difftest-types)'

And cargo difftest just points nextest to the difftest profile:

rust-gpu/.cargo/config.toml

Line 3 in 506469f

difftest = "nextest run --release -P difftests -p difftests"

The only problem is that nextest compiles all crates, regardless of the profile excluding the crate / package or not. So in CI and the difftest alias I use cargo's -p option to only select relevant packages, to compile fewer crates. See this nextest feature request.

nextest test doesn't run [difftests]

With nextest run -P difftest you can. Let me know if you'd rather have the default test everything (including compiletests as well? #318)

nextest list compiles things like wgpu, is that expected?

In that sense, yes it is expected, since nextest needs to call the test binary to figure out which tests are contained within. Although, nextest run matches previous behavior on master where example-runner-wgpu and the "old" difftest lib (now moved to the difftest workspace) would force you to compile wgpu anyway.

BTW, running cargo test from the root now appears to run difftests...

cargo test does not care for nextest profiles (obviously), not much I can do there.

Firestar99 · 2025-07-16T21:50:42Z

tests/difftests/tests/Cargo.toml

@@ -44,15 +45,18 @@ unexpected_cfgs = { level = "allow", check-cfg = [
    'cfg(target_arch, values("spirv"))'
 ] }

+# Cargo Windows bug: workspace dependencies of crates.io dependencies are resolved
+# incorrectly in the root workspace instead of this difftest workspace. So all
+# workspace dependencies here must be mirrored into the root workspace as well.


I still don't fully understand what's causing this, I tried creating a minimal repro repo but haven't been able to. How difftest calls cargo (--manifest-path or inherited env vars) seems to be important in the repro, as you can just cargo build in the tests/difftest/tests repo and it'll build just fine.

… crate `types`, massive difftest speedup 36s -> 6s

…cies

Firestar99 force-pushed the difftest_refactor branch 2 times, most recently from 1215a7c to 5b43740 Compare July 15, 2025 11:03

Firestar99 mentioned this pull request Jul 15, 2025

Add filtering for packages, not just tests nextest-rs/nextest#154

Open

Firestar99 force-pushed the difftest_refactor branch 7 times, most recently from 0677196 to a861aa2 Compare July 15, 2025 19:59

Firestar99 force-pushed the const_folding branch from d33b067 to d8ec881 Compare July 15, 2025 20:02

Firestar99 force-pushed the difftest_refactor branch 3 times, most recently from ec23891 to da0bb51 Compare July 15, 2025 21:36

Firestar99 added 3 commits July 16, 2025 00:03

difftest: use relative paths, create output directory in target

7b9e324

difftest: fix test mod paths with subdirectories

dea18ea

difftest: fix test names on windows

593c5ba

Firestar99 force-pushed the difftest_refactor branch from da0bb51 to 55e2ffb Compare July 15, 2025 22:06

Firestar99 changed the base branch from const_folding to main July 15, 2025 22:06

Firestar99 force-pushed the difftest_refactor branch from 55e2ffb to 33dbe33 Compare July 15, 2025 22:17

Firestar99 added a commit that referenced this pull request Jul 16, 2025

difftest: fixup windows dependency resolution, from PR #334

6e1980b

LegNeato reviewed Jul 16, 2025

View reviewed changes

LegNeato force-pushed the difftest_refactor branch from e74b17e to d3a1cab Compare July 16, 2025 11:53

Firestar99 commented Jul 16, 2025

View reviewed changes

Firestar99 changed the title ~~difftest refactors~~ difftest: nextest support and speedups Jul 17, 2025

Firestar99 and others added 11 commits July 17, 2025 16:06

difftest: move difftest bin to runner

108f37a

difftest: switch to mimic-libtest from tester

2465003

difftest: disable logger when listing available tests

b8cb6df

difftest: fix use-installed-tools feature

1cc4323

difftest: use nextest to properly filter tests, adjust ci

598ef14

difftest: move lib into tests workspace, move mod config to new…

f502232

… crate `types`, massive difftest speedup 36s -> 6s

difftest: compile difftest in ci before running

7c918c7

ci: do not fail-fast matrices

c2e3a1a

difftest: better io error reporting in config

af7e1be

difftest: cargo windows bug workaround, mirror all workspace dependen…

390cdcc

…cies

difftest: Unconditionally create output file

065dc99

Firestar99 force-pushed the difftest_refactor branch from 506469f to 065dc99 Compare July 17, 2025 14:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

difftest: nextest support and speedups #334

difftest: nextest support and speedups #334

Firestar99 commented Jul 14, 2025 •

edited

Loading

Uh oh!

Firestar99 commented Jul 15, 2025

Uh oh!

Firestar99 commented Jul 15, 2025 •

edited

Loading

Uh oh!

LegNeato commented Jul 16, 2025

Uh oh!

LegNeato Jul 16, 2025

Uh oh!

Firestar99 Jul 16, 2025

Uh oh!

LegNeato Jul 17, 2025

Uh oh!

Firestar99 Jul 17, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LegNeato commented Jul 16, 2025

Uh oh!

LegNeato commented Jul 16, 2025 •

edited

Loading

Uh oh!

Firestar99 commented Jul 16, 2025 •

edited

Loading

Uh oh!

Firestar99 Jul 16, 2025 •

edited

Loading

Uh oh!

Uh oh!

difftest: nextest support and speedups #334

Are you sure you want to change the base?

difftest: nextest support and speedups #334

Conversation

Firestar99 commented Jul 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Speedups

Uh oh!

Firestar99 commented Jul 15, 2025

Uh oh!

Firestar99 commented Jul 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

LegNeato commented Jul 16, 2025

Uh oh!

LegNeato Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

Firestar99 Jul 16, 2025

Choose a reason for hiding this comment

Uh oh!

LegNeato Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

Firestar99 Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

LegNeato commented Jul 16, 2025

Uh oh!

LegNeato commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Firestar99 commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Firestar99 Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Firestar99 commented Jul 14, 2025 •

edited

Loading

Firestar99 commented Jul 15, 2025 •

edited

Loading

LegNeato commented Jul 16, 2025 •

edited

Loading

Firestar99 commented Jul 16, 2025 •

edited

Loading

Firestar99 Jul 16, 2025 •

edited

Loading