Test Failure With NVC 25.3

[insertinterestingnamehere]

With the update to nvc 25.3 there's now a consistent test failure in qthread_disable_shepherd. See https://app.circleci.com/pipelines/github/insertinterestingnamehere/qthreads/830/workflows/ef89ce9f-677a-4c15-b2c5-e6b09fc84fba/jobs/65268 for an example.

[insertinterestingnamehere]

qlfqueue is also hanging with this compiler.

[insertinterestingnamehere]

The qthread_disable_shepherd error only shows up in builds with optimizations on. It can be replicated in RelWithDebInfo builds though. Could be that the nvc optimizer is doing something illegal or that it's being more aggressive and exploiting some kind of bug in the qthreads code. That's not clear yet.

Some example tracebacks:

Thread 3 (Thread 0x7f6751feb6c0 (LWP 7595) "qthread_disable"):
#0  0x00007f67551585a7 in qt_scheduler_get_thread (q=0xa90240, active=<optimized out>) at qthreads/src/threadqueues/nemesis_threadqueues.c:280
#1  0x00007f6755151d9a in qthread_master (arg=0xa5a720) at qthreads/src/qthread.c:360
#2  0x00007f675308aaa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#3  0x00007f6753117a34 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 2 (Thread 0x7f67527ec6c0 (LWP 7594) "qthread_disable"):
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007f675303327e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007f67530168ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x00000000004012dd in checkres (arg=0xffffffffffffffff) at qthreads/test/basics/qthread_disable_shepherd.c:16
#6  0x00007f67551543cb in qthread_wrapper (ptr=<optimized out>) at qthreads/src/qthread.c:1568
#7  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f6753ff6fc0 (LWP 7591) "qthread_disable"):
#0  0x00007f6753c05147 in __atomic_load () from /opt/nvidia/hpc_sdk/Linux_x86_64/25.3/compilers/lib/libnvhpcatm.so
#1  0x00007f67551585a2 in qt_scheduler_get_thread (q=0xa90040, active=<optimized out>) at qthreads/src/threadqueues/nemesis_threadqueues.c:280
#2  0x00007f6755151d9a in qthread_master (arg=0xa5a660) at qthreads/src/qthread.c:360
#3  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7fb4473eb6c0 (LWP 11537) "qthread_disable"):
#0  0x00007fb4490051a4 in __atomic_load () from /opt/nvidia/hpc_sdk/Linux_x86_64/25.3/compilers/lib/libnvhpcatm.so
#1  0x00007fb44a6365a2 in qt_scheduler_get_thread (q=0x89c240, active=<optimized out>) at qthreads/src/threadqueues/nemesis_threadqueues.c:280
#2  0x00007fb44a62fd9a in qthread_master (arg=0x866720) at qthreads/src/qthread.c:360
#3  0x00007fb44848aaa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#4  0x00007fb448517a34 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 2 (Thread 0x7fb447bec6c0 (LWP 11536) "qthread_disable"):
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007fb44843327e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007fb4484168ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x00000000004012dd in checkres (arg=0xffffffffffffffff) at qthreads/test/basics/qthread_disable_shepherd.c:16
#6  0x00007fb44a6323cb in qthread_wrapper (ptr=<optimized out>) at qthreads/src/qthread.c:1568
#7  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fb44a4d5fc0 (LWP 11533) "qthread_disable"):
#0  qt_scheduler_get_thread (q=0x89c040, active=<optimized out>) at qthreads/src/threadqueues/nemesis_threadqueues.c:290
#1  0x00007fb44a62fd9a in qthread_master (arg=0x866660) at qthreads/src/qthread.c:360
#2  0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f12debeb6c0 (LWP 11552) "qthread_disable"):
#0  0x00007f12e1d9d5a7 in qt_scheduler_get_thread (q=0x20d6240, active=<optimized out>) at qthreads/src/threadqueues/nemesis_threadqueues.c:280
#1  0x00007f12e1d96d9a in qthread_master (arg=0x20a0720) at qthreads/src/qthread.c:360
#2  0x00007f12dfc8aaa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#3  0x00007f12dfd17a34 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 2 (Thread 0x7f12df3ec6c0 (LWP 11551) "qthread_disable"):
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=<optimized out>) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=<optimized out>, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007f12dfc3327e in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007f12dfc168ff in __GI_abort () at ./stdlib/abort.c:79
#5  0x00000000004012dd in checkres (arg=0xffffffffffffffff) at qthreads/test/basics/qthread_disable_shepherd.c:16
#6  0x00007f12e1d993cb in qthread_wrapper (ptr=<optimized out>) at qthreads/src/qthread.c:1568
#7  0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f12e1c3cfc0 (LWP 11548) "qthread_disable"):
#0  0x00007f12e1d9d5a7 in qt_scheduler_get_thread (q=0x20d6040, active=<optimized out>) at qthreads/src/threadqueues/nemesis_threadqueues.c:280
#1  0x00007f12e1d96d9a in qthread_master (arg=0x20a0660) at qthreads/src/qthread.c:360

[insertinterestingnamehere]

qlfqueue hang also does not show up in debug mode, so it may have the same underlying cause. Here's an example traceback from interrupting the hung test:

Thread 2 (Thread 0x7f88e5fec6c0 (LWP 15232) "qlfqueue"):
#0  0x00007f88e740515a in __atomic_load () from /opt/nvidia/hpc_sdk/Linux_x86_64/25.3/compilers/lib/libnvhpcatm.so
#1  0x00007f88e894b5a2 in qt_scheduler_get_thread (q=0x10c4140, active=<optimized out>) at qthreads/src/threadqueues/nemesis_threadqueues.c:280
#2  0x00007f88e8944d9a in qthread_master (arg=0x108e5a0) at qthreads/src/qthread.c:360
#3  0x00007f88e688aaa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#4  0x00007f88e6917a34 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 1 (Thread 0x7f88e8818fc0 (LWP 15229) "qlfqueue"):
#0  qlfqueue_dequeue (q=0x10ac1d0) at qthreads/src/ds/qlfqueue.c:131
#1  0x000000000040159d in main (argc=<optimized out>, argv=<optimized out>) at qthreads/test/features/qlfqueue.c:66

Thread 2 (Thread 0x7f91dd7ec6c0 (LWP 15274) "qlfqueue"):
#0  0x00007f91e01938d0 in __atomic_load@plt () from /qthreads/build/src/libqthread.so
#1  0x00007f91e01ba5a2 in qt_scheduler_get_thread (q=0x221d140, active=<optimized out>) at /qthreads/src/threadqueues/nemesis_threadqueues.c:280
#2  0x00007f91e01b3d9a in qthread_master (arg=0x21e75a0) at /qthreads/src/qthread.c:360
#3  0x00007f91de08aaa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#4  0x00007f91de117a34 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 1 (Thread 0x7f91e0059fc0 (LWP 15271) "qlfqueue"):
#0  hazardous_ptr (which=0, ptr=0x224f010) at /qthreads/src/hazardptrs.c:89
#1  0x00007f91e01c6f2d in qlfqueue_dequeue (q=0x22051d0) at /qthreads/src/ds/qlfqueue.c:131
#2  0x000000000040159d in main (argc=<optimized out>, argv=<optimized out>) at /qthreads/test/features/qlfqueue.c:66

Thread 2 (Thread 0x7ff5df7ec6c0 (LWP 15288) "qlfqueue"):
#0  0x00007ff5e221f5a7 in qt_scheduler_get_thread (q=0x788140, active=<optimized out>) at qthreads/src/threadqueues/nemesis_threadqueues.c:280
#1  0x00007ff5e2218d9a in qthread_master (arg=0x7525a0) at qthreads/src/qthread.c:360
#2  0x00007ff5e008aaa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#3  0x00007ff5e0117a34 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 1 (Thread 0x7ff5e20befc0 (LWP 15285) "qlfqueue"):
#0  0x00007ff5e21fcc5c in hazardous_ptr (which=0, ptr=0x7ba010) at qthreads/src/hazardptrs.c:65
#1  0x00007ff5e222bf2d in qlfqueue_dequeue (q=0x7701d0) at qthreads/src/ds/qlfqueue.c:131
#2  0x000000000040159d in main (argc=<optimized out>, argv=<optimized out>) at qthreads/test/features/qlfqueue.c:66

Generally the qt_scheduler_get_thread and qlfqueue_dequeue calls are generally present in all tracebacks, but the stuff further in varies. Note however, that this failure occurs independent of the threadqueue used. These tracebacks were all from a nemesis/no topology build.

Interesting note: it looks like the nvhpc libraries provide their own implementation of libatomic now. That may or may not be relevant, but it's certainly interesting and unique to this config. Most other compilers just call out to libatomic.

[insertinterestingnamehere]

Here's a traceback from the qlfqueue hang in a sherwood build:

Thread 2 (Thread 0x7f3ac7dec6c0 (LWP 17148) "qlfqueue"):
#0  0x00007f3ac86fc7db in __GI_sched_yield () at ../sysdeps/unix/syscall-template.S:120
#1  0x00007f3aca770776 in qthread_steal (thief_shepherd=0xbdc478) at qthreads/src/threadqueues/sherwood_threadqueues.c:505
#2  qt_scheduler_get_thread (q=0xc12080, active=1 '\001') at qthreads/src/threadqueues/sherwood_threadqueues.c:266
#3  0x00007f3aca769dda in qthread_master (arg=0xbdc5a0) at qthreads/src/qthread.c:360
#4  0x00007f3ac868aaa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#5  0x00007f3ac8717a34 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 1 (Thread 0x7f3aca60ffc0 (LWP 17145) "qlfqueue"):
#0  0x00007f3aca77d5b0 in qlfqueue_dequeue (q=0xbfa1d0) at qthreads/src/ds/qlfqueue.c:132
#1  0x000000000040159d in main (argc=<optimized out>, argv=<optimized out>) at qthreads/test/features/qlfqueue.c:66

And with distrib:

#0  0x00007fc91f6fc7db in __GI_sched_yield () at ../sysdeps/unix/syscall-template.S:120
#1  0x00007fc9217f9776 in qthread_steal (thief_shepherd=0xb23478) at qthreads/src/threadqueues/sherwood_threadqueues.c:505
#2  qt_scheduler_get_thread (q=0xb59080, active=1 '\001') at qthreads/src/threadqueues/sherwood_threadqueues.c:266
#3  0x00007fc9217f2dda in qthread_master (arg=0xb235a0) at qthreads/src/qthread.c:360
#4  0x00007fc91f68aaa4 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447
#5  0x00007fc91f717a34 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Thread 1 (Thread 0x7fc921698fc0 (LWP 19007) "qlfqueue"):
#0  hazardous_ptr (which=1, ptr=0xb8b020) at qthreads/src/hazardptrs.c:89
#1  0x00007fc9218065c7 in qlfqueue_dequeue (q=0xb411d0) at qthreads/src/ds/qlfqueue.c:137
#2  0x000000000040159d in main (argc=<optimized out>, argv=<optimized out>) at qthreads/test/features/qlfqueue.c:66

[insertinterestingnamehere]

Fixed in #361. It was a compiler bug. It was also really easy to work around once I realized what was going on. See 362df1a for exact details. I also reported it upstream at https://forums.developer.nvidia.com/t/bug-report-sync-val-compare-and-swap-erroneously-optimized-away/332608.