SYCL Replay Error

[steverangel]

Error replaying kernel using SYCL and Intel(R) oneAPI DPC++/C++ Compiler 2025.0.4 (2025.0.4.20241205)

Pointers to device memory input do not get processed by the Python run.py script correctly.

python3.10 --version
Python 3.10.10

python3.10 run.py
Traceback (most recent call last):
File "/home/rangel/CLIntercept_Dump/sycl_axpy/Replay/Enqueue_2__ZTS11AXPYFunctor/run.py", line 75, in
input_buffer_ptrs[arguments[idx]].append(idx)
KeyError: 1

Enqueue_2__ZTS11AXPYFunctor.tar.gz

readme.txt

[bashbaug]

Good timing, I was actually working on a similar issue / request earlier this week!

Currently, the capture and replay functionality doesn't support SVM or USM, which is why this SYCL case doesn't work. There's no technical reason for this; it is only because the original use-case motivating this feature only used OpenCL buffers and images, so support for SVM and USM hasn't been implemented yet.

It's pretty easy to support basic SVM and USM cases though, see #419. This will work for many SVM and USM cases, but there are two limitations: First, it only works when the SVM or USM pointer passed to the kernel is the base of the SVM or USM allocation, and second, it doesn't handle cases where an SVM or USM allocation is used "indirectly" versus being passed as a kernel argument. The first limitation could be overcome with a bit more work, but the second is much more challenging, and I'm not sure how to solve it.

Do your use-cases pass pointers to the middle of your SYCL USM allocations? Do they access USM allocations indirectly? If not, I suspect the changes in #419 will work for you, and I'll try to get them merged shortly. I'm able to successfully run the tester attached to this issue with these changes, for example:

$ CLI_SuppressLogging=1 ONEAPI_DEVICE_SELECTOR=opencl:* python3 ~/git/CLIntercept/scripts/capture_and_validate.py -c cliloader -p ./test -k _ZTS11AXPYFunctor


Running test application: cliloader ./test
Running on Intel(R) Arc(TM) B580 Graphics
AXPY successful.
Total Enqueues: 4

Done!


Looking for output in directory: /home/bashbaug/CLIntercept_Dump/test/Replay
Using replay information from: /home/bashbaug/CLIntercept_Dump/test/Replay/Enqueue_2__ZTS11AXPYFunctor


Replaying kernel:
Running on platform: Intel(R) OpenCL Graphics
Running on device: Intel(R) Arc(TM) B580 Graphics
Using build options: [' -cl-kernel-arg-info']
Using kernel device binary
/usr/lib/python3/dist-packages/pyopencl/invoker.py:376: UserWarning: pytools.persistent_dict 'pyopencl-invoker-cache-v41': enabling safe_sync as default. This provides strong protection against data loss, but can be unnecessarily expensive for use cases such as caches.Pass 'safe_sync=False' if occasional data loss is tolerable. Pass 'safe_sync=True' to suppress this warning.
  invoker_cache = WriteOncePersistentDict(
Successfully loaded kernel device binary file: ./DeviceBinary0.bin
Global Work Size: [1024]
Local Work Size: [0]
Global Work Offsets: [0]
Writing buffer output to file: ./Test/Enqueue_0002_Kernel__ZTS11AXPYFunctor_Arg_1_Buffer.bin
Writing buffer output to file: ./Test/Enqueue_0002_Kernel__ZTS11AXPYFunctor_Arg_0_Buffer.bin
Done!


Now starting validation...
Check: Argument 0 is equal.
Check: Argument 1 is equal.

Replayed standalone kernel produces correct results.

[steverangel]

Indeed I was able to run the reproducer using the capture-replay-svm-usm branch. It seems to work as expected:

CLI_OpenCLFileName=/opt/aurora/24.347.0/oneapi/compiler/latest/lib/libOpenCL.so.1 CLI_SuppressLogging=1 ONEAPI_DEVICE_SELECTOR=opencl:* python3.10 /###/software/branch/opencl-intercept-layer/scripts/capture_and_validate.py -c /###/software/branch/opencl-intercept-layer_install/bin/cliloader -p ./sycl_axpy -k _ZTS11AXPYFunctor


Running test application: /###/software/branch/opencl-intercept-layer_install/bin/cliloader ./sycl_axpy
Running on Intel(R) Data Center GPU Max 1550
AXPY successful.
Total Enqueues: 4

Done!


Looking for output in directory: /###/CLIntercept_Dump/sycl_axpy/Replay
Using replay information from: /###/CLIntercept_Dump/sycl_axpy/Replay/Enqueue_2__ZTS11AXPYFunctor


Replaying kernel:
Choose platform:
[0] <pyopencl.Platform 'Intel(R) OpenCL Graphics' at 0x55f34f321a00>
Choice [0]:
Choose device(s):
[0] <pyopencl.Device 'Intel(R) Data Center GPU Max 1550' on 'Intel(R) OpenCL Graphics' at 0x55f34f321ad0>
[1] <pyopencl.Device 'Intel(R) Data Center GPU Max 1550' on 'Intel(R) OpenCL Graphics' at 0x55f34f32d200>
[2] <pyopencl.Device 'Intel(R) Data Center GPU Max 1550' on 'Intel(R) OpenCL Graphics' at 0x55f34f337260>
[3] <pyopencl.Device 'Intel(R) Data Center GPU Max 1550' on 'Intel(R) OpenCL Graphics' at 0x55f34f340c40>
[4] <pyopencl.Device 'Intel(R) Data Center GPU Max 1550' on 'Intel(R) OpenCL Graphics' at 0x55f34f34b2b0>
[5] <pyopencl.Device 'Intel(R) Data Center GPU Max 1550' on 'Intel(R) OpenCL Graphics' at 0x55f34f354c90>
Choice, comma-separated [0]:
Set the environment variable PYOPENCL_CTX=':' to avoid being asked again.
Running on platform: Intel(R) OpenCL Graphics
Running on device: Intel(R) Data Center GPU Max 1550
Using build options: []
Using kernel device binary
Successfully loaded kernel device binary file: ./DeviceBinary1.bin
Global Work Size: [1024]
Local Work Size: [0]
Global Work Offsets: [0]
Writing buffer output to file: ./Test/Enqueue_0002_Kernel__ZTS11AXPYFunctor_Arg_1_Buffer.bin
Writing buffer output to file: ./Test/Enqueue_0002_Kernel__ZTS11AXPYFunctor_Arg_0_Buffer.bin
Done!


Now starting validation...
Check: Argument 0 is equal.
Check: Argument 1 is equal.

Replayed standalone kernel produces correct results.

I was also able to record and replay a kernel taken from a single process of a full MPI application that uses kernel arguments that are base pointers, like reproducer. The run.py script in the recorded directory ran without error. The use-case I am interested in, however, is the pointer-to-the-middle of a SYCL USM allocation as a kernel argument. This currently fails a couple different ways:

python3.10 run.py
Choose platform:
[0] <pyopencl.Platform 'Intel(R) OpenCL Graphics' at 0x5637adbf9100>
Choice [0]:0
Choose device(s):
[0] <pyopencl.Device 'Intel(R) Data Center GPU Max 1550' on 'Intel(R) OpenCL Graphics' at 0x5637adbf91d0>
[1] <pyopencl.Device 'Intel(R) Data Center GPU Max 1550' on 'Intel(R) OpenCL Graphics' at 0x5637adc04900>
[2] <pyopencl.Device 'Intel(R) Data Center GPU Max 1550' on 'Intel(R) OpenCL Graphics' at 0x5637adc0e960>
[3] <pyopencl.Device 'Intel(R) Data Center GPU Max 1550' on 'Intel(R) OpenCL Graphics' at 0x5637adc18340>
[4] <pyopencl.Device 'Intel(R) Data Center GPU Max 1550' on 'Intel(R) OpenCL Graphics' at 0x5637adc229b0>
[5] <pyopencl.Device 'Intel(R) Data Center GPU Max 1550' on 'Intel(R) OpenCL Graphics' at 0x5637adc2c390>
Choice, comma-separated [0]:
Set the environment variable PYOPENCL_CTX='0:' to avoid being asked again.
Running on platform: Intel(R) OpenCL Graphics
Running on device: Intel(R) Data Center GPU Max 1550
Using build options: ['']
Using kernel device binary
Successfully loaded kernel device binary file: ./DeviceBinary0.bin
Segmentation fault

OR

 python3.10 run.py
Traceback (most recent call last):
  File "/###/CLIntercept_Dump/hacc_tpm/Replay/Enqueue_94__ZTSZZN5kutil4impl6launchI14updateGeometryJPKiS4_PKfS6_S6_S6_S6_PKtS4_S4_S4_S4_S4_iPfEEEvNS_12LaunchParamsEDpT0_ENKUlRN4sycl3_V17handlerEE_clESG_EUlNSE_7nd_itemILi1EEEE_/run.py", line 75, in <module>
    input_buffer_ptrs[arguments[idx]].append(idx)
KeyError: 10

[bashbaug]

Cool, thanks for confirming that the basic case is working now.

I think I know what's needed to handle the pointer-to-the-middle case, but this isn't going to be a trivial fix unfortunately. I'll keep this issue open for tracking, and I'll try to have a solution in a couple of weeks.