cuda-gdb gets "The CUDA driver has hit an internal error." at first CUDA usage.

Developer Tools CUDA Developer Tools CUDA-GDB
1

The program runs properly under regular gdb and under valgrind. (The kernel just doesn’t produce the values I expect.)

But cuda-gdb fails on the very first cuda call:

#0  0x00007ffff41e4440 in cudbgReportDriverInternalError () from /usr/lib64/nvidia/libcuda.so.1
#1  0x00007ffff41e8743 in cudbgReportDriverInternalError () from /usr/lib64/nvidia/libcuda.so.1
#2  0x00007ffff42769c4 in cuEGLApiInit () from /usr/lib64/nvidia/libcuda.so.1
#3  0x00007ffff42c556e in cuInit () from /usr/lib64/nvidia/libcuda.so.1
#4  0x00007ffff7b8a19a in __cudaInitModule () from /usr/local/cuda-9.1/targets/x86_64-linux/lib/libcudart.so.9.1
#5  0x00007ffff7b8a1f1 in __cudaInitModule () from /usr/local/cuda-9.1/targets/x86_64-linux/lib/libcudart.so.9.1
#6  0x00007ffff753ce20 in pthread_once () from /lib64/libpthread.so.0
#7  0x00007ffff7bc2389 in cudaVDPAUSetVDPAUDevice () from /usr/local/cuda-9.1/targets/x86_64-linux/lib/libcudart.so.9.1
#8  0x00007ffff7b8645a in __cudaInitModule () from /usr/local/cuda-9.1/targets/x86_64-linux/lib/libcudart.so.9.1
#9  0x00007ffff7b8b67b in __cudaInitModule () from /usr/local/cuda-9.1/targets/x86_64-linux/lib/libcudart.so.9.1
#10 0x00007ffff7baf37a in cudaFree () from /usr/local/cuda-9.1/targets/x86_64-linux/lib/libcudart.so.9.1

(I have a cudaFree(0) call up front to force initialization.)

CentOS 7, CUDA 9.1.85, gcc 4.8
Driver version is 390.12

There are two GPU cards, the program uses device 0. deviceQuery reports:
Device 0: “GeForce GTX 1080 Ti”
Device 1: “Quadro P400”

2

Hi, dlevi

Is this specific to your application?

If yes, please provide us the app you used and list your usage of cuda-gdb.

Thanks !

3

It appears the first CUDA call must come from the main thread.