I’m having this weird issue: I have a program that uses CUPTI callbackAPI to monitor the kernels in the program. It runs well when it’s directly launched; but when I put it under cuda-gdb and run, it failed with the following error:
error: function cuptiSubscribe(&subscriber, (CUpti_CallbackFunc)getStackTraceCallback, NULL) failed with error CUPTI_ERROR_NOT_INITIALIZED
I’ve tried CUDA 8.0, 9.0, but failed with the same error. Any thoughts?
Update#3:
I’ve tried all examples in CUPTI/samples and concluded that programs that use callbackAPI and activityAPI will fail under cuda-gdb. (They are all well-behaved without cuda-gdb) But the fail reason differs:
If I have calls from activityAPI, then once run it under cuda-gdb, it’ll hang for a minute then exit with error:
The CUDA driver has hit an internal error.
Error code: 0x100ff00000001c
Further execution or debugging is unreliable.
Please ensure that your temporary directory is mounted with write and exec permissions.
If I have calls from callbackAPI, then it’ll fail out much sooner with error:
error: function cuptiSubscribe(&subscriber, (CUpti_CallbackFunc)getStackTraceCallback, NULL) failed with error CUPTI_ERROR_NOT_INITIALIZED
we don’t have support for running two cuda tools together. That is we can have only one of CUPTI or nvprof or cuda-gdb or cuda-memcheck applied to an application at one time.
I appreciate the information.
Besides, I found there’s also a Debugger API but very limited documentation and no samples. Is that what you use to build cuda-gdb? Basically, you can’t have calls from CUPTI and Debugger API in the same program right?