The CUDA driver hit an error while forking off the debugger process.

Hi all, on attaching cuda gdb to existing process i am getting following error
[b]The CUDA driver hit an error while forking off the debugger process.

An error occurred while in a function called from GDB.
Evaluation of the expression containing the function
(cudbgApiAttach) will be abandoned.
When the function is done executing, GDB will silently stop.
(cuda-gdb)[/b]

I am able to debug a standalone application with no issues, but on attaching cuda-gdb to existing process it is showing above error.(cuda toolkit 8 device k80)
Can anybody help me on this.
thank you

Hi, raziuddinmohd

In order to enable the attach feature of CUDA debugger, either cuda-gdb should be laucnhed as root, or /proc/sys/kernel/yama/ptrace_scope should be set to zero, using the following command:
$ sudo sh -c “echo 0 >/proc/sys/kernel/yama/ptrace_scope”

Have you done this setting before?

And are you using following command to attach ?
cuda-gdbserver --attach :2345 PID_of_app

Hi veraj
I have verified cuda-gdb is running as root
root 10476 0.0 0.0 25616 8988 pts/0 S+ 04:17 0:00 cuda-gdb

Machine is having rhel 7.0 installed there is no /proc/sys/kernel/yama/ptrace_scope present.
I am debugging on machine where application is running there is not remote debugging.
Please let me know if any inputs is needed.
thank you

Would you tell me the toolkit and driver version on your machine ?
I will try to reproduce my side

Also have you tried other samples like SDK samples 2_Graphics/simpleGL, also reproduce the issue ?

PS: Are you using the command cuda-gdb --pid=XXX to attach ?

Hi veraj,
I am able to debug SDK samples 2_Graphics/simpleGL by attaching through cuda-gdb --pid=XXX. I am able to debug running process with cuda-gdb by attaching to it.
It looks like my application specific problem.
thank you for help.

Cuda-gdb after giving the below error upon continue(continue command in cuda-gdb) is killing the process attached to it.
We have no idea why it is happening to this process only.

An error occurred while in a function called from GDB.
Evaluation of the expression containing the function
(cudbgApiAttach) will be abandoned.
When the function is done executing, GDB will silently stop.
(cuda-gdb)

The backtrace before exiting the process.

#0 0x00007fffebc75730 in cudbgReportDriverInternalError () from /lib64/libcuda.so.1
#1 0x00007fffebc7a849 in cudbgReportDriverInternalError () from /lib64/libcuda.so.1
#2
#3 0x00007fffecb404c7 in ioctl () from /lib64/libc.so.6
#4 0x00007ffff2872bd0 in syscalldriver (ioctl_code=ioctl_code@entry=3223324737, paramp=paramp@entry=0x7fffffffd400) at sys/syscall.c:265
#5 0x00007ffff2871a4d in sysusersleep (resultp=resultp@entry=0x7fffffffd42c, reason=reason@entry=1, timeout=)
at sys/syslib.c:132
#6 0x00007ffff2865610 in tsksleep (secs=) at tsk/tskuser.c:1741
#7 0x00007ffff69e1e0b in timhang (resultp=0x7ffff4796320 , deciseconds=) at …/…/lib/tim/timlib.c:444
#8 0x00007ffff4065bd6 in actmain (argc=1, argv=0x7fffffffdf58) at …/…/amp/act/actmain.c:2722
#9 0x0000000000400e9a in main (argc=1, argv=0x7fffffffdf58) at …/…/amp/act/actmainmain.c:51

Cuda-gdb not able to attach the process is one problem. And the major problem now is that it is killing the process attached to it.
And also do cuda-gdb have traces or debug statements to debug this at lower level.
Can anybody help me on this.
thank you

Hi, raziuddinmohd

Which sample has this problem ?
Your specific sample or sdk sample ?

What your operations exactly

  1. cuda-gdb --pid=XXX
  2. press ‘c’ in cuda-gdb then the process killed automatically ?

Hello, veraj

I met a different problem when attaching cuda-gdb to a running process.
It did attach to the process, but I couldn’t do any valid gdb commands like bt, p, etc. It’ll print out this error:

cuda-gdb/7.12/gdb/cuda-coords.c:1093: internal-error: cuda_current_device: Assertion `cuda_focus_is_device ()’ failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.

I’ve tried on both my own application and cuda samples that comes with the release and all failed on this error. How can I do with it? The normal “launch program under cuda-gdb” way works but I need to attach it dynamically.

Thanks!

Hi, hzhang86

Which toolkit version are you using ?
Which GPU ? Is this also for display ?
Which SDK sample have your tried ?

Hello, Veraj

I’m using the latest version V9.0.176.

Single GPU at Tesla k20c. What do you mean by “also for display”? (sorry I’m new to gpu)

I’ve tried many samples, but I don’t know how to set arguments to those samples (no instructions found in each readme.txt) so I basically just do ‘./program’ under samples/bin/x86_64/linux/release. But since I need it to run long enough in order to attach cuda-gdb to it before it’s done so I tried many and found “c++11_cuda” that will let me do that and it failed. Also, I worked on lulesh-cuda_2.0 downloaded from LLNL website and it failed with the same error as I posted.

Thanks a lot for you help!

I’m basically using GPU for the general computing acceleration purpose, not rendering or other graph-specific processng

Hi, hzhang86

Let’s discuss the issue in the topic you raised. attach cuda-gdb to a running process failed - CUDA-GDB - NVIDIA Developer Forums