Can you debug kernels on Pascal, esp on the P100?

Hi there folks,

I’m doing development on a machine with 4 P100s (don’t ask). Nsight seems to catch on breakpoints, but not attach when debugging kernels - that is, in debug mode, when hitting a breakpoint, the program appears to freeze and VS never gets variable information. I never had this problem with K80s or other older chips.

The program runs fine, I just can’t debug it on the P100 box. Can you all debug on Pascal hardware, especially the P100? If so, any hints on how to start troubleshooting this problem? Further, do I have to compile it for some specific compute capacity to be able to debug it on Pascal?

Setup on Troubled Box: MSVC2015, Nsight 5.3, Cuda compilation tools 8.0.60, Win 7 Enterprise, Driver 376.51

If you’re running on P100, you should compile for compute_60,sm_60.
And you should compile with debug switches (-g -G) if you want to debug.

Your driver mode should be TCC. I don’t think it is possible to set WDDM mode on P100.

Note that with Nsight VSE, it’s not possible to debug host code (set host breakpoints) and device code (set device breakpoints) in the same session. You pick one or the other, based on how you start the debug session. Choose “Start CUDA debugging…” to start a device code debug session. If you don’t you won’t be able to stop on device breakpoints.

Confirmed using TCC, with -g and -G set. MSVC recognizes the breakpoint in the kernel code as valid and the breakpoints in host code as invalid, as expected, when using the CUDA debugger.

Indeed, if I don’t have the breakpoint in the kernel code and start the debugger, it’ll happily proceed through the program, only when the breakpoint is present does it stop. Further, putting a breakpoint in an unused kernel also causes it to run straight through, as expected. Swapped to compute/sm 60 and no change in behavior.

It does give me a warning about connecting to the debugger insecurely, which I’m guessing doesn’t matter?

On a broader question, should this be possible? EG, can you all debug kernels on a 1080 or P100?