Unable to debug CUDA samples in GTX1080
Hello, I have some trouble debugging some CUDA code using Nsight. so I've tried to debug one of the CUDA sample projects (sortingNetworks) using Nsight as well. I reach the desired breakpoint, but when I continue execution I have an exception thrown: CUDA module loaded: 03045ec0 oddEvenMergeSort.cu CUDA grid launch failed: CUcontext: 40233744 CUmodule: 50617792 Function: _Z18bitonicSortShared1PjS_S_S_ and the program halts. under normal visual studio debugger the code executes well. I did not tinker at all with the project's/files' properties. some background: Nsight version: 5.2.0.16321 CUDA driver: 378.49 CUDA toolkit: 8.0.44 Visual studio 2013, under update 5 GPU: GTX-1080
Hello,

I have some trouble debugging some CUDA code using Nsight.
so I've tried to debug one of the CUDA sample projects (sortingNetworks) using Nsight as well.
I reach the desired breakpoint, but when I continue execution I have an exception thrown:

CUDA module loaded: 03045ec0 oddEvenMergeSort.cu
CUDA grid launch failed: CUcontext: 40233744 CUmodule: 50617792 Function: _Z18bitonicSortShared1PjS_S_S_

and the program halts.

under normal visual studio debugger the code executes well.

I did not tinker at all with the project's/files' properties.

some background:
Nsight version: 5.2.0.16321
CUDA driver: 378.49
CUDA toolkit: 8.0.44
Visual studio 2013, under update 5

GPU: GTX-1080

#1
Posted 02/09/2017 07:56 AM   
Hi matanw, This issue always caused by the driver, could you please try the driver 376.33, I'm aware that 378.49 doesn't work well with cuda. Best Regards Harry
Hi matanw,

This issue always caused by the driver, could you please try the driver 376.33, I'm aware that 378.49 doesn't work well with cuda.

Best Regards
Harry

#2
Posted 02/10/2017 03:37 AM   
Hi Harry, is this a known issue? I've looked it up and didn't find anything about it... is this a driver issue with gtx-1080, or the pascal architecture? I have the same code that runs well on previous generation Titan-X? thanks, Matan
Hi Harry,

is this a known issue? I've looked it up and didn't find anything about it...

is this a driver issue with gtx-1080, or the pascal architecture?
I have the same code that runs well on previous generation Titan-X?

thanks,
Matan

#3
Posted 02/11/2017 09:17 PM   
just installed driver 376.33, got the same exception... tried to revert to each version from current, down to 375.95, but got the same exception every time... tried to install CUDA Toolkit 8.0.61, got the same exception... any thoughts? TIA Matan
just installed driver 376.33, got the same exception...
tried to revert to each version from current, down to 375.95, but got the same exception every time...

tried to install CUDA Toolkit 8.0.61, got the same exception...

any thoughts?

TIA
Matan

#4
Posted 02/12/2017 09:18 AM   
Hi matanw, I met the same issue before, reinstalling the driver solved it, so it's hard to say why. Could you tell me which os you use? Do you enable the memcheck in nsight? Best Regards Harry
Hi matanw,

I met the same issue before, reinstalling the driver solved it, so it's hard to say why.

Could you tell me which os you use? Do you enable the memcheck in nsight?

Best Regards
Harry

#5
Posted 02/13/2017 06:42 AM   
I'm on Maxwell (980 Ti), and have the same issue. Also tried to revert driver versions and CUDA versions to no avail. The lack of "debugability" of this problem is very frustrating :(
I'm on Maxwell (980 Ti), and have the same issue. Also tried to revert driver versions and CUDA versions to no avail. The lack of "debugability" of this problem is very frustrating :(

#6
Posted 02/13/2017 07:18 AM   
Hi Harry, My OS is windows 7. I've tried debugging both with enabled memcheck and disabled, to similar effect. I have a separate GTX-1080, tried to install it as well on a another machine with similar environment, got the same result. moreover, the second GTX-1080 I've just specified was installed onto a machine that ran and debugged the sample with a previous generation TITAN-X successfully, and now fails to debug with the GTX-1080. Is there a way to collect valuable error data (a log file, a dump file, whatever) that will assist in detecting the error and fixing it in the upcoming drivers / cuda toolkit? Matan
Hi Harry,

My OS is windows 7.
I've tried debugging both with enabled memcheck and disabled, to similar effect.

I have a separate GTX-1080, tried to install it as well on a another machine with similar environment,
got the same result.

moreover, the second GTX-1080 I've just specified was installed onto a machine that
ran and debugged the sample with a previous generation TITAN-X successfully, and now fails to debug with the GTX-1080.

Is there a way to collect valuable error data (a log file, a dump file, whatever)
that will assist in detecting the error and fixing it in the upcoming drivers / cuda toolkit?

Matan

#7
Posted 02/13/2017 08:05 AM   
Hi all, I'm experiencing the same issues as well. Recalling debugging issues I had with Nsight 5.2 RC2, https://devtalk.nvidia.com/default/topic/975742/nsight-5-2-rc2-debugging-issues/#5026457 I've reverted to that version and voila, problem (temporarily) solved! By the way, I'm still using 375.95 that driver. HTH, Avishay
Hi all,

I'm experiencing the same issues as well.

Recalling debugging issues I had with Nsight 5.2 RC2,
https://devtalk.nvidia.com/default/topic/975742/nsight-5-2-rc2-debugging-issues/#5026457
I've reverted to that version and voila, problem (temporarily) solved!
By the way, I'm still using 375.95 that driver.

HTH,
Avishay

#8
Posted 02/13/2017 10:47 AM   
My bad. Seems that if I use no breakpoints at all, running under the debugger eventually ends up with grid launch failed. - Avishay
My bad.
Seems that if I use no breakpoints at all, running under the debugger eventually ends up with grid launch failed.
- Avishay

#9
Posted 02/13/2017 10:56 AM   
But it also seems a problem, there should be no grid launch failure, please update the state if you meet it again.
But it also seems a problem, there should be no grid launch failure, please update the state if you meet it again.

#10
Posted 02/14/2017 03:44 AM   
Hello Harry, I get grid launch failure repeatedly, each time I try to debug using Nsight. I ran my code, and some of the CUDA sample code, mostly sortingNetworks. 1. I am able to hit a breakpoint if I set it early enough, but when I continue to the next kernel I get grid launch failure exception 2. If I try to debug using Nsight without any breakpoints, I also get the grid launch failure (at the same point in the same kernel as thrown in (1)) and the program halts. 3. all samples and my code work well if not trying to debug the kernels using Nsight. this was tested on two separate PCs with two different GTX-1080s, running windows 7 with VS2013-update 5. different drivers were tested from 375.95 up to the recent one. all of the above works and debugs well on TITAN-X (Maxwell). Matan
Hello Harry,

I get grid launch failure repeatedly, each time I try to debug using Nsight.

I ran my code, and some of the CUDA sample code, mostly sortingNetworks.

1. I am able to hit a breakpoint if I set it early enough, but when I continue to the next kernel I get grid launch failure exception

2. If I try to debug using Nsight without any breakpoints, I also get the grid launch failure (at the same point in the same kernel as thrown in (1)) and the program halts.

3. all samples and my code work well if not trying to debug the kernels using Nsight.

this was tested on two separate PCs with two different GTX-1080s, running windows 7 with VS2013-update 5.
different drivers were tested from 375.95 up to the recent one.

all of the above works and debugs well on TITAN-X (Maxwell).

Matan

#11
Posted 02/14/2017 07:58 AM   
I'll try to repo it today.
I'll try to repo it today.

#12
Posted 02/15/2017 02:39 AM   
Hi matanw, Bad news, I cannot repo it on both win7 and win10 with the same configuration, I've also tried the 1070 and 1060 and they both work well, maybe I should involve the dev to see what they can find out. BTY I just see the latest driver is 378.66, I think you should have a try, maybe Custom(Advanced) -> "perform clean installation" can beat this issue. If clean installation and 378.66 still cannot solve it, I will raise a bug to track it.
Hi matanw,

Bad news, I cannot repo it on both win7 and win10 with the same configuration, I've also tried the 1070 and 1060 and they both work well, maybe I should involve the dev to see what they can find out.

BTY I just see the latest driver is 378.66, I think you should have a try, maybe Custom(Advanced) -> "perform clean installation" can beat this issue.

If clean installation and 378.66 still cannot solve it, I will raise a bug to track it.

#13
Posted 02/15/2017 07:02 AM   
Hi Harry, Thanks for addressing the issue. I have tried a clean installation for 375.95, 376.33, 378.49 in the past. Now I've tried a clean installation for 378.66 (which is from yesterday) and got the same error. Is there any additional information you require, regarding the GPU, PC or environment settings? TIA, Matan
Hi Harry,

Thanks for addressing the issue.

I have tried a clean installation for 375.95, 376.33, 378.49 in the past.
Now I've tried a clean installation for 378.66 (which is from yesterday) and got the same error.

Is there any additional information you require, regarding the GPU, PC or environment settings?

TIA,
Matan

#14
Posted 02/15/2017 07:54 AM   
Thank you for trying this, yeah, I need your system information including GPU info, windows and CUDA verions. BTW, did you try to debug the x86 or x64 app?
Thank you for trying this, yeah, I need your system information including GPU info, windows and CUDA verions.

BTW, did you try to debug the x86 or x64 app?

#15
Posted 02/16/2017 03:20 AM   
Scroll To Top

Add Reply