[RESOLVED] Profiling error 4168:999
When attempting to run either "kernel profile - Instruction Execution" or "kernel compute" analysis of my kernel, I get the following messages: nvprof log: C:\Users\alexa\nvvp_workspace\.metadata\.plugins\com.nvidia.viper\launch\12\nvprof_19020.log nvprof log: C:\Users\alexa\nvvp_workspace\.metadata\.plugins\com.nvidia.viper\launch\12\nvprof_10800.log ======== Error: CUDA profiling error. ==19020== Error: Internal profiling error 4168:999. I am not certain what to do with this. Occasionally when running "kernel compute" I get the basic graph of Function Unit Utilization. I am running a 1050Ti with Cuda 9.0 installed.
When attempting to run either "kernel profile - Instruction Execution" or "kernel compute" analysis of my kernel, I get the following messages:

nvprof log: C:\Users\alexa\nvvp_workspace\.metadata\.plugins\com.nvidia.viper\launch\12\nvprof_19020.log
nvprof log: C:\Users\alexa\nvvp_workspace\.metadata\.plugins\com.nvidia.viper\launch\12\nvprof_10800.log
======== Error: CUDA profiling error.
==19020== Error: Internal profiling error 4168:999.

I am not certain what to do with this.

Occasionally when running "kernel compute" I get the basic graph of Function Unit Utilization.

I am running a 1050Ti with Cuda 9.0 installed.

#1
Posted 10/16/2017 02:18 AM   
I have some additional information: another kernel in the same process does not have the crash, but it does not do much computation, only memory movement.
I have some additional information: another kernel in the same process does not have the crash, but it does not do much computation, only memory movement.

#2
Posted 10/16/2017 02:26 PM   
For anyone who comes across this problem, I needed to increase the timeout. To ensure the computer is still usable, the OS (Windows 10) was killing my kernel. I fixed this by: [olist] [.]Running regedit from the search bar.[/.] [.]Navigating to "Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers"[/.] [.]Creating a new registry value called "TdrDelay" and setting it to 30 seconds.[/.] [/olist]
For anyone who comes across this problem, I needed to increase the timeout. To ensure the computer is still usable, the OS (Windows 10) was killing my kernel.

I fixed this by:
  1. Running regedit from the search bar.
  2. Navigating to "Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\GraphicsDrivers"
  3. Creating a new registry value called "TdrDelay" and setting it to 30 seconds.

#3
Posted 10/21/2017 02:40 AM   
Hi, rennietherocket Have your issue resolved now ?
Hi, rennietherocket

Have your issue resolved now ?

#4
Posted 10/23/2017 04:48 AM   
Yes, marked resolved.
Yes, marked resolved.

#5
Posted 10/23/2017 12:03 PM   
I am having the same issue on Ubuntu 16.04. nvprof --version: nvprof: NVIDIA (R) Cuda command line profiler Copyright (c) 2012 - 2017 NVIDIA Corporation Release version 9.0.176 (21) Trying to profile my program gives the following output: /usr/local/cuda/bin/nvprof -f -o EncodingBenchmark.nvprof ./E ncodingBenchmark -d 0 ==3160== NVPROF is profiling process 3160, command: ./EncodingBenchmark -d 0 Using device Quadro P4000 ==3160== Error: Internal profiling error 4168:999. ======== Error: CUDA profiling error.
I am having the same issue on Ubuntu 16.04. nvprof --version:

nvprof: NVIDIA (R) Cuda command line profiler
Copyright (c) 2012 - 2017 NVIDIA Corporation
Release version 9.0.176 (21)

Trying to profile my program gives the following output:

/usr/local/cuda/bin/nvprof -f -o EncodingBenchmark.nvprof ./E ncodingBenchmark -d 0
==3160== NVPROF is profiling process 3160, command: ./EncodingBenchmark -d 0
Using device Quadro P4000
==3160== Error: Internal profiling error 4168:999.
======== Error: CUDA profiling error.

#6
Posted 12/03/2017 03:36 PM   
Just noticed it fails only with Pascal GPUs but works fine with Kepler.
Just noticed it fails only with Pascal GPUs but works fine with Kepler.

#7
Posted 12/03/2017 03:54 PM   
Do you know what the timeout is for both GPUs? That was what resolved my issue.
Do you know what the timeout is for both GPUs? That was what resolved my issue.

#8
Posted 12/03/2017 04:11 PM   
I read your comment and tried several options for the timeout, even 10s, but that did not change anything sadly.
I read your comment and tried several options for the timeout, even 10s, but that did not change anything sadly.

#9
Posted 12/03/2017 04:30 PM   
Could this be some kind of driver / profiler version incompatibility? I am running driver 384.90.
Could this be some kind of driver / profiler version incompatibility? I am running driver 384.90.

#10
Posted 12/03/2017 04:35 PM   
Are you sure you are not overrunning the generous 10sec timeout? Do you have a way to try and reduce the amount of time in the kernel? Does your code run to completion outside of the profiler, if so how long does it take? I am asking this as the error I got was related to the timeout on the GPU, so I am assuming that error code means that the GPU timedout.
Are you sure you are not overrunning the generous 10sec timeout? Do you have a way to try and reduce the amount of time in the kernel? Does your code run to completion outside of the profiler, if so how long does it take?

I am asking this as the error I got was related to the timeout on the GPU, so I am assuming that error code means that the GPU timedout.

#11
Posted 12/03/2017 04:36 PM   
Its possible, I don't know.
Its possible, I don't know.

#12
Posted 12/03/2017 04:37 PM   
Driver 384.98 shows the same behavior. Tried using 384.81, which is bundled with CUDA 9.0 toolkit, same issue. Is there any possibility to enable some extended logging to understand what causes this 4168 error?
Driver 384.98 shows the same behavior. Tried using 384.81, which is bundled with CUDA 9.0 toolkit, same issue. Is there any possibility to enable some extended logging to understand what causes this 4168 error?

#13
Posted 12/03/2017 04:49 PM   
Hi, rbundulis Have you tried other sdk sample ? If others work OK, can you provide us your sample ?
Hi, rbundulis

Have you tried other sdk sample ?

If others work OK, can you provide us your sample ?

#14
Posted 12/04/2017 06:13 AM   
Hi, I get the same error (Profiling error 4168:999). Card: GTX1060 on Windows 10 (driver 388.19) with 9.0/9.1, and on Linux (drivers 387.34, 384.98) with 9.0/9.1. Profiling my application with 8.0 works.
Hi,

I get the same error (Profiling error 4168:999).

Card: GTX1060 on Windows 10 (driver 388.19) with 9.0/9.1, and on Linux (drivers 387.34, 384.98) with 9.0/9.1.

Profiling my application with 8.0 works.

#15
Posted 12/13/2017 09:11 AM   
Scroll To Top

Add Reply