GPU Performance

GeoffBil · September 18, 2015, 2:00pm

Hi All,

Just purchased the Telsa K80 GPU. Have the driver installed, but need to be able to see some sort of performance metrics on the card. I’ve tried other GPU performance apps to see what kind of processing the card is doing, but it doesn’t seem to be supported for the K80.

Need some sort of app/monitor to show how hard the card is working (if at all).

Can someone please help, it was a super expensive card and need to know it’s worth.

-Geoff.

Robert_Crovella · September 18, 2015, 2:12pm

nvidia-smi can be used to monitor GPU activity and it is installed with the driver.

Use nvidia-smi --help to learn about it or use the man page.

GeoffBil · September 18, 2015, 3:42pm

Thanks for the response txbob.

How do I get the NVidia-smi? I tried from the command prompt and received “is not recognized as an internal or external command…”

According to my device manager the Telsa K80’s drivers installed correctly, but there is no additional software or app.

-Geoff.

Robert_Crovella · September 18, 2015, 3:43pm

Are you on windows?

njuffa · September 18, 2015, 3:50pm

nvidia-smi may not be in your PATH, but it should be there if CUDA is installed. If on Windows, check the following location (or just search with dir):

Program Files/NVIDIA Corporation/NVSMI/nvidia-smi.exe

GeoffBil · September 18, 2015, 4:40pm

Thank you txbob and njuffa! It was in the listed directory. Now to figure out how to use the K80!

GeoffBil · September 21, 2015, 1:59pm

Hi guys,

I have installed my Telsa K80’s with the driver listed on NVIDIA site, but looks like that’s all it installed. I have used the NVidia-smi.exe and it shows the GPU sitting at 0% when my app is running. Is there no Display options/NVIDIA Control Panel, etc? Something where I right-click on the desktop and should see NVIDIA properties?

Is there something that didn’t install properly? Something I have yet to install that’s compatible with the Telsa K80?

-Geoff.

njuffa · September 21, 2015, 3:17pm

The K80 is a pure compute device, it does not provide any graphics output. So there cannot be any display options. If your app uses CUDA or OpenCL, you should see some load. Is the application configured to use either of those? Are you exercising app functionality that is documented to be GPU accelerated? Not all of the functions of an app may be accelerated by a GPU. What is the application you are running?

What kind of system did you install the K80 in? The K80 is a fan-less, passively-cooled GPU that requires the massive airflow provided by the fans of a server enclosure to force air through its cooling fins. If you install it in an ordinary PC case the K80 will overheat and switch itself off to prevent permanent damage to the card.

dyilmaz · September 23, 2015, 3:07pm

We have a GeForce GTX Titan black.

When it’s idle (0 % utilization) it stays P0 performance level.

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| No running processes found |
±----------------------------------------------------------------------------+

But if we started some gromacs jobs, then the card processing the the jobs on P2 performance level.

We could not understand why performance level of the graphics card drops from P0 to P2.
It’s working in a 4U HP server and server supplies enough power and there is no cooling problem.

Thank you in advance for your reply.

Robert_Crovella · September 24, 2015, 1:03am

run the command with:

nvidia-smi -a

and check the “Clocks Throttle Reasons” line to see if any reasons are reported (although I think there won’t be).

gromacs is not putting the card under much load (this is evident by the fact that the power only goes from 80W to 90W under load, and the GPU temp is the same at 46C) so it’s possible that the clock management system is alternating between P0 and P2 states as the gromacs load dynamically varies. It may be difficult to monitor this using nvidia-smi.

When nvidia-smi is run on an “idle” GPU in a single GPU config, it will pull that GPU (briefly) out of it’s idle state and up to P0 in order to service the request from nvidia-smi - this will be a very brief occurrence, and generally the GPU will go right back to a low power state such as P8 or P12.

However, when the GPU is under load (or in persistence mode) the nvidia-smi command is not “waking it up” or altering its state, so then what you are observing is the actual state under load. A GPU power state can vary even under load, if the load is dynamically going from ~0 to some non zero value, such as when a sequence of kernels are launched with gaps in between.

dyilmaz · September 28, 2015, 9:00am

Dear txbob,

Thanks you very much for the detailed answer. Today I realized something weird with the Nvidia Performance Levels:

If I looked that parameter with nvidia-smi then I see my card is on P2

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 49487 G /usr/bin/X 61MiB |
| 0 50022 G /usr/bin/nvidia-settings 2MiB |
±----------------------------------------------------------------------------+

And If I looked at the same parameter on the GUI then I see it is on P2 level in PowerMixer.

http://tinypic.com/view.php?pic=2prcv11&s=8 (This is the screenshoot)

There is no any conflict so far but the P0 performance level is described in nvidia-smi documentation as the maximum performance level. If we looked at the PowerMixer again P0 shown as a minimum performance level.

Maybe the reason of the working of my graphics card on P2 level could be PCIe 2. The graphics card needs PCIe 3 slot for the maximum performance.

Best,
Dogan

grovi1987 · March 2, 2019, 12:27pm

Hello, i am getting the same error. and i am on windows 10

sundara1101 · March 5, 2019, 10:27am

With quadro p5000 and using nvpipe library cant run more than 20 simultaneous encoding session in a single system when trying to run 21st encoding session nvpipe throws an error saying Out of memory. But here https://developer.nvidia.com/video-encode-decode-gpu-support-matrix it is mentioned for p5000 card there is no restriction.