The GPU FAN runs heavily after the process is done.

The GPU is 1080 TI. After the task process (e.g., one training of CNN) is done, the GPU FAN will run very heavily (also with a large noise), then slow down to a reasonable speed. What’s the problem? Did I miss some configuration for the card? Thanks in advance!

You might want to use nvidia-smi to monitor fan speed along with GPU temperature. The fan(s) will continue to run at high speed as long as the GPU is hot.

Most GPU cooling designs include a sizable chunk of metal (around one pound in some designs). Heavy GPU use will cause these metal parts to heat up over the course of six to seven minutes before leveling off, and they will take one to two minutes to cool down after the computational task has finished.

Hi @njuffa, thanks a lot for your comment.

When the task is running, the GPU FAN runs as usual, and the temperature is 78~82. The GPU FAN seems run at a smooth speed.

But after the task is done (so the GPU load is down to 0%), the GPU FAN suddenly runs at a very high speed (also with a large noise). Then this status will be last for 4~5 seconds. After that, the GPU FAN runs back to usual status at a smooth speed. And the temperature is decreasing to 45~55.

Sounds like a glitch in the fan profile which may be annoying but not something I would worry about from a technical standpoint. The noise is simply a consequence of the high RPM. Most fans used with GPUs get noticeably noisy when RPM increase above 65% to 70% of the maximum speed.

Unless you have a GPU made by NVIDIA itself (that means Tesla or Quadro, for the most part) fan profiles are created by the 3rd party GPU vendors, who each create their own cooling solutions. You might want to try updating to the latest BIOS offered by your GPU vendor, if you are comfortable with such updates. The vendor may also offer additional control software that lets users create custom fan profiles, overriding the vendor-supplied one.

I bought the GPU card from the NVIDIA official store. No cooling kits were adopted.
I’m not sure if I need to buy a 3rd cooling kit for the card.
But I think it is very strange that the FAN speed is suddenly high once after the heavy-computational task is done. So I dount if I missed some configuration or other something.

Thanks a lot for your kind answers!

I did not realize that the GTX 1080 Ti is one of the GPUs sold directly by NVIDIA. In that case:

(1) Check whether NVIDIA offers a newer VBIOS version that you can install
(2) Update the driver package to the latest available for your OS platform

I agree that the brief burst to fan RPMs at the end of heavy computation that you are observing seems like some sort of unintended behavior in the fan control, a bug in other words.

Thanks.

(1) I know how to update the BIOS for the motherboard, but I don’t know how to update the VBIOS. [http://nvidia.custhelp.com/app/answers/detail/a_id/17/~/uefi-%2F-video-bios-download]

(2) I’m using the latest Driver (381.22) and the driver contained in CUDA 8.0 toolkit package doesn’t work for the 1080 TI (as shown in others’ post in the forum).

ubuntu@master2:~$ nvidia-smi
Thu Jun  8 21:24:17 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 381.22                 Driver Version: 381.22                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 0000:01:00.0     Off |                  N/A |
| 45%   76C    P2   156W / 250W |   8565MiB / 11171MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0     26295    C   python                                        8553MiB |
+-----------------------------------------------------------------------------+

After I killed the task (26295), the output of nvidia-smi is:

Thu Jun  8 21:25:50 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 381.22                 Driver Version: 381.22                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 0000:01:00.0     Off |                  N/A |
|  0%   71C    P0    68W / 250W |      0MiB / 11171MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

The GPU FAN is 0%.

GPU fan at 0% strikes me as very unusual, even for a GPU that is idling. This may be due to the power efficient design of the GTX 1080 Ti, but it does look odd to me.

The driver included with CUDA 8.0 cannot work with the GTX 1080 Ti because the GTX 1080 Ti shipped a considerable time after CUDA 8.0 shipped. In general, a driver will be younger than the hardware it supports. If you are using the latest driver package available, you should be fine.

I haven’t updated the VBIOS of an NVIDIA GPU in something like five years. I don’t recall the necessary steps. Check if NVIDIA provides that information on their website, along with the latest VBIOS image.

OK, Thanks man.

GPU fan at zero is possible, in some circumstances, on newer GPUs.

Hi @txbob, thanks for your reply!

We also bought the TITAN X Pascal card (which I think has the same architecture as the GTX 1080 TI).

I found that the GPU FAN on TITAN X Pascal at idle time is about 22~23% with temperature of about 30 (Driver version is an old 369.30). But the GTX 1080 TI is always 0% at idle. So I’m confused.

So they have different fan curves. NVIDIA does not make every product identical to every other product.

And the specific driver can affect fan behavior, as well as the specific GPU, and GPU temperature, and previous activity patterns.

If you were expecting one NVIDIA GPU to behave precisely like a different one in this respect, you should probably just give up on that notion. Then your confusion level may go down.

I didn’t mean that. Generally speaking, I think the GPU FAN should run at a reasonable speed no matter the card is idle or busy. But as suggested in your kind comments, the 0% speed seems OK for the 1080 TI.

I only know this behavior from compute intensive GPU processes that I interrupt with CTRL-C.
Then briefly the GPU fans spin to full throttle.

Thanks for comments!

But in my case, the GPU FAN will be full throttling after either being interrupted with CTRL-C or not (i.e., waiting for the task being done correctly)

I have a fan problem. I have an Asus 1080 Ti Strix.
Most things seem to work fine except the fans.
At idle, the nvidia-smi reports 0 % and I know they are spinning slowly. This problem I can live with.
When I load the card, fans spin up and become rather noisy, also ok. Fan speed is still reported as 0 % when they spin fast. Not good.
The load is stopped and the fans cool the card but even when the card is cool, the fans still spin fast and noisy. This is my worst problem. I have tried waiting them out but they don’t stop until I reboot.

Any ideas?

I’m on Linux Mint 18 with nvidia 382.22

cat /proc/driver/nvidia/gpus/0000:05:00.0/information
Model: GeForce GTX 1080 Ti
IRQ: 50
GPU UUID: GPU-9e9f26fb-5ae3-365d-72ca-9f43e95ee678
Video BIOS: 86.02.39.00.23
Bus Type: PCIe
DMA Size: 47 bits
DMA Mask: 0x7fffffffffff
Bus Location: 0000:05:00.0
Device Minor: 0

Contact the vendor. As far as I am aware these consumer GPU models are heavily customized by the vendor (i.e. they are not anywhere close to the NVIDIA reference design), and therefore are very likely to use the vendor’s own fan control. You might want to check the GPU vendor’s forums (https://vip.asus.com/forum/), or see whether the latest BIOS they provide offers improvements. If you are lucky they may even offer tools for customizing the fan control curve.

Tried the same thing in windows with vendor recommended drivers. Same issue. Vendor has replied and recommends RMA as they themselves think the fan control is malfunctioning.

Hi, I suggest to setup one liquid cooler for the card. The following is my output: 0 is the original card without liquid cooler bought from NVIDIA, and 1 is bought from http://www.corsair.com/en-us/hydro-gfx-gtx-1080-ti-liquid-cooled-graphics-card. So I suggest NVIVIA should also provide their professional cooling product attached with the card device.

Thu Jul 20 11:45:24 2017
±----------------------------------------------------------------------------+
| NVIDIA-SMI 381.22 Driver Version: 381.22 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108… Off | 0000:01:00.0 Off | N/A |
| 50% 83C P2 127W / 250W | 8589MiB / 11171MiB | 91% Default |
±------------------------------±---------------------±---------------------+
| 1 GeForce GTX 108… Off | 0000:02:00.0 Off | N/A |
| 46% 55C P2 133W / 250W | 7565MiB / 11172MiB | 86% Default |
±------------------------------±---------------------±---------------------+

±----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 19793 C python 8577MiB |
| 1 19793 C python 7553MiB |
±----------------------------------------------------------------------------+