I am running into a problem on my Nvidia DevBox CUDA 7.5 install on Centos7 where only 3 out of the 4 total GPUs get recognized correctly.
lspci shows 4 cards nvidia correctly and I have blacklisted nouveau drivers. However, nvidia-smi gives error that says insufficient power (I actually plugged it into a more powerful socket and guess its not the issue).
nvidia-smi -L gives the first 3 graphics card correctly and returns the above error on the last graphics card.
X-serv works in both the 352.39 driver CUDA 7.5 came with as well as 352.63 driver (I thought it was a driver issue but it seems not the case).
In my nvidia settings, I can see 3 drivers correctly recognized, but I just cant get the last one to work! As a result I cant run theano…
dmesg gives rmInitadapater error on 0a:00.0 pci slot containing the titan x card.
havnt touched any internal, it was preconfigured nvidia box… I guess I will replug the problematic card onto the pcie slot and see if it resolves the issue…
I have the same issue after installing Ubuntu 16 on my dev box. In my case only half of the GPUs are powered up. How did you fix yours?
sudo nvidia-smi -L
GPU 0: GeForce GTX TITAN X (UUID: GPU-26b3d94a-384c-e532-774e-4385fde429a0)
Unable to determine the device handle for gpu 0000:06:00.0: Unable to communicate with GPU because it is insufficiently powered.
This may be because not all required external power cables are
attached, or the attached cables are not seated properly.
Unable to determine the device handle for gpu 0000:09:00.0: Unable to communicate with GPU because it is insufficiently powered.
This may be because not all required external power cables are
attached, or the attached cables are not seated properly.
GPU 3: GeForce GTX TITAN X (UUID: GPU-c29f1e36-d438-f51a-b78d-5a47e928be72)