NVIDIA: could not open the device file /dev/nvidia0 (Input/output error).

Linux: CENTOS 5.4
CUDA: 2.3
driver: 190.18…

I have two cards: Geforce8400
295GTX

PCI bus:
02:00.0 PCI bridge: nVidia Corporation PCI express bridge for GTX 295 (rev a3)
03:00.0 PCI bridge: nVidia Corporation PCI express bridge for GTX 295 (rev a3)
03:02.0 PCI bridge: nVidia Corporation PCI express bridge for GTX 295 (rev a3)
04:00.0 3D controller: nVidia Corporation Unknown device 05eb (rev a1)
05:00.0 VGA compatible controller: nVidia Corporation Unknown device 05eb (rev a1)
06:00.0 VGA compatible controller: nVidia Corporation G98 [GeForce 8400 GS] (rev a1)

nvidia-settings show the two cards but:

nvidia-smi:
NVIDIA: could not open the device file /dev/nvidia0 (Input/output error).
Failed to attach gpu
Could not allocate resources!

deviceQuery:
CUDA Device Query (Runtime API) version (CUDART static linking)
NVIDIA: could not open the device file /dev/nvidia0 (Input/output error).
There is no device supporting CUDA.

What can be the problem?

Thanks
Amir

Question should go to linux forum, but…

run the following:

ls -l /dev/nvidia*

On my system with 3 GTX295 I see:

crw-rw-rw- 1 root root 195,   0 Nov 29 09:17 /dev/nvidia0

crw-rw-rw- 1 root root 195,   1 Nov 29 09:17 /dev/nvidia1

crw-rw-rw- 1 root root 195,   2 Nov 29 09:17 /dev/nvidia2

crw-rw-rw- 1 root root 195,   3 Nov 29 09:17 /dev/nvidia3

crw-rw-rw- 1 root root 195,   4 Nov 29 09:17 /dev/nvidia4

crw-rw-rw- 1 root root 195,   5 Nov 29 09:17 /dev/nvidia5

crw-rw-rw- 1 root root 195, 255 Nov 29 09:17 /dev/nvidiactl

if you dont see those files, this might help you:

mknod -m 0666 /dev/nvidiactl c 195 255

mknod -m 0666 /dev/nvidia0 c 195 0

mknod -m 0666 /dev/nvidia1 c 195 1

mknod -m 0666 /dev/nvidia2 c 195 2

mknod -m 0666 /dev/nvidia3 c 195 3

mknod -m 0666 /dev/nvidia4 c 195 4

mknod -m 0666 /dev/nvidia5 c 195 5

As far as I remember if you have X enabled, there should not be such problem.

Take a look here:

http://forums.nvidia.com/index.php?showtopic=52629

amirbaru@boag-lab4-pc release$ ls -l /dev/nvidia*

crw-rw-rw- 1 root root 195, 0 Dec 14 16:05 /dev/nvidia0

crw-rw-rw- 1 root root 195, 1 Dec 14 16:05 /dev/nvidia1

crw-rw-rw- 1 root root 195, 2 Dec 14 16:05 /dev/nvidia2

crw-rw-rw- 1 root root 195, 255 Dec 14 16:05 /dev/nvidiactl

What now?

Thanks

Amir