Ok, so after playing around a bit, by chance I found a way to get all 4 GTX980 to work:
sudo nvidia-smi -L
GPU 0: GeForce GTX 980 (UUID: GPU-686ddb2b-bcac-a199-3918-cf5be21844cf)
GPU 1: GeForce GTX 980 (UUID: GPU-608bd7f7-83eb-95b3-f5f4-fd020d7897b9)
GPU 2: GeForce GTX 980 (UUID: GPU-546d367c-3d98-692e-ccc4-b99470c09834)
GPU 3: GeForce GTX 980 (UUID: GPU-b8bb1f03-34d8-f53b-5e72-d1fec15a4d5f)
Actually the key seems to be the initialization of both kernel modules:
lsmod | grep nvidia
nvidia_uvm 45407 0
nvidia 11057029 1 nvidia_uvm
drm 303102 1 nvidia
Note that the nvidia_uvm is not loaded at startup per default. After doing a ‘modprobe nvidia_uvm’ followed by a ‘deviceQuery’ from the cuda examples, all the cards appeared. I did some number crunching and checked with the nvidia-settings in X if all the GPUs are accessible. They are…
./deviceQuery
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 4 CUDA Capable device(s)
Device 0: "GeForce GTX 980"
CUDA Driver Version / Runtime Version 6.5 / 6.5
CUDA Capability Major/Minor version number: 5.2
Total amount of global memory: 4095 MBytes (4294246400 bytes)
(16) Multiprocessors, (128) CUDA Cores/MP: 2048 CUDA Cores
GPU Clock rate: 1216 MHz (1.22 GHz)
Memory Clock rate: 3505 Mhz
Memory Bus Width: 256-bit
L2 Cache Size: 2097152 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Bus ID / PCI location ID: 2 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
Device 1: "GeForce GTX 980"
CUDA Driver Version / Runtime Version 6.5 / 6.5
CUDA Capability Major/Minor version number: 5.2
Total amount of global memory: 4096 MBytes (4294770688 bytes)
(16) Multiprocessors, (128) CUDA Cores/MP: 2048 CUDA Cores
GPU Clock rate: 1216 MHz (1.22 GHz)
Memory Clock rate: 3505 Mhz
Memory Bus Width: 256-bit
L2 Cache Size: 2097152 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Bus ID / PCI location ID: 3 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
Device 2: "GeForce GTX 980"
CUDA Driver Version / Runtime Version 6.5 / 6.5
CUDA Capability Major/Minor version number: 5.2
Total amount of global memory: 4096 MBytes (4294770688 bytes)
(16) Multiprocessors, (128) CUDA Cores/MP: 2048 CUDA Cores
GPU Clock rate: 1216 MHz (1.22 GHz)
Memory Clock rate: 3505 Mhz
Memory Bus Width: 256-bit
L2 Cache Size: 2097152 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Bus ID / PCI location ID: 131 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
Device 3: "GeForce GTX 980"
CUDA Driver Version / Runtime Version 6.5 / 6.5
CUDA Capability Major/Minor version number: 5.2
Total amount of global memory: 4096 MBytes (4294770688 bytes)
(16) Multiprocessors, (128) CUDA Cores/MP: 2048 CUDA Cores
GPU Clock rate: 1216 MHz (1.22 GHz)
Memory Clock rate: 3505 Mhz
Memory Bus Width: 256-bit
L2 Cache Size: 2097152 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
Maximum Layered 1D Texture Size, (num) layers 1D=(16384), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(16384, 16384), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Device PCI Bus ID / PCI location ID: 132 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
> Peer access from GeForce GTX 980 (GPU0) -> GeForce GTX 980 (GPU1) : Yes
> Peer access from GeForce GTX 980 (GPU0) -> GeForce GTX 980 (GPU2) : No
> Peer access from GeForce GTX 980 (GPU0) -> GeForce GTX 980 (GPU3) : No
> Peer access from GeForce GTX 980 (GPU1) -> GeForce GTX 980 (GPU1) : No
> Peer access from GeForce GTX 980 (GPU1) -> GeForce GTX 980 (GPU2) : No
> Peer access from GeForce GTX 980 (GPU1) -> GeForce GTX 980 (GPU3) : No
> Peer access from GeForce GTX 980 (GPU2) -> GeForce GTX 980 (GPU1) : No
> Peer access from GeForce GTX 980 (GPU2) -> GeForce GTX 980 (GPU2) : No
> Peer access from GeForce GTX 980 (GPU2) -> GeForce GTX 980 (GPU3) : Yes
> Peer access from GeForce GTX 980 (GPU1) -> GeForce GTX 980 (GPU0) : Yes
> Peer access from GeForce GTX 980 (GPU1) -> GeForce GTX 980 (GPU1) : No
> Peer access from GeForce GTX 980 (GPU1) -> GeForce GTX 980 (GPU2) : No
> Peer access from GeForce GTX 980 (GPU2) -> GeForce GTX 980 (GPU0) : No
> Peer access from GeForce GTX 980 (GPU2) -> GeForce GTX 980 (GPU1) : No
> Peer access from GeForce GTX 980 (GPU2) -> GeForce GTX 980 (GPU2) : No
> Peer access from GeForce GTX 980 (GPU3) -> GeForce GTX 980 (GPU0) : No
> Peer access from GeForce GTX 980 (GPU3) -> GeForce GTX 980 (GPU1) : No
> Peer access from GeForce GTX 980 (GPU3) -> GeForce GTX 980 (GPU2) : Yes
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 6.5, CUDA Runtime Version = 6.5, NumDevs = 4, Device0 = GeForce GTX 980, Device1 = GeForce GTX 980, Device2 = GeForce GTX 980, Device3 = GeForce GTX 980
Result = PASS