libcublas.so.7.0: cannot open shared object file

Hi,

I have a GPU server rented on rapidswitch.com

Intel i7
32GB RAM
480GB SSD
2 graphic cards:

  • Nvidia 750Ti
  • on board (would be IntelA? HD Graphics 4600) (turned off)
    ubuntu server 14.04 x64

I installed cuda-7 and 346 drivers via ssh.

And configured the environment variables in /etc/environment

CUDA_HOME=/usr/local/cuda-7.0 
LD_LIBRARY_PATH=/usr/local/cuda-7.0/lib64 
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/opt/anaconda/bin:/usr/local/cuda-7.0/bin"

I compiled the cuda samples. It seems to pass the test, but BLAS related are failing:

./matrixMul
[Matrix Multiply Using CUDA] - Starting...
GPU Device 0: "GeForce GTX 750 Ti" with compute capability 5.0

MatrixA(320,320), MatrixB(640,320)
Computing result using CUDA Kernel...
done
Performance= 183.40 GFlop/s, Time= 0.715 msec, Size= 131072000 Ops, WorkgroupSize= 1024 threads/block
Checking computed result for correctness: Result = PASS

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
./matrixMulCUBLAS 


./matrixMulCUBLAS: error while loading shared libraries: libcublas.so.7.0: cannot open shared object file: No such file or directory

I found libcublas.so.7.0 and related files in two different directories:

>ls -la /usr/local/cuda-7.0/lib64/libcublas*

lrwxrwxrwx 1 root root       16 Feb 18 12:01 /usr/local/cuda-7.0/lib64/libcublas.so -> libcublas.so.7.0
lrwxrwxrwx 1 root root       19 Feb 18 12:01 /usr/local/cuda-7.0/lib64/libcublas.so.7.0 -> libcublas.so.7.0.28
-rwxr-xr-x 1 root root 31160168 Feb 18 12:01 /usr/local/cuda-7.0/lib64/libcublas.so.7.0.28
-rw-r--r-- 1 root root 26032916 Feb 18 12:38 /usr/local/cuda-7.0/lib64/libcublas_device.a
-rw-r--r-- 1 root root 35269768 Feb 18 12:38 /usr/local/cuda-7.0/lib64/libcublas_static.a


> ls -la /usr/local/cuda/lib64/libcublas*
lrwxrwxrwx 1 root root       16 Feb 18 12:01 /usr/local/cuda/lib64/libcublas.so -> libcublas.so.7.0
lrwxrwxrwx 1 root root       19 Feb 18 12:01 /usr/local/cuda/lib64/libcublas.so.7.0 -> libcublas.so.7.0.28
-rwxr-xr-x 1 root root 31160168 Feb 18 12:01 /usr/local/cuda/lib64/libcublas.so.7.0.28
-rw-r--r-- 1 root root 26032916 Feb 18 12:38 /usr/local/cuda/lib64/libcublas_device.a
-rw-r--r-- 1 root root 35269768 Feb 18 12:38 /usr/local/cuda/lib64/libcublas_static.a

Any help or advice will be welcomed.

Thanks

Your LD_LIBRARY_PATH variable is set incorrectly.
Immediately before running this command:

./matrixMulCUBLAS

run this command:

echo $LD_LIBRARY_PATH
> echo $LD_LIBRARY_PATH

/usr/local/cuda-7.0/lib64


> ls $LD_LIBRARY_PATH
libOpenCL.so         libcufft.so.7.0       libcurand.so.7.0       libnppc.so.7.0     libnvToolsExt.so.1
libcublas.so         libcufft.so.7.0.28    libcurand.so.7.0.28    libnppc.so.7.0.28  libnvToolsExt.so.1.0.0
libcublas.so.7.0     libcufft_static.a     libcurand_static.a     libnppc_static.a   libnvblas.so
libcublas.so.7.0.28  libcufftw.so          libcusolver.so         libnppi.so         libnvblas.so.7.0
libcublas_device.a   libcufftw.so.7.0      libcusolver.so.7.0     libnppi.so.7.0     libnvblas.so.7.0.28
libcublas_static.a   libcufftw.so.7.0.28   libcusolver.so.7.0.28  libnppi.so.7.0.28  libnvrtc-builtins.so
libcudadevrt.a       libcufftw_static.a    libcusolver_static.a   libnppi_static.a   libnvrtc-builtins.so.7.0
libcudart.so         libcuinj64.so         libcusparse.so         libnpps.so         libnvrtc-builtins.so.7.0.28
libcudart.so.7.0     libcuinj64.so.7.0     libcusparse.so.7.0     libnpps.so.7.0     libnvrtc.so
libcudart.so.7.0.28  libcuinj64.so.7.0.28  libcusparse.so.7.0.28  libnpps.so.7.0.28  libnvrtc.so.7.0
libcudart_static.a   libculibos.a          libcusparse_static.a   libnpps_static.a   libnvrtc.so.7.0.27
libcufft.so          libcurand.so          libnppc.so             libnvToolsExt.so   stubs
> echo $LD_LIBRARY_PATH
/usr/local/cuda-7.0/lib64

> ./matrixMulCUBLAS
./matrixMulCUBLAS: error while loading shared libraries: libcublas.so.7.0: cannot open shared object file: No such file or directory

Are you running all this as a root user or an ordinary user?

Are you using sudo for any of these commands?

As normal user.

thanks

I’ve tried with sudo and the problem is the same

You don’t want to use sudo. I was asking that to rule out a possibility.

Did you build these sample codes? That is, did you run a make command that created these?

What is the full path to the matrixMulCUBLAS program you are running?

What is the result of the following command:

ldd matrixMulCUBLAS

Yes, I try to use sudo the less the better.

Yes, I build these samples running a make command

The program is in ~/NVIDIA_CUDA-7.0_Samples/bin/x86_64/linux/release/matrixMulCUBLAS

> ldd matrixMulCUBLAS 
	linux-vdso.so.1 =>  (0x00007fff393af000)
	libcublas.so.7.0 => not found
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc1da8b3000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc1da694000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc1da490000)
	libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc1da18c000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc1d9e85000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc1d9ac0000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fc1daad4000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc1d98aa000)

You’ve pretty much stumped me. I still believe that your LD_LIBRARY_PATH variable is either not correct or not getting picked up properly (the ldd output suggests this), but I can’t see enough of what you’re doing to spot the error, and I can’t think of further questions to ask.

Another possibility is that the libcublas.so.7.0 file is simply the wrong one, e.g. for a different linux distribution or something like that. It seems unlikely to me, however. 32bit vs. 64bit doesn’t seem to be the issue.

An alternative thing you might try is to modify the runtime linker mechanism to incorporate the necessary path to search, without requiring LD_LIBRARY_PATH to be set.

You would do this as root, in a nutshell:

echo -e “/usr/local/cuda-7.0/lib64” > /etc/ld.so.conf.d/cuda.conf
ldconfig

more details:

[url]Where does Ubuntu look for shared libraries? - Ask Ubuntu

Hi txbob,

I’ve read the next: “LD_LIBRARY_PATH cannot be set in $HOME/.profile, /etc/profile, nor /etc/environment files” in How to set the environmental variable LD_LIBRARY_PATH in linux - Stack Overflow.

Thanks a lot!

Solved. Finally I did:

sudo echo “/usr/local/cuda-7.0/lib64” > /etc/ld.so.conf.d/cuda.conf
sudo ldconfig

Thanks a lot txbob!