libcublas.so.7.0: cannot open shared object file
Hi, I have a GPU server rented on rapidswitch.com Intel i7 32GB RAM 480GB SSD 2 graphic cards: - Nvidia 750Ti - on board (would be IntelA? HD Graphics 4600) (turned off) ubuntu server 14.04 x64 I installed cuda-7 and 346 drivers via ssh. And configured the environment variables in /etc/environment [code] CUDA_HOME=/usr/local/cuda-7.0 LD_LIBRARY_PATH=/usr/local/cuda-7.0/lib64 PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/opt/anaconda/bin:/usr/local/cuda-7.0/bin" [/code] I compiled the cuda samples. It seems to pass the test, but BLAS related are failing: [code] ./matrixMul [Matrix Multiply Using CUDA] - Starting... GPU Device 0: "GeForce GTX 750 Ti" with compute capability 5.0 MatrixA(320,320), MatrixB(640,320) Computing result using CUDA Kernel... done Performance= 183.40 GFlop/s, Time= 0.715 msec, Size= 131072000 Ops, WorkgroupSize= 1024 threads/block Checking computed result for correctness: Result = PASS NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled. [/code] [code] ./matrixMulCUBLAS ./matrixMulCUBLAS: error while loading shared libraries: libcublas.so.7.0: cannot open shared object file: No such file or directory [/code] I found libcublas.so.7.0 and related files in two different directories: [code] >ls -la /usr/local/cuda-7.0/lib64/libcublas* lrwxrwxrwx 1 root root 16 Feb 18 12:01 /usr/local/cuda-7.0/lib64/libcublas.so -> libcublas.so.7.0 lrwxrwxrwx 1 root root 19 Feb 18 12:01 /usr/local/cuda-7.0/lib64/libcublas.so.7.0 -> libcublas.so.7.0.28 -rwxr-xr-x 1 root root 31160168 Feb 18 12:01 /usr/local/cuda-7.0/lib64/libcublas.so.7.0.28 -rw-r--r-- 1 root root 26032916 Feb 18 12:38 /usr/local/cuda-7.0/lib64/libcublas_device.a -rw-r--r-- 1 root root 35269768 Feb 18 12:38 /usr/local/cuda-7.0/lib64/libcublas_static.a > ls -la /usr/local/cuda/lib64/libcublas* lrwxrwxrwx 1 root root 16 Feb 18 12:01 /usr/local/cuda/lib64/libcublas.so -> libcublas.so.7.0 lrwxrwxrwx 1 root root 19 Feb 18 12:01 /usr/local/cuda/lib64/libcublas.so.7.0 -> libcublas.so.7.0.28 -rwxr-xr-x 1 root root 31160168 Feb 18 12:01 /usr/local/cuda/lib64/libcublas.so.7.0.28 -rw-r--r-- 1 root root 26032916 Feb 18 12:38 /usr/local/cuda/lib64/libcublas_device.a -rw-r--r-- 1 root root 35269768 Feb 18 12:38 /usr/local/cuda/lib64/libcublas_static.a [/code] Any help or advice will be welcomed. Thanks
Hi,

I have a GPU server rented on rapidswitch.com

Intel i7
32GB RAM
480GB SSD
2 graphic cards:
- Nvidia 750Ti
- on board (would be IntelA? HD Graphics 4600) (turned off)
ubuntu server 14.04 x64

I installed cuda-7 and 346 drivers via ssh.

And configured the environment variables in /etc/environment
CUDA_HOME=/usr/local/cuda-7.0 
LD_LIBRARY_PATH=/usr/local/cuda-7.0/lib64
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/opt/anaconda/bin:/usr/local/cuda-7.0/bin"


I compiled the cuda samples. It seems to pass the test, but BLAS related are failing:

./matrixMul
[Matrix Multiply Using CUDA] - Starting...
GPU Device 0: "GeForce GTX 750 Ti" with compute capability 5.0

MatrixA(320,320), MatrixB(640,320)
Computing result using CUDA Kernel...
done
Performance= 183.40 GFlop/s, Time= 0.715 msec, Size= 131072000 Ops, WorkgroupSize= 1024 threads/block
Checking computed result for correctness: Result = PASS

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.



./matrixMulCUBLAS 


./matrixMulCUBLAS: error while loading shared libraries: libcublas.so.7.0: cannot open shared object file: No such file or directory



I found libcublas.so.7.0 and related files in two different directories:

>ls -la /usr/local/cuda-7.0/lib64/libcublas*

lrwxrwxrwx 1 root root 16 Feb 18 12:01 /usr/local/cuda-7.0/lib64/libcublas.so -> libcublas.so.7.0
lrwxrwxrwx 1 root root 19 Feb 18 12:01 /usr/local/cuda-7.0/lib64/libcublas.so.7.0 -> libcublas.so.7.0.28
-rwxr-xr-x 1 root root 31160168 Feb 18 12:01 /usr/local/cuda-7.0/lib64/libcublas.so.7.0.28
-rw-r--r-- 1 root root 26032916 Feb 18 12:38 /usr/local/cuda-7.0/lib64/libcublas_device.a
-rw-r--r-- 1 root root 35269768 Feb 18 12:38 /usr/local/cuda-7.0/lib64/libcublas_static.a


> ls -la /usr/local/cuda/lib64/libcublas*
lrwxrwxrwx 1 root root 16 Feb 18 12:01 /usr/local/cuda/lib64/libcublas.so -> libcublas.so.7.0
lrwxrwxrwx 1 root root 19 Feb 18 12:01 /usr/local/cuda/lib64/libcublas.so.7.0 -> libcublas.so.7.0.28
-rwxr-xr-x 1 root root 31160168 Feb 18 12:01 /usr/local/cuda/lib64/libcublas.so.7.0.28
-rw-r--r-- 1 root root 26032916 Feb 18 12:38 /usr/local/cuda/lib64/libcublas_device.a
-rw-r--r-- 1 root root 35269768 Feb 18 12:38 /usr/local/cuda/lib64/libcublas_static.a


Any help or advice will be welcomed.

Thanks

#1
Posted 06/16/2015 05:15 PM   
Your LD_LIBRARY_PATH variable is set incorrectly. Immediately before running this command: [code]./matrixMulCUBLAS [/code] run this command: [code]echo $LD_LIBRARY_PATH[/code]
Your LD_LIBRARY_PATH variable is set incorrectly.
Immediately before running this command:

./matrixMulCUBLAS



run this command:

echo $LD_LIBRARY_PATH

#2
Posted 06/16/2015 05:25 PM   
[code] > echo $LD_LIBRARY_PATH /usr/local/cuda-7.0/lib64 > ls $LD_LIBRARY_PATH libOpenCL.so libcufft.so.7.0 libcurand.so.7.0 libnppc.so.7.0 libnvToolsExt.so.1 libcublas.so libcufft.so.7.0.28 libcurand.so.7.0.28 libnppc.so.7.0.28 libnvToolsExt.so.1.0.0 libcublas.so.7.0 libcufft_static.a libcurand_static.a libnppc_static.a libnvblas.so libcublas.so.7.0.28 libcufftw.so libcusolver.so libnppi.so libnvblas.so.7.0 libcublas_device.a libcufftw.so.7.0 libcusolver.so.7.0 libnppi.so.7.0 libnvblas.so.7.0.28 libcublas_static.a libcufftw.so.7.0.28 libcusolver.so.7.0.28 libnppi.so.7.0.28 libnvrtc-builtins.so libcudadevrt.a libcufftw_static.a libcusolver_static.a libnppi_static.a libnvrtc-builtins.so.7.0 libcudart.so libcuinj64.so libcusparse.so libnpps.so libnvrtc-builtins.so.7.0.28 libcudart.so.7.0 libcuinj64.so.7.0 libcusparse.so.7.0 libnpps.so.7.0 libnvrtc.so libcudart.so.7.0.28 libcuinj64.so.7.0.28 libcusparse.so.7.0.28 libnpps.so.7.0.28 libnvrtc.so.7.0 libcudart_static.a libculibos.a libcusparse_static.a libnpps_static.a libnvrtc.so.7.0.27 libcufft.so libcurand.so libnppc.so libnvToolsExt.so stubs [/code]
> echo $LD_LIBRARY_PATH

/usr/local/cuda-7.0/lib64


> ls $LD_LIBRARY_PATH
libOpenCL.so libcufft.so.7.0 libcurand.so.7.0 libnppc.so.7.0 libnvToolsExt.so.1
libcublas.so libcufft.so.7.0.28 libcurand.so.7.0.28 libnppc.so.7.0.28 libnvToolsExt.so.1.0.0
libcublas.so.7.0 libcufft_static.a libcurand_static.a libnppc_static.a libnvblas.so
libcublas.so.7.0.28 libcufftw.so libcusolver.so libnppi.so libnvblas.so.7.0
libcublas_device.a libcufftw.so.7.0 libcusolver.so.7.0 libnppi.so.7.0 libnvblas.so.7.0.28
libcublas_static.a libcufftw.so.7.0.28 libcusolver.so.7.0.28 libnppi.so.7.0.28 libnvrtc-builtins.so
libcudadevrt.a libcufftw_static.a libcusolver_static.a libnppi_static.a libnvrtc-builtins.so.7.0
libcudart.so libcuinj64.so libcusparse.so libnpps.so libnvrtc-builtins.so.7.0.28
libcudart.so.7.0 libcuinj64.so.7.0 libcusparse.so.7.0 libnpps.so.7.0 libnvrtc.so
libcudart.so.7.0.28 libcuinj64.so.7.0.28 libcusparse.so.7.0.28 libnpps.so.7.0.28 libnvrtc.so.7.0
libcudart_static.a libculibos.a libcusparse_static.a libnpps_static.a libnvrtc.so.7.0.27
libcufft.so libcurand.so libnppc.so libnvToolsExt.so stubs

#3
Posted 06/16/2015 05:34 PM   
[code] > echo $LD_LIBRARY_PATH /usr/local/cuda-7.0/lib64 > ./matrixMulCUBLAS ./matrixMulCUBLAS: error while loading shared libraries: libcublas.so.7.0: cannot open shared object file: No such file or directory [/code]
> echo $LD_LIBRARY_PATH
/usr/local/cuda-7.0/lib64

> ./matrixMulCUBLAS
./matrixMulCUBLAS: error while loading shared libraries: libcublas.so.7.0: cannot open shared object file: No such file or directory

#4
Posted 06/16/2015 05:37 PM   
Are you running all this as a root user or an ordinary user? Are you using sudo for any of these commands?
Are you running all this as a root user or an ordinary user?

Are you using sudo for any of these commands?

#5
Posted 06/16/2015 05:50 PM   
As normal user. thanks
As normal user.

thanks

#6
Posted 06/16/2015 05:59 PM   
I've tried with sudo and the problem is the same
I've tried with sudo and the problem is the same

#7
Posted 06/16/2015 06:02 PM   
You don't want to use sudo. I was asking that to rule out a possibility. Did you build these sample codes? That is, did you run a make command that created these? What is the full path to the matrixMulCUBLAS program you are running? What is the result of the following command: ldd matrixMulCUBLAS
You don't want to use sudo. I was asking that to rule out a possibility.

Did you build these sample codes? That is, did you run a make command that created these?

What is the full path to the matrixMulCUBLAS program you are running?

What is the result of the following command:

ldd matrixMulCUBLAS

#8
Posted 06/16/2015 06:09 PM   
Yes, I try to use sudo the less the better. Yes, I build these samples running a make command The program is in ~/NVIDIA_CUDA-7.0_Samples/bin/x86_64/linux/release/matrixMulCUBLAS [code] > ldd matrixMulCUBLAS linux-vdso.so.1 => (0x00007fff393af000) libcublas.so.7.0 => not found librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc1da8b3000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc1da694000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc1da490000) libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc1da18c000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc1d9e85000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc1d9ac0000) /lib64/ld-linux-x86-64.so.2 (0x00007fc1daad4000) libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc1d98aa000) [/code]
Yes, I try to use sudo the less the better.

Yes, I build these samples running a make command

The program is in ~/NVIDIA_CUDA-7.0_Samples/bin/x86_64/linux/release/matrixMulCUBLAS

> ldd matrixMulCUBLAS 
linux-vdso.so.1 => (0x00007fff393af000)
libcublas.so.7.0 => not found
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fc1da8b3000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fc1da694000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fc1da490000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fc1da18c000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fc1d9e85000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fc1d9ac0000)
/lib64/ld-linux-x86-64.so.2 (0x00007fc1daad4000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc1d98aa000)

#9
Posted 06/16/2015 07:06 PM   
You've pretty much stumped me. I still believe that your LD_LIBRARY_PATH variable is either not correct or not getting picked up properly (the ldd output suggests this), but I can't see enough of what you're doing to spot the error, and I can't think of further questions to ask. Another possibility is that the libcublas.so.7.0 file is simply the wrong one, e.g. for a different linux distribution or something like that. It seems unlikely to me, however. 32bit vs. 64bit doesn't seem to be the issue. An alternative thing you might try is to modify the runtime linker mechanism to incorporate the necessary path to search, without requiring LD_LIBRARY_PATH to be set. You would do this as root, in a nutshell: echo -e "/usr/local/cuda-7.0/lib64" > /etc/ld.so.conf.d/cuda.conf ldconfig more details: [url]http://askubuntu.com/questions/350068/where-does-ubuntu-look-for-shared-libraries[/url]
You've pretty much stumped me. I still believe that your LD_LIBRARY_PATH variable is either not correct or not getting picked up properly (the ldd output suggests this), but I can't see enough of what you're doing to spot the error, and I can't think of further questions to ask.

Another possibility is that the libcublas.so.7.0 file is simply the wrong one, e.g. for a different linux distribution or something like that. It seems unlikely to me, however. 32bit vs. 64bit doesn't seem to be the issue.

An alternative thing you might try is to modify the runtime linker mechanism to incorporate the necessary path to search, without requiring LD_LIBRARY_PATH to be set.

You would do this as root, in a nutshell:

echo -e "/usr/local/cuda-7.0/lib64" > /etc/ld.so.conf.d/cuda.conf
ldconfig

more details:

http://askubuntu.com/questions/350068/where-does-ubuntu-look-for-shared-libraries

#10
Posted 06/16/2015 08:00 PM   
Hi txbob, I've read the next: "LD_LIBRARY_PATH cannot be set in $HOME/.profile, /etc/profile, nor /etc/environment files" in http://stackoverflow.com/questions/13428910/how-to-set-the-environmental-variable-ld-library-path-in-linux. Thanks a lot!
Hi txbob,

I've read the next: "LD_LIBRARY_PATH cannot be set in $HOME/.profile, /etc/profile, nor /etc/environment files" in http://stackoverflow.com/questions/13428910/how-to-set-the-environmental-variable-ld-library-path-in-linux.

Thanks a lot!

#11
Posted 06/16/2015 09:35 PM   
Solved. Finally I did: sudo echo "/usr/local/cuda-7.0/lib64" > /etc/ld.so.conf.d/cuda.conf sudo ldconfig Thanks a lot txbob!
Solved. Finally I did:

sudo echo "/usr/local/cuda-7.0/lib64" > /etc/ld.so.conf.d/cuda.conf
sudo ldconfig

Thanks a lot txbob!

#12
Posted 06/17/2015 05:27 PM   
Scroll To Top

Add Reply