Linking frustration -lcuda fails
This one should be pretty easy, since I know I am missing something simple.

I'm linking an NVCC generated .o file with another plain .o file from gcc.
My problem is that the final link can't find the cuda libraries. Which is crazy since I've checked about 12 times that they're there.
64 bit Ubuntu 9.04 with CUDA 2.3, driver 190.18.

nvcc -arch sm_13 -c -I/usr/local/cuda/include ${SRC}/kernel.cu
g++ -c main.cpp
g++ main.o kernel.o -L/usr/lib64 -L/usr/local/cuda/lib64 -lcuda

my $LD_LIBRARY_PATH has /usr/local/cuda/lib64 in it (as well as /lib, which is symlinked to /usr/lib64)

If I run the three build commands above, the link fails (with step #3) with
/usr/bin/ld: cannot find -lcuda

If I don't put -lcuda, I get about 40 errors about can't finding CUDA functions like:
tmpxft_00003c34_00000000-10_kernel.ii:(.text+0xdbbd): undefined reference to `__cudaRegisterFatBinary'

-lcudart also fails at link (not found).

Over and over I keep checking that inside /usr/lib64 there's a libcuda.so , and there is. It's a symlink (set up by the toolkit installer) to libcuda.so.1 and libcuda.so.1 is a symlink to the hard file libcuda.so.190.18


The SDK examples are installed and I can build them successfully. The complexity of the SDK makefile makes it hard to boil down to my app.

A single .cu file (with its own main() ) will work with a single nvcc invocation (and not need -lcuda)
nvcc -arch sm_11 simple.cu -o simple


What am I missing? This has got to be easy but I'm still lost in head-banging frustration.
Thanks very very much for any rescue!
This one should be pretty easy, since I know I am missing something simple.



I'm linking an NVCC generated .o file with another plain .o file from gcc.

My problem is that the final link can't find the cuda libraries. Which is crazy since I've checked about 12 times that they're there.

64 bit Ubuntu 9.04 with CUDA 2.3, driver 190.18.



nvcc -arch sm_13 -c -I/usr/local/cuda/include ${SRC}/kernel.cu

g++ -c main.cpp

g++ main.o kernel.o -L/usr/lib64 -L/usr/local/cuda/lib64 -lcuda



my $LD_LIBRARY_PATH has /usr/local/cuda/lib64 in it (as well as /lib, which is symlinked to /usr/lib64)



If I run the three build commands above, the link fails (with step #3) with

/usr/bin/ld: cannot find -lcuda



If I don't put -lcuda, I get about 40 errors about can't finding CUDA functions like:

tmpxft_00003c34_00000000-10_kernel.ii:(.text+0xdbbd): undefined reference to `__cudaRegisterFatBinary'



-lcudart also fails at link (not found).



Over and over I keep checking that inside /usr/lib64 there's a libcuda.so , and there is. It's a symlink (set up by the toolkit installer) to libcuda.so.1 and libcuda.so.1 is a symlink to the hard file libcuda.so.190.18





The SDK examples are installed and I can build them successfully. The complexity of the SDK makefile makes it hard to boil down to my app.



A single .cu file (with its own main() ) will work with a single nvcc invocation (and not need -lcuda)

nvcc -arch sm_11 simple.cu -o simple





What am I missing? This has got to be easy but I'm still lost in head-banging frustration.

Thanks very very much for any rescue!

#1
Posted 10/11/2009 07:09 AM   
From the looks of the missing symbols, you are using the runtime api, so you should be linking against libcudart rather than libcuda, which means providing -lcudart and -L<path to libcudart> to the linker. It really is that simple. Something like this (this example also links againts cublas and the SDK cutil library)
[code]avid@cuda:~/build/CUDA_CG$ /opt/cuda/bin/nvcc -c -arch sm_13 -I$HOME/NVIDIA_GPU_Computing_SDK/C/common/inc -I/opt/cuda/include fQCG.cu -o fQCG.o
avid@cuda:~/build/CUDA_CG$ /opt/cuda/bin/nvcc -c -arch sm_13 -I$HOME/NVIDIA_GPU_Computing_SDK/C/common/inc -I/opt/cuda/include dQCG.cu -o dQCG.o
avid@cuda:~/build/CUDA_CG$ gcc -c -I$HOME/NVIDIA_GPU_Computing_SDK/C/common/inc -I/opt/cuda/include main.c -o main.o
avid@cuda:~/build/CUDA_CG$ g++ -L/opt/cuda/lib64 -L$HOME/NVIDIA_GPU_Computing_SDK/C/lib -o cuda_cg dQCG.o fQCG.o main.o -lcublas -lcudart -lcutil
avid@cuda:~/build/CUDA_CG$ export LD_LIBRARY_PATH=/opt/cuda/lib64
avid@cuda:~/build/CUDA_CG$ LD_LIBRARY_PATH=/opt/cuda/lib64 ldd ./cuda_cg
linux-vdso.so.1 => (0x00007fffb01fe000)
libcublas.so.2 => /opt/cuda/lib64/libcublas.so.2 (0x00007f8fa6e9c000)
libcudart.so.2 => /opt/cuda/lib64/libcudart.so.2 (0x00007f8fa6c5c000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f8fa694f000)
libm.so.6 => /lib/libm.so.6 (0x00007f8fa66ca000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f8fa64b2000)
libc.so.6 => /lib/libc.so.6 (0x00007f8fa6140000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00007f8fa5f24000)
libdl.so.2 => /lib/libdl.so.2 (0x00007f8fa5d20000)
librt.so.1 => /lib/librt.so.1 (0x00007f8fa5b18000)
/lib64/ld-linux-x86-64.so.2 (0x00007f8fa7f86000)[/code]

You will definitely find it easier in the long run to use a makefile, but that is how it is done.
From the looks of the missing symbols, you are using the runtime api, so you should be linking against libcudart rather than libcuda, which means providing -lcudart and -L<path to libcudart> to the linker. It really is that simple. Something like this (this example also links againts cublas and the SDK cutil library)

avid@cuda:~/build/CUDA_CG$ /opt/cuda/bin/nvcc -c -arch sm_13 -I$HOME/NVIDIA_GPU_Computing_SDK/C/common/inc -I/opt/cuda/include fQCG.cu -o fQCG.o

avid@cuda:~/build/CUDA_CG$ /opt/cuda/bin/nvcc -c -arch sm_13 -I$HOME/NVIDIA_GPU_Computing_SDK/C/common/inc -I/opt/cuda/include dQCG.cu -o dQCG.o

avid@cuda:~/build/CUDA_CG$ gcc -c -I$HOME/NVIDIA_GPU_Computing_SDK/C/common/inc -I/opt/cuda/include main.c -o main.o

avid@cuda:~/build/CUDA_CG$ g++ -L/opt/cuda/lib64 -L$HOME/NVIDIA_GPU_Computing_SDK/C/lib -o cuda_cg dQCG.o fQCG.o main.o -lcublas -lcudart -lcutil

avid@cuda:~/build/CUDA_CG$ export LD_LIBRARY_PATH=/opt/cuda/lib64

avid@cuda:~/build/CUDA_CG$ LD_LIBRARY_PATH=/opt/cuda/lib64 ldd ./cuda_cg

linux-vdso.so.1 => (0x00007fffb01fe000)

libcublas.so.2 => /opt/cuda/lib64/libcublas.so.2 (0x00007f8fa6e9c000)

libcudart.so.2 => /opt/cuda/lib64/libcudart.so.2 (0x00007f8fa6c5c000)

libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007f8fa694f000)

libm.so.6 => /lib/libm.so.6 (0x00007f8fa66ca000)

libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00007f8fa64b2000)

libc.so.6 => /lib/libc.so.6 (0x00007f8fa6140000)

libpthread.so.0 => /lib/libpthread.so.0 (0x00007f8fa5f24000)

libdl.so.2 => /lib/libdl.so.2 (0x00007f8fa5d20000)

librt.so.1 => /lib/librt.so.1 (0x00007f8fa5b18000)

/lib64/ld-linux-x86-64.so.2 (0x00007f8fa7f86000)




You will definitely find it easier in the long run to use a makefile, but that is how it is done.

#2
Posted 10/11/2009 08:12 AM   
avid, thanks for the fast reply!

Unfortunately, it's not an issue of using -lcuda or -lcudart. They both fail.
libcuda.so lives in /usr/lib64 and libcudart.so lives in /usr/local/cuda/lib64.
Both of those paths are in my LD_LIBRARY_PATH. Both of those paths are being passed to the linker with -L .
But using either -lcuda or -lcudart results in "/usr/bin/ld: cannot find -lcudart" or "/usr/bin/ld: cannot find -lcudart"

I am using a makefile.. these are the build rules I'm trying to set up.

I appreciate the quoted build lines you're using! And checking them, I still don't see anything I'm missing. :-(
Since it's the linker failing, I suspect that it's my Linux/g++ setup and not CUDA specifically.

Ok, a further test. I can use "-lreadline" and it finds the libreadline library with no error. So I can link some libraries at least.
But I [b]cannot[/b] use "-lsmime3" even though libsmine3.so lives in that same /usr/lib64 directory!
But the difference is that there is a[b] libreadline.a [/b] file in the directory as well as libreadline.so . There is no libsmime3.a.
[b]So maybe this is a static versus dynamic library issue!?[/b] But my compile steps pretty much match aviday's.

libcuda does [b]not[/b] have a libcuda.a , only a libcuda.so .
Is there supposed to be a libcuda.a file too? (Please check your /lib directory.... do you have both files?)

[url="http://forums.nvidia.com/lofiversion/index.php?t66326.html"]This post[/url] says that the libcuda.a file is installed by the driver. However I only have libcuda.so.
Should I reinstall the latest driver?
avid, thanks for the fast reply!



Unfortunately, it's not an issue of using -lcuda or -lcudart. They both fail.

libcuda.so lives in /usr/lib64 and libcudart.so lives in /usr/local/cuda/lib64.

Both of those paths are in my LD_LIBRARY_PATH. Both of those paths are being passed to the linker with -L .

But using either -lcuda or -lcudart results in "/usr/bin/ld: cannot find -lcudart" or "/usr/bin/ld: cannot find -lcudart"



I am using a makefile.. these are the build rules I'm trying to set up.



I appreciate the quoted build lines you're using! And checking them, I still don't see anything I'm missing. :-(

Since it's the linker failing, I suspect that it's my Linux/g++ setup and not CUDA specifically.



Ok, a further test. I can use "-lreadline" and it finds the libreadline library with no error. So I can link some libraries at least.

But I cannot use "-lsmime3" even though libsmine3.so lives in that same /usr/lib64 directory!

But the difference is that there is a libreadline.a file in the directory as well as libreadline.so . There is no libsmime3.a.

So maybe this is a static versus dynamic library issue!? But my compile steps pretty much match aviday's.



libcuda does not have a libcuda.a , only a libcuda.so .

Is there supposed to be a libcuda.a file too? (Please check your /lib directory.... do you have both files?)



This post says that the libcuda.a file is installed by the driver. However I only have libcuda.so.

Should I reinstall the latest driver?

#3
Posted 10/11/2009 04:20 PM   
[quote name='MarcDitto' post='599437' date='Oct 11 2009, 07:20 PM']Unfortunately, it's not an issue of using -lcuda or -lcudart. They both fail.[/quote]
You have several "levels" of problems. Not linking against the correct library is one of them. The linker not finding the library is another.

[quote]libcuda.so lives in /usr/lib64 and libcudart.so lives in /usr/local/cuda/lib64.
Both of those paths are in my LD_LIBRARY_PATH.[/quote]
LD_LIBRARY_PATH is irrelevant to this discussion. It only controls the runtime linkers ability to locate shared libraries. It has no effect on compilation behaviour.

[quote]Both of those paths are being passed to the linker with -L .
But using either -lcuda or -lcudart results in "/usr/bin/ld: cannot find -lcudart" or "/usr/bin/ld: cannot find -lcudart"

I am using a makefile.. these are the build rules I'm trying to set up.[/quote]
This is undoubtedly where your problem is. Could you post the exact compilation statements or makefile you are using.

[quote]Since it's the linker failing, I suspect that it's my Linux/g++ setup and not CUDA specifically.[/quote]
I very much doubt it. The mere fact you can build the SDK examples completely disproves it.

[quote][b]So maybe this is a static versus dynamic library issue!?[/b] But my compile steps pretty much match aviday's.

libcuda does [b]not[/b] have a libcuda.a , only a libcuda.so .[/quote]
CUDA is only supplied as shared libraries.
[quote name='MarcDitto' post='599437' date='Oct 11 2009, 07:20 PM']Unfortunately, it's not an issue of using -lcuda or -lcudart. They both fail.

You have several "levels" of problems. Not linking against the correct library is one of them. The linker not finding the library is another.



libcuda.so lives in /usr/lib64 and libcudart.so lives in /usr/local/cuda/lib64.

Both of those paths are in my LD_LIBRARY_PATH.


LD_LIBRARY_PATH is irrelevant to this discussion. It only controls the runtime linkers ability to locate shared libraries. It has no effect on compilation behaviour.



Both of those paths are being passed to the linker with -L .

But using either -lcuda or -lcudart results in "/usr/bin/ld: cannot find -lcudart" or "/usr/bin/ld: cannot find -lcudart"



I am using a makefile.. these are the build rules I'm trying to set up.


This is undoubtedly where your problem is. Could you post the exact compilation statements or makefile you are using.



Since it's the linker failing, I suspect that it's my Linux/g++ setup and not CUDA specifically.


I very much doubt it. The mere fact you can build the SDK examples completely disproves it.



So maybe this is a static versus dynamic library issue!? But my compile steps pretty much match aviday's.



libcuda does not have a libcuda.a , only a libcuda.so .


CUDA is only supplied as shared libraries.

#4
Posted 10/11/2009 05:24 PM   
Boom, solved. Programmer stupidity.
Avid, thanks for insisting on a simple test case. I made a test and it failed, but then I decided to make a command-line only version to simplify even more and it WORKED.

So the problem was hidden in my makefile. I had flags of "CFLAGS= -ffast-math -static -O3"

Guess what, the -static flag means that you must have a static .a file. There is no static CUDA lib, so it was failing.
My fault. Though to be honest I wish GCC's linker had better error error messages. If it just said "-static link option cannot find libcuda.a in library paths." it would have helped enormously!

Avid, thanks very very much for jumping in to help. I'd still be screaming obscenities if you hadn't asked for the exact statements.. my makefile test was only using a couple lines and it was so easy to think there was no problem there.

Problem and solution summary: "Don't use -static when linking CUDA apps."
Boom, solved. Programmer stupidity.

Avid, thanks for insisting on a simple test case. I made a test and it failed, but then I decided to make a command-line only version to simplify even more and it WORKED.



So the problem was hidden in my makefile. I had flags of "CFLAGS= -ffast-math -static -O3"



Guess what, the -static flag means that you must have a static .a file. There is no static CUDA lib, so it was failing.

My fault. Though to be honest I wish GCC's linker had better error error messages. If it just said "-static link option cannot find libcuda.a in library paths." it would have helped enormously!



Avid, thanks very very much for jumping in to help. I'd still be screaming obscenities if you hadn't asked for the exact statements.. my makefile test was only using a couple lines and it was so easy to think there was no problem there.



Problem and solution summary: "Don't use -static when linking CUDA apps."

#5
Posted 10/11/2009 09:27 PM   
Hello! I need help about this:

I want to compile bandwidthTest.cu within makefile from sdk, and I use this: nvcc -0 bandwidthTest bandwithTest.cu -lcuda

/tmp/tmpxft_00007852_00000000-12_bandwidthTest.o: In function `testDeviceToDeviceTransfer(unsigned int)':
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe4ac): undefined reference to `cutCreateTimer'
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe555): undefined reference to `cutStartTimer'
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe5bd): undefined reference to `cutStopTimer'
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe5d6): undefined reference to `cutGetTimerValue'
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe664): undefined reference to `cutDeleteTimer'
/tmp/tmpxft_00007852_00000000-12_bandwidthTest.o: In function `testHostToDeviceTransfer(unsigned int, memoryMode)':
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe6ed): undefined reference to `cutCreateTimer'
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe7de): undefined reference to `cutStartTimer'
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe81d): undefined reference to `cutStopTimer'
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe836): undefined reference to `cutGetTimerValue'
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe84f): undefined reference to `cutResetTimer'
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe924): undefined reference to `cutDeleteTimer'
/tmp/tmpxft_00007852_00000000-12_bandwidthTest.o: In function `testDeviceToHostTransfer(unsigned int, memoryMode)':
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe9d7): undefined reference to `cutCreateTimer'
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xeab1): undefined reference to `cutStartTimer'
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xeb03): undefined reference to `cutStopTimer'
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xeb1c): undefined reference to `cutGetTimerValue'
tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xeba6): undefined reference to `cutDeleteTimer'
collect2: ld devolvió el estado de salida 1


Thanks
Hello! I need help about this:



I want to compile bandwidthTest.cu within makefile from sdk, and I use this: nvcc -0 bandwidthTest bandwithTest.cu -lcuda



/tmp/tmpxft_00007852_00000000-12_bandwidthTest.o: In function `testDeviceToDeviceTransfer(unsigned int)':

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe4ac): undefined reference to `cutCreateTimer'

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe555): undefined reference to `cutStartTimer'

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe5bd): undefined reference to `cutStopTimer'

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe5d6): undefined reference to `cutGetTimerValue'

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe664): undefined reference to `cutDeleteTimer'

/tmp/tmpxft_00007852_00000000-12_bandwidthTest.o: In function `testHostToDeviceTransfer(unsigned int, memoryMode)':

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe6ed): undefined reference to `cutCreateTimer'

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe7de): undefined reference to `cutStartTimer'

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe81d): undefined reference to `cutStopTimer'

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe836): undefined reference to `cutGetTimerValue'

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe84f): undefined reference to `cutResetTimer'

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe924): undefined reference to `cutDeleteTimer'

/tmp/tmpxft_00007852_00000000-12_bandwidthTest.o: In function `testDeviceToHostTransfer(unsigned int, memoryMode)':

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xe9d7): undefined reference to `cutCreateTimer'

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xeab1): undefined reference to `cutStartTimer'

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xeb03): undefined reference to `cutStopTimer'

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xeb1c): undefined reference to `cutGetTimerValue'

tmpxft_00007852_00000000-11_bandwidthTest.ii:(.text+0xeba6): undefined reference to `cutDeleteTimer'

collect2: ld devolvió el estado de salida 1





Thanks

#6
Posted 11/27/2009 12:03 AM   
You need to link with the cutil library from the SDK (and you need to tell nvcc where to find it)
You need to link with the cutil library from the SDK (and you need to tell nvcc where to find it)

#7
Posted 11/27/2009 12:08 AM   
Many thank avvi. Another cuestion, Is it posible mix driver library and api library in the same code #include <cutil_inline.h>
#include <cuda.h>

I´m trying to compile, but only work if I separate the code :(.
Many thank avvi. Another cuestion, Is it posible mix driver library and api library in the same code #include <cutil_inline.h>

#include <cuda.h>



I´m trying to compile, but only work if I separate the code :(.

#8
Posted 11/27/2009 03:17 AM   
Scroll To Top