CUDA 9.0 samples do not build with Jetpack 3.2

snarky · April 3, 2018, 4:44am

I flashed from scratch the latest downloaded Jetpack 3.2.
I then went to /usr/local/cuda-9.0/samples and did “make” (after making sure I could write the directory)
Unfortunately, the build process fails for several tools, with errors like:

make[1]: Entering directory '/usr/local/cuda-9.0/samples/6_Advanced/cdpLUDecomposition'
/usr/local/cuda-9.0/bin/nvcc -ccbin g++   -m64      -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_53,code=sm_53 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_62,code=sm_62 -gencode arch=compute_62,code=compute_62 -o cdpLUDecomposition cdp_lu.o cdp_lu_main.o dgetf2.o dgetrf.o dlaswp.o  -lcublas -lcublas_device -lcudadevrt
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@E@nvlink error   : Undefined reference to 'cublasIdamax_v2' in 'dgetf2.o' (target: sm_35)
@E@nvlink error   : Undefined reference to 'cublasDswap_v2' in 'dgetf2.o' (target: sm_35)
@E@nvlink error   : Undefined reference to 'cublasDscal_v2' in 'dgetf2.o' (target: sm_35)
@E@nvlink error   : Undefined reference to 'cublasDger_v2' in 'dgetf2.o' (target: sm_35)
@E@nvlink error   : Undefined reference to 'cublasDtrsm_v2' in 'dgetrf.o' (target: sm_35)
@E@nvlink error   : Undefined reference to 'cublasDgemm_v2' in 'dgetrf.o' (target: sm_35)
@E@nvlink error   : Undefined reference to 'cublasCreate_v2' in 'dgetrf.o' (target: sm_35)
@E@nvlink error   : Undefined reference to 'cublasSetPointerMode_v2' in 'dgetrf.o' (target: sm_35)
@E@nvlink error   : Undefined reference to 'cublasSetStream_v2' in 'dgetrf.o' (target: sm_35)
Makefile:296: recipe for target 'cdpLUDecomposition' failed
make[1]: *** [cdpLUDecomposition] Error 255
make[1]: Leaving directory '/usr/local/cuda-9.0/samples/6_Advanced/cdpLUDecomposition'
Makefile:52: recipe for target '6_Advanced/cdpLUDecomposition/Makefile.ph_build' failed
make: *** [6_Advanced/cdpLUDecomposition/Makefile.ph_build] Error 2

What am I missing? I installed all the NVIDIA samples / libraries in Jetpack.

AastaLLL · April 3, 2018, 6:40am

Hi,

This is a known issue and we are investigating internally.
The error is caused from libcublas not containing SM 3.5 architecture but the options is passed as supported.

Currently, please remove the SM 3.5 as a temporal solution.

diff --git a/6_Advanced/cdpLUDecomposition/Makefile b/6_Advanced/cdpLUDecomposition/Makefile
index 91cd626..3f3bcc2 100644
--- a/6_Advanced/cdpLUDecomposition/Makefile
+++ b/6_Advanced/cdpLUDecomposition/Makefile
@@ -234,7 +234,7 @@ LIBRARIES :=
 
 # Gencode arguments
 ifeq ($(TARGET_ARCH),$(filter $(TARGET_ARCH),armv7l aarch64))
-SMS ?= 35 37 50 52 53 60 61 62
+SMS ?= 62
 else
 SMS ?= 35 37 50 52 60 61
 endif

Thanks.

snarky · April 3, 2018, 6:18pm

Thanks for the answer!

ali.nehrani · April 20, 2018, 1:18pm

I have the same problem. Can you please explain your answere, I am new in working with Jetson and can’t exactly figure out how should I use your answere/

Many Thanks

Honey_Patouceul · April 20, 2018, 5:50pm

In short, you will have to edit file a Makefile. If you’re not familiar with it, a Makefile is mainly a build project, saying what sources are to be used with which build tools and options.
The diff above in @AastaLL’s post shows the differences between original file (a) and fixed file (b).
The ‘-’ is what is removed from original file, and ‘+’ is what is added in new (aka patched) file.
You’ll see that modification is only defining SMS (target CUDA archs) to only 62 instead of 35 37 50 52 53 60 61 62.
Probably just dropping the 35 arch may be enough, but as long you just run it on TX2, 62 is enough.

So you just have to edit the Makefile:

sudo su
gedit /usr/local/cuda/samples/6_Advanced/cdpLUDecomposition/Makefile

Then go around line 234, find the ‘# Gencode arguments’ comment, and two lines below, remove all arch numbers other than 62. Save and try to rebuild.

ali.nehrani · April 20, 2018, 9:50pm

Thanks for your answerer. I did exactly the same as @AastaLLL answer but I got the following error when trying to make Makefile. I attached the error logs:

ode=sm_61 -gencode arch=compute_62,code=sm_62 -gencode arch=compute_62,code=compute_62 -o simpleDevLibCUBLAS kernels.o simpleDevLibCUBLAS.o  -lcublas -lcublas_device -lcudadevrt
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@O@ptxas info    : 'device-function-maxrregcount' is a BETA feature (target: sm_35)
@E@nvlink error   : Undefined reference to 'cublasCreate_v2' in 'kernels.o' (target: sm_35)
@E@nvlink error   : Undefined reference to 'cublasSgemm_v2' in 'kernels.o' (target: sm_35)
@E@nvlink error   : Undefined reference to 'cublasDestroy_v2' in 'kernels.o' (target: sm_35)
Makefile:287: recipe for target 'simpleDevLibCUBLAS' failed
make[1]: *** [simpleDevLibCUBLAS] Error 255
make[1]: Leaving directory '/usr/local/cuda-9.0/samples/7_CUDALibraries/simpleDevLibCUBLAS'
Makefile:52: recipe for target '7_CUDALibraries/simpleDevLibCUBLAS/Makefile.ph_build' failed
make: *** [7_CUDALibraries/simpleDevLibCUBLAS/Makefile.ph_build] Error 2

I’ll appreciate for helping to solve this problem.

Honey_Patouceul · April 20, 2018, 10:06pm

Sorry I have no TX2 for checking now, but you may try

make clean

in order to clean any code compiled for 3.5 arch that would fail to link and relaunch then

make

for rebuilding from scratch.

Note that the problem in your case is not in the same project as what @snarky reported, seems to be in 7_CUDALibraries/simpleDevLibCUBLAS.
You may check if the makefile in this project has some options for 3.5 and remove these.

ali.nehrani · April 20, 2018, 11:09pm

Thanks @Honey_Patouceul, I solved 3.5 issue and made again after make clear, Now I get the following error! I tried to figure out but I dont want to mess things up.
sorry for asking a lot of question and thanks for your responses.

nvlink error   : Undefined reference to 'cublasIdamax_v2' in 'dgetf2.o'
nvlink error   : Undefined reference to 'cublasDswap_v2' in 'dgetf2.o'
nvlink error   : Undefined reference to 'cublasDscal_v2' in 'dgetf2.o'
nvlink error   : Undefined reference to 'cublasDger_v2' in 'dgetf2.o'
nvlink error   : Undefined reference to 'cublasDtrsm_v2' in 'dgetrf.o'
nvlink error   : Undefined reference to 'cublasDgemm_v2' in 'dgetrf.o'
nvlink error   : Undefined reference to 'cublasCreate_v2' in 'dgetrf.o'
nvlink error   : Undefined reference to 'cublasSetPointerMode_v2' in 'dgetrf.o'
nvlink error   : Undefined reference to 'cublasSetStream_v2' in 'dgetrf.o'
Makefile:296: recipe for target 'cdpLUDecomposition' failed
make: *** [cdpLUDecomposition] Error 255

One more thing, should I recompile Makefiles from all folders (the same part)?

linuxdev · April 21, 2018, 12:21am

I haven’t looked at this particular sample, but that means the linker didn’t find a library it was told to link against for those symbols. Somewhere in that Makefile will be an option for search path of the linker and it needs to have the path to the CUDA library location added.

snarky · April 21, 2018, 1:52am

If those are the only errors, I’m not sure it’s a link path problem – it’s not complaining about a missing library, it’s complaining about missing symbols within the libraries/sources it’s building.
Thus, this could also be a versioning problem.
Given that those symbol names are very similar to the error I started with in this thread, perhaps you need to re-define what shader models you’re building for and re-generate the makefiles to make it stick?

ali.nehrani · April 21, 2018, 7:52am

About the version I am using Jetpack 3.2,

I attached the Makefile in cuda/samples//6_Advanced/cdpLUDecomposition/

The error arises in the following lines of the makefile (I pointed them by ###…) :

cdp_lu.o:cdp_lu.cu
	$(EXEC) $(NVCC) $(INCLUDES) $(ALL_CCFLAGS) $(GENCODE_FLAGS) -o $@ -c $<

cdp_lu_main.o:cdp_lu_main.cu
	$(EXEC) $(NVCC) $(INCLUDES) $(ALL_CCFLAGS) $(GENCODE_FLAGS) -o $@ -c $<

dgetf2.o:dgetf2.cu
	$(EXEC) $(NVCC) $(INCLUDES) $(ALL_CCFLAGS) $(GENCODE_FLAGS) -o $@ -c $< #############

dgetrf.o:dgetrf.cu
	$(EXEC) $(NVCC) $(INCLUDES) $(ALL_CCFLAGS) $(GENCODE_FLAGS) -o $@ -c $< ##############

dlaswp.o:dlaswp.cu
	$(EXEC) $(NVCC) $(INCLUDES) $(ALL_CCFLAGS) $(GENCODE_FLAGS) -o $@ -c $<

cdpLUDecomposition: cdp_lu.o cdp_lu_main.o dgetf2.o dgetrf.o dlaswp.o   ###############################     
	$(EXEC) $(NVCC) $(ALL_LDFLAGS) $(GENCODE_FLAGS) -o $@ $+ $(LIBRARIES)
	$(EXEC) mkdir -p ../../bin/$(TARGET_ARCH)/$(TARGET_OS)/$(BUILD_TYPE)
	$(EXEC) cp $@ ../../bin/$(TARGET_ARCH)/$(TARGET_OS)/$(BUILD_TYPE)

run: build
	$(EXEC) ./cdpLUDecomposition

clean:
	rm -f cdpLUDecomposition cdp_lu.o cdp_lu_main.o dgetf2.o dgetrf.o dlaswp.o ###################
	rm -rf ../../bin/$(TARGET_ARCH)/$(TARGET_OS)/$(BUILD_TYPE)/cdpLUDecomposition

clobber: clean

Makefile.txt (10.4 KB)

linuxdev · April 21, 2018, 5:48pm

If you look at the Makefile and look for this line (should be line 295-296 below the cdpLUDecomposition) you’ll see the final linker step:

cdpLUDecomposition: cdp_lu.o cdp_lu_main.o dgetf2.o dgetrf.o dlaswp.o
	$(EXEC) $(NVCC) $(ALL_LDFLAGS) $(GENCODE_FLAGS) -o $@ $+ $(LIBRARIES)

To debug add an echo between those two to have it print the linker path used (even if there is a version mismatch it should point out what library path it is attempting to link against):

cdpLUDecomposition: cdp_lu.o cdp_lu_main.o dgetf2.o dgetrf.o dlaswp.o
	<b>echo "DEBUG ALL_LDFLAGS: $(ALL_LDFLAGS)"</b>
	$(EXEC) $(NVCC) $(ALL_LDFLAGS) $(GENCODE_FLAGS) -o $@ $+ $(LIBRARIES)

Be careful to note that those indents are a real tab, not spaces.

You might need to make clean first, and then make again. Look for what the “DEBUG ALL_LDFLAGS” prints.

ali.nehrani · April 21, 2018, 6:23pm

I added

echo "DEBUG ALL_LDFLAGS: $(ALL_LDFLAGS)"

did “make clean” and then make here are the resulting log:

rm -f cdpLUDecomposition cdp_lu.o cdp_lu_main.o dgetf2.o dgetrf.o dlaswp.o
rm -rf ../../bin/aarch64/linux/release/cdpLUDecomposition
root@user:/usr/local/cuda/samples/6_Advanced/cdpLUDecomposition# make
/usr/local/cuda-9.0/bin/nvcc -ccbin g++ -I../../common/inc  -m64    -dc -gencode arch=compute_62,code=sm_62 -gencode arch=compute_62,code=compute_62 -o cdp_lu.o -c cdp_lu.cu
/usr/local/cuda-9.0/bin/nvcc -ccbin g++ -I../../common/inc  -m64    -dc -gencode arch=compute_62,code=sm_62 -gencode arch=compute_62,code=compute_62 -o cdp_lu_main.o -c cdp_lu_main.cu
/usr/local/cuda-9.0/bin/nvcc -ccbin g++ -I../../common/inc  -m64    -dc -gencode arch=compute_62,code=sm_62 -gencode arch=compute_62,code=compute_62 -o dgetf2.o -c dgetf2.cu
/usr/local/cuda-9.0/bin/nvcc -ccbin g++ -I../../common/inc  -m64    -dc -gencode arch=compute_62,code=sm_62 -gencode arch=compute_62,code=compute_62 -o dgetrf.o -c dgetrf.cu
/usr/local/cuda-9.0/bin/nvcc -ccbin g++ -I../../common/inc  -m64    -dc -gencode arch=compute_62,code=sm_62 -gencode arch=compute_62,code=compute_62 -o dlaswp.o -c dlaswp.cu
echo "DEBUG ALL_LDFLAGS:   -m64     "
DEBUG ALL_LDFLAGS:   -m64     
/usr/local/cuda-9.0/bin/nvcc -ccbin g++   -m64      -gencode arch=compute_62,code=sm_62 -gencode arch=compute_62,code=compute_62 -o cdpLUDecomposition cdp_lu.o cdp_lu_main.o dgetf2.o dgetrf.o dlaswp.o 
nvlink error   : Undefined reference to 'cublasIdamax_v2' in 'dgetf2.o'
nvlink error   : Undefined reference to 'cublasDswap_v2' in 'dgetf2.o'
nvlink error   : Undefined reference to 'cublasDscal_v2' in 'dgetf2.o'
nvlink error   : Undefined reference to 'cublasDger_v2' in 'dgetf2.o'
nvlink error   : Undefined reference to 'cublasDtrsm_v2' in 'dgetrf.o'
nvlink error   : Undefined reference to 'cublasDgemm_v2' in 'dgetrf.o'
nvlink error   : Undefined reference to 'cublasCreate_v2' in 'dgetrf.o'
nvlink error   : Undefined reference to 'cublasSetPointerMode_v2' in 'dgetrf.o'
nvlink error   : Undefined reference to 'cublasSetStream_v2' in 'dgetrf.o'
Makefile:296: recipe for target 'cdpLUDecomposition' failed
make: *** [cdpLUDecomposition] Error 255

linuxdev · April 21, 2018, 6:44pm

That’s a problem:

DEBUG ALL_LDFLAGS:   -m64

…your LDFLAGS needs to tell the linker where to find libraries (ALL_LDFLAGS is constructed in part from LDFLAGS…somewhere in building that it wasn’t set). I couldn’t tell you though what library paths need to be searched for or what the setup step was which should have set that up. Someone else will need to comment.

ali.nehrani · April 22, 2018, 6:29pm

Thanks for your answer @linuxdev, Yes I need an experienced one to help me to resolve this issue. Im wondering if this is because of Jetpack 3.2 or my installation.

AastaLLL · April 24, 2018, 7:13am

Hi, ali.nehrani

We cannot reproduce this issue on our side.
Could you reflash your device, update the WAR and try it again?

Thanks.

liangrong · June 4, 2018, 4:20am

have you solved your problems at last? I met the same question on PX2