Cross-compile samples for DPX2(4.1.8.0L_SDK) fails with "cuda_runtime.h: No such file or directory"

SamJey · September 14, 2017, 1:47am

Dear Moderator,
I have followed the steps from both /usr/local/driveworks-0.3/doc/nvdwx_html/dwx_samples_building.html and https://devtalk.nvidia.com/default/topic/997207/driveworks/driveworks-0-2-1-build-sample-but-cannot-run-on-px2. My cmake runs without any issues. However, when I ran make -j command, it fails with “cuda_runtime.h: No such file or directory”. I started debugging this issue and found cuda_runtime.h do exist in the system. I am stuck with this issue for the past two days. Kindly can you help me to solve this problem.?

I will really appreciate your effort and time.

user@qpu-272:/usr/local/driveworks-0.3$ sudo mkdir build
user@qpu-272:/usr/local/driveworks-0.3$ cd build

user@qpu-272:/usr/local/driveworks-0.3/build$ pwd
/usr/local/driveworks-0.3/build
user@qpu-272:/usr/local/driveworks-0.3/build$ ls
user@qpu-272:/usr/local/driveworks-0.3/build$ sudo cmake -DCMAKE_BUILD_TYPE=Release -DVIBRANTE_PDK:STRING=/home/user/Downloads/NVIDIA/DRIVEINSTALL/VibranteSDK/vibrante-t186ref-linux -DCMAKE_TOOLCHAIN_FILE=/usr/local/driveworks-0.3//samples/cmake/SamplesInstallConfiguration.cmake /usr/local/driveworks-0.3/samples/src/hello_world
– Driveworks Samples install dir:
– Driveworks Samples install dir:
– The C compiler identification is GNU 4.8.4
– The CXX compiler identification is GNU 4.8.4
– Check for working C compiler: /usr/bin/cc
– Check for working C compiler: /usr/bin/cc – works
– Detecting C compiler ABI info
– Detecting C compiler ABI info - done
– Detecting C compile features
– Detecting C compile features - done
– Check for working CXX compiler: /usr/bin/c++
– Check for working CXX compiler: /usr/bin/c++ – works
– Detecting CXX compiler ABI info
– Detecting CXX compiler ABI info - done
– Detecting CXX compile features
– Detecting CXX compile features - done
– Configuring done
– Generating done
CMake Warning:
Manually-specified variables were not used by the project:

VIBRANTE_PDK

– Build files have been written to: /usr/local/driveworks-0.3/build

user@qpu-272:/usr/local/driveworks-0.3/build$ ls
CMakeCache.txt CMakeFiles cmake_install.cmake Makefile
user@qpu-272:/usr/local/driveworks-0.3/build$

user@qpu-272:/usr/local/driveworks-0.3/build$ sudo make -j VERBOSE=1
/usr/local/bin/cmake -H/usr/local/driveworks-0.3/samples/src/hello_world -B/usr/local/driveworks-0.3/build --check-build-system CMakeFiles/Makefile.cmake 0
/usr/local/bin/cmake -E cmake_progress_start /usr/local/driveworks-0.3/build/CMakeFiles /usr/local/driveworks-0.3/build/CMakeFiles/progress.marks
make -f CMakeFiles/Makefile2 all
make[1]: Entering directory /usr/local/driveworks-0.3/build' make -f CMakeFiles/sample_hello_world.dir/build.make CMakeFiles/sample_hello_world.dir/depend make[2]: Entering directory /usr/local/driveworks-0.3/build’
cd /usr/local/driveworks-0.3/build && /usr/local/bin/cmake -E cmake_depends “Unix Makefiles” /usr/local/driveworks-0.3/samples/src/hello_world /usr/local/driveworks-0.3/samples/src/hello_world /usr/local/driveworks-0.3/build /usr/local/driveworks-0.3/build /usr/local/driveworks-0.3/build/CMakeFiles/sample_hello_world.dir/DependInfo.cmake --color=
Dependee “/usr/local/driveworks-0.3/build/CMakeFiles/sample_hello_world.dir/DependInfo.cmake” is newer than depender “/usr/local/driveworks-0.3/build/CMakeFiles/sample_hello_world.dir/depend.internal”.
Dependee “/usr/local/driveworks-0.3/build/CMakeFiles/CMakeDirectoryInformation.cmake” is newer than depender “/usr/local/driveworks-0.3/build/CMakeFiles/sample_hello_world.dir/depend.internal”.
Scanning dependencies of target sample_hello_world
make[2]: Leaving directory /usr/local/driveworks-0.3/build' make -f CMakeFiles/sample_hello_world.dir/build.make CMakeFiles/sample_hello_world.dir/build make[2]: Entering directory /usr/local/driveworks-0.3/build’
/usr/local/bin/cmake -E cmake_progress_report /usr/local/driveworks-0.3/build/CMakeFiles 1
[100%] Building CXX object CMakeFiles/sample_hello_world.dir/main.cpp.o
/usr/bin/c++ -O3 -DNDEBUG -o CMakeFiles/sample_hello_world.dir/main.cpp.o -c /usr/local/driveworks-0.3/samples/src/hello_world/main.cpp
/usr/local/driveworks-0.3/samples/src/hello_world/main.cpp:39:26: fatal error: cuda_runtime.h: No such file or directory
#include <cuda_runtime.h>
^
compilation terminated.
make[2]: *** [CMakeFiles/sample_hello_world.dir/main.cpp.o] Error 1
make[2]: Leaving directory /usr/local/driveworks-0.3/build' make[1]: *** [CMakeFiles/sample_hello_world.dir/all] Error 2 make[1]: Leaving directory /usr/local/driveworks-0.3/build’
make: *** [all] Error 2
user@qpu-272:/usr/local/driveworks-0.3/build$

user@qpu-272:/usr/local/driveworks-0.3/build$ which nvcc
/usr/local/cuda-8.0/bin/nvcc
user@qpu-272:/usr/local/driveworks-0.3/build$ sudo find / -name nvcc > ~/t
user@qpu-272:/usr/local/driveworks-0.3/build$ cat ~/t
/home/user/Downloads/NVIDIA/SAMPLES/usr/local/cuda-8.0/bin/nvcc
/usr/local/cuda-8.0/bin/nvcc
user@qpu-272:/usr/local/driveworks-0.3/build$ echo $PATH
/usr/local/cuda-8.0/bin:/opt/ros/indigo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
user@qpu-272:/usr/local/driveworks-0.3/build$
user@qpu-272:/usr/local/driveworks-0.3/build$ find /usr/local -name cuda_runtime.h
/usr/local/cuda-8.0/targets/aarch64-linux/include/cuda_runtime.h
/usr/local/cuda-8.0/targets/x86_64-linux/include/cuda_runtime.h
user@qpu-272:/usr/local/driveworks-0.3/build$
user@qpu-272:/usr/local/driveworks-0.3/build$ echo $LD_LIBRARY_PATH
/usr/local/cuda-8.0/lib64:/opt/ros/indigo/lib
user@qpu-272:/usr/local/driveworks-0.3/build$
user@qpu-272:/usr/local/driveworks-0.3/build$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
user@qpu-272:/usr/local/driveworks-0.3/build$

SteveNV · September 14, 2017, 5:55am

Dear SamJey,

Did you setup cuda environment completely? if not, please refer to below setup process and try it again. Thanks.

$gedit ~/.bashrc
export PATH=/usr/local/cuda-8.0/bin/:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/targets/aarch64-linux/lib:$LD_LIBRARY_PATH

$ source ~/.bashrc
$ nvcc --version

$ /usr/local/cuda-8.0/bin/cuda-install-samples-8.0.sh ~/

$cd ~/NVIDIA_CUDA8.0_Samples/1_Utilities/deviceQuery
$make
$./deviceQuery

SamJey · September 15, 2017, 1:43am

Dear Steve,
Thank you very much for your kind reply.

I followed the steps provided in your reply. It still failed with the same error.

Are these CUDO 8.0 and Driver version 340.102 compatible with each other? This is because I am running driver version 340.102 with a GPU named “Graphics Device”. I have also provided the output the nvidia-smi command at the end of this reply. I googled but I am bit confused as people are saying diff answers.

Could you please provide your guidance on what went wrong?

user@qpu-286:/usr/local$ echo $LD_LIBRARY_PATH
/usr/local/cuda/targets/aarch64-linux/lib:/opt/ros/indigo/lib

user@qpu-286:/usr/local$ echo $PATH
/usr/local/cuda-8.0/bin:/opt/ros/indigo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

user@qpu-286:/usr/local$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Mon_Mar_20_16:30:19_CDT_2017
Cuda compilation tools, release 8.0, V8.0.72

user@qpu-286:/usr/local$ sudo /usr/local/cuda-8.0/bin/cuda-install-samples-8.0.sh ~/
[sudo] password for user:
Copying samples to /home/user/NVIDIA_CUDA-8.0_Samples now…
Finished copying samples.

user@qpu-286:/usr/local$ find . -name deviceQuery
./cuda-8.0/samples/1_Utilities/deviceQuery

user@qpu-286:/usr/local$ cd ./cuda-8.0/samples/1_Utilities/deviceQuery

user@qpu-286:/usr/local/cuda-8.0/samples/1_Utilities/deviceQuery$ ls
deviceQuery.cpp Makefile NsightEclipse.xml readme.txt

user@qpu-286:/usr/local/cuda-8.0/samples/1_Utilities/deviceQuery$ sudo make
/usr/local/cuda-8.0/bin/nvcc -ccbin g++ -I…/…/common/inc -m64 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_60,code=compute_60 -o deviceQuery.o -c deviceQuery.cpp
nvcc warning : The ‘compute_20’, ‘sm_20’, and ‘sm_21’ architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
/usr/local/cuda-8.0/bin/nvcc -ccbin g++ -m64 -gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_37,code=sm_37 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_60,code=compute_60 -o deviceQuery deviceQuery.o
nvcc warning : The ‘compute_20’, ‘sm_20’, and ‘sm_21’ architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
mkdir -p …/…/bin/x86_64/linux/release
cp deviceQuery …/…/bin/x86_64/linux/release
user@qpu-286:/usr/local/cuda-8.0/samples/1_Utilities/deviceQuery$ sudo ./deviceQuery
./deviceQuery Starting…

CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 35
→ CUDA driver version is insufficient for CUDA runtime version
Result = FAIL
user@qpu-286:/usr/local/cuda-8.0/samples/1_Utilities/deviceQuery$

±----------------------------------------------------------------------------+
| Compute processes: GPU Memory |
| GPU PID Process name Usage |
|=============================================================================|
| 0 Not Supported |
±----------------------------------------------------------------------------+
user@qpu-286:/usr/local/cuda-8.0/samples/1_Utilities/deviceQuery$

I will really appreciate your help.

Thanks
Sam

SamJey · September 16, 2017, 1:00am

Dear Steve,

It seems CUDA 8.0 is not compatible with driver 340.102 version. Uninstalling the old one and installing the 371 fixed the problem and I am able to run deviceQuery without any issues.

I am back to the old problem now.

user@qpu-286:/usr/local/driveworks-0.3/build$ sudo make -j
[100%] Building CXX object CMakeFiles/sample_hello_world.dir/main.cpp.o
/usr/local/driveworks-0.3/samples/src/hello_world/main.cpp:39:26: fatal error: cuda_runtime.h: No such file or directory
#include <cuda_runtime.h>
^
compilation terminated.
make[2]: *** [CMakeFiles/sample_hello_world.dir/main.cpp.o] Error 1
make[1]: *** [CMakeFiles/sample_hello_world.dir/all] Error 2
make: *** [all] Error 2
user@qpu-286:/usr/local/driveworks-0.3/build$

user@qpu-286:/usr/local/driveworks-0.3/build$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Mon_Mar_20_16:30:19_CDT_2017
Cuda compilation tools, release 8.0, V8.0.72
user@qpu-286:/usr/local/driveworks-0.3/build$

Any hint will be appreciated?

Thanks
Sam

SamJey · September 16, 2017, 1:18am

Actually there is some include path is missing. This is because, when I ran another sample program, it gave me a diff error.

user@qpu-286:/usr/local/driveworks-0.3/build$ sudo make -j
– Driveworks Samples install dir: /usr/local
– Configuring done
– Generating done
– Build files have been written to: /usr/local/driveworks-0.3/build
Scanning dependencies of target sample_ipc_socketclientserver
[100%] Building CXX object CMakeFiles/sample_ipc_socketclientserver.dir/sample_ipc_socketclientserver.cpp.o
/usr/local/driveworks-0.3/samples/src/ipc/sample_ipc_socketclientserver.cpp:31:29: fatal error: dw/core/Context.h: No such file or directory
#include <dw/core/Context.h>
^
compilation terminated.

victorhcmelo · November 22, 2019, 2:15pm

In case someone needs, besides PATH and LD_LIBRARY_PATH, you also need to set CPATH:

export CPATH=/usr/local/cuda-10.1/targets/x86_64-linux/include:$CPATH
export LD_LIBRARY_PATH=/usr/local/cuda-10.1/targets/x86_64-linux/lib:$LD_LIBRARY_PATH
export PATH=/usr/local/cuda-10.1/bin:$PATH