How to use OnnxRuntime for Jetson Nano wirh Cuda ,TensorRT ?

Hi,

I’m trying to build Onnxruntime running on Jetson Nano.

CPU builds work fine on Python but not on CUDA Build or TensorRT Build.

Is memory affected by CPU and GPU?
Is it cureable by the script description?
Are there not enough options for building?

So anybody can help me? Thank!
(I wondered where to ask questions but ask questions here)

onnxruntime-0.3.1:
No Problem

onnxruntime-gpu-0.3.1 (with CUDA Build):
An error occurs in session.run “no kernel image is available for execution on the device”

onnxruntime-gpu-tensorrt-0.3.1 (with TensorRT Build):
Sclipt Killed in InferenceSession

build opption ( BUILDTYPE=Debug )
–config ${BUILDTYPE} --arm --update --build --use_cuda --cudnn_home /usr/local/cuda --cuda_home /usr/local/cuda --use_tensorrt --tensorrt_home /usr/src/tensorrt --build_shared_lib --enable_pybind --build_wheel

Hi,

From your log, it looks like the library doesn’t support Nano GPU compute capacity, which should be 5.3.

An error occurs in session.run "no kernel image is available for execution on the device"

You can add one for it like this:
https://github.com/Microsoft/onnxruntime/blob/a4d7052aeb421f18371eab0961f9eaf495c6420c/cmake/CMakeLists.txt#L534

set(CMAKE_CUDA_FLAGS "${CMAKE_CUDA_FLAGS} -gencode=arch=compute_53,code=sm_53") # Nano

By the way, I don’t think you need to get the onnxruntime work on Jetson Nano.
You can use the pre-install TensorRT API directly. It support onnx model and optimize for the Jetson platform.
https://docs.nvidia.com/deeplearning/sdk/tensorrt-sample-support-guide/index.html#yolov3_onnx

Thanks.

Thank you,
I can do inference with onnxruntime-gpu on python, which gives me more powerful results than arm conditions.
with TensorRT build couldn’t load the onnx model and maybe something was missing during the pytorch conversion.
Thanks for the suggestion of using Tensor RT with python, try it next

I tried to confirm the additional

The problem below seems to be Sclipt Killed due to lack of memory when optimizing TensorRT.
I trIied with small file size images and ONNX models, it can be optimized and speeded up.

Onnxruntime-gpu-tensorrt-0.3.1 (with TensorRT Build):
Sclipt Killed in InferenceSession

I hope that pre-install TensorRT API avoids memory shortage during optimization ,and I try it.

Thanks very much, almost solved!

I had more success when disabling GUI on boot. (to save memory)

sudo systemctl set-default multi-user.target

Hey guys,
does anyone had a problem while building onnxruntime like it or someone knows how to solve it:
[ 25%] Building CXX object CMakeFiles/onnxruntime_util.dir/home/kamil/projects/onnxruntime/onnxruntime/core/util/math_cpu.cc.o
In file included from /home/kamil/projects/onnxruntime/cmake/external/eigen/Eigen/Core:313:0,
from /home/kamil/projects/onnxruntime/onnxruntime/core/util/math_cpuonly.h:31,
from /home/kamil/projects/onnxruntime/onnxruntime/core/util/math_cpu.cc:20:
/home/kamil/projects/onnxruntime/cmake/external/eigen/Eigen/src/Core/products/GeneralBlockPanelKernel.h: In member function ‘void Eigen::internal::gebp_traits<float, float, false, false, 4, 0>::updateRhs(const RhsScalar*, Eigen::internal::gebp_traits<float, float, false, false, 4, 0>::RhsPacketx4&) const’:
/home/kamil/projects/onnxruntime/cmake/external/eigen/Eigen/src/Core/products/GeneralBlockPanelKernel.h:1079:55: error: unused parameter ‘b’ [-Werror=unused-parameter]
EIGEN_STRONG_INLINE void updateRhs(const RhsScalar* b, RhsPacketx4& dest) const
^
/home/kamil/projects/onnxruntime/cmake/external/eigen/Eigen/src/Core/products/GeneralBlockPanelKernel.h:1079:71: error: unused parameter ‘dest’ [-Werror=unused-parameter]
EIGEN_STRONG_INLINE void updateRhs(const RhsScalar* b, RhsPacketx4& dest) const
^~~~
/home/kamil/projects/onnxruntime/cmake/external/eigen/Eigen/src/Core/products/GeneralBlockPanelKernel.h: In member function ‘void Eigen::internal::gebp_traits<double, double, false, false, 4>::updateRhs(const RhsScalar*, Eigen::internal::gebp_traits<double, double, false, false, 4>::RhsPacketx4&) const’:
/home/kamil/projects/onnxruntime/cmake/external/eigen/Eigen/src/Core/products/GeneralBlockPanelKernel.h:1148:55: error: unused parameter ‘b’ [-Werror=unused-parameter]
EIGEN_STRONG_INLINE void updateRhs(const RhsScalar* b, RhsPacketx4& dest) const
^
/home/kamil/projects/onnxruntime/cmake/external/eigen/Eigen/src/Core/products/GeneralBlockPanelKernel.h:1148:71: error: unused parameter ‘dest’ [-Werror=unused-parameter]
EIGEN_STRONG_INLINE void updateRhs(const RhsScalar* b, RhsPacketx4& dest) const
^~~~
cc1plus: all warnings being treated as errors
CMakeFiles/onnxruntime_util.dir/build.make:75: recipe for target ‘CMakeFiles/onnxruntime_util.dir/home/kamil/projects/onnxruntime/onnxruntime/core/util/math_cpu.cc.o’ failed
make[2]: *** [CMakeFiles/onnxruntime_util.dir/home/kamil/projects/onnxruntime/onnxruntime/core/util/math_cpu.cc.o] Error 1
CMakeFiles/Makefile2:609: recipe for target ‘CMakeFiles/onnxruntime_util.dir/all’ failed
make[1]: *** [CMakeFiles/onnxruntime_util.dir/all] Error 2
Makefile:140: recipe for target ‘all’ failed
make: *** [all] Error 2
Traceback (most recent call last):
File “/home/kamil/projects/onnxruntime/tools/ci_build/build.py”, line 983, in
sys.exit(main())
File “/home/kamil/projects/onnxruntime/tools/ci_build/build.py”, line 922, in main
build_targets(cmake_path, build_dir, configs, args.parallel)
File “/home/kamil/projects/onnxruntime/tools/ci_build/build.py”, line 415, in build_targets
run_subprocess(cmd_args)
File “/home/kamil/projects/onnxruntime/tools/ci_build/build.py”, line 200, in run_subprocess
completed_process = subprocess.run(args, cwd=cwd, check=True, stdout=stdout, stderr=stderr, env=my_env, shell=shell)
File “/usr/lib/python3.6/subprocess.py”, line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ‘[’/usr/local/bin/cmake’, ‘–build’, ‘/home/kamil/projects/onnxruntime/build/Linux/Debug’, ‘–config’, ‘Debug’]’ returned non-zero exit status 2.