unable to profile on jetson nano from host nsight

Hi,
I am going through all the steps of CUDA development on the Jetson Nano device.

From my host’s Nsight Eclipse Edition program I can cross-build a CUDA sample for AArch64 without issue. I can also run that sampled program on the nano device without problems.
The issue comes when I try to remote profile the program.
A pop-up saying “Obtaining list fo devices” runs for a long time and finally gives up with the following error message:

Data collection for 1 analysis stages failed
Failed to execute “/bin/sh -c “export LD_LIBRARY_PATH="/usr/local/cuda-10.0/lib64":${LD_LIBRARY_PATH};export NVPROF_TMPDIR="/tmp";"/usr/local/cuda-10.0/bin/nvprof" --query-cuda-info; export LD_LIBRARY_PATH="/usr/local/cuda-10.0/lib64":${LD_LIBRARY_PATH};export NVPROF_TMPDIR="/tmp";"/usr/local/cuda-10.0/bin/nvprof" --version””

The weird thing is that when I try to run the exact same nvprof commands directly on the Jetson nano’s command line I get no error.

Notes:

• Profiling with the same Nsight Eclipse Edition on a host program works.
• I have the same issue with the NVIDIA Visual Profiler tool

There seem to be an issue of Nsight running nvprof commands on the device, but I’m not sure why. Can you help me out ?

My system is relatively vanilla, and setup according to the documentation:
I have a host Ubuntu machine on which I have installed Jetpack 4.2
I have a Nano device on which I have flashed the corresponding linux distribution.

I am moving this issue to Nsight Eclipse Edition forum and I am also going to ping the engineering team to try to get you an answer.

Ok cool.
Will the engineering team respond here as well ?

that is my expectation, yup.

This is a known Visual Profiler issue. Remote profiling is not supported for the Android aarch64 target. The workaround is to run nvprof on the target and load the nvprof output in Visual Profiler. Refer the [url]https://docs.nvidia.com/cuda/profiler-users-guide/index.html#import-nvprof-session[/url] section in the Profiler User Guide.
It is same issue with the Nsight Eclipse Edition.

Thanks much ssatoor.

So what you’re saying is that GUI remote profiling altogether is not possible with Jetson Nano (unless as you suggest you run nvprof on the device and load the result) ?

Wow.

I had followed the tutorial on there : https://devblogs.nvidia.com/cuda-jetson-nvidia-nsight-eclipse-edition/
This tutorial was originally for the Jetson TX2, but hinted in the forum as appropriate for Jetson Nano as well https://devtalk.nvidia.com/default/topic/1049637/jetson-nano-amp-nsight-eclipse-edition-/
They are not mentioning this bug in the tutorial. Is it because only Jetson Nano has it ?

Thanks

The issue I mentioned in my earlier reply is on Android aarch64 target. You must be using L4T on the Jetson Nano. I need to confirm if we have some remote profiling issue on the Jetson Nano.

Hi ssatoor,
Yes that is correct, I’m using L4T on the Jetson Nano (installed through the nvidia tutorial).

Great, please let me know when you have the answer.

Thanks

Hi nicolas.stoi,

We could not reproduce this issue at our end. Primarily it seems some permission issues on target. Can you please check if you have any network level restrictions on your target (Jetson board)? Do you have root permission to your target? Can you please try SSH to the target machine and try manually running that command and see if it runs successfully? For example, run

$ ssh root@your-target-ip /bin/sh -c “export LD_LIBRARY_PATH="/usr/local/cuda-10.0/lib64":${LD_LIBRARY_PATH};export NVPROF_TMPDIR="/tmp";"/usr/local/cuda-10.0/bin/nvprof" --query-cuda-info; export LD_LIBRARY_PATH="/usr/local/cuda-10.0/lib64":${LD_LIBRARY_PATH};export NVPROF_TMPDIR="/tmp";"/usr/local/cuda-10.0/bin/nvprof" --version”

–
Thanks,
Ramesh

Hi Ramesh,
Thanks for the answer.

I’ve created an admin user “nicolas” that has sudo permission, but is different from the root user.

If I do (as you suggest):
$ ssh nicolas@jetson-ip /bin/sh -c “export LD_LIBRARY_PATH="/usr/local/cuda-10.0/lib64":${LD_LIBRARY_PATH};export NVPROF_TMPDIR="/tmp";"/usr/local/cuda-10.0/bin/nvprof" --query-cuda-info; export LD_LIBRARY_PATH="/usr/local/cuda-10.0/lib64":${LD_LIBRARY_PATH};export NVPROF_TMPDIR="/tmp";"/usr/local/cuda-10.0/bin/nvprof" --version”

I get this:
export DBUS_SESSION_BUS_ADDRESS=‘unix:path=/run/user/1001/bus’
export HOME=‘/home/nicolas’
export LANG=‘en_US.UTF-8’
export LC_ADDRESS=‘fr_FR.UTF-8’
export LC_CTYPE=‘en_US.UTF-8’
export LC_IDENTIFICATION=‘fr_FR.UTF-8’
export LC_MEASUREMENT=‘fr_FR.UTF-8’
export LC_MONETARY=‘fr_FR.UTF-8’
export LC_NAME=‘fr_FR.UTF-8’
export LC_NUMERIC=‘fr_FR.UTF-8’
export LC_PAPER=‘fr_FR.UTF-8’
export LC_TELEPHONE=‘fr_FR.UTF-8’
export LC_TIME=‘fr_FR.UTF-8’
export LOGNAME=‘nicolas’
export MAIL=‘/var/mail/nicolas’
export OLDPWD=‘/home/nicolas’
export PATH=‘/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games’
export PWD=‘/home/nicolas’
export SHELL=‘/bin/zsh’
export SHLVL=‘1’
export SSH_CLIENT=‘192.168.124.33 40778 22’
export SSH_CONNECTION=‘192.168.124.33 40778 192.168.124.34 22’
export USER=‘nicolas’
export XDG_RUNTIME_DIR=‘/run/user/1001’
export XDG_SESSION_ID=‘6’
export _=‘/bin/sh’
0/(NVIDIA Tegra X1 cc=5.3)
19595265/67108865,19595265/67108866,19595265/67108867,19595265/67108868,19595265/67108913,17825807/67108914,17825807/67108915,17825807/67108916,17825807/67108917,17825807/67108918,17825807/67108919,17825807/67108920,17825807/67108921,17825807/67108922,17825807/67108923,19595271/67108924,19595271/67108925,19595271/67108926,19595271/67108927,19595271/67108928,19595271/67108929,19595271/67108930,19595271/67108931,19595271/67108932,19595271/67108933,19595271/67108934,19595271/67108935,19595271/67108936,19595271/67108937,19595271/67108938,19595271/67108939,19595271/67108940,19595271/67108941,19595271/67108942,19595271/67108943,19595271/67108944,19595271/67108945,19595271/67108946,19595271/67108948,19595271/67108949,19595271/67108947,19595271/67108950,19595271/67108951,19595271/67108952,19595271/67108953,19595271/67108954,19595271/67108955,19595271/67108956,19595271/67108957,19595271/67108958,19595271/67108959,19595270/67108873,19595270/67108874,19595270/67108877,19595270/67108878,19595270/67108881,19595270/67108885,19595270/67108889,19595270/67108893,19595270/67108905,19595270/67108909,19595270/67108897,19595270/67108901,19595270/67108882,19595270/67108886,19595270/67108890,19595270/67108894,19595270/67108906,19595270/67108910,19595270/67108898,19595270/67108902,
2001,2002,2003,2004,2005,2006,2007,2008,2009,2010,2011,2012,2013,2014,2016,2017,2018,2019,2020,2022,2029,2030,2031,2032,2033,2034,2041,2048,2049,2050,2051,2052,2053,2054,2055,2056,2057,2058,2059,2060,2061,2062,2063,2064,2065,2066,2067,2068,2069,2070,2071,2074,2084,2085,2086,2087,2088,2089,2090,2091,2092,2093,2094,2095,2096,2097,2098,2099,2100,2104,2105,2106,2107,2108,2109,2110,2111,2112,2113,2114,2115,2116,2117,2118,2119,2120,2121,2122,2123,2124,2125,2126,2127,2128,2131,2132,2133,2134,2135,2136,2137,2138,2139,2140,2144,19136513,19136514,19136515,19136516,19136517,19136518,19136519,19136520,19136521,19136522,19136523,19136524,19136525,19136526,19136528,19136529,19136532,19136533,19136534,19136535,19136536,19136537,19136538,19136539,19136541,19136542,19136549,19136550,19136552,19136553,19595269,19595270,19595274,19595275,19595276,19595277,19595278,19595279,19595280,1073741827,1073741831,1073741832,
nvprof: NVIDIA (R) Cuda command line profiler
Copyright (c) 2012 - 2018 NVIDIA Corporation
Release version 10.0.166 (21)

It appears no errors is triggered there. Does that help you figure out where the problem is coming from ?

Hi nicolas.stoi,

The command output looks fine. We still suspect that this is something related to root privileges. Can you please try enabling root login and use root for your remote profiling?

Here are the steps to enable root user

1. sudo passwd root
2. sudo passwd -u root
3. vim /etc/ssh/sshd_config
   and set -PermitRootLogin yes
4. systemctl restart sshd

For more details please check
https://askubuntu.com/questions/44418/how-to-enable-root-login

Hi Ramesh,
Thanks for the answer.

Success! enabling a root user and using it for building/debugging/profiling works.

So it is indeed a privilege issue. Do you think there could be a fix to enable remote profiling for a standard user ?

Thanks much