Cross Compile for Jetson Xavier?

Hello,

I was hoping I could get some more detailed instructions on this topic?
I’ve seen this link: https://developer.nvidia.com/drive/learn/tutorial-xcompilation
as well as this one: https://devblogs.nvidia.com/cuda-jetson-nvidia-nsight-eclipse-edition/

Using nsight isn’t really an option in my case. I wish to be able to do this all from command line or invoke it via a bash script. I think I have all the correct support libraries and cross compile pieces downloaded, just not sure how exactly to connect the dots.

We have simple applications for resnet 50 exceution and inception SSD networks that use tensor RT 5 and cudNN 7 and cuda 10 that we currently compile and run to get performance metrics on both host hardware and on the target (Jetson Xavier).

How can I configure these custom to be cross compiled for jetson xavier on the x86 host machine?
Just updated the Jetson to Jetpack 4.2.

Hi,

There are some example in the JetPack installer.
Please check it for more information.

Ex. tegra_multimedia_api

CROSS_COMPILE ?= <b>aarch64-unknown-linux-gnu-</b>
AS             = $(AT) $(CROSS_COMPILE)as
LD             = $(AT) $(CROSS_COMPILE)ld
CC             = $(AT) $(CROSS_COMPILE)gcc
CPP            = $(AT) $(CROSS_COMPILE)g++
AR             = $(AT) $(CROSS_COMPILE)ar
NM             = $(AT) $(CROSS_COMPILE)nm
STRIP          = $(AT) $(CROSS_COMPILE)strip
OBJCOPY        = $(AT) $(CROSS_COMPILE)objcopy
OBJDUMP        = $(AT) $(CROSS_COMPILE)objdump
NVCC           = $(AT) $(CUDA_PATH)/bin/nvcc -ccbin $(filter-out $(AT), $(CPP))
...

Thanks.

What do you mean by Jetpack installer?
I used nvidia sdk manager to install 4.2 and all of it’s components.
But I cannot seem to find the example you listed.

Hi,

Okay, there are some change in the new sdk manager.

Would you mind to download our older JetPack-4.1.1?
[url]https://developer.nvidia.com/embedded/dlc/jetpack-l4t-4-1-1[/url]

This installer will cross-compile CUDA toolkit and MMAPI before pushing them into the device.
So you can check to the Makefile to get some information.

Thanks.

Thank you that was useful. But I’m still not fully able to get functionality. I can see all the downloads before the install phase but I am unable to cross compile a MMAPI sample on my machine.

Compiling: video_decode_csvparser.cpp
g++: error trying to exec ‘cc1plus’: execvp: No such file or directory
Makefile:48: recipe for target ‘video_decode_csvparser.o’ failed
make: *** [video_decode_csvparser.o] Error 1

I’m also not sure how the rules.mk + make file setup will be applicable to other existing applications. Or is that your original point that all cross compiling should be setup that way with those paramaters and rules.

I think maybe a more focused question might be more useful here. I am having an issue cross compiling a TensorRT sample. When I do make TARGET=aarch64 I get an error saying the CuDNN library cannot be found. I think this is an error in the Makefile.config

Ok so I also followed some instructions in the L4T MMAPI Reference to setup cross-compiling.

I made some progress thanks to this in cross compiling samples.

However, I still get these issues.

Linking: video_decode
ld: cannot find /home/drivehost/jetson/usr/lib/crt1.o: Permission denied
ld: cannot find /home/drivehost/jetson/usr/lib/crti.o: Permission denied
ld: cannot find crtbegin.o: No such file or directory
ld: video_decode_csvparser.o: Relocations in generic ELF (EM: 183)
ld: video_decode_csvparser.o: Relocations in generic ELF (EM: 183)
ld: video_decode_csvparser.o: Relocations in generic ELF (EM: 183)
video_decode_csvparser.o: error adding symbols: File in wrong format
Makefile:52: recipe for target ‘video_decode’ failed
make: *** [video_decode] Error 1

For these:

ld: cannot find /home/drivehost/jetson/usr/lib/crt1.o: Permission denied
ld: cannot find /home/drivehost/jetson/usr/lib/crti.o: Permission denied

Not sure what I’m doing wrong. The files are definitely present in the mounted Root FS.

Those files are owned by a different user. What do you see from:

sudo ls -ld /home/drivehost/jetson/usr/lib/
sudo ls -l /home/drivehost/jetson/usr/lib/crt*.o
whoami

Here is the output for the commands you gave me:

drivehost@drivehost:~$ sudo ls -ld /home/drivehost/jetson/usr/lib/
[sudo] password for drivehost:
drwxr-xr-x 2 root root 0 Apr 11 10:47 /home/drivehost/jetson/usr/lib/
drivehost@drivehost:~$ sudo ls -l /home/drivehost/jetson/usr/lib/crt*.o
-rwxr-xr-x 1 root root 1872 Apr 16 2018 /home/drivehost/jetson/usr/lib/crt1.o
-rwxr-xr-x 1 root root 1440 Apr 16 2018 /home/drivehost/jetson/usr/lib/crti.o
-rwxr-xr-x 1 root root 1016 Apr 16 2018 /home/drivehost/jetson/usr/lib/crtn.o
drivehost@drivehost:~$ whoami
drivehost
drivehost@drivehost:~$

Seems they are all owned by root. But even executing as root user gives me the same error output. Maybe I can try changing their owner and trying again.

I’m not sure that will work though, as it is a mounted FS of the Jetson Xavier over the network.

The ownership isn’t a problem since it is set to allow read and execute by anyone. These files typically should be owned by root and not writable by anyone except root. So the permission denied is just showing up here and not actually caused by those files being unaccessible.

So now we check to see what the files really are. What do you see from:

file /home/drivehost/jetson/usr/lib/crt*.o

Assuming you are cross compiling, then these should contain “ELF 64-bit LSB relocatable, ARM aarch64”. If it turns out these files are correct, and readable/executable, then it will probably be time to look at the cross-linker and make sure this is correct.

Hi, yeah I ran the file command earlier. This was the output:

drivehost@drivehost:~$ file /home/drivehost/jetson/usr/lib/crt*.o
/home/drivehost/jetson/usr/lib/crt1.o: executable, regular file, no read permission
/home/drivehost/jetson/usr/lib/crti.o: executable, regular file, no read permission
/home/drivehost/jetson/usr/lib/crtn.o: executable, regular file, no read permission

Nothing about “ELF 64-bit LSB relocatable, ARM aarch64”. These are the files directly from the mounted Jetson RootFS. I was hoping the nvidia provided instructions cleared up these issues, as we grant access to these files and create these symbolic links to them. I was looking specifically at the L4T Multimedia API Reference Documentation.

Perhaps one of the parent directories in the path is preventing read (the actual file was obviously readable by anyone with the “r-x” from “ls -l”). Or if this is a mounted partition, perhaps the mount was read-only. So do this in order and find out where it fails:

cd /home/drivehost/jetson/usr/lib/
ls -ld .

…you don’t care about owner or group names, but it should show “drwxr-xr-x.”.

Now repeat this until you get a non-readable directory, or else reach “/” (the last “r-x” would miss the “r” part if not readable):

cd ..
ls -ld .

Do this as the same user you are using for compiling.

If this all shows ok, or if this is a separate disk, then check this output to see if anything is “ro” instead of “rw” for ext4 partitions:

mount -t ext4

Hello. First of all thank you for your continued help!

Seems there are ok? Except they don’t have a “l” in the first spot. They have a “-”. That seemed strange to me.

-rwxr-xr-x 1 root root 1872 Apr 16 2018 crt1.o
-rwxr-xr-x 1 root root 1440 Apr 16 2018 crti.o
-rwxr-xr-x 1 root root 1016 Apr 16 2018 crtn.o

Outputs for ls -ld . on all directories seemed fine.

drivehost@drivehost:~/jetson/usr/lib$ ls -ld .
drwxr-xr-x 2 root root 0 Apr 11 10:47 .
drivehost@drivehost:~/jetson/usr/lib$ cd …
drivehost@drivehost:~/jetson/usr$ ls -ld .
drwxr-xr-x 2 root root 0 Mar 22 14:05 .
drivehost@drivehost:~/jetson/usr$ cd …
drivehost@drivehost:~/jetson$ ls -ld .
drwxr-xr-x 2 root root 0 Mar 22 11:31 .
drivehost@drivehost:~/jetson$ cd …
drivehost@drivehost:~$ ls -ld .
drwxr-xr-x 51 drivehost drivehost 4096 Apr 12 16:22 .
drivehost@drivehost:~$ cd …
drivehost@drivehost:/home$ ls -ld .
drwxr-xr-x 3 root root 4096 May 17 2018 .
drivehost@drivehost:/home$ cd …
drivehost@drivehost:/$ ls -ld .
drwxr-xr-x 26 root root 4096 Apr 9 11:25 .

Mount also seemed ok?

drivehost@drivehost:~/jetson/usr/lib$ mount -t ext4
/dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)

Also, I think this issue might also be related somehow, looking at the cuda-10 folder.

drivehost@drivehost:~/jetson/usr/local/cuda-10.0$ ls
bin extras lib64 NsightCompute-1.0 nvvm samples targets version.txt
doc include LICENSE nvml README share tools
drivehost@drivehost:~/jetson/usr/local/cuda-10.0$ cd lib64/
drivehost@drivehost:~/jetson/usr/local/cuda-10.0/lib64$ ls
ls: reading directory ‘.’: Permission denied
drivehost@drivehost:~/jetson/usr/local/cuda-10.0/lib64$ ls -ld .
drwxr-xr-x 2 root root 0 Mar 22 13:58 .
drivehost@drivehost:~/jetson/usr/local/cuda-10.0/lib64$

contents of Lib64 are permission locked. Perhaps there’s an issue with the setup steps, or I need to change permissions on the target itself for the rootFS?

It seems the output for ls -ld . on the target is fine as well, showing"drwxr-xr-x.".

nvidia@nvidia-jetson:~$ ls
Automated_Tests Downloads Music Public Videos
Desktop examples.desktop pbm_test_stand tegrastats.log VisionWorks-SFM-0.90-Samples
Documents memory_test Pictures Templates
nvidia@nvidia-jetson:~$ ls -ld .
drwxr-xr-x 24 nvidia nvidia 4096 Apr 11 10:46 .
nvidia@nvidia-jetson:~$ cd …
nvidia@nvidia-jetson:/home$ ls -ld .
drwxr-xr-x 3 root root 4096 Mar 22 13:36 .
nvidia@nvidia-jetson:/home$ cd …
nvidia@nvidia-jetson:/$ ls -ld .
drwxr-xr-x 22 root root 4096 Mar 22 11:31 .
nvidia@nvidia-jetson:/$

I’m not sure which “l” you mean from:

Seems there are ok? Except they don't have a "l" in the first spot. They have a "-". That seemed strange to me.

Can you give an example of which “l”?

I thought I might just give a recap of what you’ve posted on permissions so far for things which are likely ok and not a problem:

-rwxr-xr-x 1 root root 1872 Apr 16 2018 crt1.o
-rwxr-xr-x 1 root root 1440 Apr 16 2018 crti.o
-rwxr-xr-x 1 root root 1016 Apr 16 2018 crtn.o
# All above ok because anyone can read or execute the files. Perhaps not parent directories though.
drivehost@drivehost:~/jetson/usr/lib$ ls -ld .
drwxr-xr-x 2 root root 0 Apr 11 10:47 .
# Above indicates all ok because the directory allows anyone to read or execute.
drivehost@drivehost:~/jetson/usr/lib$ cd ..
drivehost@drivehost:~/jetson/usr$ ls -ld .
drwxr-xr-x 2 root root 0 Mar 22 14:05 .
# Above indicates all ok because the directory allows anyone to read or execute.
drivehost@drivehost:~/jetson/usr$ cd ..
drivehost@drivehost:~/jetson$ ls -ld .
drwxr-xr-x 2 root root 0 Mar 22 11:31 .
# Above indicates all ok because the directory allows anyone to read or execute.
drivehost@drivehost:~/jetson$ cd ..
drivehost@drivehost:~$ ls -ld .
drwxr-xr-x 51 drivehost drivehost 4096 Apr 12 16:22 .
# Above indicates all ok because the directory allows anyone to read or execute.
drivehost@drivehost:~$ cd ..
drivehost@drivehost:/home$ ls -ld .
drwxr-xr-x 3 root root 4096 May 17 2018 .
# Above indicates all ok because the directory allows anyone to read or execute.
drivehost@drivehost:/home$ cd ..
drivehost@drivehost:/$ ls -ld .
drwxr-xr-x 26 root root 4096 Apr 9 11:25 .
# Above indicates all ok because the directory allows anyone to read or execute.

On the last directory which is ok this seems to be the root of the file system. Any denial of permission is not due to the above permissions. There may be an exception if SElinux is getting involved.

I think you’ve found the problem with this excerpt from your above post:

drivehost@drivehost:~/jetson/usr/local/cuda-10.0/lib64$ ls
ls: reading directory '.': Permission denied
drivehost@drivehost:~/jetson/usr/local/cuda-10.0/lib64$ ls -ld .
drwxr-xr-x 2 root root 0 Mar 22 13:58 .

…which should be readable and executable by all if and only if the directory “~/jetson/usr/local/cuda-10.0/” is also readable by all. Since you are operating in a “~” (user home directory), then it is ok if only that user plus root can read everything, so I have to wonder if somewhere in that directory structure is something the current user can’t read, or if something in the build is using another user than the one owning that home directory. If for example you have this in some user home other than nvidia, and then user nvidia is used to access this, then it would fail. On the other hand, it shouldn’t matter which user it is home for if that user is the one doing the access.

Like before, you will need to run “ls -ld .” at each directory leading up to “~/jetson/usr/local/cuda-10.0/lib64/” until you find one that the user running things isn’t given permission to read. Notice that I didn’t even mention “execute” because the error is for “read”. Once “read” is fixed, then “execute” might also show up as an issue.

Hello,

So for these files:

-rwxr-xr-x 1 root root 1872 Apr 16 2018 crt1.o
-rwxr-xr-x 1 root root 1440 Apr 16 2018 crti.o
-rwxr-xr-x 1 root root 1016 Apr 16 2018 crtn.o

On the target if I get that same output the first dash in “-rwxr-xr-x” is an l so like this: “lrwxr-xr-x”
Because we create the symbolic links to those files. But I didn’t think that was related to permissions of reading those files.

Also I checked all the other directories inside the cuda-10.0 folder in the mounted rootFS. They are all fine and I can read them. EXCEPT for include and lib64.

drivehost@drivehost:~/jetson/usr/local/cuda-10.0$ cd include/
drivehost@drivehost:~/jetson/usr/local/cuda-10.0/include$ ls
ls: reading directory '.': Permission denied
drivehost@drivehost:~/jetson/usr/local/cuda-10.0/include$ ls -ld .
drwxr-xr-x 2 root root 0 Mar 22 13:58 .
drivehost@drivehost:~/jetson/usr/local/cuda-10.0/include$ cd ..
drivehost@drivehost:~/jetson/usr/local/cuda-10.0$ ls -ld .
drwxr-xr-x 2 root root 0 Mar 22 13:58 .
drivehost@drivehost:~/jetson/usr/local/cuda-10.0$ cd lib64/
drivehost@drivehost:~/jetson/usr/local/cuda-10.0/lib64$ ls -ld .
drwxr-xr-x 2 root root 0 Mar 22 13:58 .
drivehost@drivehost:~/jetson/usr/local/cuda-10.0/lib64$ ls
ls: reading directory '.': Permission denied
drivehost@drivehost:~/jetson/usr/local/cuda-10.0/lib64$

As for every directory leading up to it, this is the output for that:

drivehost@drivehost:/$ ls -ld .
drwxr-xr-x 26 root root 4096 Apr  9 11:25 .
drivehost@drivehost:/$ cd home/
drivehost@drivehost:/home$ ls -ld .
drwxr-xr-x 3 root root 4096 May 17  2018 .
drivehost@drivehost:/home$ cd drivehost/
drivehost@drivehost:~$ ls -ld .
drwxr-xr-x 51 drivehost drivehost 4096 Apr 15 14:14 .   
drivehost@drivehost:~$ cd jetson/
drivehost@drivehost:~/jetson$ ls -ld .
drwxr-xr-x 2 root root 0 Mar 22 11:31 .
drivehost@drivehost:~/jetson$ cd usr/
drivehost@drivehost:~/jetson/usr$ ls -ld .
drwxr-xr-x 2 root root 0 Mar 22 14:05 .
drivehost@drivehost:~/jetson/usr$ cd local
drivehost@drivehost:~/jetson/usr/local$ ls -ld .
drwxr-xr-x 2 root root 0 Mar 22 13:58 .
drivehost@drivehost:~/jetson/usr/local$ cd cuda-10.0/
drivehost@drivehost:~/jetson/usr/local/cuda-10.0$ ls -ld .
drwxr-xr-x 2 root root 0 Mar 22 13:58 .

“Notice that I didn’t even mention “execute” because the error is for “read”. Once “read” is fixed, then “execute” might also show up as an issue.”

Yeah… I guess I’ll cross that bridge when I get to it.

In the case of symbolic links the theory is that the link itself doesn’t set permissions, and so looking directly at the link it should show full access. The actual effective permission depends on what the link points at. This in turn is a chain of permissions.

Just to illustrate, if you have “/home/someone/link” and it points to “/usr/local/custom/file”, then to use the “file” application via the symbolic link you will need:

  • Permission to reach link, e.g., be "someone" with permission to that directory containing "link".
  • Permission to reach and execute "/usr", "/usr/local", "/usr/local/custom", and "/usr/local/custom/file".

You listed this, and I don’t see “l”:

-rwxr-xr-x 1 root root 1872 Apr 16 2018 crt1.o
-rwxr-xr-x 1 root root 1440 Apr 16 2018 crti.o
-rwxr-xr-x 1 root root 1016 Apr 16 2018 crtn.o

For cases where it is a symbolic link, then you have to use “ls -l” on the link, not the object pointed to by the link. Herein is a very subtle bit of trivia you need to know: If you specify a link with “link/”, then ls shows what the link points at for the case of a directory…but for specification without the trailing “/”, then “ls -l” of “link” (e.g., “ls -l link” instead of “ls -l link/”) is the link itself.

Find out where the link points. Examine permission of the containing parent directories of the link location from root to the final subdirectory. Then examine the permission of any final file pointed at by that link. If you cd to each directory in the sequence of where the link points, one by one, as the user which is being denied, and run “ls -l” each time, then eventually you will find a non-readable directory (or the end file).

The convenience of the symbolic link is to hide the actual path to the file as a kind of shortcut. The procedure for finding a bad permission implies you now need to know permissions are correct for the path to the shortcut, plus the full path of where the shortcut leads.