AWS Greengrass Core on Nvidia Jetson TX2 Requirements

AWS Greengrass Core on Nvidia Jetson TX2 Requirements

I try to install AWS Greengrass Core v1.3 (instruction https://docs.aws.amazon.com/greengrass/latest/developerguide/module1.html#setup-filter.other)

After running the dependencies check, here is the result:

==========================Checking script dependencies==============================
The device has all commands required for the script to run.

========================Dependency check report for GGC v1.3=========================
System configuration:
Kernel architecture: aarch64
Init process: /lib/systemd/systemd
Kernel version: 4.4
C library: Ubuntu GLIBC 2.23-0ubuntu10
C library version: 2.23
Directory /var/run: Present
/dev/stdin: Found
/dev/stdout: Found
/dev/stderr: Found

--------------------------------Kernel configuration--------------------------------
Kernel config file: /proc/config.gz

Namespace configs:
CONFIG_IPC_NS: Enabled
CONFIG_UTS_NS: Enabled
CONFIG_USER_NS: Not enabled
CONFIG_PID_NS: Enabled

Cgroup configs:
CONFIG_CGROUP_DEVICE: Not enabled
CONFIG_CGROUPS: Enabled
CONFIG_MEMCG: Not enabled

Other required configs:
CONFIG_POSIX_MQUEUE: Not enabled
CONFIG_OVERLAY_FS: Not enabled
CONFIG_HAVE_ARCH_SECCOMP_FILTER: Enabled
CONFIG_SECCOMP_FILTER: Enabled
CONFIG_KEYS: Not enabled
CONFIG_SECCOMP: Enabled

------------------------------------Cgroups check-----------------------------------
Cgroups mount directory: /sys/fs/cgroup

Devices cgroup: Not enabled
Memory cgroup: Not enabled

----------------------------Commands and software packages--------------------------
SQLite version: 3.11.0
Python version: 2.7.12
NodeJS 6.10: Not found
Java 8: Not found
OpenSSL version: 1.0.2
wget: Present
realpath: Present
tar: Present
readlink: Present
basename: Present
dirname: Present
pidof: Present
df: Present
grep: Present
umount: Present

---------------------------------Platform security----------------------------------
Hardlinks_protection: Enabled
Symlinks protection: Enabled

-----------------------------------User and group-----------------------------------
ggc_user: Present
ggc_group: Present

------------------------------------Results-----------------------------------------
Note:
1. It looks like the kernel uses 'systemd' as the init process. Be sure to set the
'useSystemd' field in the file 'config.json' to 'yes' when configuring Greengrass core.

Missing optional dependencies:
1. The 'devices' cgroup is not enabled on the device.

Greengrass lambdas with Local Resource Access(LRA) configurations will not be allowed
to open device files.

2. Could not find the binary 'nodejs6.10'.

If NodeJS 6.10 or later is installed on the device, name the binary 'nodejs6.10' and
add its parent directory to the PATH environment variable. NodeJS 6.10 or later is
required to execute NodeJS lambdas on Greengrass core.

3. Could not find the binary 'java8'.

If Java 8 or later is installed on the device name the binary 'java8' and add its
parent directory to the PATH environment variable. Java 8 or later is required to
execute Java lambdas on Greengrass core.

Missing required dependencies:
1. The kernel is missing the following required configs:
CONFIG_USER_NS
CONFIG_CGROUP_DEVICE
CONFIG_MEMCG
CONFIG_POSIX_MQUEUE
CONFIG_OVERLAY_FS
CONFIG_KEYS


2. The 'memory' cgroup is not enabled on the device.
Greengrass will fail to set the memory limit of user lambdas.


----------------------------------Exit status---------------------------------------
Either the script failed to verify all dependencies or the device is missing one or
more of the required dependencies for Greengrass version 1.3.

Refer to the 'Errors' and 'Missing required dependencies' sections under 'Results'
for details.

The kernel is missing the following required configs:
CONFIG_USER_NS
CONFIG_CGROUP_DEVICE
CONFIG_MEMCG
CONFIG_POSIX_MQUEUE
CONFIG_OVERLAY_FS
CONFIG_KEYS

How do I enable these kernel configurations? Thanks.

Is this R28.1? Directions may differ somewhat versus older versions…not so much on building kernels, but instead on installing your changes. R28.1 is the most recent release version.

The official docs are here (you probably need to log in, but it’s free and not a spam source), click on the production release R28.1, then download “Documentation”:
https://developer.nvidia.com/embedded/linux-tegra

Much of what is in the official docs is for cross-compile. This was actually easier to do than native compile on some of the earlier releases, but you can basically just compile directly on your Jetson if using R28.1…it’s mostly the same, but you don’t need to set up any cross-tools or architecture environment variables…it’s ordinary kernel stuff.

If you download the driver package you will find “Linux_for_Tegra/source_sync.sh”. This is how you should get the kernel source code. You can copy this one file to your Jetson, e.g., put it in “/usr/local/src/” (create the directory if you need to…or go to some directory you find more convenient, e.g., on SD card). Then:

./source_sync.sh -k tegra-l4t-r28.1

This will create a “sources/” subdirectory. Assuming this as your directory the script goes something like this:

cd /usr/local/src

./source_sync.sh -k tegra-l4t-r28.1

export SRC=/usr/local/src/sources/kernel/kernel-4.4
export STAGE=/usr/local/src/build
export TEGRA_KERNEL_OUT=${STAGE}/kernel
export TEGRA_MODULES_OUT=${STAGE}/modules
export TEGRA_FIRMWARE_OUT=${STAGE}/firmware

rm -Rf ${STAGE:-/usr/local/src/build}/*
mkdir ${STAGE:-/usr/local/src/build}/kernel
mkdir ${STAGE:-/usr/local/src/build}/modules
mkdir ${STAGE:-/usr/local/src/build}/firmware

cp /proc/config.gz ${STAGE:-/usr/local/src/build/}
gunzip ${STAGE:-/usr/local/src/build}/config.gz
export CONFIG_LOCALVERSION=$(echo $(uname -r) | gawk -F'^[0-9]+<li>[0-9]+<li>[0-9]+' '{print $2}')
export CONFIG_LOCALVERSION_TEST="${CONFIG_LOCALVERSION}_test1"
sed -i s/^CONFIG_LOCALVERSION=\"\"/CONFIG_LOCALVERSION=\"${CONFIG_LOCALVERSION_TEST}\"/ ${STAGE}/config
egrep 'CONFIG_LOCALVERSION=' ${STAGE}/config

mv ${STAGE:-/usr/local/src/build}/config ${STAGE:-/usr/local/src/build}/config${CONFIG_LOCALVERSION_TEST}
cp ${STAGE:-/usr/local/src/build}/config${CONFIG_LOCALVERSION_TEST} ${TEGRA_KERNEL_OUT}/.config

cd ${SRC}
make mrproper
make O=$TEGRA_KERNEL_OUT clean
make O=$TEGRA_KERNEL_OUT nconfig
make -j4 O=$TEGRA_KERNEL_OUT Image
make -j4 O=$TEGRA_KERNEL_OUT modules
make -j4 O=$TEGRA_KERNEL_OUT dtbs
make O=$TEGRA_KERNEL_OUT modules_install INSTALL_MOD_PATH=$TEGRA_MODULES_OUT
make O=$TEGRA_KERNEL_OUT firmware_install INSTALL_FW_PATH=$TEGRA_FIRMWARE_OUT

Note though that the “CONFIG_LOCALVERSION” stuff is just my customization. If on your Jetson you run the command “uname -r” you’ll see something like “4.4.38-tegra”. The “4.4.38” is from the kernel source code verison, the “-tegra” was the “CONFIG_LOCALVERSION” at the time of the build. A kernel will always search for its modules in “/lib/modules/$(uname -r)/”. So if you build a new kernel and do not set this correctly, then your modules won’t be found. On the other hand, if you change some of the kernel’s integrated features, then you probably don’t want to use modules build with requirements which no longer exist due to integrated kernel feature changes.

Also know that the source_sync.sh version will likely append a “+” to the CONFIG_LOCALVERSION.

Examine “/proc/config.gz”…this is not a real file, it is a RAM image of the gzip compressed configuration of the currently running kernel. This is where you always want to start. The “make nconfig” step gives you an editor which lets you search for various symbols.

Consider the features you are wanting via this examination of config.gz:

zcat /proc/config.gz | egrep '(CONFIG_USER_NS|CONFIG_CGROUP_DEVICE|CONFIG_MEMCG|CONFIG_POSIX_MQUEUE|CONFIG_OVERLAY_FS|CONFIG_KEYS)'

If you set those to “m”, then you build them as modules and can just copy them into the correct module directory and “depmod -a” and probably reboot. If you build something with a “y”, then you build as an integrated feature of the kernel and will probably need a new CONFIG_LOCALVERSION and a new module directory.

If you want a faster build, then set the clocks up and turn on the Denver cores, and use “-j6” instead of “-j4” whereever you see the “-j#” in a build. Example:

sudo ~ubuntu/jetson_clocks.sh
...whatever steps...
make -j6 O=$TEGRA_KERNEL_OUT modules
...more steps...

Build your kernel source as non-root (no sudo). You may need to use root (sudo) to create directories and give yourself permission as user nvidia or user ubuntu if you use “/usr/local/src/” like I do.

NOTE: It might be best to test with a configuration exactly matching config.gz (other than setting CONFIG_LOCALVERSION to match) before trying to edit and add new configuration.

This a very detail instruction, I really appreciate it.

I followed through all steps 3 times, there was no error; but seems like I cannot include the necessary modules, after multiple reboot.

nvidia@tegra-ubuntu:~$ zcat /proc/config.gz | egrep '(CONFIG_USER_NS|CONFIG_CGROUP_DEVICE|CONFIG_MEMCG|CONFIG_POSIX_MQUEUE|CONFIG_OVERLAY_FS|CONFIG_KEYS)'
# CONFIG_POSIX_MQUEUE is not set
# CONFIG_CGROUP_DEVICE is not set
# CONFIG_MEMCG is not set
# CONFIG_USER_NS is not set
# CONFIG_OVERLAY_FS is not set
# CONFIG_KEYS is not set

/usr/local/src/build/kernel/.config has the necessary modules config.

nvidia@tegra-ubuntu:~$ cat /usr/local/src/build/kernel/.config | egrep '(CONFIG_USER_NS|CONFIG_CGROUP_DEVICE|CONFIG_MEMCG|CONFIG_POSIX_MQUEUE|CONFIG_OVERLAY_FS|CONFIG_KEYS)'
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_CGROUP_DEVICE=y
CONFIG_MEMCG=y
CONFIG_MEMCG_SWAP=y
CONFIG_MEMCG_SWAP_ENABLED=y
CONFIG_MEMCG_KMEM=y
CONFIG_USER_NS=y
CONFIG_OVERLAY_FS=m
CONFIG_KEYS=y

I’m using R28.1 distro.

nvidia@tegra-ubuntu:~$ cat /etc/*release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04 LTS"
# R28 (release), REVISION: 1.0, GCID: 9379712, BOARD: t186ref, EABI: aarch64, DATE: Thu Jul 20 07:59:31 UTC 2017

Am I missing something?

During this step is when you have the opportunity to save those configurations:

make O=$TEGRA_KERNEL_OUT nconfig

Keep in mind that building the kernel and modules doesn’t actually put it in place…the install is to “$TEGRA_KERNEL_OUT”. For example, the Image file is here:

find $TEGRA_KERNEL_OUT -name Image

Modules will be under:

$TEGRA_MODULES_OUT/lib/modules/some_future_uname-r/

If the file with the changes was not put in place you won’t see any difference…you’d be running off the old Image. If you started with config.gz and then did not edit via “make nconfig”, then you’d also have the same Image. Note that nconfig has a search for symbols function…look at the hot key list at the bottom, search for things like CONFIG_USER_NS.

In the case of the Image file you’d want to save it with an alternate name (the “uname -r” is the best extension) in “/boot”, e.g., “Image-4.4.38-tegra+” if this is the new “uname -r”, and then add a new entry to extlinux.conf…then test by using a serial console cable to pick this entry at boot. You could just edit the new entry to be default right at the start, but in case things don’t work it’s best to still have the old one in place. After you boot check your “uname -r” right away and see if you have “/lib/modules/$(uname -r)/”.

I was missing the step replacing the Image file in /boot/Image with /usr/local/src/build/kernel/arch/arm64/boot/Image

find $TEGRA_KERNEL_OUT -name Image

After replacing the original /boot/Image with the built image, I renamed it to /boot/Image-4.4.38-tegra+

then update /boot/extlinux/extlinux.conf:

TIMEOUT 30
DEFAULT primary

MENU TITLE p2771-0000 eMMC boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/Image-4.4.38-tegra+
      APPEND ${cbootargs} root=/dev/mmcblk0p1 rw rootwait rootfstype=ext4

I reboot and ran:

nvidia@tegra-ubuntu:~$ zcat /proc/config.gz | egrep '(CONFIG_USER_NS|CONFIG_CGROUP_DEVICE|CONFIG_MEMCG|CONFIG_POSIX_MQUEUE|CONFIG_OVERLAY_FS|CONFIG_KEYS)'
CONFIG_POSIX_MQUEUE=y
CONFIG_POSIX_MQUEUE_SYSCTL=y
CONFIG_CGROUP_DEVICE=y
CONFIG_MEMCG=y
CONFIG_MEMCG_SWAP=y
CONFIG_MEMCG_SWAP_ENABLED=y
CONFIG_MEMCG_KMEM=y
CONFIG_USER_NS=y
CONFIG_OVERLAY_FS=m
CONFIG_KEYS=y

I ran AWS Greengrass dependencies check again, there was no error and I was able to start the daemon correctly.

Thank you for the instructions.