TX1, using overlayroot for read-only filesystem

I’m interested in setting up my TX1’s root filesystem to be read-only. I’ve found a package called “overlayroot” (see references below) that sounds like it will do exactly what I want. I’ve installed and configured it (with the simplest “tmpfs” settings) on my TX1 hardware, but I can’t seem to get it working correctly… I’m still able to create and edit new files under /, and my changes persist even after a reboot.

Has anyone had any experience with overlayroot, or knows of another way to make the filesystem read-only?

[1] From the Canyon Edge: Introducing overlayroot -- overlayfs + dmcrypt!
[2] http://domas.monkus.lt/posts/2015-02-17-overlayroot/
[3] Protecting the Root Filesystem on Ubuntu with Overlayroot

I’ve never worked with, nor configured overlayfs. However, I’d start by seeing what output is returned from “mount” and “df -T”. Search dmesg as well (e.g., “dmesg | less”) for anything related to mount or overlayfs. If there is a kernel module involved, see if lsmod shows the module as loaded.

I’m trying to do the same on TX2. I compiled and installed the “overlay” module on Jetson TX2 L4T27.1 kernel 4.4.15 following this tutorial:
[url]http://www.jetsonhacks.com/2017/03/25/build-kernel-and-modules-nvidia-jetson-tx2/[/url]
I was able to load the module at boot by adding it in /etc/modules, /etc/initramfs-tools/initramfs.conf and updating initramfs:

sudo update-initramfs -u.

now lsmod gives:

Module Size Used by
fuse 89120 3
bnep 15477 2
bluetooth 552900 7 bnep
bcmdhd 7637079 0
pci_tegra 72709 0
overlay 36906 0
bluedroid_pm 13874 0

I edited the /etc/overlayroot.conf with the simplest “tmpfs” setting but nothing happens; the “mount” command output remains unchanged:

/dev/mmcblk0p1 on / type ext4 (rw,relatime,data=ordered)
proc on /proc type proc (rw,relatime)
sysfs on /sys type sysfs (rw,relatime)
none on /dev type devtmpfs (rw,relatime,size=7969536k,nr_inodes=1992384,mode=755)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/debug type cgroup (rw,nosuid,nodev,noexec,relatime,debug)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=804240k,mode=700,uid=1000,gid=1000)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
tmpfs on /run/user/1001 type tmpfs (rw,nosuid,nodev,relatime,size=804240k,mode=700,uid=1001,gid=1001)

I also tried to append overlayroot=tmpfs to kernel parameters, in file /boot/extlinux/extlinux.conf

On R28.1 there is probably no difference between a TX1 and TX2, but there could be. The article you pointed at shows as R27.1, with a pointer to the R28.1 version. Are you using R27.1? This would be quite different for TX1 versus TX2 (a TX1 can’t use R27.1).

One thing you may want to do is verify your parameter made it to the command line. Examine this:

cat /proc/cmdline

FYI, by default R28.1 has an initrd file, but its size is zero and it is not used. If you made changes to the initrd it is unlikely to do as expected.

Did you ever get this working?

I got this working.

I had to do

update-initramfs -c -k $(uname -r)

I guess the initramfs might not have been matching my kernel version or something. Either way it wasn’t using it.

After doing this, I could update /boot/extlinux/extlinux.conf to point it to the new initrd just created.

With that, the boot flag, overlayfs in the kernel and overlayroot configured, my overlayroot was working.

In R28.1 initramfs is not used. This would have applied to an R24.x install. R28.1 would change many details.

Hi!
Does any body solved this problem about how to config the overlay fs to protect the rootfs?

I use this cmd

sudo ./flash -S 29318MiB jetson-tx2 mmcblk0p1

to part eth emmc to two parts, one is for lower lay, to protect the root file system, the other is for upper lay,

Hi, linuxdev
what you mean is that “update-initramfs -c -k $(uname -r)” can not use in TX2 R28.1?
When I run the cmd, I get the errors below!

sudo update-initramfs -c -k $(uname -r)
[sudo] password for nvidia: 
update-initramfs: Generating /boot/initrd.img-4.4.38-tegra
cryptsetup: WARNING: failed to detect canonical device of /dev/root
cryptsetup: WARNING: could not determine root device from /etc/fstab
Warning: /sbin/fsck.rootfs doesn't exist, can't install to initramfs, ignoring.

The init ramfs is not used by default in the later releases. This was only used in earlier releases. I’m not sure what the requirements would be to use this on releases newer than R24.x, but what is there by default in releases newer than R24.x is size 0 and not named in extlinux.conf.

Certainly default tools for an initrd do not understand the boot requirements of a Jetson. Much of this is custom. I highly doubt update-initramfs has any possibility of working with a Jetson, and that you’d have to manually create the initrd. Likely the initrd generated would be a good starting place, and then this would need editing.

I have not tried to put an initrd into a release in a long time. Keep in mind that if you have a feature needed for booting (such as needing ext4…or in your case overlay fs), then simply putting the feature in the kernel as a non-module (“=y” in config, not “=m”) removes the need for initrd. Why use an initrd if you don’t need it?

I think this linux - How to use OverlayFS to protect the root filesystem? - Unix & Linux Stack Exchange may a useful way to config the overlay fs. But this idea will use init ramfs to make a change when the system start. As now, we can’t use initramfs, so can you recommend some blog or source to change the mount point use overlay when system booting?

I have not tried to do so, I lack overlay rootfs experience.

Have you configured the kernel with “CONFIG_OVERLAY_FS=y”? See:

sudo zcat | /proc/config.gz | egrep OVERLAY_FS

If it says “=m”, then probably just rebuilt the kernel with “=y”. With this you would still need to do normal configuration for overlayfs, but you wouldn’t need to do it with an initrd.

Yes, I have set the “CONFIG_OVERLAY_FS=y” . I will try some other way

hey, I just also want to test overlayfs on tx2, lucky to search information on this page,

I also not succeed to let overayroot work as expected.

Since I have not worked on overlayfs, I’ll ask about what I would do next if it were me doing it: Have you tried to create a non-rootfs overlayfs? I don’t know if it is made more simple or if it is made more complex by using overlayfs on a subdirectory like “/home” by itself or or if entire file systems need to be used, maybe someone could comment. Just as an alternate test, if running rootfs on eMMC, perhaps you could test how to mount an SD card partition on “/usr/local/mnt/” and set that as overlayfs. Once either of those two test cases work it should be much easier to work on rootfs as a whole.

Note: Overlayfs is understood by the Linux kernel, but not by U-Boot. All overlayfs content would occur only after the Linux kernel is running and file system mount has taken place. This means it is very likely you will see a purely read-write mount initially, or read-only, and that the overlay won’t exist until right after that.

thanks for your replay~

Hi linuxdev
As you said, the overlayfs is understood by the Linux kernel , and the examples from the net use the initramfs to mount the overlayfs on rootfs after the kernel loaded. And the R28.1 version delete the initramfs, so can you show me how to make a change of mounting the rootfs.(Which file can’t be configured to change the mout fs)

Does the /boot/extlinux/extlinux.conf works when make a changes?(I don’t whether it works?

In some linux version (like OpenWrt), it use preinit to mount overlayfs after the kernel start up . So I want to no if there is a init script
in r28.1 for users to modify the mount system?

“extlinux.conf” won’t honor everything it used to use. The INITRD might work, but its content would probably differ from any other Linux system (and perhaps even from earlier L4T releases). The APPEND for kernel parameters works, but there is a big “gotcha” there…the parameters passed in via “${cbootargs}” are something inherited from previous stages and are actually required in earlier boot stages (thus a change in “extlinux.conf” might appear to show up correctly in “/proc/cmdline”, but the system will probably not behave correctly since earlier boot stages not only need this information, but might edit this between stages…older L4T releases didn’t pass this around in pre-boot stages and didn’t edit).

I couldn’t tell you exactly what to change in the initrd, but do you have an example version which you have tested? You can probably name it something like “initrd.img” and then use gzip to attach it to an existing forum post (hover your mouse over the quote icon in the upper right of an existing post and a paper clip icon will show up which allows you to attach a file). If the forum doesn’t like a particular file name extension like “.gz” then there is bzip2 or zip or 7z. Similar for the file contained if the forum doesn’t like the file name…it could be named anything, e.g., initrd.bin, initrd.dat (you might have to try a couple of names for the forum to allow attaching the file).

I might be able to see what is in the initrd you are testing and find out what needs to change. If you want to view what is inside any initrd or customize it, then this will be of interest:
[url]https://devtalk.nvidia.com/default/topic/1011027/jetson-tk1/jetson-tx1-initrd/post/5157554/#5157554[/url]

Also, if you have tested overlayfs on a non-rootfs mount or directory, let me know what configuration or steps you used to get that working (remember, I haven’t worked on overlayfs before so it’ll be new for me too).

@linuxdev, how to add a shell commander in the /boot/extlinux/extlinux.conf, or how to execute shell script before the rootfs mounted on /dev/mmcblk0p1 on TX2 ,R28.2.1 ? (change some init code?)

Does this work when I want building the initramfs into kernel in TX2 R28.2.1?

No idea on “shell commander” in U-Boot, I only know about the serial console access (which gets a shell prompt). On the other hand, within the U-Boot shell you can see environment variables (“printenv”). Some of those variables are used as simple macros and expanded to control boot. Here is a subset of some relevant environment variables (this is from a TX2 under R28.2, not a TX1, but most of this will be an exact match to a TX1 on the same release…I expect R28.2 and R28.2.1 to also be a nearly exact match…the steps prior to U-Boot are where most differences will occur and once reaching U-Boot the environment should be much simpler):

Tegra186 (P2771-0000-500) # printenv
...
boot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}
...
boot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}extlinux/extlinux.conf
...
boot_prefixes=/ /boot/
...
boot_targets=mmc1 mmc0 usb0 pxe dhcp 
<u><i><b>bootcmd=run distro_bootcmd</b></i></u>
...
bootcmd_mmc0=setenv devnum 0; run mmc_boot
bootcmd_mmc1=setenv devnum 1; run mmc_boot
bootcmd_pxe=run boot_net_usb_start; run boot_net_pci_enum; dhcp; if pxe get; then pxe boot; fi
...
bootdelay=2
...
cbootargs=root=/dev/mmcblk0p1 rw rootwait console=ttyS0,115200n8 console=tty0 OS=l4t fbcon=map:0 net.ifnames=0 memtype=0 video=tegrafb no_console_suspend=1 earlycon=uart8250,mmio32,0x03100000 nvdumper_reserved=0x2772e0000 gpt tegra_fbmem2=0x140000@0x969ec000 lut_mem2=0x2008@0x969e9000 tegraid=18.1.2.0.0 tegra_keep_boot_clocks maxcpus=6 boot.slot_suffix= boot.ratchetvalues=0.1.1 androidboot.serialno=0334916010131 bl_prof_dataptr=0x10000@0x277040000 sdhci_tegra.en_boot_part_access=1
...
distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done
...
mmc_boot=if mmc dev ${devnum}; then setenv devtype mmc; run scan_dev_for_boot_part; fi
...
scan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;run scan_dev_for_efi;
scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist $defaultdevplist; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done
...
scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}extlinux/extlinux.conf; then echo Found ${prefix}extlinux/extlinux.conf; run boot_extlinux; echo SCRIPT FAILED: continuing...; fi
scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; done
...

If you need to change something in what is booted, and if U-Boot understands relevant file system types and partitioning, then you can simply edit those environment variables. In particular, “bootcmd” environment variable is what starts when you enter the manual command “boot” from this prompt. This causes expansion of “distro_bootcmd”, and any edit to “distro_bootcmd” or its child expansions may do what you want.

You have two choices:

1. Causing a change from U-Boot: What you need to look out for is that U-Boot understands some of the file system types and partitioning, but not everything which Linux itself understands. Mostly I will guess that instead of looking for extlinux.conf you probably want to add a custom initrd and search for that. Upon finding that, then you will extract the content (which U-Boot knows how to do) and execute this instead of the extlinux.conf content of the "/boot/exltinux/" directory.

   Note: U-Boot can unpack an initrd into ext2 or ext4 formatted RAM, so it has no trouble reading modules and kernels and init script content which isn’t on the eMMC’s rootfs.

2. Causing a change from Linux: Change extlinux and/or "/etc/fstab" (or whatever it is which determines overlayroot) to do so.

   Implication: Initial mount of rootfs will reach read-write, and then throw the overlay on early during boot. You would still achieve an overlayroot, but something very early in boot could possibly get around this.

One of the reasons I ask about posting the method of creating a non-rootfs overlayfs is so I might duplicate what you are doing (I’m being lazy, having something to start with instead of reading all of the overlayfs docs would speed up the process). Same for wanting to see any initrd you have tested with…I can unpack it, use it, see what errors go on, and then edit to try something new. Since I have not worked specifically on overlayfs this would make it a lot easier to get to whatever point you are currently at. I expect I can read and work with initrd content without trouble, but what in terms of specific requirements to overlayfs I don’t know.

Btw, you can think of an initrd file as a combination of initial ramdisk and the content to populate it. It is just a cpio tar archive created under ext2 or ext4. This URL (from above) tells you how to open up an initrd, edit it, and repackage with the edits:
https://devtalk.nvidia.com/default/topic/1011027/jetson-tk1/jetson-tx1-initrd/post/5157554/#5157554