I used the sample rootfs, and the drivers package to create a rootfs for my board. When I boot it with the supplied kernel, I consistently get:
tegra-xhci tegra-xhci: failed to init firmware from filesystem: tegra21x_xusb_firmware
After this, none of the USB stuff seems to be working. Does anyone have an idea what’s going wrong here?
Did you run the “apply_binaries.sh” step? If so, was it done using sudo? Similarly, was the sample rootfs unpacked sudo?
Yes, yes, and yes.
I booted from the image on the flash storage that came with the board, extracted the driver package, mounted an ssd in that extracted tree, extracted the sample rootfs as root, and ran apply binaries. Then copied the kernel and the dtbs to the mmc, and booted that with root=/dev/sda1 . That doesn’t load that firmware. Funnily enough, if I use the new kernel with the fs on the mmc (which has aarch32 userland binaries instead of the aarch64 on the new sample rootfs), it works fine and it does load the firmware.
Any hints? Or should I really use the flash storage for the filesystem? I would prefer not, so I can also boot into aarch32 userland when I need to.
Your basic procedure is sound, but I’m wondering if the dtb file was unavailable to read (which can happen for any number of reasons). The files actually used from “/boot” would be those on eMMC, the SSD would be ignored until rootfs mount time (at rootfs mount time “/boot” is no longer required).
For debugging you should be able to mount “/dev/mmcblk0p1” somewhere (e.g., on “/mnt”) and view the “/mnt/boot/extlinux/extlinux.conf” there. For the boot entry you are booting, what is the FDT key/value pair? Does the named FDT file exist on the eMMC version? Is that FDT file copied in to eMMC from the matching root file system currently on the SSD?
Note that R23.2 is also 32-bit and had several fixes. If needed you could flash R23.2 and then do the extlinux.conf edit without ever touching your current SSD install (though you would still have to copy the files to eMMC “/boot” and edit extlinux.conf like before).
I’m having the same issue as “tegra-xhci tegra-xhci: failed to init firmware from filesystem” for USB3. I did some debugging in kernel/drivers/base/firmware_class.c and found that fw_get_filesystem_firmware function fails to open the firmware file from /lib/firmware/ whereas it can load other firmware files from /lib/firmware/tegra21x. I’m wondering where is the issue now and how to move forward?
Thanks,
Hakim
The firmware files are added via the “apply_binaries.sh” step if manually flashing. Those files are added by the “~/NVIDIA-INSTALLER/installer.sh” on a freshly arrived Jetson. Both require sudo to install correctly. Has your Jetson been flashed, and if so, what version? (see “head -n 1 /etc/nv_tegra_release”). Also, run this to validate some of the nVidia files:
sha1sum -c /etc/nv_tegra_release
We don’t same this issue with 24.1 but with 24.2. I did everything as root and I also think this issue is related to building and flashing probably. I followed same build and flashing procedure that I did for 24.1.
Following is the output of the two commands
ubuntu@tegra-ubuntu:~$ head -n 1 /etc/nv_tegra_release
ubuntu@tegra-ubuntu:~$ sha1sum -c /etc/nv_tegra_release
/usr/lib/aarch64-linux-gnu/tegra/libnvosd.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_image.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvomx.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmedia.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_utils.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libglx.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libscf.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvexif.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvrm_gpu.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvfnetstorehdfx.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmm_parser.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvrm.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmm_contentpipe.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvos.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvtnr.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvfnet.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvavp.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite_video.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvfnetstoredefog.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvodm_imager.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvtvmr.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvdc.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libtegrav4l2.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmm.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvapputil.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcameratools.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcam_imageencoder.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnveglstream_camconsumer.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmm_utils.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvomxilclient.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libargus.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvwinsys.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libargus_socketclient.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnveglstreamproducer.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvll.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libargus_socketserver.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcolorutil.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvddk_2d_v2.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvtestresults.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcamerautils.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvcamlog.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvparser.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so: OK
/usr/lib/aarch64-linux-gnu/tegra/libnvmmlite.so: OK
/usr/lib/aarch64-linux-gnu/libv4l/plugins/libv4l2_nvvidconv.so: OK
/usr/lib/aarch64-linux-gnu/libv4l/plugins/libv4l2_nvvideocodec.so: OK
When naming an alternate root partition (e.g., sda1) some of the files will depend on the original eMMC partition, while others will migrate to the new partition. During u-boot’s life anything u-boot loads prior to handing off to the kernel relies on mmcblk0p1; it isn’t until the kernel executes that migration of required files shifts to the new partition (the kernel image itself must be on eMMC). I’m betting you just didn’t have the firmware on the right partition. DTB files named in extlinux.conf (or generally any file named in extlinux.conf) must be on eMMC…files referred to by the kernel after kernel execution begins would need to be on sda1. I believe the “/lib/firmware” files will need to be on sda1, but it wouldn’t hurt to make sure both eMMC and sda1 “/lib/firmware” have all relevant files.
I even coped the firmware file into /boot directory and it did not work. The boot system tries to load xhci-tegra driver at around 2sec and the file system is up after 11 seconds. I hacked the driver to load firmware after such time and found it’s able to load firmware file. Not sure whether this is the best way but I may have to live with this for 24.2.
Hakim
What file system type is the sda partition? “df -T” should list all types.
Also, the “.dtb” file is always on eMMC “/boot”, as named in extlinux.conf, so long as flash was to “mmcblk0p1”…regardless of whether the boot entry later points to sda1 (I assume this was not flashed to mmcblk1p1, but only to mmcblk0p1). The “/lib/firmware” should be from sda1, I’m not sure why it can’t read that unless there is a file system or permission issue. Whichever file it claims to not read from “/lib/firmware”, do “ls -l” on that file and check permissions and existence.
Note that after boot you should be able to mount eMMC to examine it. For example, “sudo mount /dev/mmcblk0p1 /mnt” would mount this on “/mnt”. You could then compare on the live system what is seen in “/lib/firmware” and “/mnt/lib/firmware”.
Also, what is the output of “lsblk”?
I’ve everything on flash. Following are the output of two commands
ubuntu@tegra-ubuntu:~$ df -T
Filesystem Type 1K-blocks Used Available Use% Mounted on
/dev/root ext4 14318640 3352660 10215596 25% /
devtmpfs devtmpfs 1851416 0 1851416 0% /dev
tmpfs tmpfs 2046212 168 2046044 1% /dev/shm
tmpfs tmpfs 2046212 9472 2036740 1% /run
tmpfs tmpfs 5120 4 5116 1% /run/lock
tmpfs tmpfs 2046212 0 2046212 0% /sys/fs/cgroup
tmpfs tmpfs 409244 56 409188 1% /run/user/1000
ubuntu@tegra-ubuntu:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
mmcblk0rpmb 179:16 0 4M 0 disk
mmcblk0 179:0 0 14.7G 0 disk
ââmmcblk0p1 179:1 0 14G 0 part /
ââmmcblk0p2 179:2 0 2M 0 part
ââmmcblk0p3 179:3 0 4M 0 part
ââmmcblk0p4 179:4 0 2M 0 part
ââmmcblk0p5 179:5 0 6M 0 part
ââmmcblk0p6 179:6 0 4M 0 part
ââmmcblk0p7 179:7 0 6M 0 part
ââmmcblk0p8 179:8 0 2M 0 part
ââmmcblk0p9 179:9 0 2M 0 part
ââmmcblk0p10 179:10 0 20M 0 part
ââmmcblk0p11 179:11 0 64M 0 part
ââmmcblk0p12 179:12 0 64M 0 part
ââmmcblk0p13 179:13 0 4M 0 part
ââmmcblk0p14 179:14 0 2M 0 part
ââmmcblk0p15 179:15 0 6M 0 part
ââmmcblk0p16 259:0 0 6M 0 part
ââmmcblk0p17 259:1 0 2M 0 part
ââmmcblk0p18 259:2 0 496M 0 part
df-lsblk.pdf (95.6 KB)
Sorry, I just noticed it was the original poster who said sda1. I thought this was involving an alternate root partition. Can you post the content of “/boot/extlinux/extlinux.conf”, and output of “ls /boot”? Additionally, the logging which leads up to “tegra-xhci tegra-xhci: failed to init firmware from filesystem”…a bit of context as to what went on right before it got there would be useful.
I’m going to attach in pdf since copy/paste removes formatting. Just let you know, I tried dtb of reference board tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb with same result. We add a few entries in device tree, therefore, we use our own dtb.
lsboot.pdf (108 KB)
Things are configured to read mono-tegra210-jetson-tx1-CTI-ASG002.dtb from u-boot. Do you have serial console so you can see if u-boot reads this? Probably it does, but it needs to be confirmed, and serial console is the only way to see this during boot loader stage. See:
[url]http://elinux.org/Jetson/TX1_Serial_Console[/url]
Without a serial console I’d guess this file is read, but I wouldn’t count on it since you have firmware issues.
Next, a log is needed of not just the error message on firmware, but what came before this.
Keep in mind that even if firmware files in “/lib/firmware” are found, they won’t necessarily work unless the mono-tegra210-jetson-tx1-CTI-ASG002.dtb sets up everything correctly for that version of firmware. Since I’ve never seen that version of the dtb, can you give an idea of roughly how this dtb differs from stock?
Also, if firmware and dtb are correct relative to each other, then kernel drivers using the hardware also need to understand whatever version those files provide. There are a lot of values a dtb file can change and it won’t break a driver, but the “/lib/firmware” files most likely require a driver change as well if anything significant changes in that firmware (the dtb could be analogous to variables passed to a function, the firmware itself could be considered similar to an API…the driver operates on that API, so if the API changes so too must the driver change). Logs might help better determine why firmware files didn’t load, or if instead they loaded but failed.
As I mentioned, I also tried stock dtb tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb with same result; I’ve analyzed dmesg log and found that root file system is being loaded far later than when tegra-xhci driver looks for the file in /lib/firmware, see the dmesg for root file system at 8.725522
[ 8.725522] VFS: Mounted root (ext4 filesystem) on device 179:1.
whereas tegra-xhci driver tries to load firmware file around 2.852161
[ 2.852161] tegra-xhci tegra-xhci: XUSB device id = 0xfac (T210)
That would do it…not having a mounted file system tends to make it hard to read the files :P
It does look like starting in R24.2 an initrd file is used (/boot/initrd, specified in extlinux.conf via “INITRD /boot/initrd”). I don’t remember these being used in previous L4T versions, so it looks like something new. Perhaps the timing of file system mount isn’t really a bug, though it would explain firmware not being found on the file system.
During boot loader operation, prior to the kernel taking over, previous L4T versions required any files needed at that point to be on a readable file system…namely “/boot”. Having an initrd would make an alternate pseudo file system available with more files on it prior to partition mounts. I see “lib/firmware/tegra21x_xusb_firmware” in that initrd, so I believe this is the problem. Your Jetson is at a stage of boot where it wants that firmware and in R24.2 it would be provided by the initrd…any missing firmware was probably expected there.
Do you have an INITRD entry in your extlinux.conf?
You found the exact problem. I’m just wondering why it did not create this for me, probably a bug in utility that creates extlinux.conf.
Anyway, “linuxdev”, you’re the man, come and visit Toronto I’ll offer you a free lunch!
Hakim
Looks like on Ubuntu you need to have package “initramfs-tools” installed (that or manually use gzip and cpio). Different initial ramdisk tools are specific to different distributions…so building an initrd may not have consistent instructions other than manual steps.
For those interested, the initrd is a gzip compressed cpio archive (basically cpio serializes a file system for putting it on a tape backup, or the reverse for restore from tape backup). To view an initrd you could extract it like this:
sudo -s
# cd ...somewhere to unpack with nothing else in the directory...e.g., mkdir /tmp/initrd, cd /tmp/initrd
cp /boot/initrd initrd.cpio.gz
gunzip initrd.cpio.gz
cpio -vid < initrd.cpio
I haven’t tested this in actual use, but I believe recreating the same unpacked directory would be:
rm initrd.cpio
find . -print | cpio -ov > ../initrd.cpio
cd ..
gzip initrd.cpio
# I'd suggest an alternate name if initrd is modified.
mv initrd.cpio initrd
As the original poster: the missing initrd is the problem. It wasn’t clear that it had to be used, as none of the documentation actually mentions it, and the error seemed to occur after the rootfs was mounted. Anyway, thanks a lot!