TK1 r21.8 uboot hang on issue

Hi All,

I cloned the latest R21.8 u-boot source code form the official git as show below:

http://nv-tegra.nvidia.com/gitweb/?p=3rdparty/u-boot.git;a=commit;h=c50329da15b4c8a2bc0d176e10a8606b2417ac2a

I used the oldest 32-bit release R24.2.1 gcc tool chain to compiled it. But the system is hang on when I use the u-boot compiled by myself. The messages show on the console is

U-Boot SPL 2018.05 (Jan 22 2020 - 15:23:54 +0800)
Trying to boot from RAM

I found that the system would boot successfully after backed it to this commit “da41cbbdb4a0bc047401e85255163639247b6bb9”.
If I switched to the next commit “da0b0582522a67fdbfeca26d10d0106c9dca8fcc” and flashed it to TK1 after compiled it, then the u-boot would hang on again.

How can I fix it?
Any suggestion is appreciated!

I do not personally know the difference between those two releases. However, before going into those details (which someone else will need to answer), you will probably want to know the exact byte size of the two produced images correctly fit within the partition size. I am hoping to first rule out something simple like a bootloader too big for the partition, or a change in flash procedure. Once those are known as valid you can go on to bisect the two bootloader release differences.

Hi,

Sorry for late response. I just wonder, if you directly use r21.8 package, would you hit this issue?

If not, then might be some problem in u-boot compilation. You could also compare the uboot binary size between the file you build and the official binary.

Hi All,

Thanks for your reply!!

If I directly use r21.8 package, the u-boot work normally!

The information as show below:

official release u-boot size : 625.8 KB
u-boot be compiled by myself : 635.8 KB

If I only revert this commit “da0b0582522a67fdbfeca26d10d0106c9dca8fcc” then compiled it. The size of u-boot was 622.1KB. This u-boot could boot into system, but it would be system halted after reboot.

Partition table as show below:

[device]
type=sdmmc
instance=3

[partition]
name=BCT
id=2
type=boot_config_table
allocation_policy=sequential
filesystem_type=basic
size=8388608
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0

[partition]
name=PPT
id=3
type=data
allocation_policy=sequential
filesystem_type=basic
size=2097152
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0
filename=ppt.img

[partition]
name=PT
id=4
type=partition_table
allocation_policy=sequential
filesystem_type=basic
size=2097152
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0

[partition]
name=EBT
id=5
type=bootloader
allocation_policy=sequential
filesystem_type=basic
size=4194304
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0
filename=u-boot.bin

[partition]
name=LNX
id=6
type=data
allocation_policy=sequential
filesystem_type=basic
size=16777216
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0

[partition]
name=SOS
id=7
type=data
allocation_policy=sequential
filesystem_type=basic
size=6291456
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0
#filename=recovery.img

[partition]
name=NVC
id=8
type=data		#TEGRABOOT
allocation_policy=sequential
filesystem_type=basic
size=2097152
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0
#filename=nvtboot.bin

[partition]
name=MPB
id=9
type=data		#MTSPREBOOT
allocation_policy=sequential
filesystem_type=basic
size=6291456
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0
#filename=mts_preboot_si

[partition]
name=MBP
id=10
type=data		#MTSBOOTPACK
allocation_policy=sequential
filesystem_type=basic
size=6291456
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0
#filename=mts_si

[partition]
name=GP1
id=11
type=GP1
allocation_policy=sequential
filesystem_type=basic
size=2097152
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0

[partition]
name=APP
id=12
type=data
allocation_policy=sequential
filesystem_type=basic
size=15032385536
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0
filename=system.img

[partition]
name=DTB
id=13
type=data
allocation_policy=sequential
filesystem_type=basic
size=4194304
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0
filename=tegra124-jetson_tk1-pm375-000-c00-00.dtb

[partition]
name=EFI
id=14
type=data
allocation_policy=sequential
filesystem_type=basic
size=67108864
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0
#filename=efi.img

[partition]
name=USP
id=15
type=data
allocation_policy=sequential
filesystem_type=basic
size=4194304
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0

[partition]
name=TP1
id=16
type=data
allocation_policy=sequential
filesystem_type=basic
size=4194304
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0

[partition]
name=TP2
id=17
type=data
allocation_policy=sequential
filesystem_type=basic
size=4194304
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0

[partition]
name=TP3
id=18
type=data
allocation_policy=sequential
filesystem_type=basic
size=4194304
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0

[partition]
name=WB0
id=19
type=data	#WB0BOOT
allocation_policy=sequential
filesystem_type=basic
size=2097152
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0
#filename=nvtbootwb0.bin

[partition]
name=UDA
id=20
type=data
allocation_policy=sequential
filesystem_type=basic
size=2097152
file_system_attribute=0
partition_attribute=0
allocation_attribute=0x808
percent_reserved=0

[partition]
name=GPT
id=21
type=GPT
allocation_policy=sequential
filesystem_type=basic
size=0xFFFFFFFFFFFFFFFF
file_system_attribute=0
partition_attribute=0
allocation_attribute=8
percent_reserved=0
filename=gpt.img

Thanks!!

But the official uboot.bin should also have this commit. Could you share your steps of compiling u-boot?

Hi,

I used the oldest 32-bit release R24.2.1 gcc tool chain to compile the u-boot. It will have compiler error, but we fixed this error at this topic as below:

https://devtalk.nvidia.com/default/topic/1070186/jetson-tk1/r21-8-build-uboot-kernel-error/

The steps as show below:

$ cd u-boot/
$ export ARCH=arm
$ export CROSS_COMPILE=.../bin/arm-unknown-linux-gnueabi-
$ make distclean
$ make jetson-tk1_defconfig
$ make

I also found that if I used the Code Sourcery ARM toolchain 2009q1-203 to compile the R21.8 u-boot, it could be compiled successfully without changed any u-boot codes. But it still have same problem.

The steps as show below:

$ cd u-boot/
$ export ARCH=arm
$ export CROSS_COMPILE=.../arm-2009q1/bin/arm-none-linux-gnueabi-
$ make distclean
$ make jetson-tk1_defconfig
$ make

Could you share the steps for compiling the U-boot ?

Thanks~

I would still be curious to see a serial console boot log since this might offer hints as to where it went wrong (if you have a serial boot log for both the GNU compiler boot failure and the Code Sourcery compiler and the two match/differ in the failure reason, then this would offer even more clues). There may be some config the jetson-tk1_defconfig is missing, and a boot message might narrow the search.

Hi linuxdev,

Thanks for reply. But the serial console boot log only show that as below (Even though I used the different compilers, the u-boot logs are the same):

U-Boot SPL 2018.05-dirty (Feb 06 2020 - 10:42:11 +0800)
Trying to boot from RAM

And then the system hang on…

If I only revert this u-boot commit as show below

http://nv-tegra.nvidia.com/gitweb/?p=3rdparty/u-boot.git;a=commit;h=da0b0582522a67fdbfeca26d10d0106c9dca8fcc

And flashed it to TK1 after compiled it, the system would boot into ubuntu. The boot logs as show below :

U-Boot 2018.05-00001-g6487db0-dirty (Feb 06 2020 - 11:03:07 +0800)

TEGRA124
Model: NVIDIA Jetson TK1
Board: NVIDIA Jetson TK1
DRAM:  2 GiB
MMC:   sdhci@700b0400: 1, sdhci@700b0600: 0
Loading Environment from MMC... *** Warning - bad CRC, using default environment

Failed (-5)
In:    serial
Out:   serial
Err:   serial
Net:   No ethernet found.
Hit any key to stop autoboot:  0
MMC: no card present
mmc_init: -123, time 0
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found /boot/extlinux/extlinux.conf
Retrieving file: /boot/extlinux/extlinux.conf
827 bytes read in 319 ms (2 KiB/s)
Jetson-TK1 eMMC boot options
1:      primary kernel
Enter choice: 1:        primary kernel
Retrieving file: /boot/zImage
6230800 bytes read in 190 ms (31.3 MiB/s)
append: console=ttyS0,115200n8 console=tty1 no_console_suspend=1 lp0_vec=2064@0xf46ff000 mem=2015M@2048M memtype=255 ddr_die=2048M@2048M section=256M pmuboard=0x0177:0x0000:0x02:0x43:0x00 tsec=32M@3913M otf_key=c75e5bb91eb3bd947560357b64422f85 usbcore.old_scheme_first=1 core_edp_mv=1150 core_edp_ma=4000 tegraid=40.1.1.0.0 debug_uartport=lsport,3 power_supply=Adapter audio_codec=rt5640 modem_id=0 android.kerneltype=normal fbcon=map:1 commchip_id=0 usb_port_owner_info=0 lane_owner_info=6 emc_max_dvfs=0 touch_id=0@0 board_info=0x0177:0x0000:0x02:0x43:0x00 net.ifnames=0 root=/dev/mmcblk0p1 rw rootwait tegraboot=sdmmc gpt
Retrieving file: /boot/tegra124-jetson_tk1-pm375-000-c00-00.dtb
59966 bytes read in 281 ms (208 KiB/s)
## Flattened Device Tree blob at 82000000
   Booting using the fdt blob at 0x82000000
   Using Device Tree in place at 82000000, end 82011a3d

Starting kernel ...

I trying to add more debug message on u-boot source code for debugging.

Any suggestion is appreciated!
Thanks.

An early note, and question for NVIDIA: I find “source_synch.sh -u tegra-l4t-r21.8” does not recognize this tag. Is there a source_sync.sh tag for R21.8? I don’t know where this URL actually originates from, although there is a “u-boot_src.tbz2” within the sources under R21.8 download page:
http://nv-tegra.nvidia.com/gitweb/?p=3rdparty/u-boot.git;a=commit;h=c50329da15b4c8a2bc0d176e10a8606b2417ac2a

I’m not positive, but at the point where the unusual “Failed (-5)” occurs, this should probably be the PCIe driver, “tegra-pcie”, listing statistics about the bus and PHY. This is from R21.6, but you’ll get the idea from this excerpt:

TEGRA124
Board: NVIDIA Jetson TK1
I2C:   ready
DRAM:  2 GiB
MMC:   Tegra SD/MMC: 0, Tegra SD/MMC: 1
*** Warning - bad CRC, using default environment

[i][b]tegra-pcie: PCI regions:
tegra-pcie:   I/O: 0x12000000-0x12010000
tegra-pcie:   non-prefetchable memory: 0x13000000-0x20000000
tegra-pcie:   prefetchable memory: 0x20000000-0x40000000
tegra-pcie: 2x1, 1x1 configuration
tegra-pcie: probing port 0, using 2 lanes
tegra-pcie: link 0 down, retrying
tegra-pcie: link 0 down, retrying
tegra-pcie: link 0 down, retrying
tegra-pcie: link 0 down, ignoring
tegra-pcie: probing port 1, using 1 lanes[/b][/i]
In:    serial
Out:   serial
Err:   serial
Net:   RTL8169#0

Do you have anything at all connected to the PCIe, or anything at all different about your PCIe? Note that the “tegra-pcie” is a driver specific to the NVIDIA SoC, and that although this could be completely unrelated to the kernel never starting (the kernel and command line are named in a usual way, but nothing occurs upon handing over to that kernel), it is also possible that this is the start of any “difference” between your bootloader and the precompiled bootloader from the R21.8 distribution.

If you look at your U-Boot build area, then you should find a “.config” file at the top directory. Within this, do you see:

CONFIG_PCI_TEGRA=y

In comment 8, “Failed (-5)” refers to a failure loading a saved environment from eMMC. This is normal if the user has never manually saved an environment, and is unrelated to the main issue of this thread.

For reference, NVIDIA internally used gcc 4.5.3 compiled with crosstool-ng to create the U-Boot binary in L4T r21.x.

I have tracked down the docs re: how to build the toolchain that we use:

Download the Documentation package from the most recent release that contains docs:

Extract that package, view “Start_Linux_for_Tegra_Docs.html” in a browser, the use the menu on the left of the page to view “Building Hardfp Crosstool-ng Toolchain and glibc”.

Unfortunately the domain “crosstool-ng.org” no longer exists, so it isn’t possible to use the older official URLs for this release.

I personally have crosstool-ng all the way back to 4.8.2, but even this is rather ancient. I realize that the TK1 is EOL other than the actual SoC chip, but perhaps a method of using Linaro 4.8.x would avoid needing crosstool-ng. If NVIDIA has a way to distribute this, then this is probably the only reasonable possibility for compiling with the original tool chain. If anyone else might know where to obtain crosstool-ng 4.5.3 that would help a lot.

It looks like crosstool has moved to https://crosstool-ng.github.io/, and in particular all the releases are available at Releases · crosstool-ng/crosstool-ng · GitHub

Hi linuxdev,

Because it did not have tegra-l4t-r21.8 tag on NVIDIA git, you could see the link as below:

http://nv-tegra.nvidia.com/gitweb/?p=3rdparty/u-boot.git;a=summary

I cloned the tegra-l4t-r21.8-3 from NVIDIA git and I compared this u-boot source codes with “u-boot_src.tbz2” within the sources under R21.8 download page, I found that they are the same.

Hi Stephen,

Thanks for reply, I trying to make crosstool-ng by myself, but I meet a problem when I type the command as below :

$ ./ct-ng build.8

The error log as attached file.

Please give me some advice.
Thanks.
build.log (1.7 MB)

So the error message in that log appears to be:

[ALL  ]    *** ld does not support target alphaev4-cortex_a9-elf

I assume the machine you’re building the compiler on is a regular x86 PC, and doesn’t have a DEC Alpha CPU? Do you have any environment variables like ARCH or similar set that might confuse the build process into thinking it’s building on/for a DEC Alpha?

Which Linux distro and version are you building on, and is it a 32-bit or 64-bit OS?

@StephenWarren, in #11, you mentioned using gcc 4.5.3 (the tool used to build tools can have an effect, but I’m mentioning this because it differs from the git repo numbering). I consider it possible (considering the the age of crosstool-ng and gcc 4.5.3) that this suggests building crosstool-ng with a native x86/x86_64 gcc 4.5.3 (whereas most hosts now have a gcc 7.x or above, although that will probably work). Alternatively, this could mean that the crosstool-ng source was compiled and produced gcc 4.5.3. I just want to clarify that https://github.com/crosstool-ng/crosstool-ng/releases, which defaults to release version crosstool-ng-1.24.0 (if another tag is not stated), and is the correct source for the build. Perhaps an older crosstool release from around 2011 would be more appropriate? I don’t know.

@michaelwei9pt6c, as mentioned by @StephenWarren, the cross linker architecture is wrong (a cross tool was being created, but it was being created to run on Alphav4 to produce armhf instead of being built to run on x86/x86_64 to produce armhf). Keep in mind that when building cross tools you must specify both the architecture the tools will run on (x86/x86_64 if you have a PC), and the architecture you are going to produce by the tool. If one tool is required to build another tool, then you may need to build a series of tools. Examples:

  • Build PC tool from a PC -> produces armhf code.
  • Build Alphav4 tool from a PC -> produces armhf, but won't run on a PC.
  • Build Alphav4 tool from a PC requiring intermediate tool -> produces Alphav4 tool running on PC which in turn builds a purely Alphav4 tool to output armhf.

This latter “triple architecture” configuration case is needed if your configuration needs an intermediate tool to build the final tool. There is an extra architecture beyond x86/x86_64 and armhf: Alphav4, intended to be an intermediate host architecture (not what you want). Typical case: Configuration defaulted to something you didn’t intend.

Incidentally, there is a full instruction page here (and configuring cross tools for build is generally not an easy or simple thing, so it is worth looking at):
http://crosstool-ng.github.io/docs/

Caveat for what I see during a compile: I am using Fedora with gcc-7.3.1. No doubt there will be differences on Ubuntu 18.04.

NOTE: Build configuration usually produces multiple tools (emphasis plural), needed for both bare metal (static, e.g., bootloader or kernel) and user space (linked) applications, and sometimes one tool uses the other. If your build were only for a compiler with bare metal, then there would be no need for a linker. When the tools are built together as a set, then one tool might need to call another. The docs URL above will talk about configuring, and since you might (incorrectly, by accident) be building the tool to execute on a different architecture (neither armhf nor x86/x86_64, but instead DEC Alpha v4 minicomputer/mainframe…you want to execute on x86/x86_64), and that different architecture will output yet another architecture (armhf, regardless of execution on x86/x86_64 or Alphav4), then it is likely you need to reconfigure for just two architectures: Native x86_64/x86, foreign armhf. Skip Alphav4 (old DEC computer) through configuration.

Notice that if you download the git repo, then you will have a subdirectory:

crosstool-ng/testing/docker/

…and if you are using Ubuntu 18.04:

crosstool-ng/testing/docker/ubuntu18.04/

Within that directory is file “Dockerfile”. You will see some apt-get lines, and some packages needed for build. As individual commands this is:

sudo apt-get update
sudo apt-get install <b>-y</b> gcc g++ gperf bison flex texinfo help2man make libncurses5-dev \
    python3-dev autoconf automake libtool libtool-bin gawk wget bzip2 xz-utils unzip \
    patch libstdc++6

(perhaps you already have these…remove the “-y” if you want to manually confirm any installs)

I do not run in a container environment, and some of this is for doing so…but the actual packages needed in the apt-get command will not care if this is a container or a native Ubuntu 18.04. I am only using the Dockerfile as a list of needed packages.

The toplevel crosstool-ng directory will contain file “bootstrap”. As a regular user I ran this (“./bootstrap”). This creates some configuration files among which options can be picked. The “configure” file is a result of this, and is the main method of configuring.

Running “./configure --help” to see options helps. I always manually select “–prefix”. For my case:

./configure --prefix=/usr/local/crosstool-ng/git_1.24.0

(my user has created “/usr/local/crosstool-ng/git_1.24.0/”, and has permission to read/write there)

When I run “make” and then “sudo make install”, I get the configure utility:

/usr/local/crosstool-ng/git_1.24.0/bin/ct-ng

(this is used for other configuration steps)

I then make this available in my user’s path (and you’ll need to do this for root as well if you use sudo):

export PATH="${PATH}:/usr/local/crosstool-ng/git_1.24.0/bin"

(note that “/usr/local/crosstool-ng/git_1.24.0/bin” is my prefix plus “/bin/”)

From the top level of the source tree (this is the main key you need to fix most everything in the build):

ct-ng menuconfig

(note the “Help” menu at the bottom)

It is in “Target options —>” where the current config issue is fixed. The default, if you go in to that, is “Target Architecture (alpha) —>”. This needs to be “arm” (32-bit ARMv7-a). Within this you will need to add things like thumb “interworking”, switching back and forth between modes can be useful. All ARMv8 32-bit superset architectures support thumb, although you might not use it. EABI is already correctly set. Everything done in any ARM you might see from NVIDIA, going back a very long time, uses EABI. 32-bit little endian is correct.

Target OS can remain “bare-metal” since you are working on a bootloader. If you were to use “linux”, then possibly this would require a cross linker you don’t want to build.

Note that the “.config” will update if you exit the menuconfig, and that if you start menuconfig again, then it will start where you left off. If you find something you want to remember, then guard a copy of the “.config”.

I did not actually complete my build since it then had an issue with not finding “-lstc++” (which I am guessing might be needing 32-bit x86 instead of 64-bit x86_64…not sure). However, this would get you started and get past the alphav4 incorrect architecture.

Hi All,

Thanks for your all replies and suggestions.

My OS is ubuntu 16.04 (64-bits)

@linuxdev
I also met an issue with not finding “-lstc++” in previous time, maybe you could typed the command as below to fix it.

$sudo apt-get install g++-multilib

I complied crosstool-ng successfully, the attached file was build log after complied successfully.

Then I compiled the u-boot (tag : tegra-l4t-r21.8-3) that cloned from NVIDIA git repository with crosstool-ng.

The Steps as show below:

$export ARCH=arm
$export CROSS_COMPILE=~/crosstool/crosstool-ng/install/bin/arm-cortex_a9-eabi-
$make distclean
$make jetson-tk1_defconfig
$make

It still had compiler error that we already discussed on topic as below :
https://devtalk.nvidia.com/default/topic/1070186/jetson-tk1/r21-8-build-uboot-kernel-error/

So I fixed it and compiled it again, the u-boot was compiled successfully and its size was 609.6KB.

Then flashed it to TK1, it still have same problem…
build.log.zip (390 KB)

I couldn’t tell you what the options from your earlier diff in #8 will do, but if you compile as you just did, and configure, but do not make the changes to the config, does it boot? If you know you can compile this and boot without those changes, then the issue can be narrowed down to those specific changes.

What does the following output:

~/crosstool/crosstool-ng/install/bin/arm-cortex_a9-eabi-gcc --version

It needs to be gcc 4.5.3 to compile the U-Boot we ship in r21. I wonder if you used the latest version of crosstool-ng, which will itself probably compile a different version of gcc? I say this because if you’re using the correct version of gcc, you won’t need to patch the U-Boot source code to make it compile. You will probably need to use an old version of crosstool-ng to create this older toolchain (probably crosstool-ng-1.11.4 judging by the date in the notes below), and probably build on an old distro. I have the following notes, which imply you’ll also want to build everything using 32-bit CentOS5 (probably in a VM or container) to get an exact match to the toolchain we use:

# The hg clone needs to be done on a newer system, since CentOS5/RHEL5 don't
# include hg. The result is also archived at:
# .../path/to/crosstool-ng-hg20110628-glibc-2.11.3.tar.bz2
#
# NOTE: If you clone crosstool-ng rather than using the tar file above, you
# will need to patch it to support glibc-2.11.3. Edit ./config/libc/glibc.in,
# search for 2_11 and 2.11, and add equivalent 2_11_3/2.11.3 logic before it:
#
#+ config LIBC_GLIBC_V_2_11_3
#+     bool
#+     prompt "2.11.3 (EXPERIMENTAL)"
#+     depends on EXPERIMENTAL
#...
#+     default "2.11.3" if LIBC_GLIBC_V_2_11_3
hg clone http://crosstool-ng.org/hg/crosstool-ng