Jetson TX1 Won't Flash - Not entering recovery mode - Won't Boot - Brick

Hello All,

I am in desperate need of help, to recover my TX1 from being a brick (light and small, but still a brick). I have the TX1 Development board and have been using it successfully for a week or so until…

  1. Recompiled Kernel following THISvery helpful tutorial.
  2. Ran the Jetpack installer to flash the new (barely modified, view appendixes) kernel.
  3. Entered Recovery mode, started flashing...
  4. It hung. Flashing one of the images, it just sat there and the percentage stopped moving. After 45m+ I quit the xTerm
  5. Ran the Jetpack installer again and attempted to flash again. Fails with these main issues (full logs, appendixes) [i][ 0.2191 ] BootRom is not running ... [ 0.2386 ] Applet is not running on device. Continue with Bootloader ... [ 0.2690 ] Applet is not running on device. Continue with Bootloader ... [ 0.2946 ] Cboot is not running on device. ... Failed flashing t210ref.[/i]
  6. Panicked

Status: lsusb still shows Nvidia Corp. But the sounds my machine makes are different. I get USB device attached as soon as I power on the jetson, without holding down recovery. Nothing on HDMI output. Ethernet lights turn on and blink as if activity.

Attempts to remedy: Tried running it again many more times. Recompiled Stock kernel. Tried that. Manually started entering tegrarcm commands, get the the same place and usb communication fails. Tried running it without force recovery, tried with, tried all sorts of button combinations. Connected a UART/USB dongle to PC and wired it up to J21 P8-10 (RX TX GND), as detailed HERE with (115200 8N1, Flow control OFF). Nothing (although i will re-attempt this with a proper dongle as this was one I found lying around).

ToDo:

  1. Scan the IPs and see if anybody is home inside the Jetson.
  2. Re-check UART1 with a better setup.
  3. Cry.

Please help!

Update 14/05/2016:
Checked IPs, nothing. Picked up my logic analyser, see photo. Options left RMA or try JTAG flashing? any advice about manually mounting the device under ubuntu? I am in NZ, not even sure how RMA works here…

Appendixes:

Environment: DEV is a Ubuntu 14.04 4.2.0-36-generic x64, running within a VirtualBox VM on HOST
HOST is a Win7 SP1 x64 running on a i7 with 16GB ram and SSD
JETSON is Standard un-modified TX1 developers kit, no cameras attached, no PCIE, no nothing. not even an SD card. This environment (unchanged) has successfully flashed the Jetson TX1 at least 3 times.

Logs:

flash_os_tx1.log

Warning: missing eksfile (bootloader/eks.img), continue... 
Warning: missing fbfile (bootloader/reserved_fb.xml), continue... 
copying bctfile(/home/linux/TX1/Linux_for_Tegra_tx1/bootloader/t210ref/BCT/P2180_A00_LP4_DSC_204Mhz.cfg)... done.
	populating kernel to rootfs... done.
	populating initrd to rootfs... done.
	populating extlinux.conf.emmc to rootfs... done.
	populating /home/linux/TX1/Linux_for_Tegra_tx1/kernel/dtb/tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb to rootfs... done.
done.
copying bcffile(/home/linux/TX1/Linux_for_Tegra_tx1/bootloader/t210ref/cfg/board_config_p2597-devkit.xml)... done.
Existing sosfile(/home/linux/TX1/Linux_for_Tegra_tx1/bootloader/nvtboot_recovery.bin) reused.
copying tegraboot(/home/linux/TX1/Linux_for_Tegra_tx1/bootloader/t210ref/nvtboot.bin)... done.
Existing bpffile(/home/linux/TX1/Linux_for_Tegra_tx1/bootloader/bpmp.bin) reused.
copying wb0boot(/home/linux/TX1/Linux_for_Tegra_tx1/bootloader/t210ref/warmboot.bin)... done.
Existing tosfile(/home/linux/TX1/Linux_for_Tegra_tx1/bootloader/tos.img) reused.
copying dtbfile(/home/linux/TX1/Linux_for_Tegra_tx1/kernel/dtb/tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb)... done.
Reusing existing system.img... 
done.
Existing tbcfile(/home/linux/TX1/Linux_for_Tegra_tx1/bootloader/nvtboot_cpu.bin) reused.
copying cfgfile(/home/linux/TX1/Linux_for_Tegra_tx1/bootloader/t210ref/cfg/gnu_linux_tegraboot_emmc_full.xml) to flash.xml... done.
creating gpt(ppt.img)... 

*** GPT Parameters ***
device size -------------- 31276924928
bootpart size ------------ 8388608
userpart size ------------ 31268536320
Erase Block Size --------- 2097152
sector size -------------- 4096
Partition Config file ---- flash.xml
Visible partition flag --- GP1
Primary GPT output ------- PPT->ppt.img
Secondary GPT output ----- GPT->gpt.img
Target device name ------- none

*** PARTITION LAYOUT(23 partitions) ***
[     BCT] BH            0         8191       4.0MiB 
[     NVC] BH         8192        16383       4.0MiB nvtboot.bin
[     PPT] UH            0         4095       2.0MiB 
[     GP1] UH         4096         8191       2.0MiB 
[     APP] UH         8192     29368319   14336.0MiB system.img
[     TBC] UV     29368320     29372415       2.0MiB nvtboot_cpu.bin
[     EBT] UV     29372416     29380607       4.0MiB u-boot-dtb.bin
[     BPF] UV     29380608     29384703       2.0MiB bpmp.bin
[     WB0] UV     29384704     29396991       6.0MiB warmboot.bin
[     RP1] UV     29396992     29405183       4.0MiB tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb
[     TOS] UV     29405184     29417471       6.0MiB tos.img
[     EKS] UV     29417472     29421567       2.0MiB 
[      FX] UV     29421568     29425663       2.0MiB 
[     SOS] UV     29425664     29466623      20.0MiB 
[     EXI] UV     29466624     29597695      64.0MiB 
[     LNX] UV     29597696     29728767      64.0MiB 
[     DTB] UV     29728768     29736959       4.0MiB tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb
[     NXT] UV     29736960     29741055       2.0MiB 
[     MXB] UV     29741056     29753343       6.0MiB 
[     MXP] UV     29753344     29765631       6.0MiB 
[     USP] UV     29765632     29769727       2.0MiB 
[     UDA] UV     29769728     61067263   15282.0MiB 
[     GPT] UH     61067264     61071359       2.0MiB 
copying flasher(/home/linux/TX1/Linux_for_Tegra_tx1/bootloader/t210ref/cboot.bin)... done.
Existing flashapp(/home/linux/TX1/Linux_for_Tegra_tx1/bootloader/tegraflash.py) reused.
*** Flashing target device started. ***
./tegraflash.py --bl cboot.bin --bct P2180_A00_LP4_DSC_204Mhz.cfg --odmdata 0x84000 --bldtb tegra210-jetson-tx1-p2597-2180-a01-devkit.dtb --applet nvtboot_recovery.bin --boardconfig board_config_p2597-devkit.xml --cmd "flash;reboot" --cfg flash.xml --chip 0x21 
Welcome to Tegra Flash
version 1.0.0
Type ? or help for help and q or quit to exit
Use ! to execute system commands
 
[   0.0000 ] Generating RCM messages
[   0.0012 ] tegrarcm --listrcm rcm_list.xml --chip 0x21 --download rcm nvtboot_recovery.bin 0 0
[   0.0026 ] RCM 0 is saved as rcm_0.rcm
[   0.0035 ] RCM 1 is saved as rcm_1.rcm
[   0.0038 ] List of rcm files are saved in rcm_list.xml
[   0.0117 ] 
[   0.0117 ] Signing RCM messages
[   0.0137 ] tegrasign --key None --list rcm_list.xml --pubkeyhash pub_key.hash
[   0.0158 ] Assuming zero filled SBK key
[   0.0218 ] 
[   0.0219 ] Copying signature to RCM mesages
[   0.0248 ] tegrarcm --chip 0x21 --updatesig rcm_list_signed.xml
[   0.0344 ] 
[   0.0344 ] Parsing partition layout
[   0.0361 ] tegraparser --pt flash.xml
[   0.0379 ] 
[   0.0379 ] Creating list of images to be signed
[   0.0391 ] tegrahost --chip 0x21 --partitionlayout flash.bin --list images_list.xml
[   0.0485 ] 
[   0.0485 ] Generating signatures
[   0.0497 ] tegrasign --key None --list images_list.xml --pubkeyhash pub_key.hash
[   0.0507 ] Assuming zero filled SBK key
[   0.0877 ] 
[   0.0877 ] Parsing BCT
[   0.0893 ] tegrabct --bct P2180_A00_LP4_DSC_204Mhz.cfg --chip 0x21
[   0.0907 ] Copying Sdram info from 0 to 1 set
[   0.0932 ] Copying Sdram info from 1 to 2 set
[   0.0937 ] Copying Sdram info from 2 to 3 set
[   0.1046 ] 
[   0.1047 ] Updating boot device parameters
[   0.1061 ] tegrabct --bct P2180_A00_LP4_DSC_204Mhz.bct --chip 0x21 --updatedevparam flash.bin
[   0.1073 ] Warning: No sdram params
[   0.1156 ] 
[   0.1156 ] Updating bl info
[   0.1171 ] tegrabct --bct P2180_A00_LP4_DSC_204Mhz.bct --chip 0x21 --updateblinfo flash.bin --updatesig images_list_signed.xml
[   0.1286 ] 
[   0.1286 ] Updating secondary storage information into bct
[   0.1301 ] tegraparser --pt flash.bin --chip 0x21 --updatecustinfo P2180_A00_LP4_DSC_204Mhz.bct
[   0.1406 ] 
[   0.1406 ] Updating board information from board config into bct
[   0.1424 ] tegraparser --boardconfig board_config_p2597-devkit.xml --chip 0x21 --updatecustinfo P2180_A00_LP4_DSC_204Mhz.bct
[   0.1526 ] 
[   0.1527 ] Updating Odmdata
[   0.1544 ] tegrabct --bct P2180_A00_LP4_DSC_204Mhz.bct --chip 0x21 --updatefields Odmdata = 0x84000
[   0.1563 ] Warning: No sdram params
[   0.1646 ] 
[   0.1647 ] Get Signed section bct
[   0.1667 ] tegrabct --bct P2180_A00_LP4_DSC_204Mhz.bct --chip 0x21 --listbct bct_list.xml
[   0.1768 ] 
[   0.1768 ] Signing BCT
[   0.1781 ] tegrasign --key None --list bct_list.xml --pubkeyhash pub_key.hash
[   0.1793 ] Assuming zero filled SBK key
[   0.1871 ] 
[   0.1871 ] Updating BCT with signature
[   0.1892 ] tegrabct --bct P2180_A00_LP4_DSC_204Mhz.bct --chip 0x21 --updatesig bct_list_signed.xml
[   0.2006 ] 
[   0.2006 ] Copying signatures
[   0.2020 ] tegrahost --chip 0x21 --partitionlayout flash.bin --updatesig images_list_signed.xml
[   0.2032 ] Run tegrabct to update tboot signature in bct
[   0.2166 ] 
[   0.2166 ] Boot Rom communication
[   0.2178 ] tegrarcm --rcm rcm_list_signed.xml
[   0.2191 ] BootRom is not running
[   0.2360 ] 
[   0.2361 ] Sending BCT and pre-requisite binaries
[   0.2373 ] tegrarcm --download bct P2180_A00_LP4_DSC_204Mhz.bct
[   0.2386 ] Applet is not running on device. Continue with Bootloader
[   0.2633 ] 
[   0.2634 ] Retrieving storage infomation
[   0.2658 ] tegrarcm --oem platformdetails storage storage_info.bin
[   0.2690 ] Applet is not running on device. Continue with Bootloader
[   0.2926 ] 
[   0.2927 ] tegradevflash --oem platformdetails storage storage_info.bin
[   0.2946 ] Cboot is not running on device.
[   0.3041 ] 
Error: Return value 4
Command tegradevflash --oem platformdetails storage storage_info.bin
Failed flashing t210ref.

filesystem_tx1.log

Using rootfs directory of: /home/linux/TX1/Linux_for_Tegra_tx1/rootfs
Extracting the NVIDIA user space components to /home/linux/TX1/Linux_for_Tegra_tx1/rootfs
Extracting the BSP test tools to /home/linux/TX1/Linux_for_Tegra_tx1/rootfs
Extracting the NVIDIA gst test applications to /home/linux/TX1/Linux_for_Tegra_tx1/rootfs
Extracting the configuration files for the supplied root filesystem to /home/linux/TX1/Linux_for_Tegra_tx1/rootfs
Creating a symbolic link nvgstplayer pointing to nvgstplayer-0.10
Creating a symbolic link nvgstcapture pointing to nvgstcapture-0.10
Adding symlink libcuda.so --> libcuda.so.1.1 in target rootfs
Adding symlink libGL.so --> libGL.so.1 in target rootfs
Adding symlink libcuda.so --> tegra/libcuda.so in target rootfs
Adding symlink libEGL.so --> libEGL.so.1 in target rootfs
Extracting the firmwares and kernel modules to /home/linux/TX1/Linux_for_Tegra_tx1/rootfs
Extracting the kernel headers to /usr/src in target rootfs
tar: write error
Adding target symlink /lib/modules/3.10.67-g458d45c/build --> /usr/src/linux-headers-3.10.67-g458d45c
Installing zImage into /boot in target rootfs
Installing Image into /boot in target rootfs
Installing the board *.dtb files into /boot in target rootfs
Success!

jetson-duo.patch (modifications made to kernel)

diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile
index 0c2bc73..0ef7c9a 100644
--- a/arch/arm64/kernel/vdso32/Makefile
+++ b/arch/arm64/kernel/vdso32/Makefile
@@ -11,7 +11,7 @@ obj-vdso32 := $(addprefix $(obj)/, $(obj-vdso32))
 
 GCOV_PROFILE := n
 
-ccflags-y := -shared -fPIC -fno-common -fno-builtin -march=armv7-a
+ccflags-y := -shared -fPIC -fomit-frame-pointer -fno-common -fno-builtin -march=armv7-a
 ccflags-y += -nostdlib -Wl,-soname=linux-vdso32.so.1 \
 		$(call cc-ldoption, -Wl$(comma)--hash-style=sysv)
 asflags-y := -D__VDSO32__ -s
diff --git a/drivers/platform/tegra/tegra21_clocks.c b/drivers/platform/tegra/tegra21_clocks.c
index 0a0584f..cf0e277 100644
--- a/drivers/platform/tegra/tegra21_clocks.c
+++ b/drivers/platform/tegra/tegra21_clocks.c
@@ -1062,7 +1062,7 @@ static struct clk_ops tegra_super_ops = {
  */
 static void tegra21_cpu_clk_init(struct clk *c)
 {
-	c->state = (!is_lp_cluster() == (c->u.cpu.mode == MODE_G)) ? ON : OFF;
+	c->state = ((!is_lp_cluster()) == (c->u.cpu.mode == MODE_G)) ? ON : OFF;
 }
 
 static int tegra21_cpu_clk_enable(struct clk *c)
diff --git a/drivers/usb/host/xhci-mem.c b/drivers/usb/host/xhci-mem.c
index 677f032..032b014 100644
--- a/drivers/usb/host/xhci-mem.c
+++ b/drivers/usb/host/xhci-mem.c
@@ -1436,8 +1436,9 @@ int xhci_endpoint_init(struct xhci_hcd *xhci,
 		break;
 	case USB_SPEED_HIGH:
 		/* Some devices get this wrong */
-		if (usb_endpoint_xfer_bulk(&ep->desc))
-			max_packet = 512;
+//Duo3D Kernel Patch, applied 13/05/2016
+		//if (usb_endpoint_xfer_bulk(&ep->desc))
+		//	max_packet = 512;
 		/* bits 11:12 specify the number of additional transaction
 		 * opportunities per microframe (USB 2.0, section 9.6.6)
 		 */

Many people have had issues running on a VM…it isn’t that it can’t work, but that there are a lot of little issues you may never see which actually break the flash. One is that the underlying file system must be a native Linux system, that NTFS will not do the job (NTFS cannot reproduce Linux permissions). Others are suprised to find the VM does not have enough disk space…you’ll probably want about 30GB of free space after everything is copied into the VM (I see “tar: write error”). Sometimes there are issues related to ownership of the USB port on the VM host. The list goes on.

Hello Linuxdev,

Thanks for the suggestions, and yeah I have read many of those posts myself. However as mentioned this exact same setup has worked many times for me already, I would struggle to blame the VM, since nothing has changed. Also I have given it ample free space. Sadly I don’t have access to a native Linux machine at the moment. The only thing I can think of that maybe changed was that I updated my VirtualBox guest additions, but I will check this (I run many VMs, not sure if i did that on my jetson VM).

As mentioned I strongly think the Jetson is struggling to enter flash mode. Is there a way I could manually mount it or check? Normally the Ethernet lights do not start blinking so soon on boot, but now they do.

I think the cake is in the line:

BootRom is not running

Indicating the jetson isn’t running something it should, if I understand it correctly

Spoiler: It is working, somehow.

Right. So given my options of crying or finding a JTAG flasher, I got up, scoured the office (it is a Saturday, so no one is in) and found a few. Deciding though that this was not my area of expertise and that I’d wait until Monday I sat down. Coming to terms with the defeat I moved onto some other tasks. But unwilling to give up I decided “What the heck, I’m a gambling man, I’ll try flash it for the 30th+ time” … it worked. Flashed, booting and confirming the new kernel.

I have no idea what is reality anymore. All I know is that I won’t recover those 10 hours and it is best to move on. Maybe it was a magic combination of luck and crying …

I was able to get past this particular error by deleting everything on the host and downloading from scratch.