Is there any historic difference between the failed and working clone installs as to which L4T version which was on the Jetson just prior to the restore? There are many hidden partitions, and if one of those partitions is from a different version of L4T, or if one of those others had different install options, then I could see this happening.
As an alternate view, you could clone all of the partitions on one Jetson which works, and attempt to clone restore each of the hidden partitions one at a time into the failing system which already has the rootfs clone restored. See if one of those partitions fixes it.
A note about partitions and cloning: There is still some “bookkeeping” type information at the start of the eMMC which is not part of the hidden partitions, nor part of the rootfs, but which is needed for the system to identify where things are. If this boot record information were from a system cloned with a different partition size or layout versus what was cloned in as a restore, then it would probably imply something would fail when offsets do not match metadata. So consider that if anything were historically different not only in partition content, but also in partition size (especially rootfs since it is the first partition after the metadata), this would break boot.
On a TX2 I have here with R28.1 (using sudo) I can see output from “gdisk -l /dev/mmcblk0” as this:
GPT fdisk (gdisk) version 1.0.1
Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present
Found valid GPT with protective MBR; using GPT.
Disk /dev/mmcblk0: 61071360 sectors, 29.1 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 00000000-0000-0000-0000-000000000000
Partition table holds up to 17 entries
First usable sector is 4097, last usable sector is 61071327
Partitions will be aligned on 1-sector boundaries
Total free space is 1 sectors (512 bytes)
Number Start (sector) End (sector) Size Code Name
1 4097 41947136 20.0 GiB 0700 APP
2 41947137 41955328 4.0 MiB 0700 mts-bootpack
3 41955329 41955840 256.0 KiB 0700 cpu-bootloader
4 41955841 41956864 512.0 KiB 0700 bootloader-dtb
5 41956865 41963008 3.0 MiB 0700 secure-os
6 41963009 41963012 2.0 KiB 0700 eks
7 41963013 41964220 604.0 KiB 0700 bpmp-fw
8 41964221 41965220 500.0 KiB 0700 bpmp-fw-dtb
9 41965221 41969316 2.0 MiB 0700 sce-fw
10 41969317 41981604 6.0 MiB 0700 sc7
11 41981605 41985700 2.0 MiB 0700 FBNAME
12 41985701 42247844 128.0 MiB 0700 BMP
13 42247845 42313380 32.0 MiB 0700 SOS
14 42313381 42444452 64.0 MiB 0700 kernel
15 42444453 42445476 512.0 KiB 0700 kernel-dtb
16 42445477 42969764 256.0 MiB 0700 CAC
17 42969765 61071326 8.6 GiB 0700 UDA
APP is the rootfs and is the first partition. It begins at byte 4097, which is a 4096 byte offset or one sector offset when sector size is 4096 bytes. An older BIOS style partition would reserve 512 bytes for MBR, plus backup MBR…UEFI moves some of the firmware into the start of the disk and out of the BIOS. If this were a working and bootable system you could use dd to write into the initial disk metadata using a copy of the metadata from dd reading the first 4096 bytes of a working system, but since you can’t boot you can’t use dd for write. I don’t know if the clone software is capable of cloning this unlabeled raw byte offset (you could on the TK1, but it has different flasher options).
Can anyone from NVIDIA suggest if the R28.1 driver package clone can copy via exact byte offset (perhaps with a patch)? Is it mandatory to clone only by partition label? Is there a way to clone offset byte zero through byte 4096, and then to write this by offset into one of the failed units? This would provide a true backup and restore mechanism even on customized installs.