Our Jetson TX1 stopped working in the middle of a ssh session. We have tried to re-flash L4T and JetPack with no success. In the L4T case, the output of flash.sh was:
On the serial console we recieve the following output:
Is there anything else we could try before sending our board for repair?
Best,
Siniša
That amount of log does not show enough information to call it hardware failure. In the case of a Jetson suddenly failing, more information might also be useful about what was happening, and what steps were taken to flash (including if a VM host is involved).
One thing I often ask regarding systems mysteriously going down in the middle of normal operation is whether it was accessible by network to the outside world (the internet in general)? Unless your Jetson is isolated against remote logins via firewall or a router, then I’d expect the standard known user name and password of “ubuntu/ubuntu” to cause someone port scanning to find and vandalize the system within seconds to hours. Is it possible someone had remote access from an unprotected internet connection?
If possible, also describe your method of flashing, including if it was JetPack, which version of whatever flash method, and how much disk space is available on your host from where the flash program was run (e.g., “df -H -T” describes this).
The output I’ve pasted is all we get in minicom. When we press the res button, we get the same output again.
We were attempting to flash from two different native Ubuntu laptops following the instructions at:http://developer.download.nvidia.com/embedded/L4T/r24_Release_v2.0/BSP/l4t_quick_start_guide.txt?autho=1477393718_89ce252d13f3333f862705619a0d2a15&file=l4t_quick_start_guide.txt
The command where the process hang was:
One of the two laptops had only 10 GB space (however, I successfully flashed the latest Jetpack from that system on Friday) so we retried from another laptop which had hundreds of gigabytes free. The results were the same.
The system was on internet, so an intrusion is possible (I had checked with nmap, the only open port was ssh). However, I had changed the password so that would not be trivial. Anyway, I would not expect a hardware failure due to software exploit.
10 GB would result in a truncated image, but hundreds of GB would be more than enough.
It’s interesting that when I build a new system and connect it to the internet with a directly accessible IP address, within 30 seconds I start seeing port scans and root login attempts, along with dictionary login attempts to common login names. I agree that with the password changed you should be ok from any kind of simple attack.
For the flash command, could you try with size specified:
sudo ./flash.sh -S 14580MiB jetson-tx1 mmcblk0p1
If you want to log this, you can instead:
sudo ./flash.sh -S 14580MiB jetson-tx1 mmcblk0p1 2>&1 | tee flash_log.txt
…but expect the “progress” indicators for a logged flash session to be a huge amount of text in the log; you may have issues removing the progress indicator (its a single enormous line) before posting. This might not be important though, the real point is to see if it makes it far enough to have any initial transfer to the Jetson from the point of view of the flash script.
To avoid any extraneous noise from the serial console (just to be thorough), remove this before starting the flash. Keep monitoring the byte size of “bootloader/system.img” (which gets moved to system.img.raw at some point, so you need to see that size as well), and hopefully the raw image will be exactly 15288238080 bytes at some point. Use “df -H -T” from the flash directory to verify you still have disk space left and to also verify it is ext4. If it does work, expect creating the system.img and system.img.raw takes significant time prior to actual flash starting.
We have tried the flash command with the specified size on an Ubuntu laptop with a lot of free disk space. Unfortunately, the process still freezes. The flash log and the serial console log are pasted below.
===== flash log =====
Warning: missing eksfile (bootloader/eks.img), continue…
---- Raw to Sparse Image Converter v1.0 ----------------------------
Looks like the log truncated during post to the forum (post length was probably at max). The point at which it truncated it was still compressing the raw image into a sparse image. Would you post an excerpt of the log via:
cat flash_log.txt | gawk '/CHUNK/,/Writing partition TBC/' | sed 's/\r.*100[%]/SNIP/'
What this does is show the portion of the log at the end of creating sparse image and beyond, and cut out the long progress line (replacing it with “SNIP”). The “\r” is used because the flash.sh script has a windows style end of line on just that part of the script and can be used to mark the beginning of that progress line, while “100[%]” is the end of that progress line. Without this the progress line itself can be enormous. The part of the flash which is interesting really starts when something is written to the Jetson (we already know the image is created, the question is how it writes).
ssegvic
November 3, 2016, 11:19am
7
You were right, the log got truncated during posting. Sorry!
I have placed the output on our web server, you can access it at the following address:http://www.zemris.fer.hr/~ssegvic/flash_log.txt
The output of your command is pasted below:
done.
*** GPT Parameters ***
*** PARTITION LAYOUT(23 partitions) ***
[ 0.0000 ] Generating RCM messages
It looks like it partially works. ebt and rp1 are successfully sent, so up until that point you can be sure that at least the Jetson is in recovery mode and that it was able to receive and write data. I’m not sure where storage_info.bin is; this could be on the Jetson’s eMMC (I don’t see it on any of my host’s flashes’ after using them…I have R19.2 through R24.2 and “locate” does not show this anywhere on the host at all…seems it must be a file on eMMC).
Then things halt. If you were using a VM I’d suspect this (they’ve been known to suddenly fail like this by halting USB in the middle of doing something), but since you are not, I have to conclude that wherever the storage_info.bin is the memory never completes write. This is probably a hardware failure for RMA.
If you go to this URL there is info on RMA near the top:https://devtalk.nvidia.com/default/topic/793798/embedded-systems/some-jetson-web-links/[/url]