Stability issue with DMA-memory transfer while GPU activity

I’m having a serious stability issue when tranfering data from a Xilinx Kintex-7 FPGA over PCIe to DMA-mapped memory on the Jetson TX1. After doing countless tests, my observations basically break down to that heavy GPU-utilization breaks the DMA-functionality. The basic test for reproducing the problem looks like this:

  1. The FPGA infinitely writes data to DMA-mapped memory at high rates (>50% of the theoretical bandwidth of the 4x Gen2-link)
  2. As long as there isn’t any GPU-activity it works flawlessly (heavy CPU- or other memory-activity of other processes doesn’t seem to matter)
  3. I’m lauching a CUDA-sample application (like 5_Simulations/particles) and after a few seconds or immediately (or very rarely never) the DMA-memory transfer stops and the system gets very unstable and typically crashes completely after a few more seconds.

The stop of the ability to transmit data is indicated by the signals of the Xilinx PCIe-ip: The tx_buf_av- and s_axis_tx_tready-signals indicate that there aren’t any transmit-buffers available and transmission isn’t ready. But user_lnk_up and cfg_pcie_link_state still indicate that the PCIe-link is up and running and in the L0-state. After the DMA is broken there seems to be a few seconds where my driver still can iowrite32 successfully to the FPGA, which also gives some indication that it’s not the PCIe-link which fails, but rather something like the memory-controller responsible for the bus arbitration.

Do you have any idea what could be the problem?

Hi mrJazz, is the behavior impacted if you maximize EMC memory controller clocks (by running ~/jetson_clocks.sh)? Does it happen with other CUDA samples or applications? Are you able to call a simple kernel from your process or otherwise perform GPU activity - starting with smaller loads and working up. Also are there any kernel debug or oops during instability, and is the FPGA PCIe driver receiving error? Thank you.

Also, is the FPGA card externally powered? The devkit power supply is not rated for high power PCIe cards. This may become a factor when the GPU starts consuming power.

Running jetson_clocks.sh doesn’t seem to change the behavior.

I tried many sample-applications, but it seems to happen only with particles and
smokeParticles.

I came across this problem while running my own cuda-kernels. At first I thought it would be related to the memory access in my app until I realized that even an unrelated foreign app can let the DMA-transfer break, while I let my app not do anything besides initializing the memory and DMA-mapping. But when letting my cuda-kernels run (instead of a foreign app), the load has influence on the duration until the DMA stops working.

There are no errors thrown in my app or written into the kern.log-file. It seems like the OS is not aware that something is going wrong.

Yes, the FPGA is externally powered. The FPGA doesn’t seem to get into an inconsistent state as all other functionality of it continues to work. It even continues to succesfully receive BAR-write accesses from my driver after the DMA has stopped (indicated by some status leds), so the PCIe-link is at least not breaking fully.

Are CUDA applications you are launching trying to access the same data that is being dumped by FPGA’s DMA?

Are CUDA applications you are launching trying to access the same data that is being dumped by FPGA’s DMA?

In the case of my own app running as intended, yes. But not in the case, when I launch the particles-app in the background and my app just sits there after having initialized the DMA and kicked off the data transmission from the FPGA. That’s why I conclude that it’s not the read-access to the DMA-mapped memory, which leads to problems, but something else.

Couple of follow-up questions…

  1. in #1, it is mentioned that FPGA infinately writes data to DMA-mapped region. How is it done? I mean, is it like a fixed region in memory is reserved for this purpose and FPGA starts dumping data to it starting from beginning and when it reaches end, it goes back to starting point and does it in loops? (or) everytime new regions are allocated?
  2. If you let your app map memory and configure DMA to kick off and let it run infinitely and keep dumping data and don’t start any CUDA app like particles, do you still see the issue? (wondering if continuous data dumping by FPGA alone has any issue?)
  3. Can you please share the logs when it crashes? would like to see where it is crashing

Also, Is it possible to repro it without really having the FPGA? (by some hacks or instrumenting the code). That would greatly help us to speed up debugging.

  1. There are 16 image sensors running in a synchronized manner. For each of them there is a specific range within the DMA-mapped memory for writing their pixels to. Through a structure of fifos their pixels are written to that memory. This process is going on forever.
  2. If the FPGA is writing data to the memory, but it’s not read and no process is utilizing GPU time, it runs forever without crashing. If I throttle the bandwidth to about a quarter of full utilization of the PCIe-link (by slowing down the pixel-clock of the sensors), it doesn’t crash either no matter what the GPU load is and if cuda-kernels are reading from that memory.
  3. There are no log-messages written to the kern.log-file when the system crashes. If you know other places where to look for log-messages, please let me know.
  4. Without having the FPGA generating DMA-activity this problem is not occurring. I don’t know about any way of reproducing the issue without having the hardware.

Hi,

Regarding point 3, When you say that system crashes, what actually is the HW behavior? Does it reboots or hangs?Is there a debug UART port in your HW from where you can collect runtime kernel logs?
Set the log level to max “dmesg -n 8” and then run your usecase.

You say, no data is there in kern.log, does that means, you are able to access the filesystem of the HW after the issue occurred?
If the system is rebooting on its own after the issue is reproduced, can you store the files mentioned below after the reboot and share?
/sys/fs/pstore/*

regards
Bibek

Could you share the tegrastats log (sudo /home/ubuntu/tegrastats) when all running ?

I control the Jetson through SSH. I don’t know what happened, but the behaviour has changed. When the error (DMA stops working) now happens, the system doens’t crash and I can continue to use it. But when I try either to map the DMA again, run lspci or try to remove the pci_tegra-module (rmmod pci_tegra) after the error occurred I loose control over the Jetson respectively I can’t connect again to it until I reset it. I don’t know what exactly is breaking the SSH-connection, but I assume that the system is gone by then.

I now waited a little longer after my loose of control (after calling lspci in this case) before restarting the system and there shows now something up in the kern.log-file:

Dec 29 15:58:18 tegra-ubuntu kernel: [  141.156188] Bad mode in Error handler detected, code 0xbf000002
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.162144] CPU: 0 PID: 571 Comm: NetworkManager Tainted: G           O 3.10.96+ #2
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.169790] task: ffffffc0f30cf100 ti: ffffffc0f1bf0000 task.ti: ffffffc0f1bf0000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.177261] PC is at 0x7fa9762e64
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.180567] LR is at 0x7fa9762e44
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.183874] pc : [<0000007fa9762e64>] lr : [<0000007fa9762e44>] pstate: 80000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.191254] sp : ffffffc0f1bf3ff0
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.194560] x29: 0000007fc9d2c870 x28: 0000000000000001 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.199879] x27: 0000000000000000 x26: 0000000000000001 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.205195] x25: 0000000000000001 x24: 0000007fa991c000 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.210511] x23: 0000007fa98732a0 x22: 0000000000001753 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.215827] x21: 0000007fc9d2c8a0 x20: 00000000007f1b10 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.221144] x19: 000000000000000a x18: 0000000018fd6694 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.226461] x17: 0000007fa9762d98 x16: 0000007fa991d220 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.231778] x15: 003b9aca00000000 x14: 0021b59ffe000000 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.237096] x13: ffffffffa79acdfd x12: 0000000000000018 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.242412] x11: 0000000001c91ebc x10: 000000000000007d 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.247730] x9 : 000000000000590c x8 : 0000000000000049 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.253048] x7 : 0000000000000000 x6 : 0000007fa742f6f0 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.258364] x5 : 0000000000000000 x4 : 0000000000000000 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.263680] x3 : 0000000000000000 x2 : 0000007fc9d2c8a0 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.268998] x1 : 000000000000000a x0 : 0000000000000001 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.274314] 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.275811] Bad mode in Synchronous Abort handler detected, code 0x8600000f
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.282761] CPU: 0 PID: 571 Comm: NetworkManager Tainted: G           O 3.10.96+ #2
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.290400] task: ffffffc0f30cf100 ti: ffffffc0f1bf0000 task.ti: ffffffc0f1bf0000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.297866] PC is at 0x7fa9762e44
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.301172] LR is at 0x7fa9762e44
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.304478] pc : [<0000007fa9762e44>] lr : [<0000007fa9762e44>] pstate: 800003c5
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.311857] sp : ffffffc0f1bf3ed0
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.315162] x29: 0000007fc9d2c870 x28: 0000000000000001 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.320479] x27: 0000000000000000 x26: 0000000000000001 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.325796] x25: 0000000000000001 x24: 0000007fa991c000 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.331112] x23: 0000000080000000 x22: 0000007fa9762e64 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.336429] x21: ffffffc0f1bf3ff0 x20: 00000000007f1b10 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.341744] x19: 000000000000000a x18: 0000000018fd6694 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.347061] x17: 0000007fa9762d98 x16: 000000000000000a 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.352379] x15: 0000000000000020 x14: 000000000000005d 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.357694] x13: 0000000000000034 x12: 0000000000000031 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.363011] x11: 0000000000000033 x10: 0000000000000034 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.368328] x9 : 0000000000000037 x8 : 0000000000000006 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.373645] x7 : ffffffc0000860e0 x6 : ffffffc0f1bf3e90 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.378959] x5 : 0000000000000007 x4 : 0000000000000012 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.384275] x3 : 0000000000000001 x2 : 0000000000000001 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.389591] x1 : ffffffc0f1bf3dc0 x0 : 0000000000000000 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.394908] 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.394908] SP: 0xffffffc0f1bf3e50:
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.399861] 3e50  007f1b10 00000000 f1bf3ff0 ffffffc0 a9762e64 0000007f 80000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.408106] 3e70  a991c000 0000007f 00000001 00000000 00000001 00000000 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.416347] 3e90  00000001 00000000 c9d2c870 0000007f a9762e44 0000007f f1bf3ed0 ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.424587] 3eb0  a9762e44 0000007f 800003c5 00000000 0000000a 00000000 007f1b10 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.432826] 3ed0  00000001 00000000 0000000a 00000000 c9d2c8a0 0000007f 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.441067] 3ef0  00000000 00000000 00000000 00000000 a742f6f0 0000007f 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.449307] 3f10  00000049 00000000 0000590c 00000000 0000007d 00000000 01c91ebc 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.457548] 3f30  00000018 00000000 a79acdfd ffffffff fe000000 0021b59f 00000000 003b9aca
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.465791] 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.465791] X1: 0xffffffc0f1bf3d40:
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.470743] 3d40  f1bf3d90 ffffffc0 000bc6c0 ffffffc0 f1bf3e50 ffffffc0 f30cf100 ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.478986] 3d60  00000004 00000000 00000000 00000000 f1bf3e50 ffffffc0 a991c000 0000007f
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.487229] 3d80  f1bf3dc0 ffffffc0 00b358d0 ffffffc0 000003c0 00000000 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.495471] 3da0  00000004 00000000 000003c0 00000000 00000000 00000000 f1bf3dc0 ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.503713] 3dc0  00000001 00000000 00000001 00000000 00000012 00000000 00000007 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.511954] 3de0  f1bf3e90 ffffffc0 000860e0 ffffffc0 00000006 00000000 00000037 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.520199] 3e00  00000034 00000000 00000033 00000000 00000031 00000000 00000034 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.528440] 3e20  0000005d 00000000 00000020 00000000 0000000a 00000000 a9762d98 0000007f
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.536683] 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.536683] X6: 0xffffffc0f1bf3e10:
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.541636] 3e10  00000031 00000000 00000034 00000000 0000005d 00000000 00000020 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.549879] 3e30  0000000a 00000000 a9762d98 0000007f 18fd6694 00000000 0000000a 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.558120] 3e50  007f1b10 00000000 f1bf3ff0 ffffffc0 a9762e64 0000007f 80000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.566359] 3e70  a991c000 0000007f 00000001 00000000 00000001 00000000 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.574597] 3e90  00000001 00000000 c9d2c870 0000007f a9762e44 0000007f f1bf3ed0 ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.582837] 3eb0  a9762e44 0000007f 800003c5 00000000 0000000a 00000000 007f1b10 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.591077] 3ed0  00000001 00000000 0000000a 00000000 c9d2c8a0 0000007f 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.599318] 3ef0  00000000 00000000 00000000 00000000 a742f6f0 0000007f 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.607560] 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.607560] X7: 0xffffffc000086060:
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.612513] 6060  f9400000 36080040 942ab8af a8c17bfd d65f03c0 a9be7bfd 910003fd a90153f3
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.620755] 6080  52800020 94016166 52800060 910003e2 9272c441 940a81fb 34000260 910003e0
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.628996] 60a0  9272c414 f9400a93 91150273 aa1303e0 9400005c d53b4221 d50342df d0008940
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.637235] 60c0  91008000 b9804e83 90009c02 912de042 f8637842 f8226813 d51b4221 940ade01
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.645474] 60e0  b9021260 52800020 94016181 910003e1 9272c420 f9400000 36080040 942ab88a
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.653715] 6100  a94153f3 a8c27bfd d65f03c0 a9be7bfd 910003fd f9000bf3 aa0003f3 52800020
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.661957] 6120  9401613f aa1303e0 9400003e 52800060 910003e2 9272c441 940a81d2 34000220
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.670201] 6140  910003e0 9272c401 f9400833 d53b4222 d50342df d0008940 91008000 b9804c23
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.678446] 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.678446] X21: 0xffffffc0f1bf3f70:
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.683485] 3f70  007f1b10 00000000 c9d2c8a0 0000007f 00001753 00000000 a98732a0 0000007f
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.691727] 3f90  a991c000 0000007f 00000001 00000000 00000001 00000000 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.699968] 3fb0  00000001 00000000 c9d2c870 0000007f a9762e44 0000007f f1bf3ff0 ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.708207] 3fd0  a9762e64 0000007f 80000000 00000000 007f1b10 00000000 00000049 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.716448] 3ff0  0028b5aa 00000000 00000102 00000016 00000006 00000574 31797474 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.724690] 4010  00000000 00000000 00000000 00000000 00000000 00000000 31797474 49474f4c
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.732930] 4030  0000004e 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.741173] 4050  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.749418] 
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.750906] Internal error: Oops - bad mode: 0 [#1] PREEMPT SMP
Dec 29 15:58:18 tegra-ubuntu kernel: [  141.756813] Enter nvdumper_crash_setup_regs
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684741] nvdumper: all registers are saved.
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684743] nvdumper: all registers are saved.
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684789] nvdumper: all registers are saved.
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684909] Bad mode in Error handler detected, code 0xbf000002
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684913] CPU: 1 PID: 776 Comm: Xorg Tainted: G           O 3.10.96+ #2
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684916] task: ffffffc0e62aaec0 ti: ffffffc0e62ac000 task.ti: ffffffc0e62ac000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684920] PC is at 0x7f93956888
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684922] LR is at 0x7f9395686c
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684924] pc : [<0000007f93956888>] lr : [<0000007f9395686c>] pstate: 80000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684925] sp : ffffffc0e62afff0
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684930] x29: 0000007fe8082220 x28: 000000555c64f620 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684934] x27: 000000555ca88e20 x26: 0000007fe8082318 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684937] x25: 000000555c649f58 x24: 000000555c62c000 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684941] x23: 000000555c62c000 x22: 000000555c640000 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684944] x21: 000000555c62c000 x20: 0000000000000200 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684947] x19: 0000007fe8082308 x18: 0000000000000099 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684950] x17: 0000007f939567b8 x16: 000000555c62d400 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684954] x15: 003b9aca00000000 x14: 0021b59ffe000000 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684957] x13: ffffffffa79acdfd x12: 0000000000000018 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684961] x11: 000000000d6557bb x10: 000000000000008d 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684964] x9 : 000000555c62c000 x8 : 0000000000000048 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684967] x7 : 0000000000000000 x6 : 0000000000000000 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684971] x5 : 0000000000000000 x4 : 0000007fe8082260 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684974] x3 : 0000000000000000 x2 : 0000000000000000 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684977] x1 : 000000555c649fe0 x0 : 0000000000000000 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684978] 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684986] Bad mode in Synchronous Abort handler detected, code 0x8600000f
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684989] CPU: 1 PID: 776 Comm: Xorg Tainted: G           O 3.10.96+ #2
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684992] task: ffffffc0e62aaec0 ti: ffffffc0e62ac000 task.ti: ffffffc0e62ac000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684994] PC is at 0x7f9395686c
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684996] LR is at 0x7f9395686c
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684998] pc : [<0000007f9395686c>] lr : [<0000007f9395686c>] pstate: 600003c5
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.684999] sp : ffffffc0e62afed0
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685003] x29: 0000007fe8082220 x28: 000000555c64f620 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685007] x27: 000000555ca88e20 x26: 0000007fe8082318 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685010] x25: 000000555c649f58 x24: 000000555c62c000 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685014] x23: 0000000080000000 x22: 0000007f93956888 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685017] x21: ffffffc0e62afff0 x20: 0000000000000200 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685020] x19: 0000007fe8082308 x18: 0000000000000099 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685024] x17: 0000007f939567b8 x16: 000000000000000a 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685027] x15: 0000000000000030 x14: 0000000000000030 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685031] x13: 0000000000000030 x12: 0000000000000030 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685034] x11: 0000000000000030 x10: 0000000000000030 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685037] x9 : 0000000000000030 x8 : 0000000000000007 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685041] x7 : ffffffc0000860e0 x6 : ffffffc0e62afe90 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685044] x5 : 0000000000000000 x4 : 0000000000000018 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685047] x3 : 0000000000000001 x2 : 0000000000000001 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685050] x1 : ffffffc0e62afdc0 x0 : 0000000000000000 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685053] 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685053] SP: 0xffffffc0e62afe50:
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685062] fe50  00000200 00000000 e62afff0 ffffffc0 93956888 0000007f 80000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685071] fe70  5c62c000 00000055 5c649f58 00000055 e8082318 0000007f 5ca88e20 00000055
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685080] fe90  5c64f620 00000055 e8082220 0000007f 9395686c 0000007f e62afed0 ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685088] feb0  9395686c 0000007f 600003c5 00000000 e8082308 0000007f 00000200 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685096] fed0  00000000 00000000 5c649fe0 00000055 00000000 00000000 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685104] fef0  e8082260 0000007f 00000000 00000000 00000000 00000000 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685113] ff10  00000048 00000000 5c62c000 00000055 0000008d 00000000 0d6557bb 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685121] ff30  00000018 00000000 a79acdfd ffffffff fe000000 0021b59f 00000000 003b9aca
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685124] 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685124] X1: 0xffffffc0e62afd40:
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685132] fd40  e62afd90 ffffffc0 000bc6c0 ffffffc0 e62afe50 ffffffc0 e62aaec0 ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685140] fd60  00000004 00000000 00000000 00000000 e62afe50 ffffffc0 5c62c000 00000055
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685148] fd80  e62afdc0 ffffffc0 00b358d0 ffffffc0 000003c0 00000000 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685156] fda0  00000004 00000000 000003c0 00000000 00000000 00000000 e62afdc0 ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685164] fdc0  00000001 00000000 00000001 00000000 00000018 00000000 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685172] fde0  e62afe90 ffffffc0 000860e0 ffffffc0 00000007 00000000 00000030 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685180] fe00  00000030 00000000 00000030 00000000 00000030 00000000 00000030 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685189] fe20  00000030 00000000 00000030 00000000 0000000a 00000000 939567b8 0000007f
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685192] 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685192] X6: 0xffffffc0e62afe10:
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685200] fe10  00000030 00000000 00000030 00000000 00000030 00000000 00000030 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685208] fe30  0000000a 00000000 939567b8 0000007f 00000099 00000000 e8082308 0000007f
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685217] fe50  00000200 00000000 e62afff0 ffffffc0 93956888 0000007f 80000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685225] fe70  5c62c000 00000055 5c649f58 00000055 e8082318 0000007f 5ca88e20 00000055
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685234] fe90  5c64f620 00000055 e8082220 0000007f 9395686c 0000007f e62afed0 ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685242] feb0  9395686c 0000007f 600003c5 00000000 e8082308 0000007f 00000200 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685251] fed0  00000000 00000000 5c649fe0 00000055 00000000 00000000 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685259] fef0  e8082260 0000007f 00000000 00000000 00000000 00000000 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685260] 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685260] X7: 0xffffffc000086060:
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685269] 6060  f9400000 36080040 942ab8af a8c17bfd d65f03c0 a9be7bfd 910003fd a90153f3
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685277] 6080  52800020 94016166 52800060 910003e2 9272c441 940a81fb 34000260 910003e0
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685285] 60a0  9272c414 f9400a93 91150273 aa1303e0 9400005c d53b4221 d50342df d0008940
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685293] 60c0  91008000 b9804e83 90009c02 912de042 f8637842 f8226813 d51b4221 940ade01
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685302] 60e0  b9021260 52800020 94016181 910003e1 9272c420 f9400000 36080040 942ab88a
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685310] 6100  a94153f3 a8c27bfd d65f03c0 a9be7bfd 910003fd f9000bf3 aa0003f3 52800020
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685318] 6120  9401613f aa1303e0 9400003e 52800060 910003e2 9272c441 940a81d2 34000220
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685326] 6140  910003e0 9272c401 f9400833 d53b4222 d50342df d0008940 91008000 b9804c23
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685331] 
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685331] X21: 0xffffffc0e62aff70:
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685340] ff70  00000200 00000000 5c62c000 00000055 5c640000 00000055 5c62c000 00000055
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685348] ff90  5c62c000 00000055 5c649f58 00000055 e8082318 0000007f 5ca88e20 00000055
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685357] ffb0  5c64f620 00000055 e8082220 0000007f 9395686c 0000007f e62afff0 ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685365] ffd0  93956888 0000007f 80000000 00000000 00000200 00000000 00000048 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685374] fff0  00000000 00000000 00000000 00000000 e62c1000 ffffffc0 e62a9000 ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685382] 0010  00000088 00000000 e62b0088 ffffffc0 00000016 ffffffff 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685391] 0030  00000001 00000002 00000003 00000004 00000005 00000006 00000007 00000008
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685399] 0050  00000009 0000000a 0000000b 0000000c 0000000d 0000000e 0000000f 00000010
Dec 29 15:58:18 tegra-ubuntu kernel: [  151.685402] 
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286839] nvdumper: all registers are saved.
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286866] INFO: rcu_preempt self-detected stall on CPU { 2}  (t=2166 jiffies g=737 c=736 q=141)
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286871] CPU: 2 PID: 1177 Comm: gnome-session-b Tainted: G           O 3.10.96+ #2
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286873] Call trace:
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286881] [<ffffffc000089dd4>] dump_backtrace+0x0/0xf4
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286885] [<ffffffc00008a0d4>] show_stack+0x14/0x1c
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286890] [<ffffffc0003287bc>] dump_stack+0x20/0x28
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286895] [<ffffffc00012b27c>] print_cpu_stall+0xd8/0x140
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286899] [<ffffffc00012b76c>] check_cpu_stall+0x64/0x9c
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286903] [<ffffffc00012b7d0>] __rcu_pending+0x2c/0x128
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286906] [<ffffffc00012b91c>] rcu_pending+0x50/0x84
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286911] [<ffffffc00012e154>] rcu_check_callbacks+0xd8/0x164
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286915] [<ffffffc0000ba374>] update_process_times+0x44/0x70
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286920] [<ffffffc000102b34>] tick_sched_handle+0x54/0x60
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286924] [<ffffffc000102b94>] tick_sched_timer+0x54/0x7c
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286927] [<ffffffc0000d5768>] __run_hrtimer+0x168/0x27c
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286930] [<ffffffc0000d613c>] hrtimer_interrupt+0x15c/0x278
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286936] [<ffffffc0007e34f8>] tegra210_timer_isr+0x3c/0x4c
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286940] [<ffffffc000122c34>] handle_irq_event_percpu+0xd0/0x21c
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286944] [<ffffffc000122dc8>] handle_irq_event+0x48/0x74
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286948] [<ffffffc000125f9c>] handle_fasteoi_irq+0xc4/0x10c
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286952] [<ffffffc0001220f4>] generic_handle_irq+0x28/0x40
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286956] [<ffffffc0000859dc>] handle_IRQ+0x94/0xc8
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286959] [<ffffffc0000813d4>] gic_handle_irq+0x74/0x184
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286961] Exception stack(0xffffffc0d033bb80 to 0xffffffc0d033bca0)
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286966] bb80: 00000000 066e0000 00000000 00000000 d033bce0 ffffffc0 000de6ec ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286971] bba0: 80000145 00000000 00000001 00000000 f7552df0 ffffffc0 f7552dd0 ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286974] bbc0: 00000001 00000000 00000004 00000000 00000003 00000000 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286978] bbe0: 00000000 00000000 000003be 00000000 00fc8668 ffffffc0 011b7897 ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286982] bc00: 0000000b 00000000 00000000 00000000 00000fd3 00600000 00100173 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286985] bc20: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286989] bc40: d2058430 ffffffc0 9666fc30 0000007f ffffff00 00000000 00000000 066e0000
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286993] bc60: 00000000 00000000 f6a1d380 ffffffc0 00000000 00000000 e78df8c8 ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.286997] bc80: eb1a7c80 ffffffc0 f7552df0 ffffffc0 f7552dd0 ffffffc0 e78df208 ffffffc0
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.287001] [<ffffffc000084e04>] el1_irq+0x84/0xf0
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.287005] [<ffffffc0000a557c>] dup_mm+0x84/0x108
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.287009] [<ffffffc0000a5664>] copy_mm+0x64/0x94
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.287012] [<ffffffc0000a5a24>] copy_process+0x390/0xad4
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.287016] [<ffffffc0000a61f4>] do_fork+0x8c/0x2d4
Dec 29 15:58:18 tegra-ubuntu kernel: [  152.287019] [<ffffffc0000a6574>] SyS_clone+0xb0/0x140
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.530591] Modules linked in: pci_tegra ql2pcie(O) bnep bcmdhd cfg80211 bluedroid_pm
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.538478] CPU: 0 PID: 571 Comm: NetworkManager Tainted: G           O 3.10.96+ #2
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.546119] task: ffffffc0f30cf100 ti: ffffffc0f1bf0000 task.ti: ffffffc0f1bf0000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.553586] PC is at 0x7fa9762e44
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.556892] LR is at 0x7fa9762e44
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.560198] pc : [<0000007fa9762e44>] lr : [<0000007fa9762e44>] pstate: 800003c5
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.567577] sp : ffffffc0f1bf3ed0
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.570881] x29: 0000007fc9d2c870 x28: 0000000000000001 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.576199] x27: 0000000000000000 x26: 0000000000000001 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.581517] x25: 0000000000000001 x24: 0000007fa991c000 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.586833] x23: 0000000080000000 x22: 0000007fa9762e64 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.592149] x21: ffffffc0f1bf3ff0 x20: 00000000007f1b10 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.597465] x19: 000000000000000a x18: 0000000018fd6694 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.602782] x17: 0000007fa9762d98 x16: 000000000000000a 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.608098] x15: 0000000000000020 x14: 000000000000005d 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.613412] x13: 0000000000000034 x12: 0000000000000031 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.618728] x11: 0000000000000033 x10: 0000000000000034 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.624044] x9 : 0000000000000037 x8 : 0000000000000006 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.629358] x7 : ffffffc0000860e0 x6 : ffffffc0f1bf3e90 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.634672] x5 : 0000000000000007 x4 : 0000000000000012 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.639987] x3 : 0000000000000001 x2 : 0000000000000001 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.645303] x1 : ffffffc0f1bf3dc0 x0 : 0000000000000000 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.650622] 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.650622] SP: 0xffffffc0f1bf3e50:
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.655574] 3e50  007f1b10 00000000 f1bf3ff0 ffffffc0 a9762e64 0000007f 80000000 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.663815] 3e70  a991c000 0000007f 00000001 00000000 00000001 00000000 00000000 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.672052] 3e90  00000001 00000000 c9d2c870 0000007f a9762e44 0000007f f1bf3ed0 ffffffc0
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.680293] 3eb0  a9762e44 0000007f 800003c5 00000000 0000000a 00000000 007f1b10 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.688533] 3ed0  00000001 00000000 0000000a 00000000 c9d2c8a0 0000007f 00000000 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.696772] 3ef0  00000000 00000000 00000000 00000000 a742f6f0 0000007f 00000000 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.705008] 3f10  00000049 00000000 0000590c 00000000 0000007d 00000000 01c91ebc 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.713243] 3f30  00000018 00000000 a79acdfd ffffffff fe000000 0021b59f 00000000 003b9aca
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.721483] 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.721483] X1: 0xffffffc0f1bf3d40:
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.726435] 3d40  a9762e44 0000007f 000bc6c0 ffffffc0 f1bf3e50 ffffffc0 f30cf100 ffffffc0
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.734675] 3d60  00000004 00000000 00000000 00000000 f1bf3e50 ffffffc0 a991c000 0000007f
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.742913] 3d80  f1bf3dc0 ffffffc0 00b358d0 ffffffc0 000003c0 00000000 00000000 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.751154] 3da0  00000004 00000000 000003c0 00000000 00000000 00000000 f1bf3dc0 ffffffc0
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.759393] 3dc0  00000001 00000000 00000001 00000000 00000012 00000000 00000007 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.767629] 3de0  f1bf3e90 ffffffc0 000860e0 ffffffc0 00000006 00000000 00000037 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.775865] 3e00  00000034 00000000 00000033 00000000 00000031 00000000 00000034 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.784101] 3e20  0000005d 00000000 00000020 00000000 0000000a 00000000 a9762d98 0000007f
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.792346] 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.792346] X6: 0xffffffc0f1bf3e10:
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.797298] 3e10  00000031 00000000 00000034 00000000 0000005d 00000000 00000020 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.805536] 3e30  0000000a 00000000 a9762d98 0000007f 18fd6694 00000000 0000000a 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.813775] 3e50  007f1b10 00000000 f1bf3ff0 ffffffc0 a9762e64 0000007f 80000000 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.822017] 3e70  a991c000 0000007f 00000001 00000000 00000001 00000000 00000000 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.830253] 3e90  00000001 00000000 c9d2c870 0000007f a9762e44 0000007f f1bf3ed0 ffffffc0
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.838490] 3eb0  a9762e44 0000007f 800003c5 00000000 0000000a 00000000 007f1b10 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.846728] 3ed0  00000001 00000000 0000000a 00000000 c9d2c8a0 0000007f 00000000 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.854967] 3ef0  00000000 00000000 00000000 00000000 a742f6f0 0000007f 00000000 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.863205] 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.863205] X7: 0xffffffc000086060:
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.868157] 6060  f9400000 36080040 942ab8af a8c17bfd d65f03c0 a9be7bfd 910003fd a90153f3
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.876397] 6080  52800020 94016166 52800060 910003e2 9272c441 940a81fb 34000260 910003e0
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.884635] 60a0  9272c414 f9400a93 91150273 aa1303e0 9400005c d53b4221 d50342df d0008940
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.892872] 60c0  91008000 b9804e83 90009c02 912de042 f8637842 f8226813 d51b4221 940ade01
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.901112] 60e0  b9021260 52800020 94016181 910003e1 9272c420 f9400000 36080040 942ab88a
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.909350] 6100  a94153f3 a8c27bfd d65f03c0 a9be7bfd 910003fd f9000bf3 aa0003f3 52800020
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.917588] 6120  9401613f aa1303e0 9400003e 52800060 910003e2 9272c441 940a81d2 34000220
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.925826] 6140  910003e0 9272c401 f9400833 d53b4222 d50342df d0008940 91008000 b9804c23
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.934069] 
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.934069] X21: 0xffffffc0f1bf3f70:
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.939107] 3f70  007f1b10 00000000 c9d2c8a0 0000007f 00001753 00000000 a98732a0 0000007f
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.947348] 3f90  a991c000 0000007f 00000001 00000000 00000001 00000000 00000000 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.955586] 3fb0  00000001 00000000 c9d2c870 0000007f a9762e44 0000007f f1bf3ff0 ffffffc0
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.963827] 3fd0  a9762e64 0000007f 80000000 00000000 007f1b10 00000000 00000049 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.972064] 3ff0  0028b5aa 00000000 00000102 00000016 00000006 00000574 31797474 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.980298] 4010  00000000 00000000 00000000 00000000 00000000 00000000 31797474 49474f4c
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.988537] 4030  0000004e 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  152.996775] 4050  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.005017] 
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.006505] Process NetworkManager (pid: 571, stack limit = 0xffffffc0f1bf0058)
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.013800] Stack: (0xffffffc0f1bf3ed0 to 0xffffffc0f1bf4000)
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.019534] 3ec0:                                     00000001 00000000 0000000a 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.027696] 3ee0: c9d2c8a0 0000007f 00000000 00000000 00000000 00000000 00000000 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.035858] 3f00: a742f6f0 0000007f 00000000 00000000 00000049 00000000 0000590c 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.044021] 3f20: 0000007d 00000000 01c91ebc 00000000 00000018 00000000 a79acdfd ffffffff
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.052181] 3f40: fe000000 0021b59f 00000000 003b9aca a991d220 0000007f a9762d98 0000007f
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.060344] 3f60: 18fd6694 00000000 0000000a 00000000 007f1b10 00000000 c9d2c8a0 0000007f
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.068505] 3f80: 00001753 00000000 a98732a0 0000007f a991c000 0000007f 00000001 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.076666] 3fa0: 00000001 00000000 00000000 00000000 00000001 00000000 c9d2c870 0000007f
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.084826] 3fc0: a9762e44 0000007f f1bf3ff0 ffffffc0 a9762e64 0000007f 80000000 00000000
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.092988] 3fe0: 007f1b10 00000000 00000049 00000000 0028b5aa 00000000 00000102 00000016
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.101148] Call trace:
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.103590] Code: 34fffde2 aa0103f3 aa0003f4 9400517c (2a0003e5) 
Dec 29 15:58:19 tegra-ubuntu kernel: [  153.109674] Internal error: Oops - bad mode: 0 [#2] PREEMPT SMP
Dec 29 15:58:30 tegra-ubuntu kernel: [  153.115590] Enter nvdumper_crash_setup_regs
Dec 29 15:58:30 tegra-ubuntu kernel: [  153.116623] ---[ end trace ce8d0e06a10d34dd ]---
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.648291] nvdumper: all registers are saved.
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.648320] nvdumper: all registers are saved.
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.648326] nvdumper: all registers are saved.
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661607] nvdumper: all registers are saved.
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661747] Bad mode in Error handler detected, code 0xbf000002
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661754] CPU: 2 PID: 873 Comm: rtkit-daemon Tainted: G      D    O 3.10.96+ #2
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661758] task: ffffffc0fdd051c0 ti: ffffffc0d74a0000 task.ti: ffffffc0d74a0000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661763] PC is at 0x7f8ec0ee64
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661765] LR is at 0x7f8ec0ee44
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661768] pc : [<0000007f8ec0ee64>] lr : [<0000007f8ec0ee44>] pstate: 80000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661769] sp : ffffffc0d74a3ff0
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661774] x29: 0000007f8e0be780 x28: 0000000000025325 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661777] x27: 000000000041e000 x26: 000000000041e000 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661781] x25: 0000000000000260 x24: 0000000000004c00 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661784] x23: 0000000000000000 x22: 0000000000000001 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661787] x21: 0000007f8e0be7b0 x20: 0000007f8e0be868 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661791] x19: 0000000000000002 x18: 0000000000000014 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661797] x17: 0000007f8ec0ed98 x16: 000000000041e110 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661807] x15: 003b9aca00000000 x14: 0021b59ffe000000 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661816] x13: ffffffffa79acdfd x12: 0000000000000018 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661824] x11: 00000000154c7921 x10: 0000000000000098 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661833] x9 : 0000000000005d02 x8 : 0000000000000049 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661841] x7 : 0000000000000000 x6 : 0000007f8e0bf780 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661849] x5 : 0000000000000000 x4 : 0000000000000000 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661854] x3 : 0000000000000000 x2 : 0000007f8e0be7b0 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661857] x1 : 0000000000000002 x0 : 0000000000000001 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661858] 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661867] Bad mode in Synchronous Abort handler detected, code 0x8600000f
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661870] CPU: 2 PID: 873 Comm: rtkit-daemon Tainted: G      D    O 3.10.96+ #2
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661873] task: ffffffc0fdd051c0 ti: ffffffc0d74a0000 task.ti: ffffffc0d74a0000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661875] PC is at 0x7f8ec0ee44
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661876] LR is at 0x7f8ec0ee44
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661878] pc : [<0000007f8ec0ee44>] lr : [<0000007f8ec0ee44>] pstate: 600003c5
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661880] sp : ffffffc0d74a3ed0
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661883] x29: 0000007f8e0be780 x28: 0000000000025325 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661886] x27: 000000000041e000 x26: 000000000041e000 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661889] x25: 0000000000000260 x24: 0000000000004c00 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661893] x23: 0000000080000000 x22: 0000007f8ec0ee64 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661896] x21: ffffffc0d74a3ff0 x20: 0000007f8e0be868 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661899] x19: 0000000000000002 x18: 0000000000000014 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661902] x17: 0000007f8ec0ed98 x16: 000000000000000a 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661905] x15: 0000000000000031 x14: 0000000000000030 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661909] x13: 0000000000000030 x12: 0000000000000030 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661912] x11: 0000000000000030 x10: 0000000000000030 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661915] x9 : 0000000000000030 x8 : 0000000000000007 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661918] x7 : ffffffc0000860e0 x6 : ffffffc0d74a3e90 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661924] x5 : 0000000000000000 x4 : 0000000000000018 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661932] x3 : 0000000000000001 x2 : 0000000000000001 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661941] x1 : ffffffc0d74a3dc0 x0 : 0000000000000000 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661945] 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661945] SP: 0xffffffc0d74a3e50:
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661971] 3e50  8e0be868 0000007f d74a3ff0 ffffffc0 8ec0ee64 0000007f 80000000 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.661996] 3e70  00004c00 00000000 00000260 00000000 0041e000 00000000 0041e000 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662005] 3e90  00025325 00000000 8e0be780 0000007f 8ec0ee44 0000007f d74a3ed0 ffffffc0
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662013] 3eb0  8ec0ee44 0000007f 600003c5 00000000 00000002 00000000 8e0be868 0000007f
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662021] 3ed0  00000001 00000000 00000002 00000000 8e0be7b0 0000007f 00000000 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662029] 3ef0  00000000 00000000 00000000 00000000 8e0bf780 0000007f 00000000 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662037] 3f10  00000049 00000000 00005d02 00000000 00000098 00000000 154c7921 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662045] 3f30  00000018 00000000 a79acdfd ffffffff fe000000 0021b59f 00000000 003b9aca
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662048] 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662048] X1: 0xffffffc0d74a3d40:
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662056] 3d40  d74a3d90 ffffffc0 000bc6c0 ffffffc0 d74a3e50 ffffffc0 fdd051c0 ffffffc0
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662064] 3d60  00000004 00000000 00000000 00000000 d74a3e50 ffffffc0 00004c00 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662073] 3d80  d74a3dc0 ffffffc0 00b358d0 ffffffc0 000003c0 00000000 00000000 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662081] 3da0  00000004 00000000 000003c0 00000000 00000000 00000000 d74a3dc0 ffffffc0
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662093] 3dc0  00000001 00000000 00000001 00000000 00000018 00000000 00000000 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662120] 3de0  d74a3e90 ffffffc0 000860e0 ffffffc0 00000007 00000000 00000030 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662132] 3e00  00000030 00000000 00000030 00000000 00000030 00000000 00000030 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662140] 3e20  00000030 00000000 00000031 00000000 0000000a 00000000 8ec0ed98 0000007f
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662143] 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662143] X6: 0xffffffc0d74a3e10:
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662151] 3e10  00000030 00000000 00000030 00000000 00000030 00000000 00000031 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662159] 3e30  0000000a 00000000 8ec0ed98 0000007f 00000014 00000000 00000002 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662168] 3e50  8e0be868 0000007f d74a3ff0 ffffffc0 8ec0ee64 0000007f 80000000 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662176] 3e70  00004c00 00000000 00000260 00000000 0041e000 00000000 0041e000 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662198] 3e90  00025325 00000000 8e0be780 0000007f 8ec0ee44 0000007f d74a3ed0 ffffffc0
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662208] 3eb0  8ec0ee44 0000007f 600003c5 00000000 00000002 00000000 8e0be868 0000007f
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662216] 3ed0  00000001 00000000 00000002 00000000 8e0be7b0 0000007f 00000000 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662224] 3ef0  00000000 00000000 00000000 00000000 8e0bf780 0000007f 00000000 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662226] 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662226] X7: 0xffffffc000086060:
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662238] 6060  f9400000 36080040 942ab8af a8c17bfd d65f03c0 a9be7bfd 910003fd a90153f3
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662253] 6080  52800020 94016166 52800060 910003e2 9272c441 940a81fb 34000260 910003e0
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662261] 60a0  9272c414 f9400a93 91150273 aa1303e0 9400005c d53b4221 d50342df d0008940
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662269] 60c0  91008000 b9804e83 90009c02 912de042 f8637842 f8226813 d51b4221 940ade01
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662283] 60e0  b9021260 52800020 94016181 910003e1 9272c420 f9400000 36080040 942ab88a
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662293] 6100  a94153f3 a8c27bfd d65f03c0 a9be7bfd 910003fd f9000bf3 aa0003f3 52800020
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662304] 6120  9401613f aa1303e0 9400003e 52800060 910003e2 9272c441 940a81d2 34000220
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662315] 6140  910003e0 9272c401 f9400833 d53b4222 d50342df d0008940 91008000 b9804c23
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662322] 
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662322] X21: 0xffffffc0d74a3f70:
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662334] 3f70  8e0be868 0000007f 8e0be7b0 0000007f 00000001 00000000 00000000 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662345] 3f90  00004c00 00000000 00000260 00000000 0041e000 00000000 0041e000 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662356] 3fb0  00025325 00000000 8e0be780 0000007f 8ec0ee44 0000007f d74a3ff0 ffffffc0
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662367] 3fd0  8ec0ee64 0000007f 80000000 00000000 8e0be868 0000007f 00000049 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662377] 3ff0  00000000 00000000 00000000 00000000 000081ed 000006b5 5842513c 5842513c
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662387] 4010  546b3f55 00000000 00010000 00000008 00080000 00000001 0001f30a 00000004
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662397] 4030  00000000 00000000 00000001 001082ae 00000000 00000000 00000000 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662407] 4050  00000000 00000000 00000000 00000000 00000000 8fc9b953 00000000 00000000
Dec 29 15:58:30 tegra-ubuntu kernel: [  163.662411] 
Dec 29 15:58:31 tegra-ubuntu kernel: [  164.256249] Modules linked in: pci_tegra ql2pcie(O) bnep bcmdhd cfg80211 bluedroid_pm
Dec 29 15:58:31 tegra-ubuntu kernel: [  164.264135] CPU: 1 PID: 776 Comm: Xorg Tainted: G      D    O 3.10.96+ #2
Dec 29 15:58:31 tegra-ubuntu kernel: [  164.270909] task: ffffffc0e62aaec0 ti: ffffffc0e62ac000 task.ti: ffffffc0e62ac000
Dec 29 15:58:31 tegra-ubuntu kernel: [  164.278377] PC is at 0x7f9395686c
Dec 29 15:58:31 tegra-ubuntu kernel: [  164.281682] LR is at 0x7f9395686c
Dec 29 15:58:31 tegra-ubuntu kernel: [  164.284987] pc : [<0000007f9395686c>] lr : [<0000007f9395686c>] pstate: 600003c5
Dec 29 15:58:31 tegra-ubuntu kernel: [  164.292365] sp : ffffffc0e62afed0
Dec 29 15:58:31 tegra-ubuntu kernel: [  164.295671] x29: 0000007fe8082220 x28: 000000555c64f620 
Dec 29 15:58:31 tegra-ubuntu kernel: [  164.300987] x27: 000000555ca88e20 x26: 0000007fe8082318 
Dec 29 15:58:31 tegra-ubuntu kernel: [  164.306304] x25: 000000555c649f58 x24: 000000555c62c000 
Dec 29 1Dec 29 15:57:33 tegra-ubuntu kernel: [    0.000000] Initializing cgroup subsys cpuset

Does this make any sense to you?

Here is the output of tegrastats where I did a full walkthrough from idle to setting up fpga, pci, starting my app (for initializing the DMA and holding the memory), running particles until error, stopping it, stopping my app:

RAM 924/3995MB (lfb 620x4MB) cpu [0%,0%,0%,0%]@1555 EMC 1%@1331 AVP 7%@115 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 924/3995MB (lfb 620x4MB) cpu [2%,0%,1%,0%]@102 EMC 3%@408 AVP 68%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 924/3995MB (lfb 620x4MB) cpu [3%,0%,1%,0%]@102 EMC 3%@408 AVP 68%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 924/3995MB (lfb 620x4MB) cpu [5%,0%,0%,0%]@102 EMC 3%@408 AVP 68%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 926/3995MB (lfb 620x4MB) cpu [7%,2%,2%,5%]@102 EMC 3%@408 AVP 68%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 926/3995MB (lfb 620x4MB) cpu [7%,0%,1%,2%]@102 EMC 3%@408 AVP 75%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 935/3995MB (lfb 618x4MB) cpu [13%,15%,3%,7%]@102 EMC 3%@408 AVP 82%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 935/3995MB (lfb 618x4MB) cpu [19%,4%,2%,0%]@102 EMC 3%@408 AVP 84%@13 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 935/3995MB (lfb 618x4MB) cpu [9%,6%,1%,0%]@102 EMC 3%@408 AVP 84%@13 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 928/3995MB (lfb 617x4MB) cpu [10%,10%,6%,5%]@204 EMC 2%@665 AVP 84%@14 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 928/3995MB (lfb 617x4MB) cpu [4%,4%,4%,6%]@102 EMC 2%@665 AVP 82%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1351/3995MB (lfb 512x4MB) cpu [11%,10%,3%,77%]@102 EMC 10%@665 AVP 37%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1351/3995MB (lfb 512x4MB) cpu [10%,7%,2%,0%]@102 EMC 4%@665 AVP 80%@12 NVDEC 268 MSENC 268 GR3D 3%@76 EDP limit 1734
RAM 1358/3995MB (lfb 511x4MB) cpu [45%,2%,1%,0%]@1734 EMC 2%@1600 AVP 53%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1358/3995MB (lfb 511x4MB) cpu [100%,0%,0%,0%]@1734 EMC 3%@1600 AVP 19%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1358/3995MB (lfb 511x4MB) cpu [100%,0%,0%,0%]@1734 EMC 3%@1600 AVP 5%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1357/3995MB (lfb 511x4MB) cpu [100%,0%,0%,0%]@1734 EMC 3%@1600 AVP 5%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1427/3995MB (lfb 495x4MB) cpu [85%,23%,14%,43%]@1734 EMC 12%@1600 AVP 5%@12 NVDEC 268 MSENC 268 GR3D 89%@998 EDP limit 1734
RAM 1428/3995MB (lfb 495x4MB) cpu [16%,30%,66%,100%]@1734 EMC 23%@1600 AVP 5%@12 NVDEC 268 MSENC 268 GR3D 71%@998 EDP limit 1734
RAM 1428/3995MB (lfb 495x4MB) cpu [6%,35%,63%,100%]@1734 EMC 27%@1600 AVP 5%@12 NVDEC 268 MSENC 268 GR3D 90%@998 EDP limit 1734
RAM 1428/3995MB (lfb 495x4MB) cpu [21%,20%,59%,100%]@1734 EMC 29%@1600 AVP 5%@12 NVDEC 268 MSENC 268 GR3D 90%@998 EDP limit 1734
RAM 1428/3995MB (lfb 495x4MB) cpu [22%,19%,63%,100%]@1734 EMC 27%@1600 AVP 5%@12 NVDEC 268 MSENC 268 GR3D 89%@998 EDP limit 1734
RAM 1428/3995MB (lfb 495x4MB) cpu [22%,28%,62%,100%]@1734 EMC 27%@1600 AVP 5%@12 NVDEC 268 MSENC 268 GR3D 89%@998 EDP limit 1734
RAM 1428/3995MB (lfb 495x4MB) cpu [30%,30%,61%,100%]@1734 EMC 27%@1600 AVP 5%@12 NVDEC 268 MSENC 268 GR3D 90%@998 EDP limit 1734
RAM 1410/3995MB (lfb 495x4MB) cpu [18%,10%,29%,100%]@1734 EMC 14%@1600 AVP 5%@12 NVDEC 268 MSENC 268 GR3D 0%@153 EDP limit 1734
RAM 1410/3995MB (lfb 495x4MB) cpu [0%,0%,0%,100%]@1734 EMC 4%@1600 AVP 5%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1410/3995MB (lfb 495x4MB) cpu [0%,1%,0%,100%]@1734 EMC 2%@1600 AVP 5%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1410/3995MB (lfb 495x4MB) cpu [1%,0%,0%,100%]@1734 EMC 1%@1600 AVP 5%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1410/3995MB (lfb 495x4MB) cpu [0%,0%,0%,100%]@1734 EMC 1%@1600 AVP 5%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1387/3995MB (lfb 495x4MB) cpu [9%,10%,9%,55%]@102 EMC 9%@665 AVP 12%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1387/3995MB (lfb 495x4MB) cpu [5%,3%,5%,4%]@102 EMC 5%@665 AVP 54%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1387/3995MB (lfb 495x4MB) cpu [5%,4%,3%,3%]@102 EMC 3%@665 AVP 61%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1387/3995MB (lfb 495x4MB) cpu [3%,0%,2%,0%]@102 EMC 3%@665 AVP 61%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1387/3995MB (lfb 495x4MB) cpu [13%,2%,5%,7%]@1132 EMC 1%@800 AVP 54%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1387/3995MB (lfb 495x4MB) cpu [5%,1%,0%,0%]@102 EMC 2%@665 AVP 60%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734
RAM 1387/3995MB (lfb 495x4MB) cpu [6%,0%,1%,0%]@102 EMC 2%@665 AVP 67%@12 NVDEC 268 MSENC 268 GR3D 0%@76 EDP limit 1734

Can you please try with the following patch and let us know the repro rate?

diff --git a/arch/arm64/boot/dts/tegra210-jetson-cv-base-p2597-2180-a00.dts b/arch/arm64/boot/dts/tegra210-jetson-cv-base-p2597-2180-a00.dts
index dc24ce8..69b2b2c 100644
--- a/arch/arm64/boot/dts/tegra210-jetson-cv-base-p2597-2180-a00.dts
+++ b/arch/arm64/boot/dts/tegra210-jetson-cv-base-p2597-2180-a00.dts
@@ -231,8 +231,6 @@
        vddio-pex-ctl-supply = <&max77620_sd3>;
        status = "okay";
 
-       iommus = <&smmu TEGRA_SWGROUP_AFI>;
-
        pci@1,0 {
            status = "okay";
        };

Also, with the existing setup and without #13, can you execute following commands to disable SMMU for GPU (keeping SMMU enabled for PCIe… which is default)

root@tegra-ubuntu:/home/ubuntu# echo Y > /sys/kernel/debug/gpu.0/disable_bigpage 
root@tegra-ubuntu:/home/ubuntu# echo Y > /sys/kernel/debug/gpu.0/bypass_smmu

and update your observations?

Doing the things from #14 didn’t make any difference. I’m still working on getting #13 to work.

Thanks for the update on #14.
May I know if the time it takes to repro the issue is same as before or does it take more time to repro with workaround in #14?

I didn’t oberve any difference. It failed to work in the same manner as without it.
But the issue now gone after implementing the smmu-less DMA which you proposed in #13.