Jetson hangs until power cycled

I’m trying to prototype a camera server in Python and GStreamer (0.10, I know) that displays multiple cameras with both a local stream, as well as pushing out an h264 stream for remote viewing. However, I’m experiencing an issue where my Jetson dev kit will sometimes hang the entire system (requiring a power cycle). To make the issue more fun, it seems to render the serial port unavailable for debugging. In addition, until recently, I’ve been unable to find anything in the logs I’ve scoured to suggest what might be the cause. Once I started pushing the syslog to a syslog server, I started to get a bit of information (perhaps it wasn’t yet written to disk when push wasn’t setup).

I’m trying to replicate the issue with a script, but so far am not having much luck. So, I thought I’d check here to see if anyone might have any possible ideas as to what could be going on. Also, I should mention I’m new to working with the GPU, so I may be doing something foolish (I use the GPU in multiple Python threads, which I am unsure if that’s safe). From the logs, it looks like the hw encoder is getting stuck, but not entirely certain. And if that’s the case, I’m not certain why.

Below is an example of the GStreamer pipeline, along with the only clue I’ve found in the log. All thoughts, ideas and advice are extremely welcome.

L4T 19.3
CUDA 6 (cuda-repo-l4t-r19.2_6.0-42_armhf.deb)
OpenCV4Tegra v2.4.8.2 (libopencv4tegra_2.4.8.2_armhf.deb, libopencv4tegra-dev_2.4.8.2_armhf.deb)
Python 2.7.6
GStreamer 0.10

Example pipeline:

gst-launch v4l2src device=/dev/video0 queue-size=8 decimate=2
! videorate max-rate=15
! image/jpeg,width=1024,height=768
! jpegdec
! ‘video/x-raw-yuv,format=(fourcc)I420,width=1024,height=768’
! tee name=rawdattee
! queue max-size-buffers=5
! appsink max-buffers=1 emit-signals=true rawdattee.
! queue max-size-buffers=15 leaky=2
! nv_omx_h264enc temporal-tradeoff=0 bit-packetization=true bitrate=300000 rc-mode=0
! video/x-h264,stream-format=byte-stream
! h264parse
! flvmux streamable=true
! rtmpsink max-lateness=3000000000 location=rtmp://TARGET_IP:TARGET_PORT/STREAM

Kernel log:

INFO | ps_stats = {‘pid’: 1057, ‘cpu’: 146, ‘mem’: 28.1, ‘vsz’: 756880, ‘rss’: 503084, ‘stat’: Sl, ‘time’: 2051:45}
kernel:[83979.090984] host1x host1x: python: syncpoint id 23 (mpe) stuck waiting 1018988, timeout=2147481647
kernel:[83979.091015] host1x host1x: id 5 (disp0d) min 1 max 0
kernel:[83979.091034] host1x host1x: id 8 (disp0) min 1 max 0
kernel:[83979.091058] host1x host1x: id 18 (vic) min 12 max 12
kernel:[83979.091074] host1x host1x: id 20 (disp0b) min 1 max 0
kernel:[83979.091090] host1x host1x: id 23 (mpe) min 1018987 max 1018988
kernel:[83979.091104] host1x host1x: id 24 (disp0c) min 1 max 0
kernel:[83979.091119] host1x host1x: id 26 (vblank0) min 5030735 max 0
kernel:[83979.091133] host1x host1x: id 27 (vblank1) min 8 max 0
kernel:[83979.091149] host1x host1x: id 31 (dsi) min 3 max 0
kernel:[83981.090952] host1x host1x: python: syncpoint id 23 (mpe) stuck waiting 1018988, timeout=2147479647
kernel:[83981.090983] host1x host1x: id 5 (disp0d) min 1 max 0
kernel:[83981.091001] host1x host1x: id 8 (disp0) min 1 max 0
kernel:[83981.091025] host1x host1x: id 18 (vic) min 12 max 12
kernel:[83981.091041] host1x host1x: id 20 (disp0b) min 1 max 0
kernel:[83981.091057] host1x host1x: id 23 (mpe) min 1018987 max 1018988
kernel:[83981.091071] host1x host1x: id 24 (disp0c) min 1 max 0
kernel:[83981.091087] host1x host1x: id 26 (vblank0) min 5030855 max 0
kernel:[83981.091544] host1x host1x: id 27 (vblank1) min 8 max 0
kernel:[83981.091562] host1x host1x: id 31 (dsi) min 3 max 0
kernel:[83983.090951] host1x host1x: python: syncpoint id 23 (mpe) stuck waiting 1018988, timeout=2147477647
kernel:[83983.090982] host1x host1x: id 5 (disp0d) min 1 max 0
kernel:[83983.091001] host1x host1x: id 8 (disp0) min 1 max 0
kernel:[83983.091025] host1x host1x: id 18 (vic) min 12 max 12
kernel:[83983.091041] host1x host1x: id 20 (disp0b) min 1 max 0
kernel:[83983.091058] host1x host1x: id 23 (mpe) min 1018987 max 1018988
kernel:[83983.091072] host1x host1x: id 24 (disp0c) min 1 max 0
kernel:[83983.091087] host1x host1x: id 26 (vblank0) min 5030975 max 0
kernel:[83983.091101] host1x host1x: id 27 (vblank1) min 8 max 0
kernel:[83983.091118] host1x host1x: id 31 (dsi) min 3 max 0
kernel:[83985.091006] host1x host1x: python: syncpoint id 23 (mpe) stuck waiting 1018988, timeout=2147475647
kernel:[83985.091038] host1x host1x: id 5 (disp0d) min 1 max 0
kernel:[83985.091284] host1x host1x: id 8 (disp0) min 1 max 0
kernel:[83985.091309] host1x host1x: id 18 (vic) min 12 max 12
kernel:[83985.091325] host1x host1x: id 20 (disp0b) min 1 max 0
kernel:[83985.091341] host1x host1x: id 23 (mpe) min 1018987 max 1018988
kernel:[83985.091356] host1x host1x: id 24 (disp0c) min 1 max 0
kernel:[83985.091370] host1x host1x: id 26 (vblank0) min 5031094 max 0
kernel:[83985.091384] host1x host1x: id 27 (vblank1) min 8 max 0
kernel:[83985.091400] host1x host1x: id 31 (dsi) min 3 max 0
kernel:[83987.090969] host1x host1x: python: syncpoint id 23 (mpe) stuck waiting 1018988, timeout=2147473647
kernel:[83987.091000] host1x host1x: id 5 (disp0d) min 1 max 0
kernel:[83987.091019] host1x host1x: id 8 (disp0) min 1 max 0
kernel:[83987.091043] host1x host1x: id 18 (vic) min 12 max 12
kernel:[83987.091059] host1x host1x: id 20 (disp0b) min 1 max 0
kernel:[83987.091075] host1x host1x: id 23 (mpe) min 1018987 max 1018988
kernel:[83987.091089] host1x host1x: id 24 (disp0c) min 1 max 0
kernel:[83987.091104] host1x host1x: id 26 (vblank0) min 5031214 max 0
kernel:[83987.091118] host1x host1x: id 27 (vblank1) min 8 max 0
kernel:[83987.091135] host1x host1x: id 31 (dsi) min 3 max 0
kernel:[83987.113963] ---- mlocks ----
kernel:[83987.113987] 0: unlocked
kernel:[83987.114001] 1: unlocked
kernel:[83987.114012] 2: unlocked
kernel:[83987.114024] 3: unlocked
kernel:[83987.114035] 4: unlocked
kernel:[83987.114045] 5: unlocked
kernel:[83987.114056] 6: unlocked
kernel:[83987.114066] 7: unlocked
kernel:[83987.114077] 8: unlocked
kernel:[83987.114087] 9: unlocked
kernel:[83987.114099] 10: unlocked
kernel:[83987.114109] 11: unlocked
kernel:[83987.114121] 12: unlocked
kernel:[83987.114131] 13: unlocked
kernel:[83987.114143] 14: unlocked
kernel:[83987.114153] 15: unlocked
kernel:[83987.114163]
kernel:[83987.114176] ---- syncpts ----
kernel:[83987.114196] id 5 (disp0d) min 1 max 0
kernel:[83987.114213] id 8 (disp0) min 1 max 0
kernel:[83987.114241] id 18 (vic) min 12 max 12
kernel:[83987.114257] id 20 (disp0b) min 1 max 0
kernel:[83987.114274] id 23 (mpe) min 1018987 max 1018988
kernel:[83987.114287] id 24 (disp0c) min 1 max 0
kernel:[83987.114302] id 26 (vblank0) min 5031216 max 0
kernel:[83987.114315] id 27 (vblank1) min 8 max 0
kernel:[83987.114333] id 31 (dsi) min 3 max 0
kernel:[83987.114680]
kernel:[83987.114691] ---- channels ----
kernel:[83987.114706] 4: fifo:
kernel:[83987.114719] FIFOSTAT 80100840
kernel:[83987.114719] [empty]
kernel:[83987.114738] 4-msenc (1): active class 21, offset 0000, val 00000117
kernel:[83987.114762] DMAPUT 80000a20, DMAGET 80000a20, DMACTL 00000000
kernel:[83987.114774] CBREAD 00000117, CBSTAT 00210000
kernel:[83987.114795]
kernel:[83987.114795] f18c0000: JOB, syncpt_id=23, syncpt_val=1018988, first_get=80000a08, timeout=10000, ctx= (null), num_slots=3, num_handles=12
kernel:[83987.114838] GATHER at 800e4000+0638, 57 words
kernel:[83987.114850] 00000840 100b0001 00000013 10100002 000001c0 12001103 10100002 000001c1 000f8c6b 10100002 00000080 00000001 10100002 000001c4 00800ad0 10100002 000001c6 00821000 10100002 000001c9 00800bd0 10100002 000001c7 00827000 10100002 000001cd 00815000 10100002 000001d0 00815e00 10100002 000001d1 00816200 10100002 000001cb 00804c00 10100002 000001cc 0082a000 10100002 000001c5 00803480 10100002 000001ce 00800f40 10100002 000001cf 00800ec0 10100002 00000100 0081f000 10100002 00000101 0082a000 10100002 000000c0 00000100
kernel:[83987.115413] GATHER at 800e4000+071c, 2 words
kernel:[83987.115424] 20000001 00000117
kernel:[83987.115445]
kernel:[83987.118305] 5: fifo:
kernel:[83987.118325] FIFOSTAT 80100840
kernel:[83987.118325] [empty]
kernel:[83987.118346] 5-tsec (0): inactive
kernel:[83987.118346]
kernel:[83987.118633] 0: fifo:
kernel:[83987.118649] FIFOSTAT 80100840
kernel:[83987.118649] [empty]
kernel:[83987.118667] 0-isp.0 (0): inactive
kernel:[83987.118667]
kernel:[83987.118717] 1: fifo:
kernel:[83987.118730] FIFOSTAT 80100840
kernel:[83987.118730] [empty]
kernel:[83987.118748] 1-isp.1 (0): inactive
kernel:[83987.118748]
kernel:[83987.118782] 7: fifo:
kernel:[83987.118795] FIFOSTAT 80100840
kernel:[83987.118795] [empty]
kernel:[83987.118812] 7-vic03.0 (0): inactive
kernel:[83987.118812]
kernel:[83987.120665] tegra_simon_gpu_grading_cb: graded gpu: v = 800, t = 39000, grade = 0
kernel:[83987.140008] gk20a.0 pbdma 0: id: 0 (channel), next_id: 56 (channel) status: invalid
kernel:[83987.140052] PUT: 000000deb56feefc GET: 000000fdffeffffc FETCH: 8311b4e5 HEADER: e1d73fda
kernel:[83987.140067]
kernel:[83987.140080] gk20a.0 eng 0: id: 0 (channel), next_id: 0 (channel), ctx: invalid
kernel:[83987.140106] gk20a.0 eng 1: id: 0 (channel), next_id: 0 (channel), ctx: invalid
kernel:[83987.140130]
kernel:[83987.140319] 2: fifo:
kernel:[83987.140334] FIFOSTAT 80100840
kernel:[83987.140334] [empty]
kernel:[83987.140352] 2-vi.0 (0): inactive
kernel:[83987.140352]
kernel:[83987.140538] 3: fifo:
kernel:[83987.140552] FIFOSTAT 80100840
kernel:[83987.140552] [empty]
kernel:[83987.140569] 3-vi (0): inactive
kernel:[83987.140569]
kernel:[83989.090981] host1x host1x: python: syncpoint id 23 (mpe) stuck waiting 1018988, timeout=2147471647
kernel:[83989.091012] host1x host1x: id 5 (disp0d) min 1 max 0
kernel:[83989.091030] host1x host1x: id 8 (disp0) min 1 max 0
kernel:[83989.091054] host1x host1x: id 18 (vic) min 12 max 12
kernel:[83989.091069] host1x host1x: id 20 (disp0b) min 1 max 0
kernel:[83989.091086] host1x host1x: id 23 (mpe) min 1018987 max 1018988
kernel:[83989.091100] host1x host1x: id 24 (disp0c) min 1 max 0
kernel:[83989.091116] host1x host1x: id 26 (vblank0) min 5031334 max 0
kernel:[83989.091129] host1x host1x: id 27 (vblank1) min 8 max 0
kernel:[83989.091146] host1x host1x: id 31 (dsi) min 3 max 0
kernel:[83991.091210] host1x host1x: python: syncpoint id 23 (mpe) stuck waiting 1018988, timeout=2147469647
kernel:[83991.091242] host1x host1x: id 5 (disp0d) min 1 max 0
kernel:[83991.091260] host1x host1x: id 8 (disp0) min 1 max 0
kernel:[83991.091284] host1x host1x: id 18 (vic) min 12 max 12
kernel:[83991.091526] host1x host1x: id 20 (disp0b) min 1 max 0
kernel:[83991.091544] host1x host1x: id 23 (mpe) min 1018987 max 1018988
kernel:[83991.091558] host1x host1x: id 24 (disp0c) min 1 max 0
kernel:[83991.091574] host1x host1x: id 26 (vblank0) min 5031454 max 0
kernel:[83991.091588] host1x host1x: id 27 (vblank1) min 8 max 0
kernel:[83991.091604] host1x host1x: id 31 (dsi) min 3 max 0
kernel:[83993.090990] host1x host1x: python: syncpoint id 23 (mpe) stuck waiting 1018988, timeout=2147467647
kernel:[83993.091021] host1x host1x: id 5 (disp0d) min 1 max 0
kernel:[83993.091039] host1x host1x: id 8 (disp0) min 1 max 0
kernel:[83993.091064] host1x host1x: id 18 (vic) min 12 max 12
kernel:[83993.091080] host1x host1x: id 20 (disp0b) min 1 max 0
kernel:[83993.091096] host1x host1x: id 23 (mpe) min 1018987 max 1018988
kernel:[83993.091110] host1x host1x: id 24 (disp0c) min 1 max 0
kernel:[83993.091125] host1x host1x: id 26 (vblank0) min 5031574 max 0
kernel:[83993.091139] host1x host1x: id 27 (vblank1) min 8 max 0
kernel:[83993.091155] host1x host1x: id 31 (dsi) min 3 max 0
kernel:[83995.091217] host1x host1x: python: syncpoint id 23 (mpe) stuck waiting 1018988, timeout=2147465647
kernel:[83995.091249] host1x host1x: id 5 (disp0d) min 1 max 0
kernel:[83995.091267] host1x host1x: id 8 (disp0) min 1 max 0
kernel:[83995.091291] host1x host1x: id 18 (vic) min 12 max 12
kernel:[83995.091307] host1x host1x: id 20 (disp0b) min 1 max 0
kernel:[83995.091324] host1x host1x: id 23 (mpe) min 1018987 max 1018988
kernel:[83995.091338] host1x host1x: id 24 (disp0c) min 1 max 0
kernel:[83995.091353] host1x host1x: id 26 (vblank0) min 5031694 max 0
kernel:[83995.091366] host1x host1x: id 27 (vblank1) min 8 max 0
kernel:[83995.091383] host1x host1x: id 31 (dsi) min 3 max 0
kernel:[83997.091019] host1x host1x: python: syncpoint id 23 (mpe) stuck waiting 1018988, timeout=2147463647
kernel:[83997.091050] host1x host1x: id 5 (disp0d) min 1 max 0
kernel:[83997.091068] host1x host1x: id 8 (disp0) min 1 max 0
kernel:[83997.091092] host1x host1x: id 18 (vic) min 12 max 12
kernel:[83997.091109] host1x host1x: id 20 (disp0b) min 1 max 0
kernel:[83997.091125] host1x host1x: id 23 (mpe) min 1018987 max 1018988
kernel:[83997.091139] host1x host1x: id 24 (disp0c) min 1 max 0
kernel:[83997.091155] host1x host1x: id 26 (vblank0) min 5031813 max 0
kernel:[83997.091168] host1x host1x: id 27 (vblank1) min 8 max 0
kernel:[83997.091185] host1x host1x: id 31 (dsi) min 3 max 0
kernel:[83997.162052] ---- mlocks ----
kernel:[83997.162076] 0: unlocked
kernel:[83997.162089] 1: unlocked
kernel:[83997.162101] 2: unlocked
kernel:[83997.162113] 3: unlocked
kernel:[83997.162124] 4: unlocked
kernel:[83997.162134] 5: unlocked
kernel:[83997.162146] 6: unlocked
kernel:[83997.162157] 7: unlocked
kernel:[83997.162167] 8: unlocked
kernel:[83997.162180] 9: unlocked
kernel:[83997.162190] 10: unlocked
kernel:[83997.162200] 11: unlocked
kernel:[83997.162210] 12: unlocked
kernel:[83997.162222] 13: unlocked
kernel:[83997.162234] 14: unlocked
kernel:[83997.162244] 15: unlocked
kernel:[83997.162255]
kernel:[83997.162265] ---- syncpts ----
kernel:[83997.162289] id 5 (disp0d) min 1 max 0
kernel:[83997.162307] id 8 (disp0) min 1 max 0
kernel:[83997.162335] id 18 (vic) min 12 max 12
kernel:[83997.162351] id 20 (disp0b) min 1 max 0
kernel:[83997.162369] id 23 (mpe) min 1018987 max 1018988
kernel:[83997.162384] id 24 (disp0c) min 1 max 0
kernel:[83997.162398] id 26 (vblank0) min 5031818 max 0
kernel:[83997.162412] id 27 (vblank1) min 8 max 0
kernel:[83997.162429] id 31 (dsi) min 3 max 0
kernel:[83997.162792]
kernel:[83997.162803] ---- channels ----
kernel:[83997.162818] 4: fifo:
kernel:[83997.162832] FIFOSTAT 80100840
kernel:[83997.162832] [empty]
kernel:[83997.162852] 4-msenc (1): active class 21, offset 0000, val 00000117
kernel:[83997.162875] DMAPUT 80000a20, DMAGET 80000a20, DMACTL 00000000
kernel:[83997.162889] CBREAD 00000117, CBSTAT 00210000
kernel:[83997.162910]
kernel:[83997.162910] f18c0000: JOB, syncpt_id=23, syncpt_val=1018988, first_get=80000a08, timeout=10000, ctx= (null), num_slots=3, num_handles=12
kernel:[83997.162971] GATHER at 800e4000+0638, 57 words
kernel:[83997.162982] 00000840 100b0001 00000013 10100002 000001c0 12001103 10100002 000001c1 000f8c6b 10100002 00000080 00000001 10100002 000001c4 00800ad0 10100002 000001c6 00821000 10100002 000001c9 00800bd0 10100002 000001c7 00827000 10100002 000001cd 00815000 10100002 000001d0 00815e00 10100002 000001d1 00816200 10100002 000001cb 00804c00 10100002 000001cc 0082a000 10100002 000001c5 00803480 10100002 000001ce 00800f40 10100002 000001cf 00800ec0 10100002 00000100 0081f000 10100002 00000101 0082a000 10100002 000000c0 00000100
kernel:[83997.163477] GATHER at 800e4000+071c, 2 words
kernel:[83997.163487] 20000001 00000117
kernel:[83997.163512]
kernel:[83997.164837] 5: fifo:
kernel:[83997.164854] FIFOSTAT 80100840
kernel:[83997.164854] [empty]
kernel:[83997.164876] 5-tsec (0): inactive
kernel:[83997.164876]
kernel:[83997.165018] 0: fifo:
kernel:[83997.165032] FIFOSTAT 80100840
kernel:[83997.165032] [empty]
kernel:[83997.165049] 0-isp.0 (0): inactive
kernel:[83997.165049]
kernel:[83997.165097] 1: fifo:
kernel:[83997.165118] FIFOSTAT 80100840
kernel:[83997.165118] [empty]
kernel:[83997.165135] 1-isp.1 (0): inactive
kernel:[83997.165135]
kernel:[83997.165166] 7: fifo:
kernel:[83997.165196] FIFOSTAT 80100840
kernel:[83997.165196] [empty]
kernel:[83997.165212] 7-vic03.0 (0): inactive
kernel:[83997.165212]
kernel:[83997.173595] gk20a.0 pbdma 0: id: 0 (channel), next_id: 56 (channel) status: invalid
kernel:[83997.173636] PUT: 000000deb5efeefc GET: 000000fdffeffffc FETCH: 8311b4e5 HEADER: e1d73fda
kernel:[83997.173649]
kernel:[83997.173661] gk20a.0 eng 0: id: 0 (channel), next_id: 0 (channel), ctx: invalid
kernel:[83997.173686] gk20a.0 eng 1: id: 0 (channel), next_id: 0 (channel), ctx: invalid
kernel:[83997.173711]
kernel:[83997.174019] 2: fifo:
kernel:[83997.174041] FIFOSTAT 80100840
kernel:[83997.174041] [empty]
kernel:[83997.174063] 2-vi.0 (0): inactive
kernel:[83997.174063]
kernel:[83997.174313] 3: fifo:
kernel:[83997.174328] FIFOSTAT 80100840
kernel:[83997.174328] [empty]
kernel:[83997.174347] 3-vi (0): inactive
kernel:[83997.174347]
kernel:[83997.174467] host1x host1x: cdma_timeout_handler: timeout: 23 (mpe) ctx 0x (null), HW thresh 1018987, done 1018988
kernel:[83999.687232] [sched_delayed] sched: RT throttling activated
kernel:[84003.208880] Host read timeout at address 544c1118
kernel:[84003.718187] host1x host1x: python: syncpoint id 23 (mpe) stuck waiting 1018988, timeout=2147461647
kernel:[84003.718213] host1x host1x: id 5 (disp0d) min 1 max 0
kernel:[84003.718226] host1x host1x: id 8 (disp0) min 1 max 0
kernel:[84003.718257] host1x host1x: id 18 (vic) min 12 max 12
kernel:[84003.718267] host1x host1x: id 20 (disp0b) min 1 max 0
kernel:[84003.718279] host1x host1x: id 23 (mpe) min 1018987 max 1018988
kernel:[84003.718288] host1x host1x: id 24 (disp0c) min 1 max 0
kernel:[84003.718298] host1x host1x: id 26 (vblank0) min 5032211 max 0
kernel:[84003.718307] host1x host1x: id 27 (vblank1) min 8 max 0
kernel:[84003.718319] host1x host1x: id 31 (dsi) min 3 max 0
kernel:[84003.720251] Host read timeout at address 544c104c
kernel:[84006.236783] host1x host1x: python: syncpoint id 23 (mpe) stuck waiting 1018988, timeout=2147459647
kernel:[84006.236806] host1x host1x: id 5 (disp0d) min 1 max 0
kernel:[84006.236820] host1x host1x: id 8 (disp0) min 1 max 0
kernel:[84006.236843] host1x host1x: id 18 (vic) min 12 max 12
kernel:[84006.236854] host1x host1x: id 20 (disp0b) min 1 max 0
kernel:[84006.236882] host1x host1x: id 23 (mpe) min 1018987 max 1018988
kernel:[84006.236892] host1x host1x: id 24 (disp0c) min 1 max 0
kernel:[84006.236907] host1x host1x: id 26 (vblank0) min 5032361 max 0
kernel:[84006.236916] host1x host1x: id 27 (vblank1) min 8 max 0
kernel:[84006.236927] host1x host1x: id 31 (dsi) min 3 max 0
kernel:[84006.237226] tegra-i2c tegra12-i2c.0: — register dump for debugging ----
kernel:[84006.238907] uvcvideo: Non-zero status (-75) in status completion handler.
kernel:[84006.239041] Host read timeout at address 544c104c
kernel:[84006.243287] uvcvideo: Non-zero status (-75) in status completion handler.
kernel:[84006.751383] tegra-i2c tegra12-i2c.0: I2C_CNFG - 0x2c00
kernel:[84006.751388] tegra-i2c tegra12-i2c.0: I2C_PACKET_TRANSFER_STATUS - 0xff0001
kernel:[84006.751392] tegra-i2c tegra12-i2c.0: I2C_FIFO_CONTROL - 0xe0
kernel:[84006.751397] tegra-i2c tegra12-i2c.0: I2C_FIFO_STATUS - 0x800080
kernel:[84006.751401] tegra-i2c tegra12-i2c.0: I2C_INT_MASK - 0xec

Anyone happen to have an idea as to which kernel tracing hex mask to enable?

The kernel log is not complete. The first lines of the errors might reveal something, the later lines are just more stuff getting stuck.

What are you doing with python in the above example? The kernel log shows it being stuck as well.

Have you tried a newer release? 19r3 is already a bit old and 21.x has many fixes. That said, I did have some issues with my USB camera on 21.x and thus I’m still using 19r3.

digitstil,

I have a repro of “RT Throttling”, but not a permanent hang, if function tracer is enabled.
but the system responds back after 4-5 mins

It seems that system is busy in your case, as it says sched_delayed

Can you provide data of, if the system responds back after 5-6 mins of time?
Just to see/confirm if our issues coincide

cd /sys/kernel/debug/tracing/
cat tracing_on
cat current_tracer
echo function >  current_tracer
cat trace

dmesg | tail 
[ 8647.355242] NOHZ: local_softirq_pending 202
<b>[ 8682.978774] [sched_delayed] sched: RT throttling activated</b>
[ 8940.483573] NOHZ: local_softirq_pending 100

I have the same problem on

L4T 21.4
CUDA 6.5
GStreamer 1.0
customed kernel

Anyone knows where is the problem? Thanks!

log:
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.164236] host1x host1x: pool: syncpoint id 16 (msenc_0) stuck waiting 17164, timeout=2147481647
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.164245] host1x host1x: id 1 (disp0_a) min 1 max 0
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.164250] host1x host1x: id 2 (disp0_b) min 1 max 0
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.164473] host1x host1x: id 3 (disp0_c) min 1 max 0
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.164485] host1x host1x: id 4 (disp0_d) min 1 max 0
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.164491] host1x host1x: id 5 (dsi) min 1 max 0
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.164497] host1x host1x: id 6 (disp1_a) min 11908 max 11909
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.167173] host1x host1x: id 7 (disp1_b) min 1 max 0
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.167646] host1x host1x: id 8 (disp1_c) min 1 max 0
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.167657] host1x host1x: id 9 (gk20a.0_1) min 111890 max 111890
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.167914] host1x host1x: id 11 (gk20a.0_2) min 11 max 11
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.167919] host1x host1x: id 12 (gk20a.0_4) min 30187 max 30187
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.168157] host1x host1x: id 13 (gk20a.0_5) min 306047 max 306047
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.168162] host1x host1x: id 14 (gk20a.0_6) min 7605 max 7605
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.168872] host1x host1x: id 15 (gk20a.0_7) min 5751 max 5751
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.168878] host1x host1x: id 16 (msenc_0) min 17163 max 17164
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.168884] host1x host1x: id 18 (vic03.0_0) min 34330 max 34330
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.168893] host1x host1x: id 26 (vblank0) min 523 max 0
May 20 12:33:09 tegra-ubuntu kernel: [ 1269.169126] host1x host1x: id 27 (vblank1) min 76038 max 0
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172726] host1x host1x: pool: syncpoint id 16 (msenc_0) stuck waiting 17164, timeout=2147479647
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172736] host1x host1x: id 1 (disp0_a) min 1 max 0
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172741] host1x host1x: id 2 (disp0_b) min 1 max 0
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172745] host1x host1x: id 3 (disp0_c) min 1 max 0
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172749] host1x host1x: id 4 (disp0_d) min 1 max 0
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172752] host1x host1x: id 5 (dsi) min 1 max 0
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172756] host1x host1x: id 6 (disp1_a) min 11937 max 11938
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172760] host1x host1x: id 7 (disp1_b) min 1 max 0
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172764] host1x host1x: id 8 (disp1_c) min 1 max 0
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172768] host1x host1x: id 9 (gk20a.0_1) min 112108 max 112108
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172773] host1x host1x: id 11 (gk20a.0_2) min 11 max 11
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172777] host1x host1x: id 12 (gk20a.0_4) min 30259 max 30260
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172781] host1x host1x: id 13 (gk20a.0_5) min 307587 max 307587
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172785] host1x host1x: id 14 (gk20a.0_6) min 7645 max 7645
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172789] host1x host1x: id 15 (gk20a.0_7) min 5781 max 5781
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172793] host1x host1x: id 16 (msenc_0) min 17163 max 17164
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172798] host1x host1x: id 18 (vic03.0_0) min 34330 max 34330
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172806] host1x host1x: id 26 (vblank0) min 523 max 0
May 20 12:33:11 tegra-ubuntu kernel: [ 1271.172811] host1x host1x: id 27 (vblank1) min 76158 max 0
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172468] host1x host1x: pool: syncpoint id 16 (msenc_0) stuck waiting 17164, timeout=2147477647
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172485] host1x host1x: id 1 (disp0_a) min 1 max 0
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172496] host1x host1x: id 2 (disp0_b) min 1 max 0
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172506] host1x host1x: id 3 (disp0_c) min 1 max 0
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172515] host1x host1x: id 4 (disp0_d) min 1 max 0
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172524] host1x host1x: id 5 (dsi) min 1 max 0
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172690] host1x host1x: id 6 (disp1_a) min 11977 max 11977
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172700] host1x host1x: id 7 (disp1_b) min 1 max 0
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172708] host1x host1x: id 8 (disp1_c) min 1 max 0
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172717] host1x host1x: id 9 (gk20a.0_1) min 112489 max 112489
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172708] host1x host1x: id 8 (disp1_c) min 1 max 0
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172717] host1x host1x: id 9 (gk20a.0_1) min 112489 max 112489
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172727] host1x host1x: id 11 (gk20a.0_2) min 11 max 11
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172734] host1x host1x: id 12 (gk20a.0_4) min 30359 max 30359
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172741] host1x host1x: id 13 (gk20a.0_5) min 309127 max 309127
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172750] host1x host1x: id 14 (gk20a.0_6) min 7685 max 7685
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172758] host1x host1x: id 15 (gk20a.0_7) min 5811 max 5811
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172901] host1x host1x: id 16 (msenc_0) min 17163 max 17164
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172909] host1x host1x: id 18 (vic03.0_0) min 34330 max 34330
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172925] host1x host1x: id 26 (vblank0) min 523 max 0
May 20 12:33:13 tegra-ubuntu kernel: [ 1273.172934] host1x host1x: id 27 (vblank1) min 76278 max 0
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174407] host1x host1x: pool: syncpoint id 16 (msenc_0) stuck waiting 17164, timeout=2147475647
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174421] host1x host1x: id 1 (disp0_a) min 1 max 0
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174429] host1x host1x: id 2 (disp0_b) min 1 max 0
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174437] host1x host1x: id 3 (disp0_c) min 1 max 0
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174447] host1x host1x: id 4 (disp0_d) min 1 max 0
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174461] host1x host1x: id 5 (dsi) min 1 max 0
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174473] host1x host1x: id 6 (disp1_a) min 12004 max 12005
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174484] host1x host1x: id 7 (disp1_b) min 1 max 0
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174494] host1x host1x: id 8 (disp1_c) min 1 max 0
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174505] host1x host1x: id 9 (gk20a.0_1) min 112714 max 112714
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174511] host1x host1x: id 11 (gk20a.0_2) min 11 max 11
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174524] host1x host1x: id 12 (gk20a.0_4) min 30429 max 30430
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174536] host1x host1x: id 13 (gk20a.0_5) min 310403 max 310450
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174546] host1x host1x: id 14 (gk20a.0_6) min 7718 max 7718
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174557] host1x host1x: id 15 (gk20a.0_7) min 5836 max 5836
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174567] host1x host1x: id 16 (msenc_0) min 17163 max 17164
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174583] host1x host1x: id 18 (vic03.0_0) min 34330 max 34330
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174605] host1x host1x: id 26 (vblank0) min 523 max 0
May 20 12:33:15 tegra-ubuntu kernel: [ 1275.174617] host1x host1x: id 27 (vblank1) min 76398 max 0
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174448] host1x host1x: pool: syncpoint id 16 (msenc_0) stuck waiting 17164, timeout=2147473647
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174467] host1x host1x: id 1 (disp0_a) min 1 max 0
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174477] host1x host1x: id 2 (disp0_b) min 1 max 0
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174485] host1x host1x: id 3 (disp0_c) min 1 max 0
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174493] host1x host1x: id 4 (disp0_d) min 1 max 0
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174501] host1x host1x: id 5 (dsi) min 1 max 0
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174509] host1x host1x: id 6 (disp1_a) min 12033 max 12034
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174518] host1x host1x: id 7 (disp1_b) min 1 max 0
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174525] host1x host1x: id 8 (disp1_c) min 1 max 0
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174532] host1x host1x: id 9 (gk20a.0_1) min 112948 max 112948
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174540] host1x host1x: id 11 (gk20a.0_2) min 11 max 11
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174550] host1x host1x: id 12 (gk20a.0_4) min 30502 max 30503
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174559] host1x host1x: id 13 (gk20a.0_5) min 311906 max 311990
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174567] host1x host1x: id 14 (gk20a.0_6) min 7758 max 7758
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174577] host1x host1x: id 15 (gk20a.0_7) min 5866 max 5866
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174586] host1x host1x: id 16 (msenc_0) min 17163 max 17164
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174595] host1x host1x: id 18 (vic03.0_0) min 34330 max 34330
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174608] host1x host1x: id 26 (vblank0) min 523 max 0
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.174616] host1x host1x: id 27 (vblank1) min 76518 max 0
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.191422] ---- mlocks ----
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.191437]
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.191442] ---- syncpts ----
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.191457] id 1 (disp0_a) min 1 max 0
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.191463] id 2 (disp0_b) min 1 max 0
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.191469] id 3 (disp0_c) min 1 max 0
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192771]
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192782] 1-gk20a.0, pid 880:
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192792] in use pending busy
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192810] TOP: 800000000810a350 PUT: 000000000810a350 GET: 000000000810a350 FETCH: 000000000810a350
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192810] HEADER: 60400000 COUNT: 80000000
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192810] SYNCPOINT 00000000 00000901 SEMAPHORE 00000004 05100000 00000000 01100002
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192820]
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192833] 2-gk20a.0, pid 880:
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192841] in use idle not busy
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192857] TOP: 80000000082c3080 PUT: 00000000082c3080 GET: 00000000082c3080 FETCH: 00000000082c3080
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192857] HEADER: 60400000 COUNT: 80000000
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192857] SYNCPOINT 00000000 00000b01 SEMAPHORE 00000000 00000000 00000000 00000000
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192865]
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192877] 3-gk20a.0, pid 2023:
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192884] in use idle not busy
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192903] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192903] HEADER: 60400000 COUNT: 00000000
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192903] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192913]
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192925] 4-gk20a.0, pid 2023:
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192934] in use pending_acquire busy
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192951] TOP: 8000000401e05290 PUT: 0000000401e05294 GET: 0000000401e05290 FETCH: 00000c0401e05290
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192951] HEADER: 2006002c COUNT: 80018001
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192951] SYNCPOINT 00002f02 00000610 SEMAPHORE 00000004 01a40600 0000f653 00081004
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192962] Waiting on syncpt 6 (disp1_a) val 12034
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192971]
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192984] 5-gk20a.0, pid 2461:
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.192991] in use on_pbdma_and_eng busy
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193002] TOP: 800000008e084e90 PUT: 000000008e0851ac GET: 000000008e084e90 FETCH: 0005c0008e085500
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193002] HEADER: 20010180 COUNT: 01110002
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193002] SYNCPOINT 00000000 00000d01 SEMAPHORE 00000000 8d114744 0000032e 00080004
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193008]
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193013] 6-gk20a.0, pid 2461:
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193017] in use pending busy
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193029] TOP: 800000008da5e5b8 PUT: 000000008da5e5cc GET: 000000008da5e5b8 FETCH: 000000010010d9a0
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193029] HEADER: 20000018 COUNT: 00110002
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193029] SYNCPOINT 00000000 00000e01 SEMAPHORE 00000000 8eb1dfe0 00046bef 00080004
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193037]
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193047] 7-gk20a.0, pid 2461:
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193054] in use idle not busy
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193068] TOP: 80000001001efa10 PUT: 00000001001efa10 GET: 00000001001efa10 FETCH: 00000001001efa10
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193068] HEADER: 60400000 COUNT: 80000000
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193068] SYNCPOINT 00000000 00000f01 SEMAPHORE 00000000 8d1159e4 0000000e 00001004
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193075]
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193084] 8-gk20a.0, pid 2461:
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193090] in use idle not busy
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193103] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193103] HEADER: 60400000 COUNT: 00000000
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193103] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
May 20 12:33:17 tegra-ubuntu kernel: [ 1277.193111]
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175430] host1x host1x: pool: syncpoint id 16 (msenc_0) stuck waiting 17164, timeout=2147471647
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175442] host1x host1x: id 1 (disp0_a) min 1 max 0
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175449] host1x host1x: id 2 (disp0_b) min 1 max 0
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175455] host1x host1x: id 3 (disp0_c) min 1 max 0
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175462] host1x host1x: id 4 (disp0_d) min 1 max 0
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175474] host1x host1x: id 6 (disp1_a) min 12067 max 12068
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175480] host1x host1x: id 7 (disp1_b) min 1 max 0
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175486] host1x host1x: id 8 (disp1_c) min 1 max 0
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175493] host1x host1x: id 9 (gk20a.0_1) min 113262 max 113262
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175500] host1x host1x: id 11 (gk20a.0_2) min 11 max 11
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175507] host1x host1x: id 12 (gk20a.0_4) min 30588 max 30589
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175513] host1x host1x: id 13 (gk20a.0_5) min 313439 max 313439
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175520] host1x host1x: id 14 (gk20a.0_6) min 7795 max 7795
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175526] host1x host1x: id 15 (gk20a.0_7) min 5893 max 5893
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175533] host1x host1x: id 16 (msenc_0) min 17163 max 17164
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175541] host1x host1x: id 18 (vic03.0_0) min 34330 max 34330
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175553] host1x host1x: id 26 (vblank0) min 523 max 0
May 20 12:33:19 tegra-ubuntu kernel: [ 1279.175560] host1x host1x: id 27 (vblank1) min 76638 max 0
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175532] host1x host1x: pool: syncpoint id 16 (msenc_0) stuck waiting 17164, timeout=2147469647
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175554] host1x host1x: id 1 (disp0_a) min 1 max 0
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175564] host1x host1x: id 2 (disp0_b) min 1 max 0
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175574] host1x host1x: id 3 (disp0_c) min 1 max 0
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175583] host1x host1x: id 4 (disp0_d) min 1 max 0
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175592] host1x host1x: id 5 (dsi) min 1 max 0
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175602] host1x host1x: id 6 (disp1_a) min 12098 max 12098
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175610] host1x host1x: id 7 (disp1_b) min 1 max 0
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175617] host1x host1x: id 8 (disp1_c) min 1 max 0
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175625] host1x host1x: id 9 (gk20a.0_1) min 113489 max 113489
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175635] host1x host1x: id 11 (gk20a.0_2) min 11 max 11
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175644] host1x host1x: id 12 (gk20a.0_4) min 30665 max 30665
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175654] host1x host1x: id 13 (gk20a.0_5) min 314979 max 314979
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175663] host1x host1x: id 14 (gk20a.0_6) min 7837 max 7837
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175672] host1x host1x: id 15 (gk20a.0_7) min 5925 max 5925
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175680] host1x host1x: id 16 (msenc_0) min 17163 max 17164
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175690] host1x host1x: id 18 (vic03.0_0) min 34330 max 34330
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175706] host1x host1x: id 26 (vblank0) min 523 max 0
May 20 12:33:21 tegra-ubuntu kernel: [ 1281.175715] host1x host1x: id 27 (vblank1) min 76758 max 0
May 20 12:35:24 tegra-ubuntu kernel: [ 0.000000] Booting Linux on physical CPU 0x0

Hi zhuying20002,

Please help to try to repro this issue again the kernel we provided instead of the custom kernel, and also provide the pipeline you used, more info regarding your use case.

It’s not easy to judge and provide the suggestion with the log only from custom kernel.

Thanks