Receiving half the framerate

I’m having a problem with the framerate. I’m sending the frames through a custom camera with an FPGA, where I can control the line valid pause:

Controlling this Line Valid Pause time will directly influence the Frame pause. So lets put two real cases:

<b><u>Case 1</u></b>
Framerate: 50fps
Frame time: 20ms
Line Valid pause: 11.76us
Frame pause: 3.24ms

<b><u>Case 2</u></b>
Framerate: 50fps
Frame time: 20ms
Line Valid pause: 11.06us
Frame pause: 4ms

With these two setups, Case 1 will be giving 50 fps in the TX2, and Case 2 will give half of it. Analyzing the frames in the TX2, it receives a frame and ignores next one, so it captures frames 0,2,4,6,8,10…

The bad point is that this is a custom test, where I’m able to modify the line valid pause time. In the original project I won’t be able to control this pause time.

Anyway, it makes no sense that increasing the frame pause will cause lost odd frames… I’m pretty sure it has something to do with some configuration, buffer overflow,…

I’ve read documentation about v4l2 and videobuf2, but I don’t see a way to relate this problem to the frameworks.

Any advice or suggestion on how to proceed with this problem?

Please dump VI notify trace in 2 cases to compare if any different;
and is the MIPI clock changed when line valid pause is reduced?

Here are the vi_notify traces for both cases
CASE 1: Line valid pause smaller. 1920x1080 @ 25fps. 15 frames

nvidia@nvidia-desktop:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=GRAY16_LE --set-ctrl bypass_mode=0 --stream-mmap --stream-count=15 --stream-to=test.raw
<<<<<<<<<<<<<< 12.62 fps
<
root@nvidia-desktop:/home/nvidia# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 157/157   #P:4
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
     kworker/5:2-1812  [005] ....   901.134596: rtos_queue_peek_from_isr_failed: tstamp:28556180898 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   901.134601: rtcpu_start: tstamp:28556182763
     kworker/5:2-1812  [005] ....   901.134603: rtos_queue_send_from_isr_failed: tstamp:28556191808 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....   901.134604: rtos_queue_send_from_isr_failed: tstamp:28556191916 queue:0x0b4aad68
     kworker/5:2-1812  [005] ....   901.134605: rtos_queue_send_from_isr_failed: tstamp:28556192023 queue:0x0b4ac998
     kworker/5:2-1812  [005] ....   901.134606: rtos_queue_send_from_isr_failed: tstamp:28556192131 queue:0x0b4ae518
     kworker/5:2-1812  [005] ....   901.134606: rtos_queue_send_from_isr_failed: tstamp:28556192236 queue:0x0b4af2d8
     kworker/5:2-1812  [005] ....   901.134607: rtos_queue_send_from_isr_failed: tstamp:28556192341 queue:0x0b4b0098
     kworker/5:2-1812  [005] ....   901.134608: rtos_queue_send_from_isr_failed: tstamp:28556192445 queue:0x0b4b0e58
     kworker/5:2-1812  [005] ....   901.134609: rtos_queue_send_from_isr_failed: tstamp:28556192551 queue:0x0b4b1c18
     kworker/5:2-1812  [005] ....   901.134610: rtos_queue_send_failed: tstamp:28556193022 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....   901.134611: rtos_queue_send_from_isr_failed: tstamp:28556195230 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....   901.134612: rtos_queue_send_from_isr_failed: tstamp:28556195343 queue:0x0b4aad68
     kworker/5:2-1812  [005] ....   901.134612: rtos_queue_send_from_isr_failed: tstamp:28556195449 queue:0x0b4ac998
     kworker/5:2-1812  [005] ....   901.134613: rtos_queue_send_from_isr_failed: tstamp:28556195556 queue:0x0b4ae518
     kworker/5:2-1812  [005] ....   901.134614: rtos_queue_send_from_isr_failed: tstamp:28556195660 queue:0x0b4af2d8
     kworker/5:2-1812  [005] ....   901.134614: rtos_queue_send_from_isr_failed: tstamp:28556195764 queue:0x0b4b0098
     kworker/5:2-1812  [005] ....   901.134615: rtos_queue_send_from_isr_failed: tstamp:28556195869 queue:0x0b4b0e58
     kworker/5:2-1812  [005] ....   901.134616: rtos_queue_send_from_isr_failed: tstamp:28556195978 queue:0x0b4b1c18
     kworker/5:2-1812  [005] ....   901.134616: rtos_queue_send_failed: tstamp:28556196966 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....   901.134618: rtcpu_vinotify_event: tstamp:28556774746 tag:CSIMUX_STREAM channel:0xff frame:0 vi_tstamp:28556774344 data:0x00000001
     kworker/5:2-1812  [005] ....   901.190636: rtcpu_vinotify_event: tstamp:28558024877 tag:CHANSEL_PXL_SOF channel:0x00 frame:9242 vi_tstamp:28558024346 data:0x00000001
     kworker/5:2-1812  [005] ....   901.190643: rtcpu_vinotify_event: tstamp:28558025087 tag:ATOMP_FS channel:0x00 frame:9242 vi_tstamp:28558024350 data:0x00000000
     kworker/5:2-1812  [005] ....   901.190645: rtcpu_vinotify_event: tstamp:28558028560 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9242 vi_tstamp:28558028177 data:0x08000000
     kworker/5:2-1812  [005] ....   901.190647: rtcpu_vinotify_event: tstamp:28558384827 tag:CHANSEL_PXL_EOF channel:0x00 frame:9242 vi_tstamp:28558384137 data:0x04370002
     kworker/5:2-1812  [005] ....   901.190650: rtcpu_vinotify_event: tstamp:28558384981 tag:ATOMP_FE channel:0x00 frame:9242 vi_tstamp:28558384211 data:0x00000000
     kworker/5:2-1812  [005] ....   901.246650: rtcpu_vinotify_event: tstamp:28560524873 tag:CHANSEL_PXL_SOF channel:0x00 frame:9244 vi_tstamp:28560524348 data:0x00000001
     kworker/5:2-1812  [005] ....   901.246658: rtcpu_vinotify_event: tstamp:28560525091 tag:ATOMP_FS channel:0x00 frame:9244 vi_tstamp:28560524352 data:0x00000000
     kworker/5:2-1812  [005] ....   901.246661: rtcpu_vinotify_event: tstamp:28560529475 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9244 vi_tstamp:28560529094 data:0x08000000
     kworker/5:2-1812  [005] ....   901.302675: rtcpu_vinotify_event: tstamp:28560884841 tag:CHANSEL_PXL_EOF channel:0x00 frame:9244 vi_tstamp:28560884138 data:0x04370002
     kworker/5:2-1812  [005] ....   901.302681: rtcpu_vinotify_event: tstamp:28560885010 tag:ATOMP_FE channel:0x00 frame:9244 vi_tstamp:28560884213 data:0x00000000
     kworker/5:2-1812  [005] ....   901.302686: rtos_queue_peek_from_isr_failed: tstamp:28561181289 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   901.358620: rtcpu_vinotify_event: tstamp:28563024875 tag:CHANSEL_PXL_SOF channel:0x00 frame:9246 vi_tstamp:28563024349 data:0x00000001
     kworker/5:2-1812  [005] ....   901.358625: rtcpu_vinotify_event: tstamp:28563025089 tag:ATOMP_FS channel:0x00 frame:9246 vi_tstamp:28563024353 data:0x00000000
     kworker/5:2-1812  [005] ....   901.358626: rtcpu_vinotify_event: tstamp:28563030406 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9246 vi_tstamp:28563030019 data:0x08000000
     kworker/5:2-1812  [005] ....   901.358627: rtcpu_vinotify_event: tstamp:28563384828 tag:CHANSEL_PXL_EOF channel:0x00 frame:9246 vi_tstamp:28563384141 data:0x04370002
     kworker/5:2-1812  [005] ....   901.358629: rtcpu_vinotify_event: tstamp:28563384978 tag:ATOMP_FE channel:0x00 frame:9246 vi_tstamp:28563384214 data:0x00000000
     kworker/5:2-1812  [005] ....   901.414610: rtcpu_vinotify_event: tstamp:28565524887 tag:CHANSEL_PXL_SOF channel:0x00 frame:9248 vi_tstamp:28565524350 data:0x00000001
     kworker/5:2-1812  [005] ....   901.414616: rtcpu_vinotify_event: tstamp:28565525110 tag:ATOMP_FS channel:0x00 frame:9248 vi_tstamp:28565524354 data:0x00000000
     kworker/5:2-1812  [005] ....   901.414617: rtcpu_vinotify_event: tstamp:28565528827 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9248 vi_tstamp:28565528446 data:0x08000000
     kworker/5:2-1812  [005] ....   901.414617: rtcpu_vinotify_event: tstamp:28565884836 tag:CHANSEL_PXL_EOF channel:0x00 frame:9248 vi_tstamp:28565884142 data:0x04370002
     kworker/5:2-1812  [005] ....   901.414618: rtcpu_vinotify_event: tstamp:28565884986 tag:ATOMP_FE channel:0x00 frame:9248 vi_tstamp:28565884216 data:0x00000000
     kworker/5:2-1812  [005] ....   901.470656: rtos_queue_peek_from_isr_failed: tstamp:28566181252 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   901.526700: rtcpu_vinotify_event: tstamp:28568024890 tag:CHANSEL_PXL_SOF channel:0x00 frame:9250 vi_tstamp:28568024352 data:0x00000001
     kworker/5:2-1812  [005] ....   901.526708: rtcpu_vinotify_event: tstamp:28568025125 tag:ATOMP_FS channel:0x00 frame:9250 vi_tstamp:28568024356 data:0x00000000
     kworker/5:2-1812  [005] ....   901.526711: rtcpu_vinotify_event: tstamp:28568029530 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9250 vi_tstamp:28568029147 data:0x08000000
     kworker/5:2-1812  [005] ....   901.526713: rtcpu_vinotify_event: tstamp:28568384837 tag:CHANSEL_PXL_EOF channel:0x00 frame:9250 vi_tstamp:28568384143 data:0x04370002
     kworker/5:2-1812  [005] ....   901.526715: rtcpu_vinotify_event: tstamp:28568384986 tag:ATOMP_FE channel:0x00 frame:9250 vi_tstamp:28568384218 data:0x00000000
     kworker/5:2-1812  [005] ....   901.582586: rtcpu_vinotify_event: tstamp:28570524877 tag:CHANSEL_PXL_SOF channel:0x00 frame:9252 vi_tstamp:28570524355 data:0x00000001
     kworker/5:2-1812  [005] ....   901.582589: rtcpu_vinotify_event: tstamp:28570525113 tag:ATOMP_FS channel:0x00 frame:9252 vi_tstamp:28570524359 data:0x00000000
     kworker/5:2-1812  [005] ....   901.582590: rtcpu_vinotify_event: tstamp:28570529208 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9252 vi_tstamp:28570528827 data:0x08000000
     kworker/5:2-1812  [005] ....   901.582590: rtcpu_vinotify_event: tstamp:28570884844 tag:CHANSEL_PXL_EOF channel:0x00 frame:9252 vi_tstamp:28570884145 data:0x04370002
     kworker/5:2-1812  [005] ....   901.582590: rtcpu_vinotify_event: tstamp:28570884995 tag:ATOMP_FE channel:0x00 frame:9252 vi_tstamp:28570884219 data:0x00000000
     kworker/5:2-1812  [005] ....   901.582593: rtos_queue_peek_from_isr_failed: tstamp:28571181263 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   901.694746: rtcpu_vinotify_event: tstamp:28573024877 tag:CHANSEL_PXL_SOF channel:0x00 frame:9254 vi_tstamp:28573024357 data:0x00000001
     kworker/5:2-1812  [005] ....   901.694755: rtcpu_vinotify_event: tstamp:28573025099 tag:ATOMP_FS channel:0x00 frame:9254 vi_tstamp:28573024360 data:0x00000000
     kworker/5:2-1812  [005] ....   901.694758: rtcpu_vinotify_event: tstamp:28573030017 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9254 vi_tstamp:28573029635 data:0x08000000
     kworker/5:2-1812  [005] ....   901.694761: rtcpu_vinotify_event: tstamp:28573384832 tag:CHANSEL_PXL_EOF channel:0x00 frame:9254 vi_tstamp:28573384147 data:0x04370002
     kworker/5:2-1812  [005] ....   901.694763: rtcpu_vinotify_event: tstamp:28573384984 tag:ATOMP_FE channel:0x00 frame:9254 vi_tstamp:28573384221 data:0x00000000
     kworker/5:2-1812  [005] ....   901.750593: rtcpu_vinotify_event: tstamp:28575524878 tag:CHANSEL_PXL_SOF channel:0x00 frame:9256 vi_tstamp:28575524357 data:0x00000001
     kworker/5:2-1812  [005] ....   901.750597: rtcpu_vinotify_event: tstamp:28575525099 tag:ATOMP_FS channel:0x00 frame:9256 vi_tstamp:28575524362 data:0x00000000
     kworker/5:2-1812  [005] ....   901.750598: rtcpu_vinotify_event: tstamp:28575530354 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9256 vi_tstamp:28575529971 data:0x08000000
     kworker/5:2-1812  [005] ....   901.750598: rtcpu_vinotify_event: tstamp:28575884836 tag:CHANSEL_PXL_EOF channel:0x00 frame:9256 vi_tstamp:28575884149 data:0x04370002
     kworker/5:2-1812  [005] ....   901.750599: rtcpu_vinotify_event: tstamp:28575884988 tag:ATOMP_FE channel:0x00 frame:9256 vi_tstamp:28575884223 data:0x00000000
     kworker/5:2-1812  [005] ....   901.750602: rtos_queue_peek_from_isr_failed: tstamp:28576181268 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   901.806621: rtcpu_vinotify_event: tstamp:28578024884 tag:CHANSEL_PXL_SOF channel:0x00 frame:9258 vi_tstamp:28578024360 data:0x00000001
     kworker/5:2-1812  [005] ....   901.806626: rtcpu_vinotify_event: tstamp:28578025101 tag:ATOMP_FS channel:0x00 frame:9258 vi_tstamp:28578024363 data:0x00000000
     kworker/5:2-1812  [005] ....   901.806627: rtcpu_vinotify_event: tstamp:28578028752 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9258 vi_tstamp:28578028372 data:0x08000000
     kworker/5:2-1812  [005] ....   901.862617: rtcpu_vinotify_event: tstamp:28578384838 tag:CHANSEL_PXL_EOF channel:0x00 frame:9258 vi_tstamp:28578384151 data:0x04370002
     kworker/5:2-1812  [005] ....   901.862624: rtcpu_vinotify_event: tstamp:28578384987 tag:ATOMP_FE channel:0x00 frame:9258 vi_tstamp:28578384225 data:0x00000000
     kworker/5:2-1812  [005] ....   901.918597: rtcpu_vinotify_event: tstamp:28580524888 tag:CHANSEL_PXL_SOF channel:0x00 frame:9260 vi_tstamp:28580524361 data:0x00000001
     kworker/5:2-1812  [005] ....   901.918602: rtcpu_vinotify_event: tstamp:28580525121 tag:ATOMP_FS channel:0x00 frame:9260 vi_tstamp:28580524365 data:0x00000000
     kworker/5:2-1812  [005] ....   901.918604: rtcpu_vinotify_event: tstamp:28580529241 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9260 vi_tstamp:28580528857 data:0x08000000
     kworker/5:2-1812  [005] ....   901.918605: rtcpu_vinotify_event: tstamp:28580884842 tag:CHANSEL_PXL_EOF channel:0x00 frame:9260 vi_tstamp:28580884152 data:0x04370002
     kworker/5:2-1812  [005] ....   901.918606: rtcpu_vinotify_event: tstamp:28580884995 tag:ATOMP_FE channel:0x00 frame:9260 vi_tstamp:28580884226 data:0x00000000
     kworker/5:2-1812  [005] ....   901.918611: rtos_queue_peek_from_isr_failed: tstamp:28581181288 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   901.974608: rtcpu_vinotify_event: tstamp:28583024889 tag:CHANSEL_PXL_SOF channel:0x00 frame:9262 vi_tstamp:28583024363 data:0x00000001
     kworker/5:2-1812  [005] ....   901.974613: rtcpu_vinotify_event: tstamp:28583025102 tag:ATOMP_FS channel:0x00 frame:9262 vi_tstamp:28583024367 data:0x00000000
     kworker/5:2-1812  [005] ....   901.974614: rtcpu_vinotify_event: tstamp:28583030270 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9262 vi_tstamp:28583029889 data:0x08000000
     kworker/5:2-1812  [005] ....   901.974615: rtcpu_vinotify_event: tstamp:28583384845 tag:CHANSEL_PXL_EOF channel:0x00 frame:9262 vi_tstamp:28583384154 data:0x04370002
     kworker/5:2-1812  [005] ....   901.974616: rtcpu_vinotify_event: tstamp:28583384996 tag:ATOMP_FE channel:0x00 frame:9262 vi_tstamp:28583384228 data:0x00000000
     kworker/5:2-1812  [005] ....   902.086615: rtcpu_vinotify_event: tstamp:28585524894 tag:CHANSEL_PXL_SOF channel:0x00 frame:9264 vi_tstamp:28585524365 data:0x00000001
     kworker/5:2-1812  [005] ....   902.086621: rtcpu_vinotify_event: tstamp:28585525140 tag:ATOMP_FS channel:0x00 frame:9264 vi_tstamp:28585524369 data:0x00000000
     kworker/5:2-1812  [005] ....   902.086622: rtcpu_vinotify_event: tstamp:28585529037 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9264 vi_tstamp:28585528654 data:0x08000000
     kworker/5:2-1812  [005] ....   902.086623: rtcpu_vinotify_event: tstamp:28585884852 tag:CHANSEL_PXL_EOF channel:0x00 frame:9264 vi_tstamp:28585884156 data:0x04370002
     kworker/5:2-1812  [005] ....   902.086624: rtcpu_vinotify_event: tstamp:28585885003 tag:ATOMP_FE channel:0x00 frame:9264 vi_tstamp:28585884230 data:0x00000000
     kworker/5:2-1812  [005] ....   902.086628: rtos_queue_peek_from_isr_failed: tstamp:28586181280 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   902.142578: rtcpu_vinotify_event: tstamp:28588024930 tag:CHANSEL_PXL_SOF channel:0x00 frame:9266 vi_tstamp:28588024367 data:0x00000001
     kworker/5:2-1812  [005] ....   902.142581: rtcpu_vinotify_event: tstamp:28588025153 tag:ATOMP_FS channel:0x00 frame:9266 vi_tstamp:28588024370 data:0x00000000
     kworker/5:2-1812  [005] ....   902.142582: rtcpu_vinotify_event: tstamp:28588029136 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9266 vi_tstamp:28588028757 data:0x08000000
     kworker/5:2-1812  [005] ....   902.142583: rtcpu_vinotify_event: tstamp:28588384846 tag:CHANSEL_PXL_EOF channel:0x00 frame:9266 vi_tstamp:28588384158 data:0x04370002
     kworker/5:2-1812  [005] ....   902.142583: rtcpu_vinotify_event: tstamp:28588384998 tag:ATOMP_FE channel:0x00 frame:9266 vi_tstamp:28588384232 data:0x00000000
     kworker/5:2-1812  [005] ....   902.254637: rtcpu_vinotify_event: tstamp:28590524900 tag:CHANSEL_PXL_SOF channel:0x00 frame:9268 vi_tstamp:28590524368 data:0x00000001
     kworker/5:2-1812  [005] ....   902.254647: rtcpu_vinotify_event: tstamp:28590525132 tag:ATOMP_FS channel:0x00 frame:9268 vi_tstamp:28590524372 data:0x00000000
     kworker/5:2-1812  [005] ....   902.254649: rtcpu_vinotify_event: tstamp:28590528965 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9268 vi_tstamp:28590528581 data:0x08000000
     kworker/5:2-1812  [005] ....   902.254652: rtcpu_vinotify_event: tstamp:28590884847 tag:CHANSEL_PXL_EOF channel:0x00 frame:9268 vi_tstamp:28590884160 data:0x04370002
     kworker/5:2-1812  [005] ....   902.254654: rtcpu_vinotify_event: tstamp:28590885002 tag:ATOMP_FE channel:0x00 frame:9268 vi_tstamp:28590884234 data:0x00000000
     kworker/5:2-1812  [005] ....   902.254661: rtos_queue_peek_from_isr_failed: tstamp:28591181291 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   902.310653: rtcpu_vinotify_event: tstamp:28593024898 tag:CHANSEL_PXL_SOF channel:0x00 frame:9270 vi_tstamp:28593024370 data:0x00000001
     kworker/5:2-1812  [005] ....   902.310661: rtcpu_vinotify_event: tstamp:28593025134 tag:ATOMP_FS channel:0x00 frame:9270 vi_tstamp:28593024374 data:0x00000000
     kworker/5:2-1812  [005] ....   902.310663: rtcpu_vinotify_event: tstamp:28593028188 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9270 vi_tstamp:28593027805 data:0x08000000
     kworker/5:2-1812  [005] ....   902.310665: rtcpu_vinotify_event: tstamp:28593384851 tag:CHANSEL_PXL_EOF channel:0x00 frame:9270 vi_tstamp:28593384161 data:0x04370002
     kworker/5:2-1812  [005] ....   902.310670: rtcpu_vinotify_event: tstamp:28593385001 tag:ATOMP_FE channel:0x00 frame:9270 vi_tstamp:28593384235 data:0x00000000
     kworker/5:2-1812  [005] ....   902.366636: rtcpu_vinotify_event: tstamp:28595524909 tag:CHANSEL_PXL_SOF channel:0x00 frame:9272 vi_tstamp:28595524372 data:0x00000001
     kworker/5:2-1812  [005] ....   902.366643: rtcpu_vinotify_event: tstamp:28595525142 tag:ATOMP_FS channel:0x00 frame:9272 vi_tstamp:28595524376 data:0x00000000
     kworker/5:2-1812  [005] ....   902.366644: rtcpu_vinotify_event: tstamp:28595530234 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9272 vi_tstamp:28595529851 data:0x08000000
     kworker/5:2-1812  [005] ....   902.422647: rtcpu_vinotify_event: tstamp:28595884856 tag:CHANSEL_PXL_EOF channel:0x00 frame:9272 vi_tstamp:28595884163 data:0x04370002
     kworker/5:2-1812  [005] ....   902.422656: rtcpu_vinotify_event: tstamp:28595885006 tag:ATOMP_FE channel:0x00 frame:9272 vi_tstamp:28595884236 data:0x00000000
     kworker/5:2-1812  [005] ....   902.422663: rtos_queue_peek_from_isr_failed: tstamp:28596181296 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   902.478608: rtcpu_vinotify_event: tstamp:28598024911 tag:CHANSEL_PXL_SOF channel:0x00 frame:9274 vi_tstamp:28598024374 data:0x00000001
     kworker/5:2-1812  [005] ....   902.478614: rtcpu_vinotify_event: tstamp:28598025148 tag:ATOMP_FS channel:0x00 frame:9274 vi_tstamp:28598024378 data:0x00000000
     kworker/5:2-1812  [005] ....   902.478615: rtcpu_vinotify_event: tstamp:28598028779 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9274 vi_tstamp:28598028399 data:0x08000000
     kworker/5:2-1812  [005] ....   902.478616: rtcpu_vinotify_event: tstamp:28598384856 tag:CHANSEL_PXL_EOF channel:0x00 frame:9274 vi_tstamp:28598384165 data:0x04370002
     kworker/5:2-1812  [005] ....   902.478617: rtcpu_vinotify_event: tstamp:28598385006 tag:ATOMP_FE channel:0x00 frame:9274 vi_tstamp:28598384239 data:0x00000000
     kworker/5:2-1812  [005] ....   902.534640: rtcpu_vinotify_event: tstamp:28600524912 tag:CHANSEL_PXL_SOF channel:0x00 frame:9276 vi_tstamp:28600524375 data:0x00000001
     kworker/5:2-1812  [005] ....   902.534649: rtcpu_vinotify_event: tstamp:28600525148 tag:ATOMP_FS channel:0x00 frame:9276 vi_tstamp:28600524379 data:0x00000000
     kworker/5:2-1812  [005] ....   902.534651: rtcpu_vinotify_event: tstamp:28600532006 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:9276 vi_tstamp:28600531626 data:0x08000000
     kworker/5:2-1812  [005] ....   902.534653: rtcpu_vinotify_event: tstamp:28600884855 tag:CHANSEL_PXL_EOF channel:0x00 frame:9276 vi_tstamp:28600884166 data:0x04370002
     kworker/5:2-1812  [005] ....   902.534656: rtcpu_vinotify_event: tstamp:28600885008 tag:ATOMP_FE channel:0x00 frame:9276 vi_tstamp:28600884241 data:0x00000000
     kworker/5:2-1812  [005] ....   902.590654: rtos_queue_peek_from_isr_failed: tstamp:28601181326 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   902.646583: rtcpu_vinotify_event: tstamp:28603024905 tag:CHANSEL_PXL_SOF channel:0x00 frame:9278 vi_tstamp:28603024377 data:0x00000001
     kworker/5:2-1812  [005] ....   902.646586: rtcpu_vinotify_event: tstamp:28603025119 tag:ATOMP_FS channel:0x00 frame:9278 vi_tstamp:28603024381 data:0x00000000
     kworker/5:2-1812  [005] ....   902.646587: rtcpu_vinotify_event: tstamp:28603384863 tag:CHANSEL_PXL_EOF channel:0x00 frame:9278 vi_tstamp:28603384168 data:0x04370002
     kworker/5:2-1812  [005] ....   902.646587: rtcpu_vinotify_event: tstamp:28603385015 tag:ATOMP_FE channel:0x00 frame:9278 vi_tstamp:28603384243 data:0x00000000
     kworker/5:2-1812  [005] ....   902.646590: rtos_queue_send_from_isr_failed: tstamp:28603390881 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....   902.646590: rtos_queue_send_from_isr_failed: tstamp:28603391025 queue:0x0b4aad68
     kworker/5:2-1812  [005] ....   902.646591: rtos_queue_send_from_isr_failed: tstamp:28603391166 queue:0x0b4ac998
     kworker/5:2-1812  [005] ....   902.646592: rtos_queue_send_from_isr_failed: tstamp:28603391306 queue:0x0b4ae518
     kworker/5:2-1812  [005] ....   902.646592: rtos_queue_send_from_isr_failed: tstamp:28603391444 queue:0x0b4af2d8
     kworker/5:2-1812  [005] ....   902.646593: rtos_queue_send_from_isr_failed: tstamp:28603391582 queue:0x0b4b0098
     kworker/5:2-1812  [005] ....   902.646594: rtos_queue_send_from_isr_failed: tstamp:28603391725 queue:0x0b4b0e58
     kworker/5:2-1812  [005] ....   902.646594: rtos_queue_send_from_isr_failed: tstamp:28603391862 queue:0x0b4b1c18
     kworker/5:2-1812  [005] ....   902.646596: rtos_queue_send_failed: tstamp:28603392669 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....   902.646596: rtos_queue_send_from_isr_failed: tstamp:28603398832 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....   902.646597: rtos_queue_send_from_isr_failed: tstamp:28603398971 queue:0x0b4aad68
     kworker/5:2-1812  [005] ....   902.646597: rtos_queue_send_from_isr_failed: tstamp:28603399113 queue:0x0b4ac998
     kworker/5:2-1812  [005] ....   902.646598: rtos_queue_send_from_isr_failed: tstamp:28603399253 queue:0x0b4ae518
     kworker/5:2-1812  [005] ....   902.646599: rtos_queue_send_from_isr_failed: tstamp:28603399391 queue:0x0b4af2d8
     kworker/5:2-1812  [005] ....   902.646599: rtos_queue_send_from_isr_failed: tstamp:28603399528 queue:0x0b4b0098
     kworker/5:2-1812  [005] ....   902.646600: rtos_queue_send_from_isr_failed: tstamp:28603399666 queue:0x0b4b0e58
     kworker/5:2-1812  [005] ....   902.646601: rtos_queue_send_from_isr_failed: tstamp:28603399802 queue:0x0b4b1c18
     kworker/5:2-1812  [005] ....   902.646601: rtos_queue_send_failed: tstamp:28603400366 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....   902.702566: rtos_queue_peek_from_isr_failed: tstamp:28606181330 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   902.870559: rtos_queue_peek_from_isr_failed: tstamp:28611181347 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   903.038709: rtos_queue_peek_from_isr_failed: tstamp:28616181382 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   903.206626: rtos_queue_peek_from_isr_failed: tstamp:28621181392 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   903.374662: rtos_queue_peek_from_isr_failed: tstamp:28626181395 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   903.542738: rtos_queue_peek_from_isr_failed: tstamp:28631181392 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   903.710648: rtos_queue_peek_from_isr_failed: tstamp:28636181386 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   903.822686: rtos_queue_peek_from_isr_failed: tstamp:28641181378 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   903.990646: rtos_queue_peek_from_isr_failed: tstamp:28646181396 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   904.158680: rtos_queue_peek_from_isr_failed: tstamp:28651181430 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   904.326667: rtos_queue_peek_from_isr_failed: tstamp:28656181401 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   904.494635: rtos_queue_peek_from_isr_failed: tstamp:28661181409 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   904.662644: rtos_queue_peek_from_isr_failed: tstamp:28666181413 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   904.830742: rtos_queue_peek_from_isr_failed: tstamp:28671181427 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....   904.942658: rtos_queue_peek_from_isr_failed: tstamp:28676181429 queue:0x0b4b4500

CASE 2: Line valid pause higher. 1920x1080 @ 25fps. 30 frames

nvidia@nvidia-desktop:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=GRAY16_LE --set-ctrl bypass_mode=0 --stream-mmap --stream-count=30 --stream-to=test.raw
<<<<<<<<<<<<<<<<<<<<<<<<<<< 25.24 fps
<<<
root@nvidia-desktop:/home/nvidia# cat /sys/kernel/debug/tracing/trace
# tracer: nop
#
# entries-in-buffer/entries-written: 255/255   #P:4
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
     kworker/5:2-1812  [005] ....  1168.020746: rtos_queue_peek_from_isr_failed: tstamp:36896878483 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....  1168.020753: rtcpu_start: tstamp:36896879329
     kworker/5:2-1812  [005] ....  1168.020756: rtos_queue_send_from_isr_failed: tstamp:36896887824 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....  1168.020757: rtos_queue_send_from_isr_failed: tstamp:36896887933 queue:0x0b4aad68
     kworker/5:2-1812  [005] ....  1168.020759: rtos_queue_send_from_isr_failed: tstamp:36896888040 queue:0x0b4ac998
     kworker/5:2-1812  [005] ....  1168.020760: rtos_queue_send_from_isr_failed: tstamp:36896888148 queue:0x0b4ae518
     kworker/5:2-1812  [005] ....  1168.020762: rtos_queue_send_from_isr_failed: tstamp:36896888252 queue:0x0b4af2d8
     kworker/5:2-1812  [005] ....  1168.020763: rtos_queue_send_from_isr_failed: tstamp:36896888355 queue:0x0b4b0098
     kworker/5:2-1812  [005] ....  1168.020764: rtos_queue_send_from_isr_failed: tstamp:36896888459 queue:0x0b4b0e58
     kworker/5:2-1812  [005] ....  1168.020765: rtos_queue_send_from_isr_failed: tstamp:36896888564 queue:0x0b4b1c18
     kworker/5:2-1812  [005] ....  1168.020768: rtos_queue_send_failed: tstamp:36896889022 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....  1168.020769: rtos_queue_send_from_isr_failed: tstamp:36896891359 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....  1168.020770: rtos_queue_send_from_isr_failed: tstamp:36896891465 queue:0x0b4aad68
     kworker/5:2-1812  [005] ....  1168.020772: rtos_queue_send_from_isr_failed: tstamp:36896891571 queue:0x0b4ac998
     kworker/5:2-1812  [005] ....  1168.020773: rtos_queue_send_from_isr_failed: tstamp:36896891679 queue:0x0b4ae518
     kworker/5:2-1812  [005] ....  1168.020774: rtos_queue_send_from_isr_failed: tstamp:36896891784 queue:0x0b4af2d8
     kworker/5:2-1812  [005] ....  1168.020775: rtos_queue_send_from_isr_failed: tstamp:36896891899 queue:0x0b4b0098
     kworker/5:2-1812  [005] ....  1168.020777: rtos_queue_send_from_isr_failed: tstamp:36896892007 queue:0x0b4b0e58
     kworker/5:2-1812  [005] ....  1168.020778: rtos_queue_send_from_isr_failed: tstamp:36896892112 queue:0x0b4b1c18
     kworker/5:2-1812  [005] ....  1168.020779: rtos_queue_send_failed: tstamp:36896893040 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....  1168.076724: rtcpu_vinotify_event: tstamp:36897471426 tag:CSIMUX_STREAM channel:0xff frame:0 vi_tstamp:36897471037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.076727: rtcpu_vinotify_event: tstamp:36898721554 tag:CHANSEL_PXL_SOF channel:0x00 frame:547 vi_tstamp:36898721037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.076728: rtcpu_vinotify_event: tstamp:36898721735 tag:ATOMP_FS channel:0x00 frame:547 vi_tstamp:36898721041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.076728: rtcpu_vinotify_event: tstamp:36898726664 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:547 vi_tstamp:36898726294 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.132782: rtcpu_vinotify_event: tstamp:36899306266 tag:CHANSEL_PXL_EOF channel:0x00 frame:547 vi_tstamp:36899305619 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.132787: rtcpu_vinotify_event: tstamp:36899306396 tag:ATOMP_FE channel:0x00 frame:547 vi_tstamp:36899305693 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.132790: rtcpu_vinotify_event: tstamp:36899971548 tag:CHANSEL_PXL_SOF channel:0x00 frame:548 vi_tstamp:36899971037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.132792: rtcpu_vinotify_event: tstamp:36899971725 tag:ATOMP_FS channel:0x00 frame:548 vi_tstamp:36899971041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.132794: rtcpu_vinotify_event: tstamp:36899974537 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:548 vi_tstamp:36899974165 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.132796: rtcpu_vinotify_event: tstamp:36900556264 tag:CHANSEL_PXL_EOF channel:0x00 frame:548 vi_tstamp:36900555619 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.132798: rtcpu_vinotify_event: tstamp:36900556393 tag:ATOMP_FE channel:0x00 frame:548 vi_tstamp:36900555694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.188755: rtcpu_vinotify_event: tstamp:36901221554 tag:CHANSEL_PXL_SOF channel:0x00 frame:549 vi_tstamp:36901221037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.188759: rtcpu_vinotify_event: tstamp:36901221733 tag:ATOMP_FS channel:0x00 frame:549 vi_tstamp:36901221041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.188760: rtcpu_vinotify_event: tstamp:36901226720 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:549 vi_tstamp:36901226347 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.188761: rtcpu_vinotify_event: tstamp:36901806249 tag:CHANSEL_PXL_EOF channel:0x00 frame:549 vi_tstamp:36901805619 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.188762: rtcpu_vinotify_event: tstamp:36901806378 tag:ATOMP_FE channel:0x00 frame:549 vi_tstamp:36901805693 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.188765: rtos_queue_peek_from_isr_failed: tstamp:36901878828 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....  1168.188766: rtcpu_vinotify_event: tstamp:36902471549 tag:CHANSEL_PXL_SOF channel:0x00 frame:550 vi_tstamp:36902471037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.188767: rtcpu_vinotify_event: tstamp:36902471729 tag:ATOMP_FS channel:0x00 frame:550 vi_tstamp:36902471041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.188768: rtcpu_vinotify_event: tstamp:36902477696 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:550 vi_tstamp:36902477326 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.244771: rtcpu_vinotify_event: tstamp:36903056252 tag:CHANSEL_PXL_EOF channel:0x00 frame:550 vi_tstamp:36903055619 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.244779: rtcpu_vinotify_event: tstamp:36903056374 tag:ATOMP_FE channel:0x00 frame:550 vi_tstamp:36903055693 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.244781: rtcpu_vinotify_event: tstamp:36903721576 tag:CHANSEL_PXL_SOF channel:0x00 frame:551 vi_tstamp:36903721037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.244783: rtcpu_vinotify_event: tstamp:36903721776 tag:ATOMP_FS channel:0x00 frame:551 vi_tstamp:36903721041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.244785: rtcpu_vinotify_event: tstamp:36903724456 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:551 vi_tstamp:36903724085 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.244787: rtcpu_vinotify_event: tstamp:36904306248 tag:CHANSEL_PXL_EOF channel:0x00 frame:551 vi_tstamp:36904305619 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.244789: rtcpu_vinotify_event: tstamp:36904306363 tag:ATOMP_FE channel:0x00 frame:551 vi_tstamp:36904305694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.300768: rtcpu_vinotify_event: tstamp:36904971548 tag:CHANSEL_PXL_SOF channel:0x00 frame:552 vi_tstamp:36904971037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.300773: rtcpu_vinotify_event: tstamp:36904971727 tag:ATOMP_FS channel:0x00 frame:552 vi_tstamp:36904971041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.300774: rtcpu_vinotify_event: tstamp:36904977728 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:552 vi_tstamp:36904977353 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.300775: rtcpu_vinotify_event: tstamp:36905556255 tag:CHANSEL_PXL_EOF channel:0x00 frame:552 vi_tstamp:36905555620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.300776: rtcpu_vinotify_event: tstamp:36905556371 tag:ATOMP_FE channel:0x00 frame:552 vi_tstamp:36905555694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.356731: rtcpu_vinotify_event: tstamp:36906221551 tag:CHANSEL_PXL_SOF channel:0x00 frame:553 vi_tstamp:36906221037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.356734: rtcpu_vinotify_event: tstamp:36906221738 tag:ATOMP_FS channel:0x00 frame:553 vi_tstamp:36906221041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.356735: rtcpu_vinotify_event: tstamp:36906225335 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:553 vi_tstamp:36906224964 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.356735: rtcpu_vinotify_event: tstamp:36906806246 tag:CHANSEL_PXL_EOF channel:0x00 frame:553 vi_tstamp:36906805620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.356736: rtcpu_vinotify_event: tstamp:36906806363 tag:ATOMP_FE channel:0x00 frame:553 vi_tstamp:36906805694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.356738: rtos_queue_peek_from_isr_failed: tstamp:36906878835 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....  1168.356738: rtcpu_vinotify_event: tstamp:36907471552 tag:CHANSEL_PXL_SOF channel:0x00 frame:554 vi_tstamp:36907471037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.356739: rtcpu_vinotify_event: tstamp:36907471730 tag:ATOMP_FS channel:0x00 frame:554 vi_tstamp:36907471041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.356739: rtcpu_vinotify_event: tstamp:36907474702 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:554 vi_tstamp:36907474330 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.412842: rtcpu_vinotify_event: tstamp:36908056245 tag:CHANSEL_PXL_EOF channel:0x00 frame:554 vi_tstamp:36908055620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.412851: rtcpu_vinotify_event: tstamp:36908056367 tag:ATOMP_FE channel:0x00 frame:554 vi_tstamp:36908055694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.412854: rtcpu_vinotify_event: tstamp:36908721560 tag:CHANSEL_PXL_SOF channel:0x00 frame:555 vi_tstamp:36908721037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.412856: rtcpu_vinotify_event: tstamp:36908721753 tag:ATOMP_FS channel:0x00 frame:555 vi_tstamp:36908721041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.412858: rtcpu_vinotify_event: tstamp:36908726517 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:555 vi_tstamp:36908726146 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.412860: rtcpu_vinotify_event: tstamp:36909306247 tag:CHANSEL_PXL_EOF channel:0x00 frame:555 vi_tstamp:36909305620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.412862: rtcpu_vinotify_event: tstamp:36909306364 tag:ATOMP_FE channel:0x00 frame:555 vi_tstamp:36909305694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.468751: rtcpu_vinotify_event: tstamp:36909971548 tag:CHANSEL_PXL_SOF channel:0x00 frame:556 vi_tstamp:36909971037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.468755: rtcpu_vinotify_event: tstamp:36909971727 tag:ATOMP_FS channel:0x00 frame:556 vi_tstamp:36909971041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.468755: rtcpu_vinotify_event: tstamp:36909977472 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:556 vi_tstamp:36909977104 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.468756: rtcpu_vinotify_event: tstamp:36910556246 tag:CHANSEL_PXL_EOF channel:0x00 frame:556 vi_tstamp:36910555620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.468757: rtcpu_vinotify_event: tstamp:36910556364 tag:ATOMP_FE channel:0x00 frame:556 vi_tstamp:36910555694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.468758: rtcpu_vinotify_event: tstamp:36911221549 tag:CHANSEL_PXL_SOF channel:0x00 frame:557 vi_tstamp:36911221037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.468758: rtcpu_vinotify_event: tstamp:36911221729 tag:ATOMP_FS channel:0x00 frame:557 vi_tstamp:36911221041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.468759: rtcpu_vinotify_event: tstamp:36911228230 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:557 vi_tstamp:36911227859 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.524805: rtcpu_vinotify_event: tstamp:36911806246 tag:CHANSEL_PXL_EOF channel:0x00 frame:557 vi_tstamp:36911805620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.524814: rtcpu_vinotify_event: tstamp:36911806360 tag:ATOMP_FE channel:0x00 frame:557 vi_tstamp:36911805694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.524820: rtos_queue_peek_from_isr_failed: tstamp:36911878837 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....  1168.524822: rtcpu_vinotify_event: tstamp:36912471559 tag:CHANSEL_PXL_SOF channel:0x00 frame:558 vi_tstamp:36912471038 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.524824: rtcpu_vinotify_event: tstamp:36912471747 tag:ATOMP_FS channel:0x00 frame:558 vi_tstamp:36912471042 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.524826: rtcpu_vinotify_event: tstamp:36912474888 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:558 vi_tstamp:36912474517 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.524828: rtcpu_vinotify_event: tstamp:36913056256 tag:CHANSEL_PXL_EOF channel:0x00 frame:558 vi_tstamp:36913055620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.524830: rtcpu_vinotify_event: tstamp:36913056371 tag:ATOMP_FE channel:0x00 frame:558 vi_tstamp:36913055694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.580768: rtcpu_vinotify_event: tstamp:36913721562 tag:CHANSEL_PXL_SOF channel:0x00 frame:559 vi_tstamp:36913721037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.580776: rtcpu_vinotify_event: tstamp:36913721741 tag:ATOMP_FS channel:0x00 frame:559 vi_tstamp:36913721041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.580779: rtcpu_vinotify_event: tstamp:36913724951 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:559 vi_tstamp:36913724583 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.580781: rtcpu_vinotify_event: tstamp:36914306247 tag:CHANSEL_PXL_EOF channel:0x00 frame:559 vi_tstamp:36914305620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.580783: rtcpu_vinotify_event: tstamp:36914306362 tag:ATOMP_FE channel:0x00 frame:559 vi_tstamp:36914305694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.636731: rtcpu_vinotify_event: tstamp:36914971551 tag:CHANSEL_PXL_SOF channel:0x00 frame:560 vi_tstamp:36914971038 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.636734: rtcpu_vinotify_event: tstamp:36914971732 tag:ATOMP_FS channel:0x00 frame:560 vi_tstamp:36914971041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.636735: rtcpu_vinotify_event: tstamp:36914974322 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:560 vi_tstamp:36914973954 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.636735: rtcpu_vinotify_event: tstamp:36915556247 tag:CHANSEL_PXL_EOF channel:0x00 frame:560 vi_tstamp:36915555620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.636736: rtcpu_vinotify_event: tstamp:36915556366 tag:ATOMP_FE channel:0x00 frame:560 vi_tstamp:36915555694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.636737: rtcpu_vinotify_event: tstamp:36916221547 tag:CHANSEL_PXL_SOF channel:0x00 frame:561 vi_tstamp:36916221038 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.636737: rtcpu_vinotify_event: tstamp:36916221731 tag:ATOMP_FS channel:0x00 frame:561 vi_tstamp:36916221042 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.636738: rtcpu_vinotify_event: tstamp:36916225557 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:561 vi_tstamp:36916225187 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.692798: rtcpu_vinotify_event: tstamp:36916806254 tag:CHANSEL_PXL_EOF channel:0x00 frame:561 vi_tstamp:36916805620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.692805: rtcpu_vinotify_event: tstamp:36916806376 tag:ATOMP_FE channel:0x00 frame:561 vi_tstamp:36916805694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.692810: rtos_queue_peek_from_isr_failed: tstamp:36916878842 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....  1168.692811: rtcpu_vinotify_event: tstamp:36917471550 tag:CHANSEL_PXL_SOF channel:0x00 frame:562 vi_tstamp:36917471038 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.692815: rtcpu_vinotify_event: tstamp:36917471730 tag:ATOMP_FS channel:0x00 frame:562 vi_tstamp:36917471042 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.692817: rtcpu_vinotify_event: tstamp:36917477936 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:562 vi_tstamp:36917477559 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.692818: rtcpu_vinotify_event: tstamp:36918056257 tag:CHANSEL_PXL_EOF channel:0x00 frame:562 vi_tstamp:36918055620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.692820: rtcpu_vinotify_event: tstamp:36918056372 tag:ATOMP_FE channel:0x00 frame:562 vi_tstamp:36918055694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.748728: rtcpu_vinotify_event: tstamp:36918721552 tag:CHANSEL_PXL_SOF channel:0x00 frame:563 vi_tstamp:36918721037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.748733: rtcpu_vinotify_event: tstamp:36918721742 tag:ATOMP_FS channel:0x00 frame:563 vi_tstamp:36918721041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.748733: rtcpu_vinotify_event: tstamp:36918725916 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:563 vi_tstamp:36918725543 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.748734: rtcpu_vinotify_event: tstamp:36919306248 tag:CHANSEL_PXL_EOF channel:0x00 frame:563 vi_tstamp:36919305620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.748735: rtcpu_vinotify_event: tstamp:36919306363 tag:ATOMP_FE channel:0x00 frame:563 vi_tstamp:36919305694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.748735: rtcpu_vinotify_event: tstamp:36919971542 tag:CHANSEL_PXL_SOF channel:0x00 frame:564 vi_tstamp:36919971037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.748736: rtcpu_vinotify_event: tstamp:36919971704 tag:ATOMP_FS channel:0x00 frame:564 vi_tstamp:36919971041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.748737: rtcpu_vinotify_event: tstamp:36919974187 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:564 vi_tstamp:36919973824 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.804760: rtcpu_vinotify_event: tstamp:36920556250 tag:CHANSEL_PXL_EOF channel:0x00 frame:564 vi_tstamp:36920555620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.804765: rtcpu_vinotify_event: tstamp:36920556372 tag:ATOMP_FE channel:0x00 frame:564 vi_tstamp:36920555694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.804766: rtcpu_vinotify_event: tstamp:36921221549 tag:CHANSEL_PXL_SOF channel:0x00 frame:565 vi_tstamp:36921221037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.804766: rtcpu_vinotify_event: tstamp:36921221728 tag:ATOMP_FS channel:0x00 frame:565 vi_tstamp:36921221041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.804767: rtcpu_vinotify_event: tstamp:36921224664 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:565 vi_tstamp:36921224292 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.804768: rtcpu_vinotify_event: tstamp:36921806251 tag:CHANSEL_PXL_EOF channel:0x00 frame:565 vi_tstamp:36921805620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.804769: rtcpu_vinotify_event: tstamp:36921806367 tag:ATOMP_FE channel:0x00 frame:565 vi_tstamp:36921805694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.804771: rtos_queue_peek_from_isr_failed: tstamp:36921878851 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....  1168.860774: rtcpu_vinotify_event: tstamp:36922471552 tag:CHANSEL_PXL_SOF channel:0x00 frame:566 vi_tstamp:36922471037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.860779: rtcpu_vinotify_event: tstamp:36922471741 tag:ATOMP_FS channel:0x00 frame:566 vi_tstamp:36922471041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.860780: rtcpu_vinotify_event: tstamp:36922474390 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:566 vi_tstamp:36922474020 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.860782: rtcpu_vinotify_event: tstamp:36923056265 tag:CHANSEL_PXL_EOF channel:0x00 frame:566 vi_tstamp:36923055620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.860782: rtcpu_vinotify_event: tstamp:36923056389 tag:ATOMP_FE channel:0x00 frame:566 vi_tstamp:36923055694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.916725: rtcpu_vinotify_event: tstamp:36923721549 tag:CHANSEL_PXL_SOF channel:0x00 frame:567 vi_tstamp:36923721037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.916728: rtcpu_vinotify_event: tstamp:36923721728 tag:ATOMP_FS channel:0x00 frame:567 vi_tstamp:36923721041 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.916728: rtcpu_vinotify_event: tstamp:36923725077 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:567 vi_tstamp:36923724705 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.916729: rtcpu_vinotify_event: tstamp:36924306257 tag:CHANSEL_PXL_EOF channel:0x00 frame:567 vi_tstamp:36924305620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.916729: rtcpu_vinotify_event: tstamp:36924306391 tag:ATOMP_FE channel:0x00 frame:567 vi_tstamp:36924305694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.916730: rtcpu_vinotify_event: tstamp:36924971557 tag:CHANSEL_PXL_SOF channel:0x00 frame:568 vi_tstamp:36924971037 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.916730: rtcpu_vinotify_event: tstamp:36924971736 tag:ATOMP_FS channel:0x00 frame:568 vi_tstamp:36924971042 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.916730: rtcpu_vinotify_event: tstamp:36924975157 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:568 vi_tstamp:36924974785 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.972815: rtcpu_vinotify_event: tstamp:36925556247 tag:CHANSEL_PXL_EOF channel:0x00 frame:568 vi_tstamp:36925555620 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.972821: rtcpu_vinotify_event: tstamp:36925556369 tag:ATOMP_FE channel:0x00 frame:568 vi_tstamp:36925555694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.972822: rtcpu_vinotify_event: tstamp:36926221550 tag:CHANSEL_PXL_SOF channel:0x00 frame:569 vi_tstamp:36926221038 data:0x00000001
     kworker/5:2-1812  [005] ....  1168.972823: rtcpu_vinotify_event: tstamp:36926221728 tag:ATOMP_FS channel:0x00 frame:569 vi_tstamp:36926221042 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.972824: rtcpu_vinotify_event: tstamp:36926224470 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:569 vi_tstamp:36926224097 data:0x08000000
     kworker/5:2-1812  [005] ....  1168.972826: rtcpu_vinotify_event: tstamp:36926806256 tag:CHANSEL_PXL_EOF channel:0x00 frame:569 vi_tstamp:36926805621 data:0x04370002
     kworker/5:2-1812  [005] ....  1168.972827: rtcpu_vinotify_event: tstamp:36926806378 tag:ATOMP_FE channel:0x00 frame:569 vi_tstamp:36926805694 data:0x00000000
     kworker/5:2-1812  [005] ....  1168.972831: rtos_queue_peek_from_isr_failed: tstamp:36926878867 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....  1169.028746: rtcpu_vinotify_event: tstamp:36927471541 tag:CHANSEL_PXL_SOF channel:0x00 frame:570 vi_tstamp:36927471038 data:0x00000001
     kworker/5:2-1812  [005] ....  1169.028749: rtcpu_vinotify_event: tstamp:36927471703 tag:ATOMP_FS channel:0x00 frame:570 vi_tstamp:36927471042 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.028750: rtcpu_vinotify_event: tstamp:36927473940 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:570 vi_tstamp:36927473576 data:0x08000000
     kworker/5:2-1812  [005] ....  1169.028751: rtcpu_vinotify_event: tstamp:36928056257 tag:CHANSEL_PXL_EOF channel:0x00 frame:570 vi_tstamp:36928055620 data:0x04370002
     kworker/5:2-1812  [005] ....  1169.028751: rtcpu_vinotify_event: tstamp:36928056372 tag:ATOMP_FE channel:0x00 frame:570 vi_tstamp:36928055694 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.028752: rtcpu_vinotify_event: tstamp:36928721549 tag:CHANSEL_PXL_SOF channel:0x00 frame:571 vi_tstamp:36928721038 data:0x00000001
     kworker/5:2-1812  [005] ....  1169.028752: rtcpu_vinotify_event: tstamp:36928721722 tag:ATOMP_FS channel:0x00 frame:571 vi_tstamp:36928721042 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.028753: rtcpu_vinotify_event: tstamp:36928724483 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:571 vi_tstamp:36928724121 data:0x08000000
     kworker/5:2-1812  [005] ....  1169.084757: rtcpu_vinotify_event: tstamp:36929306245 tag:CHANSEL_PXL_EOF channel:0x00 frame:571 vi_tstamp:36929305620 data:0x04370002
     kworker/5:2-1812  [005] ....  1169.084761: rtcpu_vinotify_event: tstamp:36929306363 tag:ATOMP_FE channel:0x00 frame:571 vi_tstamp:36929305694 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.084762: rtcpu_vinotify_event: tstamp:36929971558 tag:CHANSEL_PXL_SOF channel:0x00 frame:572 vi_tstamp:36929971038 data:0x00000001
     kworker/5:2-1812  [005] ....  1169.084763: rtcpu_vinotify_event: tstamp:36929971752 tag:ATOMP_FS channel:0x00 frame:572 vi_tstamp:36929971042 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.084763: rtcpu_vinotify_event: tstamp:36929975315 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:572 vi_tstamp:36929974943 data:0x08000000
     kworker/5:2-1812  [005] ....  1169.084764: rtcpu_vinotify_event: tstamp:36930556252 tag:CHANSEL_PXL_EOF channel:0x00 frame:572 vi_tstamp:36930555620 data:0x04370002
     kworker/5:2-1812  [005] ....  1169.084765: rtcpu_vinotify_event: tstamp:36930556375 tag:ATOMP_FE channel:0x00 frame:572 vi_tstamp:36930555694 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.140769: rtcpu_vinotify_event: tstamp:36931221550 tag:CHANSEL_PXL_SOF channel:0x00 frame:573 vi_tstamp:36931221038 data:0x00000001
     kworker/5:2-1812  [005] ....  1169.140774: rtcpu_vinotify_event: tstamp:36931221730 tag:ATOMP_FS channel:0x00 frame:573 vi_tstamp:36931221042 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.140775: rtcpu_vinotify_event: tstamp:36931226618 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:573 vi_tstamp:36931226245 data:0x08000000
     kworker/5:2-1812  [005] ....  1169.140776: rtcpu_vinotify_event: tstamp:36931806254 tag:CHANSEL_PXL_EOF channel:0x00 frame:573 vi_tstamp:36931805620 data:0x04370002
     kworker/5:2-1812  [005] ....  1169.140777: rtcpu_vinotify_event: tstamp:36931806377 tag:ATOMP_FE channel:0x00 frame:573 vi_tstamp:36931805694 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.140781: rtos_queue_peek_from_isr_failed: tstamp:36931878872 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....  1169.196742: rtcpu_vinotify_event: tstamp:36932471560 tag:CHANSEL_PXL_SOF channel:0x00 frame:574 vi_tstamp:36932471038 data:0x00000001
     kworker/5:2-1812  [005] ....  1169.196745: rtcpu_vinotify_event: tstamp:36932471748 tag:ATOMP_FS channel:0x00 frame:574 vi_tstamp:36932471042 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.196745: rtcpu_vinotify_event: tstamp:36932474856 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:574 vi_tstamp:36932474488 data:0x08000000
     kworker/5:2-1812  [005] ....  1169.196746: rtcpu_vinotify_event: tstamp:36933056251 tag:CHANSEL_PXL_EOF channel:0x00 frame:574 vi_tstamp:36933055620 data:0x04370002
     kworker/5:2-1812  [005] ....  1169.196746: rtcpu_vinotify_event: tstamp:36933056373 tag:ATOMP_FE channel:0x00 frame:574 vi_tstamp:36933055695 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.196747: rtcpu_vinotify_event: tstamp:36933721574 tag:CHANSEL_PXL_SOF channel:0x00 frame:575 vi_tstamp:36933721037 data:0x00000001
     kworker/5:2-1812  [005] ....  1169.196747: rtcpu_vinotify_event: tstamp:36933721771 tag:ATOMP_FS channel:0x00 frame:575 vi_tstamp:36933721042 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.196748: rtcpu_vinotify_event: tstamp:36933724586 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:575 vi_tstamp:36933724215 data:0x08000000
     kworker/5:2-1812  [005] ....  1169.252747: rtcpu_vinotify_event: tstamp:36934306250 tag:CHANSEL_PXL_EOF channel:0x00 frame:575 vi_tstamp:36934305621 data:0x04370002
     kworker/5:2-1812  [005] ....  1169.252751: rtcpu_vinotify_event: tstamp:36934306365 tag:ATOMP_FE channel:0x00 frame:575 vi_tstamp:36934305695 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.252751: rtcpu_vinotify_event: tstamp:36934971561 tag:CHANSEL_PXL_SOF channel:0x00 frame:576 vi_tstamp:36934971038 data:0x00000001
     kworker/5:2-1812  [005] ....  1169.252752: rtcpu_vinotify_event: tstamp:36934971756 tag:ATOMP_FS channel:0x00 frame:576 vi_tstamp:36934971042 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.252753: rtcpu_vinotify_event: tstamp:36934974193 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:576 vi_tstamp:36934973824 data:0x08000000
     kworker/5:2-1812  [005] ....  1169.252753: rtcpu_vinotify_event: tstamp:36935556250 tag:CHANSEL_PXL_EOF channel:0x00 frame:576 vi_tstamp:36935555621 data:0x04370002
     kworker/5:2-1812  [005] ....  1169.252754: rtcpu_vinotify_event: tstamp:36935556369 tag:ATOMP_FE channel:0x00 frame:576 vi_tstamp:36935555695 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.308752: rtcpu_vinotify_event: tstamp:36936221540 tag:CHANSEL_PXL_SOF channel:0x00 frame:577 vi_tstamp:36936221038 data:0x00000001
     kworker/5:2-1812  [005] ....  1169.308755: rtcpu_vinotify_event: tstamp:36936221711 tag:ATOMP_FS channel:0x00 frame:577 vi_tstamp:36936221042 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.308756: rtcpu_vinotify_event: tstamp:36936225193 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:577 vi_tstamp:36936224828 data:0x08000000
     kworker/5:2-1812  [005] ....  1169.308757: rtcpu_vinotify_event: tstamp:36936806300 tag:CHANSEL_PXL_EOF channel:0x00 frame:577 vi_tstamp:36936805621 data:0x04370002
     kworker/5:2-1812  [005] ....  1169.308757: rtcpu_vinotify_event: tstamp:36936806423 tag:ATOMP_FE channel:0x00 frame:577 vi_tstamp:36936805695 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.308760: rtos_queue_peek_from_isr_failed: tstamp:36936878879 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....  1169.308761: rtcpu_vinotify_event: tstamp:36937471548 tag:CHANSEL_PXL_SOF channel:0x00 frame:578 vi_tstamp:36937471038 data:0x00000001
     kworker/5:2-1812  [005] ....  1169.308761: rtcpu_vinotify_event: tstamp:36937471727 tag:ATOMP_FS channel:0x00 frame:578 vi_tstamp:36937471042 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.308762: rtcpu_vinotify_event: tstamp:36937474784 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:578 vi_tstamp:36937474414 data:0x08000000
     kworker/5:3-9506  [005] ....  1169.364803: rtcpu_vinotify_event: tstamp:36938056268 tag:CHANSEL_PXL_EOF channel:0x00 frame:578 vi_tstamp:36938055621 data:0x04370002
     kworker/5:3-9506  [005] ....  1169.364812: rtcpu_vinotify_event: tstamp:36938056388 tag:ATOMP_FE channel:0x00 frame:578 vi_tstamp:36938055695 data:0x00000000
     kworker/5:3-9506  [005] ....  1169.364815: rtcpu_vinotify_event: tstamp:36938721549 tag:CHANSEL_PXL_SOF channel:0x00 frame:579 vi_tstamp:36938721038 data:0x00000001
     kworker/5:3-9506  [005] ....  1169.364817: rtcpu_vinotify_event: tstamp:36938721729 tag:ATOMP_FS channel:0x00 frame:579 vi_tstamp:36938721042 data:0x00000000
     kworker/5:3-9506  [005] ....  1169.364819: rtcpu_vinotify_event: tstamp:36938725228 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:579 vi_tstamp:36938724859 data:0x08000000
     kworker/5:3-9506  [005] ....  1169.364821: rtcpu_vinotify_event: tstamp:36939306265 tag:CHANSEL_PXL_EOF channel:0x00 frame:579 vi_tstamp:36939305620 data:0x04370002
     kworker/5:3-9506  [005] ....  1169.364823: rtcpu_vinotify_event: tstamp:36939306409 tag:ATOMP_FE channel:0x00 frame:579 vi_tstamp:36939305695 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.420761: rtcpu_vinotify_event: tstamp:36939971551 tag:CHANSEL_PXL_SOF channel:0x00 frame:580 vi_tstamp:36939971038 data:0x00000001
     kworker/5:2-1812  [005] ....  1169.420767: rtcpu_vinotify_event: tstamp:36939971734 tag:ATOMP_FS channel:0x00 frame:580 vi_tstamp:36939971042 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.420768: rtcpu_vinotify_event: tstamp:36940556291 tag:CHANSEL_PXL_EOF channel:0x00 frame:580 vi_tstamp:36940555621 data:0x04370002
     kworker/5:2-1812  [005] ....  1169.420769: rtcpu_vinotify_event: tstamp:36940556420 tag:ATOMP_FE channel:0x00 frame:580 vi_tstamp:36940555694 data:0x00000000
     kworker/5:2-1812  [005] ....  1169.420773: rtos_queue_send_from_isr_failed: tstamp:36940561632 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....  1169.420774: rtos_queue_send_from_isr_failed: tstamp:36940561756 queue:0x0b4aad68
     kworker/5:2-1812  [005] ....  1169.420775: rtos_queue_send_from_isr_failed: tstamp:36940561877 queue:0x0b4ac998
     kworker/5:2-1812  [005] ....  1169.420776: rtos_queue_send_from_isr_failed: tstamp:36940561996 queue:0x0b4ae518
     kworker/5:2-1812  [005] ....  1169.420777: rtos_queue_send_from_isr_failed: tstamp:36940562114 queue:0x0b4af2d8
     kworker/5:2-1812  [005] ....  1169.420778: rtos_queue_send_from_isr_failed: tstamp:36940562231 queue:0x0b4b0098
     kworker/5:2-1812  [005] ....  1169.420779: rtos_queue_send_from_isr_failed: tstamp:36940562348 queue:0x0b4b0e58
     kworker/5:2-1812  [005] ....  1169.420780: rtos_queue_send_from_isr_failed: tstamp:36940562464 queue:0x0b4b1c18
     kworker/5:2-1812  [005] ....  1169.420782: rtos_queue_send_failed: tstamp:36940563144 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....  1169.420783: rtos_queue_send_from_isr_failed: tstamp:36940568940 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....  1169.420784: rtos_queue_send_from_isr_failed: tstamp:36940569072 queue:0x0b4aad68
     kworker/5:2-1812  [005] ....  1169.420785: rtos_queue_send_from_isr_failed: tstamp:36940569193 queue:0x0b4ac998
     kworker/5:2-1812  [005] ....  1169.420786: rtos_queue_send_from_isr_failed: tstamp:36940569311 queue:0x0b4ae518
     kworker/5:2-1812  [005] ....  1169.420787: rtos_queue_send_from_isr_failed: tstamp:36940569427 queue:0x0b4af2d8
     kworker/5:2-1812  [005] ....  1169.420788: rtos_queue_send_from_isr_failed: tstamp:36940569544 queue:0x0b4b0098
     kworker/5:2-1812  [005] ....  1169.420788: rtos_queue_send_from_isr_failed: tstamp:36940569661 queue:0x0b4b0e58
     kworker/5:2-1812  [005] ....  1169.420789: rtos_queue_send_from_isr_failed: tstamp:36940569778 queue:0x0b4b1c18
     kworker/5:2-1812  [005] ....  1169.420790: rtos_queue_send_failed: tstamp:36940570252 queue:0x0b4a7258
     kworker/5:2-1812  [005] ....  1169.476742: rtos_queue_peek_from_isr_failed: tstamp:36941878885 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....  1169.644757: rtos_queue_peek_from_isr_failed: tstamp:36946878902 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....  1169.812752: rtos_queue_peek_from_isr_failed: tstamp:36951878931 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....  1169.924741: rtos_queue_peek_from_isr_failed: tstamp:36956878915 queue:0x0b4b4500
     kworker/5:2-1812  [005] ....  1170.092740: rtos_queue_peek_from_isr_failed: tstamp:36961878936 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1170.260826: rtos_queue_peek_from_isr_failed: tstamp:36966878949 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1170.428863: rtos_queue_peek_from_isr_failed: tstamp:36971878933 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1170.596865: rtos_queue_peek_from_isr_failed: tstamp:36976878944 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1170.764815: rtos_queue_peek_from_isr_failed: tstamp:36981878949 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1170.932817: rtos_queue_peek_from_isr_failed: tstamp:36986878943 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1171.044868: rtos_queue_peek_from_isr_failed: tstamp:36991878959 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1171.212800: rtos_queue_peek_from_isr_failed: tstamp:36996878983 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1171.380840: rtos_queue_peek_from_isr_failed: tstamp:37001878995 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1171.548830: rtos_queue_peek_from_isr_failed: tstamp:37006878977 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1171.716849: rtos_queue_peek_from_isr_failed: tstamp:37011878986 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1171.884862: rtos_queue_peek_from_isr_failed: tstamp:37016878996 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1172.052846: rtos_queue_peek_from_isr_failed: tstamp:37021878991 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1172.164836: rtos_queue_peek_from_isr_failed: tstamp:37026879006 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1172.332841: rtos_queue_peek_from_isr_failed: tstamp:37031879029 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1172.500855: rtos_queue_peek_from_isr_failed: tstamp:37036879012 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1172.668845: rtos_queue_peek_from_isr_failed: tstamp:37041879018 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1172.836871: rtos_queue_peek_from_isr_failed: tstamp:37046879054 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1173.004862: rtos_queue_peek_from_isr_failed: tstamp:37051879092 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1173.172799: rtos_queue_peek_from_isr_failed: tstamp:37056879049 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1173.284832: rtos_queue_peek_from_isr_failed: tstamp:37061879059 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1173.452874: rtos_queue_peek_from_isr_failed: tstamp:37066879059 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1173.620931: rtos_queue_peek_from_isr_failed: tstamp:37071879064 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1173.788850: rtos_queue_peek_from_isr_failed: tstamp:37076879082 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1173.956877: rtos_queue_peek_from_isr_failed: tstamp:37081879085 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1174.124861: rtos_queue_peek_from_isr_failed: tstamp:37086879082 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1174.292868: rtos_queue_peek_from_isr_failed: tstamp:37091879088 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1174.404794: rtos_queue_peek_from_isr_failed: tstamp:37096879111 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1174.572896: rtos_queue_peek_from_isr_failed: tstamp:37101879162 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1174.740920: rtos_queue_peek_from_isr_failed: tstamp:37106879119 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1174.908828: rtos_queue_peek_from_isr_failed: tstamp:37111879114 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1175.076877: rtos_queue_peek_from_isr_failed: tstamp:37116879123 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1175.244877: rtos_queue_peek_from_isr_failed: tstamp:37121879146 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1175.412889: rtos_queue_peek_from_isr_failed: tstamp:37126879187 queue:0x0b4b4500
     kworker/5:3-9506  [005] ....  1175.412898: rtos_queue_peek_from_isr_failed: tstamp:37127787331 queue:0x0b4b4500

Regarding the MIPI clock, it is not changing, it is in both cases the same (600MHz DDR, declared as 1.2GHz in DT. Anyway I’ve also tried to set 600MHz in the mclk_khz property, and yet the same result)

I’ve seen in some posts and in the Sensor Driver Programming Guide that some cameras may require the property “serdes_pix_clk_hz” to be defined (cameras that use Serializer/Deserializer). In my case, I have an FPGA outputting data in LVDS format, and a METICOM chip (MC20902) to gather HS and LP signals into SLVS format. Taking this into account, should I declare “serdes_pix_clk_hz”? If so, which value (same as pxl_clk_hz)? And should it be declared only in the DT?

MISC INFO

  • pxl_clk_hz is set to 480000000 (1200MHz*4lanes/10bpp).
  • I've increased arbitrarily the line_length - from 1920 to 2000 (it was set the same as active_w as the image is sent from the FPGA without any kind of blanking or dummy pixels). It keeps working as before. Shouldn't it stop working if the line_length is incorrectly set?
  • For your case serdes_pix_clk_hz should be used and it should be way bigger than the pxl_clk_hz.

    So should I arbitrarily set it? Is there any rule for it, or even a maximum value?
    At the moment, I’ve fixed the pxl_clk_hz based on data rate, n. lanes and bpp (480MHz), but I have no criteria to do so with the serdes_pxl_clk_hz.

    Moreover, regarding the bpp, I’m using in both DT and pxl_clk_hz calculations 10 bpp because I’m using RAW10 format. Anyway, this raw format packs one pixel in 2 bytes (with padding and repeated bits). Should I be using 10bpp or 16bpp?

    The serdes_pxl_clk_hz was looking for calculate the NVCSI/VI clock, if these clock too slow to handle the sensor output that could cause the frame drop issue. Theoretically it’s bigger than the pxl_clk_hz could be 1.5x or twice of the pxl_clk_hz.

    That’s really good to know, thanks!

    And regarding the bpp to be used both in DT and pxl_clk_hz calculations for RAW10 (or even other formats that are packed in more bits than the useful ones), should I be using 10bpp or 16bpp?

    Below property also for calculate the bandwidth. I think RAW10 and RAW16 are the same set the bpp as 2(vi_peak_byte_per_pixel)

    num_csi_lanes = <2>;
     		max_lane_speed = <1500000>;
     		min_bits_per_pixel = <10>;
     		vi_peak_byte_per_pixel = <2>;
     		vi_bw_margin_pct = <25>;
     		max_pixel_rate = <160000>;
     		isp_peak_byte_per_pixel = <5>;
     		isp_bw_margin_pct = <25>;
    

    I’ve set serdes_pxl_clk_hz 1.5x times pxl_clk_hz, and yet the same issue in receiving half the framerate.


    Regarding the bpp, I mean the following:
    RAW10, 10 bits per pixel, packed in 16 bits.
    RAW12, 12 bits per pixel, packed in 16 bits.

    When defining them in the DT, are they declared as 10 and 12 bpp respectively, or both 16?
    When doing the calculations for pxl_clk_hz, is 10/12 bpp considered, or 16?


    Regarding the flashing procedure, I do the following:

    1. Set custom camera as module in menuconfig
    2. Copy generated image in /boot/ and set it as default Image
    3. Flash kernel dtb (sudo ./flash.sh -k kernel-dtb jetson-tx2 mmcblk0p1)
    4. Generate module binary in jetson with the following Makefile
    ifneq ($(KERNELRELEASE),)
    obj-m	:= customcam.o
    else
    KDIR	:= /lib/modules/$(shell uname -r)/build
    PWD	:= $(shell pwd)
    all:
    	$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules
    install:
    	$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules_install
    %:
    	$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) $@
    endif
    
    1. Load the module and run the v4l2-ctl pipeline

    Is this procedure correct?

    All things should be fine for the driver due you already can capture the data from the bus well.
    And you have already boost the nvcsi/vi clocks to try that tell the property to calculate the bandwidth doesn’t matter with this case. Why not set the frame pause to 4ms as solution.

    Because I’m not able to do it when grabbing images from the sensor. I can only do it in a testing project where a ramp is being sent.

    Im lacking of ideas to solve this, but I’ll keep trying, because it is a really strange behaviour that I’m pretty sure can be solved.

    If you have any idea or suggestion that can help in solving this will be really appreciated!

    Hi!
    I’m still having the same issue, and I see no way how could I solve it. I’d like to know if there’s a way to see more helpful messages, as trace is giving no information at the moment. It just gives the information that it has received frame n and frame n+2, but nothing about frame n+1.

    So, when generating the image, in menuconfig there are many things that can be configured. Is any of those useful to get more info about the lost frames that trace log is not giving?

    P.D. There is another topic I created long time ago related to this problem also, with more tests… Just in case it is useful. Should I create a topic summarizing both of the topics and close both of them, or should I keep in this one? Thanks!

    Hi,
    I’m still trying to solve the half framerate issue, and with the trace log messages and dmesg, I get no useful information, as shown in comments above.So,
    Is there a way to get deeper and more detailed messages to know why the frames are not being captured? Maybe activating something in the menuconfig before generating an image, or any other way. Thanks in advance!

    There’s no other way to know the capture status except trace.

    So how could I solve this problem? Isn’t there any way to keep debugging and get a better insight to know what to change? Because actually I don’t know how to proceed…

    Same issue here.

    Have any solution, euskadi?