syncSensor crashed on full res(3840x2160)
Hi Nvidia, I found problem. syncSensor from samples crashed when resolution is 3840x2160 after 3-5 seconds. RAM is over too. Cameras are 2x LI imx274. argus_daemon errorlog: Growing thread pool to 1 threads NvRmChannelSubmit: NvError_IoctlFailed with error code 22 NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 18, SyncPointValue = 0) SCF: Error IoctlFailed: (propagating from src/services/capture/CaptureServiceDeviceIsp.cpp, function startCapture(), line 461) SCF: Error IoctlFailed: (propagating from src/services/capture/CaptureServiceCore.cpp, function doMEMtoISPCapture(), line 538) SCF: Error IoctlFailed: (propagating from src/services/capture/CaptureServiceCore.cpp, function issueCapture(), line 346) SCF: Error IoctlFailed: (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1087) SCF: Error IoctlFailed: (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 913) SCF: Error IoctlFailed: (propagating from src/common/Utils.cpp, function workerThread(), line 183) SCF: Error IoctlFailed: Worker function failed (in src/common/Utils.cpp, function workerThread(), line 199) SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59) Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc On 1920x1080 syncSensor works fine. 2x argus_camera works fine at full res in the same time. Best regards, Viktor.
Answer Accepted by Original Poster
Hi Nvidia,

I found problem.
syncSensor from samples crashed when resolution is 3840x2160 after 3-5 seconds. RAM is over too.
Cameras are 2x LI imx274.

argus_daemon errorlog:

Growing thread pool to 1 threads
NvRmChannelSubmit: NvError_IoctlFailed with error code 22
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 18, SyncPointValue = 0)
SCF: Error IoctlFailed: (propagating from src/services/capture/CaptureServiceDeviceIsp.cpp, function startCapture(), line 461)
SCF: Error IoctlFailed: (propagating from src/services/capture/CaptureServiceCore.cpp, function doMEMtoISPCapture(), line 538)
SCF: Error IoctlFailed: (propagating from src/services/capture/CaptureServiceCore.cpp, function issueCapture(), line 346)
SCF: Error IoctlFailed: (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 1087)
SCF: Error IoctlFailed: (propagating from src/services/capture/CaptureServiceDevice.cpp, function issueCaptures(), line 913)
SCF: Error IoctlFailed: (propagating from src/common/Utils.cpp, function workerThread(), line 183)
SCF: Error IoctlFailed: Worker function failed (in src/common/Utils.cpp, function workerThread(), line 199)
SCF: Error Timeout: (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc


On 1920x1080 syncSensor works fine.
2x argus_camera works fine at full res in the same time.

Best regards, Viktor.

#1
Posted 01/08/2018 10:34 PM   
@vsw Did you check any time out message form the kernel message?
@vsw
Did you check any time out message form the kernel message?

#2
Posted 01/09/2018 02:58 AM   
Here is dmesg last messages: [ 100.797696] imx274 7-001a: imx274_set_frame_rate: val: 50331648 [ 100.797706] imx274 7-001a: imx274_set_frame_length length = 23076 [ 100.798357] imx274 7-001a: imx274_set_frame_length: val: 23076 [ 100.800199] imx274 7-001a: imx274_set_gain - val = 92274688 [ 100.800207] imx274 7-001a: input gain value: 5632 [ 100.800213] imx274 7-001a: imx274_set_gain: gain 1600 val: 07a3 [ 100.801509] imx274 7-001a: imx274_set_exposure: val: 349525 [ 100.801516] imx274 7-001a: imx274_set_coarse_time [ 100.801521] imx274 7-001a: imx274_set_coarse_time: input val: 2080 [ 100.801526] imx274 7-001a: imx274_set_coarse_time: set val: 20996 [ 100.816559] imx274 8-001a: imx274_set_frame_rate: val: 50331648 [ 100.816569] imx274 8-001a: imx274_set_frame_length length = 23076 [ 100.817257] imx274 8-001a: imx274_set_frame_length: val: 23076 [ 100.819126] imx274 8-001a: imx274_set_gain - val = 92274688 [ 100.819133] imx274 8-001a: input gain value: 5632 [ 100.819140] imx274 8-001a: imx274_set_gain: gain 1600 val: 07a3 [ 100.820375] imx274 8-001a: imx274_set_exposure: val: 349525 [ 100.820381] imx274 8-001a: imx274_set_coarse_time [ 100.820388] imx274 8-001a: imx274_set_coarse_time: input val: 2080 [ 100.820394] imx274 8-001a: imx274_set_coarse_time: set val: 20996 [ 104.099379] isp isp.0: nvhost_ioctl_channel_submit: failed with err -12 [ 104.107911] isp isp.0: nvhost_ioctl_channel_submit: failed with err -22
Here is dmesg last messages:

[ 100.797696] imx274 7-001a: imx274_set_frame_rate: val: 50331648
[ 100.797706] imx274 7-001a: imx274_set_frame_length length = 23076
[ 100.798357] imx274 7-001a: imx274_set_frame_length: val: 23076
[ 100.800199] imx274 7-001a: imx274_set_gain - val = 92274688
[ 100.800207] imx274 7-001a: input gain value: 5632
[ 100.800213] imx274 7-001a: imx274_set_gain: gain 1600 val: 07a3
[ 100.801509] imx274 7-001a: imx274_set_exposure: val: 349525
[ 100.801516] imx274 7-001a: imx274_set_coarse_time
[ 100.801521] imx274 7-001a: imx274_set_coarse_time: input val: 2080
[ 100.801526] imx274 7-001a: imx274_set_coarse_time: set val: 20996
[ 100.816559] imx274 8-001a: imx274_set_frame_rate: val: 50331648
[ 100.816569] imx274 8-001a: imx274_set_frame_length length = 23076
[ 100.817257] imx274 8-001a: imx274_set_frame_length: val: 23076
[ 100.819126] imx274 8-001a: imx274_set_gain - val = 92274688
[ 100.819133] imx274 8-001a: input gain value: 5632
[ 100.819140] imx274 8-001a: imx274_set_gain: gain 1600 val: 07a3
[ 100.820375] imx274 8-001a: imx274_set_exposure: val: 349525
[ 100.820381] imx274 8-001a: imx274_set_coarse_time
[ 100.820388] imx274 8-001a: imx274_set_coarse_time: input val: 2080
[ 100.820394] imx274 8-001a: imx274_set_coarse_time: set val: 20996
[ 104.099379] isp isp.0: nvhost_ioctl_channel_submit: failed with err -12

[ 104.107911] isp isp.0: nvhost_ioctl_channel_submit: failed with err -22

#3
Posted 01/09/2018 03:27 AM   
Looks like the same error: https://devtalk.nvidia.com/default/topic/1018971/memory-leaks-reading-images-from-native-buffer/

#4
Posted 01/09/2018 04:02 AM   
Did you modify the sample code? I can't repo the problem with two sensors with resolution 5632x3168
Did you modify the sample code? I can't repo the problem with two sensors with resolution 5632x3168

#5
Posted 01/09/2018 04:36 AM   
Yes. I changed capture time to 15s and stream size to 4k.
Yes. I changed capture time to 15s and stream size to 4k.

#6
Posted 01/09/2018 04:41 AM   
From argus_camera MultiSession 3x cameras on 3840x2160@60 are working, but from syncSensor(2x cameras) are not. But i think capture to gl buffer from sensors is not the same as in syncSensor EGLfarme capturing: ScopedCudaEGLStreamFrameAcquire left(m_cuStreamLeft); ScopedCudaEGLStreamFrameAcquire right(m_cuStreamRight); I wrote to LeopardImaging support too. Summary: syncSensor with 1920x1080 and LI imx274 - OK syncSensor with 3840x2160(any fps) and LI imx274 - Crashed after few seconds due to a memory leaks up to 100%. It is Nvidia libargus bug or camera/driver provider bug?
From argus_camera MultiSession 3x cameras on 3840x2160@60 are working, but from syncSensor(2x cameras) are not.
But i think capture to gl buffer from sensors is not the same as in syncSensor EGLfarme capturing:
ScopedCudaEGLStreamFrameAcquire left(m_cuStreamLeft);
ScopedCudaEGLStreamFrameAcquire right(m_cuStreamRight);

I wrote to LeopardImaging support too.
Summary:
syncSensor with 1920x1080 and LI imx274 - OK
syncSensor with 3840x2160(any fps) and LI imx274 - Crashed after few seconds due to a memory leaks up to 100%.
It is Nvidia libargus bug or camera/driver provider bug?

#7
Posted 01/13/2018 07:50 PM   
@vsw Could you help to check if any memory leak with 1080p. Make the APP run in a while loop to check if the memory is increased by the time.
@vsw
Could you help to check if any memory leak with 1080p. Make the APP run in a while loop to check if the memory is increased by the time.

#8
Posted 01/17/2018 03:05 AM   
Yes, of course. But it is already done. 2x constructor ScopedCudaEGLStreamFrameAcquire left(m_cuStreamLeft); execute in while loop. After 50-100 calls the Tx1 RAM ends. If i remove frame accuire constructors then all is ok. And it is only at 4k. I am right? I wrote to LI support too.
Yes, of course. But it is already done.
2x constructor ScopedCudaEGLStreamFrameAcquire left(m_cuStreamLeft); execute in while loop. After 50-100 calls the Tx1 RAM ends.
If i remove frame accuire constructors then all is ok. And it is only at 4k.
I am right?
I wrote to LI support too.

#9
Posted 01/17/2018 06:17 AM   
@vsw I am not understand "If i remove frame accuire constructors then all is ok. And it is only at 4k." Could you have more explanation and post what you modify here.
@vsw
I am not understand "If i remove frame accuire constructors then all is ok. And it is only at 4k."
Could you have more explanation and post what you modify here.

#10
Posted 01/18/2018 08:26 AM   
I mean that if the code with frame acquiring is commented then syncSensor works; while (true) { [b]continue;[/b] ScopedCudaEGLStreamFrameAcquire left(m_cuStreamLeft); ScopedCudaEGLStreamFrameAcquire right(m_cuStreamRight); if (!left.hasValidFrame() || !right.hasValidFrame()) break; // Calculate histograms. float time = 0.0f; if (left.generateHistogram(histogramLeft, &time) && right.generateHistogram(histogramRight, &time)) { // Calculate KL distance. float distance = 0.0f; Size2D<uint32_t> size = right.getSize(); float dTime = computeKLDistance(histogramRight, histogramLeft, HISTOGRAM_BINS, size.width() * size.height(), &distance); CONSUMER_PRINT("KL distance of %6.3f with %5.2f ms computing histograms and " "%5.2f ms spent computing distance\n", distance, time, dTime); } }
I mean that if the code with frame acquiring is commented then syncSensor works;
while (true)
{
continue;
ScopedCudaEGLStreamFrameAcquire left(m_cuStreamLeft);
ScopedCudaEGLStreamFrameAcquire right(m_cuStreamRight);

if (!left.hasValidFrame() || !right.hasValidFrame())
break;

// Calculate histograms.
float time = 0.0f;
if (left.generateHistogram(histogramLeft, &time) &&
right.generateHistogram(histogramRight, &time))
{
// Calculate KL distance.
float distance = 0.0f;
Size2D<uint32_t> size = right.getSize();
float dTime = computeKLDistance(histogramRight,
histogramLeft,
HISTOGRAM_BINS,
size.width() * size.height(),
&distance);
CONSUMER_PRINT("KL distance of %6.3f with %5.2f ms computing histograms and "
"%5.2f ms spent computing distance\n",
distance, time, dTime);
}
}

#11
Posted 01/18/2018 09:40 AM   
Scroll To Top

Add Reply