My custom app based on deepstream-test3 gets lots of NvMapMemCacheMaint:1075334668 failed [14] error...

I have a deepstream app based on deepstream-test3. I have added logic so that on an object detection new elements are dynamically added to the pipeline to record the video stream to a file.

The pipeline is like this:

rtsp sources ! decodebin ! streammux ! pgie ! tracker ! streamdemux ! queue ! nvvideoconvert ! nvdsosd ! TEE ! queue ! fakesink

On a detection event I add another tee pad to TEE above with the following:

new teepad ! queue ! convert ! caps_filter ! encoder ! parser ! muxer ! filesink

After a 15 second timer this dynamic file saving pipeline is removed.

It seems to be working well but I continually see these messages: NvMapMemCacheMaint:1075334668 failed [14] ?

Will attach log output from GST_DEBUG=3. I can see a lot of v4l2 buffer errors?!?
debug.log (2.87 MB)

An update on this. I’ve noticed that if I increase the queue size from the default 200 buffers to 500 (in the two queues following the TEE) it reduces the number of these NvMapMemCacheMaint errrors that I see.

What element raises these and why? Adding to the sizes of your queues - is this the typical way to solve the issue??

Anyone at nvidia able to describe what this error is all about; what causes it and how to resolve. I’m unable to continue with jetson and deepstream with this issue. It’s continually raised and also leads to occasion segfaults.

My app is based on deepstream-test3. I have extended it to dynamically add some file recording elements when a person is detected in the probe.
There is also a large memory leak that I’m thinking is associated with the encoder as memory usage goes up a couple hundred MB each time I add then remove he encoder, muxer and filesink.

I also wish to know what exactly does this error mean

Before “NvMapMemCacheMaint” failed, there is error “v4l2 provided buffer that is too big for the memory it was writing into. v4l2 claims 64 bytes used but memory is only 0B. This is probably a driver bug.” I think the error is caused by this.

Hi Chris,
Yes I could see that v4l2 issues as well when I run the app in Valgrind.

What should I do - I imagine its coming from the nvv4l2h265enc element.

I am dynamically adding this to the pipeline when there is a detection so that I can record the stream to an mp4 file and then after 15 seconds I unlink it. Because I’m doing this dynamic changes quite a bit and re-creating the encoder over and over its probably exacerbating the underlying bug.

Any suggestions? Maybe I can try to use the standard gstreamer encoder?!?

“standard gstreamer encoder” does not receive data from gpu buffer, it needs nvvideoconvert to transfer gpu buffer to cpu buffer.

Yes - understand that. Might have no choice than to do that which will likely be a significant hit to performance.

You have any other suggestions? Does nvidia raise a bug with v4l2 or should I do something here?

@ChrisDing or anyone else - ideas on how to get around this v4l2 memory bug?
Nvidia has not confirmed the component that issues this message yet - my testing is point toward the encoder though - any updates?

I’d like advice from NVidia that the encoder can’t be used and we’d need to drop back to the omx or software-based gstreamer encoders?

Also - have tested with the OMX encoder (both h264 and h265) omxh265enc and I get the same issues. The logs shows that its using NVENC for the encoding anyway so you get exactly the same issues as using nvv4l2h265enc.

Can anyone at nvidia help with this. It seems that when you dynamically add a deepstream encoder to your pipeline then unlink and eos it I get these memory errors if I add the encoder again.
I can provide working sample source code for you to test with.

Have upgraded to latest jetpack and deepstream 4.02 but there is no change.

Hi,
Please share a sample that we can reproduce the issue. Probably certain corner cases are not handled well enough. We can reproduce it first and do further check.

For information, please also share which jetson platform you are using.

Can I send sample code offline via email?

I have this too with my latest app if you want another sample.

0:00:02.633547680  5722   0x7f58060680 WARN          v4l2bufferpool gstv4l2bufferpool.c:1482:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder0:pool:sink> v4l2 provided buffer that is too big for the memory it was writing into.  v4l2 claims 64 bytes used but memory is only 0B.  This is probably a driver bug.
0:00:02.698939820  5722   0x7f58060680 WARN          v4l2bufferpool gstv4l2bufferpool.c:1482:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder0:pool:sink> v4l2 provided buffer that is too big for the memory it was writing into.  v4l2 claims 64 bytes used but memory is only 0B.  This is probably a driver bug.
0:00:02.765807669  5722   0x7f58060680 WARN          v4l2bufferpool gstv4l2bufferpool.c:1482:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder0:pool:sink> v4l2 provided buffer that is too big for the memory it was writing into.  v4l2 claims 64 bytes used but memory is only 0B.  This is probably a driver bug.
0:00:02.766389165  5722   0x7f58060680 WARN          v4l2bufferpool gstv4l2bufferpool.c:1482:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder0:pool:sink> v4l2 provided buffer that is too big for the memory it was writing into.  v4l2 claims 64 bytes used but memory is only 0B.  This is probably a driver bug.
0:00:02.766985253  5722   0x7f58060680 WARN          v4l2bufferpool gstv4l2bufferpool.c:1482:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder0:pool:sink> v4l2 provided buffer that is too big for the memory it was writing into.  v4l2 claims 64 bytes used but memory is only 0B.  This is probably a driver bug.
0:00:02.767885482  5722   0x7f58060680 WARN          v4l2bufferpool gstv4l2bufferpool.c:1482:gst_v4l2_buffer_pool_dqbuf:<nvv4l2decoder0:pool:sink> v4l2 provided buffer that is too big for the memory it was writing into.  v4l2 claims 64 bytes used but memory is only 0B.  This is probably a driver bug.

Using Jetson Xavier. The log is full of this at --gst-debug-level=4

.dot files are dumped to ~/.nvalhalla/ as well on the dev branch.

Hi,

Have sent you ftp site through message. Please upload a sample so that we can reproduce

NvMapMemCacheMaint:1075334668 failed [14]