drop-frame-interval on rtsp-stream causes the stream to get stuck?

I was currently trying to work with variable streaming speed with deepstream. I was able to set drop-frame-interval per source and when I ran on the recorded video. The app ran as expected. One stream getting processed early and the other a bit late. However when running the same on the nano causes the display to be stuck. Any idea as to how to fix this issue?

The log I get on the terminal,

WARNING from element nvvideo-renderer: A lot of buffers are being dropped.
Warning: A lot of buffers are being dropped.

But when running the same on a Jetson nano this happens with a pre recorded video also. Any idea as to how to fix this?

Hi,
Please share your release version of DeepStream SDK( latest is 4.0.2 ). And config file so that we can try to reproduce the issue.

Deepstream Version: 4.0.2

Config file

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-file=../../../../samples/models/Primary_Detector/resnet10.caffemodel
proto-file=../../../../samples/models/Primary_Detector/resnet10.prototxt
labelfile-path=../../../../samples/models/Primary_Detector/labels.txt
int8-calib-file=../../../../samples/models/Primary_Detector/cal_trt.bin
batch-size=1
process-mode=1
model-color-format=0
network-mode=1
num-detected-classes=4
interval=0
gie-unique-id=1
output-blob-names=conv2d_bbox;conv2d_cov/Sigmoid

[class-attrs-all]
threshold=0.2
eps=0.2
group-threshold=1

The code change which I did on the deepstream-test3 app i.e setting drop-frame-interval on other streams,

static void
decodebin_child_added (GstChildProxy * child_proxy, GObject * object,
    gchar * name, gpointer user_data)
{
  g_print ("Decodebin child added: %s\n", name);
  if (g_strrstr (name, "decodebin") == name) {
    g_signal_connect (G_OBJECT (object), "child-added",
        G_CALLBACK (decodebin_child_added), user_data);
  }
  if (g_strstr_len (name, -1, "nvv4l2decoder") == name) {
    g_print ("Seting bufapi_version: %s\n", name);
    g_object_set (object, "bufapi-version", TRUE, NULL);
    gboolean src_res = g_str_has_suffix(name, "0");
    if (!src_res) {
      g_object_set(object, "drop-frame-interval", 20, NULL);
    }
  }
}

Here I am setting the drop-frame-interval param to streams other than 0, i.e if I run 3 streams, I get I’m dropping frames on 1 and 2 by checking the boolean src_res.

And I faced the same issue when I ran the test3 app with 3 sources, I see the video getting stuck in between but on two video sources it works fine. The same when run on three streams was causing the same issue
And on rtsp, immediately as the display opens up I can see the frame number getting displayed on the logs but it was always stuck.

Hi,
There is an issue in configuring drop-frame-interval property in nvv4l2decoder. Please get gst-v4l2 source code, apply the patch, and rebuild/replace libgstnvvideo4linux2.so.
Source code is in
https://developer.nvidia.com/embedded/dlc/r32-3-1_Release_v1.0/Sources/T210/public_sources.tbz2
Patch is in
https://devtalk.nvidia.com/default/topic/1065372/deepstream-sdk/jetson-nano-shows-100-cpu-usage-after-30-minutes-with-deepstream-app-demo/post/5412417/#5412417

Hi @Danell, I tried the above patch and it didn’t seem to fix the issue, here are the steps I followed,

cd Linux_for_Tegra/source/public/gst-v4l2/
 patch < ~/Downloads/0001-gstv4l2dec-Fix-high-CPU-usage-in-drop-frame.patch
 make
 sudo make install

I had removed the older .so file and when I ran the above it generated a new so file in the gst-plugins folder. And compiled and ran the test3 app with drop-frame-interval (same code changes as listed above) and I see the video stutter. Any idea as to how to proceed?

Hi,
Please ensure you overwrites

/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideo4linux2.so

Also if you dynamically add/remove source, please refer to this sample:

@Danell, I’m not adding dynamic sources on runtime. Yes I did replace the libgstnvvideo4linux2.so file. So far I’m testing on a dgpu machine not a nano. Hence I replaced the file in

/opt/nvidia/deepstream/deepstream-4.0/lib/gst-plugins/libgstnvvideo4linux2.so

But that didn’t change anything. Could you please give it a try on your end and see if it works?

Apart from this, regarding the drop-frame-interval, I also did give another thing a try, I modified few lines in deepstream_test3_app.c, Here’s the modified segment,

375c375,376
<   transform = gst_element_factory_make ("nvegltransform", "nvegl-transform");
---
>   // transform = gst_element_factory_make ("nvegltransform", "nvegl-transform");
>   transform = gst_element_factory_make ("queue", "queue");

377c378,379
<   sink = gst_element_factory_make ("nveglglessink", "nvvideo-renderer");
---
>   // sink = gst_element_factory_make ("nveglglessink", "nvvideo-renderer");
>   sink = gst_element_factory_make ("nvoverlaysink", "nvvideo-renderer");
406a409,410
> 
>   g_object_set(G_OBJECT(sink), "sync", FALSE, NULL);

I made the above change and I did not see the video rendering stutter on either nano or the dgpu device. But with this I did come across another thing on the nano, usually on a nano a single stream processes at 24-30(or higher) fps. But with 4 streams, out of which 3 streams were set to drop 20 frames on them and the other at full fps, when I ran it I saw a lag which was getting accumulated. But mathematically in terms of processing, it should ideally be processing 24(stream-0) + 1 or 2 fps on other sources accumulating to 30fps. But the processing wasn’t that much causing a lag. Any idea on this?

So far I’ve tried two different solutions for the drop-frame-interval. It would really help if either one works.

Hi,
We modify test3 app and run on Jetson Nano. The result looks fine. The modifications are
1 Apply this to config file

2 Modify test3 app:
https://devtalk.nvidia.com/default/topic/1058597/deepstream-sdk/-nano-deepstream-test3-app-not-working-as-expected-for-multiple-video-source/post/5368694/#5368694

static void
decodebin_child_added (GstChildProxy * child_proxy, GObject * object,
    gchar * name, gpointer user_data)
{
  g_print ("Decodebin child added: %s\n", name);
  if (g_strrstr (name, "decodebin") == name) {
    g_signal_connect (G_OBJECT (object), "child-added",
        G_CALLBACK (decodebin_child_added), user_data);
  }
  if (g_strstr_len (name, -1, "nvv4l2decoder") == name) {
    g_print ("Seting bufapi_version: %s\n", name);
    g_object_set (object, "bufapi-version", TRUE, NULL);
    [b]gboolean src_res = g_str_has_suffix(name, "0");
    if (!src_res) {
      g_object_set(object, "drop-frame-interval", 20, NULL);[/b]
    }
  }
}

3 Run

$ ./deepstream-test3-app file:///home/nvidia/deepstream_sdk_v4.0.2_jetson/samples/streams/sample_1080p_h264.mp4 file:///home/nvidia/deepstream_sdk_v4.0.2_jetson/samples/streams/sample_1080p_h264.mp4 file:///home/nvidia/deepstream_sdk_v4.0.2_jetson/samples/streams/sample_1080p_h265.mp4 file:///home/nvidia/deepstream_sdk_v4.0.2_jetson/samples/streams/sample_720p.mp4

Hi @Danell, when the same modifications were done on the nano, I was expecting the the source0 to process more number of frames and run smoothly and the other 3 streams to process as though the frames were missing. While the latter scenario of the frames missing (due to drop-frame-interval is true) but the source 0 is still running slower. Or is it different on your end?

Hi,
It looks to be expected since nvstreammux collects frames from each source and make into surface list. If the sources are not in identical frame rate, you will see some sources end and the other sources are running still.

@DaneLLL, I had a doubt regarding this. So According to this fix, the first source shouldn’t the first source run smoothly and process a lot of frames and there there shouldn’t be any lag which should build up right??