Crop rtsp streams before inferencing

So, I have written an app on top of deepstream reference app to infer on live rtsp streams. I am not concerned with the final dump on sink pads. I have modified the code of gst_dsexample plugin to extract the information I need and process on them using something like this -

for (NvDsMetaList * l_frame = batch_meta->frame_meta_list; l_frame != NULL;
		  l_frame = l_frame->next) {
		NvDsFrameMeta *frame_meta = l_frame->data;
		guint stream_id = frame_meta->pad_index;
		guint source_id = frame_meta->source_id;

            { POST PROCESSING CODE }

I am not concerned with rest of the pipeline after this.

Now I wish to preprocess on the input frames specifically I wish to crop them and then feed into the inferencing pipeline. I wish to then get the results on the cropped frames in gst_dsexample and then apply my own post processing logic on this.

What would be the best way to do this ?

Can you refer to this solution: https://devtalk.nvidia.com/default/topic/1062748/deepstream-sdk/nvvideoconvert-crashes-on-rtsp-input-src-crop-x-y-w-h-pipeline/3

Hi,
I am having difficulty understanding the discussion in the above mentioned link. From what I can understand that pipeline is quite different from that which is employed by the deepstream reference app. Also instead of using multiuri, I use RTSP as input. Can you clarify further ?

Hi,
You may link one more nvvideoconvert before nvinfer and set src-crop according to your usecase. There are examples in #2

HI, DaneLL. I am able to successfully crop the images to a certain size and then feed it to the inference pipeline. Now I wish to do is something like this -
1.) Take input from live rtsp streams
2.) divide the input to multiple crops and then resize the crops. i.e a single input image will get cropped to multiple parts so and then each crop will be fed for inference.

I have worked with gstdsexample plugin and added my post processing logic.

I wish to know how to go on about doing this and what are the things should I keep into consideration.

Should I play with the source code of nvinfer plugin ?

Also, is the source code of streammux available ?

Why do you need streammux source code ?
As of now there is no plan to open source existing streammux in ds 5.0

IN https://devtalk.nvidia.com/default/topic/1061492/deepstream-sdk/deepstream-sdk-faq/
, it’s mentioned that FrameMeta and BatchMeta is filled here. So, I thought one way to solve the problem mentioned in #5 would be to modify the code here. If there is any other way to do it, I would like to know about it.

Hi,
The architecture of deepstream-app is shown in:
https://docs.nvidia.com/metropolis/deepstream/dev-guide/index.html#page/DeepStream%2520Development%2520Guide%2Fdeepstream_app_architecture.html%23

The sample config file is source4_1080p_dec_infer-resnet_tracker_sgie_tiled_display_int8.txt
It is for desktop GPUs and Xavier since the loading is heavy. However, if we disable primary detection and use secondary classifiers, it probably can work. Does your mode do classification?

HI, I am using the pipeline for primary detection only. I am not concerned with tiled display as given in the config file but with tiled detections i.e I want to chop up my input image into several crops (primarily 2) and then pass them through nvinfer plugin as individual images.

On a secondary note, I would also like to have control over the internal resizing of images which are to be put in nvinfer plugin. For example - I wish to resize them by keeping the aspect ratio preserved and padding the image.

https://docs.nvidia.com/metropolis/deepstream/plugin-manual/index.html#page/DeepStream_Plugin_Manual%2Fdeepstream_plugin_details.02.03.html → enable-padding : Maintains aspect ratio by padding with black borders when scaling input frames.

Thanks ChrisDing for the input. What about tiled crop detections ? I wish to crop an inputimage into separate tiles, Feed them into the pipeline for inference.

Hi,
If your RTSP source can run with multiple clients, one solution is to connect the same source multiple times and use nvvideoconvert to crop the frames. You may run

$ ./deepstream-test3-app rtsp://_STREAM_ADDR_ rtsp://_STREAM_ADDR_

And customize create_source_bin() to run uridecodebin ! nvvideoconvert src-crop=X:Y:W:H

Default deepstream-test3 is not for Jetson Nano. You may refer to this post to customize it:
https://devtalk.nvidia.com/default/topic/1058597/deepstream-sdk/-nano-deepstream-test3-app-not-working-as-expected-for-multiple-video-source/post/5368352/#5368352

Hi Neophyte1, I need to do the exact same thing with an Xavier AGX. I also need to be able to stream metadata over Kafka. Were you able to solve this issue? If so, could you kindly guide me?

Thank you in advance,
Ramya.