gstreamer input to opencv process and send back to gstreamer show

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hi all ,

I have successfully open my own video device with gstreamer then process by opencv .https://devtalk.nvidia.com/default/topic/1031458/opencv-videocapture-with-gstreamer-v4l2/#5247684

But for opencv imshow is very slow for me . So I refer https://github.com/wzyy2/gstreamer-opencv
http://blog.iotwrt.com/media/2017/08/23/opencv-gstreamer/

This project is the final result that I want to realize .

for this project his main pipleline

"filesrc location=/usr/local/test.mp4 ! qtdemux ! h264parse ! mppvideodec \
    ! rgaconvert output-io-mode=dmabuf capture-io-mode=dmabuf ! \
    video/x-raw,format=BGR,width=(int)1920,height=(int)1080 ! \
    appsink caps=video/x-raw,format=BGR name=sink"

    "appsrc caps=video/x-raw,format=(string)BGR,width=(int)1920,height=(int)1080,framerate=(fraction)30/1 \
    block=true name=src ! rkximagesink sync=false";

But in nvidia TX2 have no rgaconvert which send image data to DMA . then realize zero-copy .
So My question is How can I realize the same problem with nvidia DMA data copy with gst-launch-1.0 cmd?

2

Please refer to [url]https://devtalk.nvidia.com/default/topic/1024245/jetson-tx2/opencv-3-3-and-integrated-camera-problems-/post/5210735/#5210735[/url]

It shall improve performance by sending I420 or NV12 to appsink.

3

Hi DaneLLL,
you mean opencv 3.3.0 can support I420/NV12 but I have problem when input I420 to appsink.
Here is my opencv version (ps.I changed to opencv 3.4.0 because I can not open the video )

General configuration for OpenCV 3.4.0-rc =====================================
  Version control:               unknown

  Platform:
    Timestamp:                   2018-03-28T11:50:09Z
    Host:                        Linux 4.4.38-tegra aarch64
    CMake:                       3.5.1
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               Release

  CPU/HW features:
    Baseline:                    NEON FP16
      required:                  NEON
      disabled:                  VFPV3

  C/C++:
    Built as dynamic libs?:      YES
    C++ Compiler:                /usr/bin/c++  (ver 5.4.0)
    C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):
    Linker flags (Debug):
    ccache:                      NO
    Precompiled headers:         NO
    Extra dependencies:          dl m pthread rt /usr/lib/aarch64-linux-gnu/libtbb.so cudart nppc nppial nppicc nppicom nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cufft -L/usr/local/cuda-9.0/lib64
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 calib3d core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev dnn features2d flann highgui imgcodecs imgproc ml objdetect photo python2 python_bindings_generator shape stitching superres ts video videoio videostab
    Disabled:                    js world
    Disabled by dependency:      -
    Unavailable:                 java python3 viz
    Applications:                tests perf_tests examples apps
    Documentation:               NO
    Non-free algorithms:         NO

  GUI: 
    GTK+:                        YES (ver 2.24.30)
      GThread :                  YES (ver 2.48.1)
      GtkGlExt:                  NO

  Media I/O: 
    ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.8)
    JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver )
    WEBP:                        build (ver encoder: 0x020e)
    PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.2.54)
    TIFF:                        /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.0.6)
    JPEG 2000:                   /usr/lib/aarch64-linux-gnu/libjasper.so (ver 1.900.1)

  Video I/O:
    FFMPEG:                      YES
      avcodec:                   YES (ver 56.60.100)
      avformat:                  YES (ver 56.40.101)
      avutil:                    YES (ver 54.31.100)
      swscale:                   YES (ver 3.1.101)
      avresample:                NO
    GStreamer:                   
      base:                      YES (ver 1.8.3)
      video:                     YES (ver 1.8.3)
      app:                       YES (ver 1.8.3)
      riff:                      YES (ver 1.8.3)
      pbutils:                   YES (ver 1.8.3)
    libv4l/libv4l2:              NO
    v4l/v4l2:                    linux/videodev2.h
    gPhoto2:                     NO

  Parallel framework:            TBB (ver 4.4 interface 9002)

  Trace:                         YES (built-in)

  Other third-party libraries:
    Lapack:                      NO
    Eigen:                       YES (ver 3.2.92)
    Custom HAL:                  YES (carotene (ver 0.0.1))

  NVIDIA CUDA:                   YES (ver 9.0, CUFFT CUBLAS)
    NVIDIA GPU arch:             62
    NVIDIA PTX archs:

  Python 2:
    Interpreter:                 /usr/bin/python2.7 (ver 2.7.12)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.12)
    numpy:                       /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.11.0)
    packages path:               lib/python2.7/dist-packages

  Python (for build):            /usr/bin/python2.7

  Java:
    ant:                         NO
    JNI:                         NO
    Java wrappers:               NO
    Java tests:                  NO

  Matlab:                        NO

  Install to:                    /usr
-----------------------------------------------------------------

But I can opencv video successfully with videoconvert to BGR format both in 3.3.0 and 3.4.0 with the following code ,but for videoconvert is so slow for me that I can’t tolerant.

VideoCapture capture("v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080,format=(string)UYVY ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080,format=(string)I420 ! nvvidconv! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink");

But both error with

VideoCapture capture("v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080,format=UYVY ! nvvidconv ! video/x-raw(memory:NVMM),width=1920,height=1080,format=I420 ! nvvidconv! video/x-raw, format=I420 ! appsink");

Here is the error information

(run_tld:19118): GStreamer-CRITICAL **: gst_mini_object_copy: assertion 'mini_object != NULL' failed

(run_tld:19118): GStreamer-CRITICAL **: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(run_tld:19118): GStreamer-CRITICAL **: gst_structure_copy: assertion 'structure != NULL' failed

(run_tld:19118): GStreamer-CRITICAL **: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(run_tld:19118): GStreamer-CRITICAL **: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(run_tld:19118): GStreamer-CRITICAL **: gst_structure_copy: assertion 'structure != NULL' failed

(run_tld:19118): GStreamer-CRITICAL **: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(run_tld:19118): GStreamer-CRITICAL **: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(run_tld:19118): GStreamer-CRITICAL **: gst_mini_object_ref: assertion 'mini_object != NULL' failed
OpenCV Error: Unspecified error (GStreamer: unable to start pipeline
) in icvStartPipeline, file /home/nvidia/opencv/opencv-3.4.0-rc/modules/videoio/src/cap_gstreamer.cpp, line 450
terminate called after throwing an instance of 'cv::Exception'
  what():  /home/nvidia/opencv/opencv-3.4.0-rc/modules/videoio/src/cap_gstreamer.cpp:450: error: (-2) GStreamer: unable to start pipeline
 in function icvStartPipeline

Aborted (core dumped)

Any suggestions would be appreciate!

4

Hi ChrisRiz,
We have verified the sample at [url]https://devtalk.nvidia.com/default/topic/1024245/jetson-tx2/opencv-3-3-and-integrated-camera-problems-/post/5210735/#5210735[/url]

Are you able to run this successfully?