Processing video with EGLImages?

micjan · May 2, 2017, 10:51am

Hi,

I came across this post https://devtalk.nvidia.com/default/topic/983587/jetson-tx1/gst-omx-plugin-build-error/post/5136394/ trying to build the “testegl” example under gstomx1_src/gst-omx1/examples/egl.

At first, I had the same problem as zeitgeist, but then I downloaded the gstomx1_src_quick_fix_r24_2_1 zip from post #8, and followed the steps from the README in it. With the modifications in place, ./configure worked for me.

But, when I try to make the library source, I’m getting:

ubuntu@tegra-ubuntu:~/gstomx1_src/gst-omx1$ make
make  all-recursive
make[1]: Entering directory `/home/ubuntu/gstomx1_src/gst-omx1'
Making all in common
make[2]: Entering directory `/home/ubuntu/gstomx1_src/gst-omx1/common'
Making all in m4
make[3]: Entering directory `/home/ubuntu/gstomx1_src/gst-omx1/common/m4'
make[3]: Nothing to be done for `all'.
make[3]: Leaving directory `/home/ubuntu/gstomx1_src/gst-omx1/common/m4'
make[3]: Entering directory `/home/ubuntu/gstomx1_src/gst-omx1/common'
make[3]: Nothing to be done for `all-am'.
make[3]: Leaving directory `/home/ubuntu/gstomx1_src/gst-omx1/common'
make[2]: Leaving directory `/home/ubuntu/gstomx1_src/gst-omx1/common'
Making all in omx
make[2]: Entering directory `/home/ubuntu/gstomx1_src/gst-omx1/omx'
  CC       libgstomx_la-gstomxvideodec.lo
gstomxvideodec.c: In function 'gst_omx_video_dec_set_property':
gstomxvideodec.c:831:11: error: 'GstOMXVideoDec' has no member named 'output_buffers'
       self->output_buffers = g_value_get_uint (value);
           ^
gstomxvideodec.c: In function 'gst_omx_video_dec_get_property':
gstomxvideodec.c:864:36: error: 'GstOMXVideoDec' has no member named 'output_buffers'
       g_value_set_uint (value, self->output_buffers);
                                    ^
gstomxvideodec.c: In function 'gst_omx_video_dec_init':
gstomxvideodec.c:977:7: error: 'GstOMXVideoDec' has no member named 'output_buffers'
   self->output_buffers = DEFAULT_OUTPUT_BUFFER;
       ^
gstomxvideodec.c:979:7: error: 'GstOMXVideoDec' has no member named 'enable_frame_type_reporting'
   self->enable_frame_type_reporting = FALSE;
       ^
gstomxvideodec.c:980:7: error: 'GstOMXVideoDec' has no member named 'cpu_dec_buf'
   self->cpu_dec_buf = FALSE;
       ^
gstomxvideodec.c: In function 'gst_omx_video_dec_open':
gstomxvideodec.c:1049:41: error: 'GstOMXVideoDec' has no member named 'enable_frame_type_reporting'
     if (self->enable_error_check || self->enable_frame_type_reporting) {
                                         ^
gstomxvideodec.c:1053:13: error: 'GstOMXVideoDec' has no member named 'enable_frame_type_reporting'
         self->enable_frame_type_reporting = FALSE;
             ^
gstomxvideodec.c: In function 'gst_omx_video_dec_loop':
gstomxvideodec.c:2606:9: error: 'NVX_VIDEO_DEC_OUTPUT_PARAMS_FLAG_FRAME_DEC_ERR_REPORT' undeclared (first use in this function)
         NVX_VIDEO_DEC_OUTPUT_PARAMS_FLAG_FRAME_DEC_ERR_REPORT)
         ^
gstomxvideodec.c:2606:9: note: each undeclared identifier is reported only once for each function it appears in
gstomxvideodec.c:2613:40: error: 'NVX_VIDEO_DEC_OUTPUT_EXTRA_DATA' has no member named 'sDecErrReport'
         if (pNvxVideoDecOutputExtraData->sDecErrReport.nDecodeError &
                                        ^
gstomxvideodec.c:2614:13: error: 'VIDEO_DEC_DECODED_ERROR_FATAL' undeclared (first use in this function)
             VIDEO_DEC_DECODED_ERROR_FATAL) {
             ^
gstomxvideodec.c:2619:40: error: 'NVX_VIDEO_DEC_OUTPUT_EXTRA_DATA' has no member named 'sDecErrReport'
         if (pNvxVideoDecOutputExtraData->sDecErrReport.nDecodeError &
                                        ^
gstomxvideodec.c:2620:13: error: 'VIDEO_DEC_DECODED_ERROR_MB_SYNTAX' undeclared (first use in this function)
             VIDEO_DEC_DECODED_ERROR_MB_SYNTAX) {
             ^
gstomxvideodec.c:2626:40: error: 'NVX_VIDEO_DEC_OUTPUT_EXTRA_DATA' has no member named 'sDecErrReport'
         if (pNvxVideoDecOutputExtraData->sDecErrReport.nDecodeError &
                                        ^
gstomxvideodec.c:2627:9: error: 'VIDEO_DEC_DECODED_ERROR_MISSING_SLICE' undeclared (first use in this function)
         VIDEO_DEC_DECODED_ERROR_MISSING_SLICE) {
         ^
gstomxvideodec.c:2632:40: error: 'NVX_VIDEO_DEC_OUTPUT_EXTRA_DATA' has no member named 'sDecErrReport'
         if (pNvxVideoDecOutputExtraData->sDecErrReport.nDecodeError &
                                        ^
gstomxvideodec.c:2633:13: error: 'VIDEO_DEC_DECODED_ERROR_PREV_FRAME_LOST' undeclared (first use in this function)
             VIDEO_DEC_DECODED_ERROR_PREV_FRAME_LOST) {
             ^
gstomxvideodec.c:2639:40: error: 'NVX_VIDEO_DEC_OUTPUT_EXTRA_DATA' has no member named 'sDecErrReport'
         if (pNvxVideoDecOutputExtraData->sDecErrReport.nDecodeError ==
                                        ^
gstomxvideodec.c:2640:13: error: 'VIDEO_DEC_DECODED_ERROR_NONE' undeclared (first use in this function)
             VIDEO_DEC_DECODED_ERROR_NONE) {
             ^
gstomxvideodec.c:2647:44: error: 'NVX_VIDEO_DEC_OUTPUT_EXTRA_DATA' has no member named 'sDecErrReport'
                 pNvxVideoDecOutputExtraData->sDecErrReport.nDecodeError,
                                            ^
gstomxvideodec.c:2650:44: error: 'NVX_VIDEO_DEC_OUTPUT_EXTRA_DATA' has no member named 'sDecErrReport'
                 pNvxVideoDecOutputExtraData->sDecErrReport.nDecodedMBs,
                                            ^
gstomxvideodec.c:2652:44: error: 'NVX_VIDEO_DEC_OUTPUT_EXTRA_DATA' has no member named 'sDecErrReport'
                 pNvxVideoDecOutputExtraData->sDecErrReport.nConcealedMBs,
                                            ^
gstomxvideodec.c:2654:44: error: 'NVX_VIDEO_DEC_OUTPUT_EXTRA_DATA' has no member named 'sDecErrReport'
                 pNvxVideoDecOutputExtraData->sDecErrReport.nFrameDecodeTime, NULL);
                                            ^
gstomxvideodec.c:2665:12: error: 'GstOMXVideoDecClass' has no member named 'video_dec_loop'
   if (klass->video_dec_loop) {
            ^
gstomxvideodec.c:2666:10: error: 'GstOMXVideoDecClass' has no member named 'video_dec_loop'
     klass->video_dec_loop (buf);
          ^
gstomxvideodec.c:2715:42: error: 'GstOMXBuffer' has no member named 'Video_Meta'
     gst_buffer_add_iva_meta(outbuf, &(buf->Video_Meta.VideoDecMeta));
                                          ^
gstomxvideodec.c:2743:58: error: 'GstOMXBuffer' has no member named 'Video_Meta'
       gst_buffer_add_iva_meta(frame->output_buffer, &(buf->Video_Meta.VideoDecMeta));
                                                          ^
gstomxvideodec.c:2765:60: error: 'GstOMXBuffer' has no member named 'Video_Meta'
         gst_buffer_add_iva_meta(frame->output_buffer, &(buf->Video_Meta.VideoDecMeta));
                                                            ^
gstomxvideodec.c: In function 'gst_omx_video_dec_decide_allocation':
gstomxvideodec.c:3890:11: error: 'GstOMXVideoDec' has no member named 'output_buffers'
   if (self->output_buffers > min)
           ^
gstomxvideodec.c:3891:15: error: 'GstOMXVideoDec' has no member named 'output_buffers'
     min = self->output_buffers;
               ^
make[2]: *** [libgstomx_la-gstomxvideodec.lo] Error 1
make[2]: Leaving directory `/home/ubuntu/gstomx1_src/gst-omx1/omx'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/home/ubuntu/gstomx1_src/gst-omx1'
make: *** [all] Error 2

Anyway, there is a binary of this library already included in the L4T filesystem I’m using:
ubuntu@tegra-ubuntu:~$ ls -l /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/ | grep omx
-rwxrwxr-x 1 root root 256664 Apr 26 10:59 libgstomx.so

However, I would like to be able to build the “testegl” example under gstomx1_src/gst-omx1/examples/egl.

Because ./configure worked, I got a Makefile under gstomx1_src/gst-omx1/examples/egl.

But, when I try to make the example, I’m getting:

ubuntu@tegra-ubuntu:~/gstomx1_src/gst-omx1/examples/egl$ make
  CC       testegl-testegl.o
testegl.c:57:22: fatal error: bcm_host.h: No such file or directory
 #include "bcm_host.h"
                      ^
compilation terminated.
make: *** [testegl-testegl.o] Error 1

I can’t see this file anywhere on the system:

ubuntu@tegra-ubuntu:~$ sudo find / -name bcm_host.h
ubuntu@tegra-ubuntu:~$

I’m using a Jetson TK1 with L4T release 21 revision 5, and I downloaded the source packages from here: https://developer.nvidia.com/linux-tegra-r215.

Please could you help?

Essentially, I’m looking for a way to do a zero copy pass of a decoded video frame from the CPU address space into the GPU address space.
I’m used to doing this on Android where external EGL images are wrapped around ion / dma_buf buffers.

DaneLLL · May 3, 2017, 1:31am

Hi micjan,
For Tk1, please download, build and install gstegl and gstomx1 in sequence.
https://developer.nvidia.com/embedded/dlc/l4t-Jetson-TK1-GstEGL-Sources-R21-5
https://developer.nvidia.com/embedded/dlc/l4t-Jetson-TK1-gstreamer-1-R21-5

gstomx1 depends on gstegl.

micjan · May 3, 2017, 11:20am

Hi DaneLLL,

I can ‘configure’, ‘make’, and ‘make install’ the source in gstegl_src OK.

What ‘make’ does for me is it builds libgstegl-1.0.so and libgstnveglglessink.so.

What ‘make install’ does for me is it puts the first library in /usr/local/lib, and the second library in /usr/local/lib/gstreamer-1.0.

gstreamer appears to expect to load the plugin library from /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstnveglglessink.so, but I could work around that by symlinking my library as /usr/lib/arm-linux-gnueabihf/gstreamer-1.0/libgstnveglglessink.so.

With the modifications to libgstnveglglessink.so I’ve done, it looks like I can achieve what I needed - I can process the video frames with OpenGL (ES) and display them on the screen.

I still get the same error as before when I try to build the source in gstomx1_src, however, it looks like I’ve answered my own question for now.

I know NVidia recommends using gst-nvivafilter with CUDA for that kind of task (as seen for example here: https://devtalk.nvidia.com/default/topic/937850/jetson-tk1/gpu-accelerated-gstreamer-filters-/), but that capability is not available on the TK1?

Is my approach aligned with what NVidia recommends to process video on the GPU on the TK1? The reason I ask is I can see glTexImage2D being used in eglgles/gsteglglessink.c and the comments in the file mention a ‘slow copy-over rendering path’ - it looks like every video frame is copied into GPU memory?

I thought there would be a way to process EGLImages as external textures directly on the TK1. Which is why I was trying to build the example in gstomx1_src/gst-omx1/examples/egl, as that seems to be testing EGLImage use. However, there are references there to ARM Mali (libump) and Broadcom VideoCore (the bcm_host.h header), so that’s confusing.

DaneLLL · May 4, 2017, 3:41am

Hi micjan,
For your case you can process EGLImage in nveglglessink and no need to build gstomx1_src, just like what you have done now. The decoder can output EGLImage:

gst-launch-1.0 filesrc location=test.h264 ! h264parse ! omxh264dec ! nveglglessink

‘slow copy-over rendering path’ is there in original file

github.com

rawoul/gst-plugins-bad/blob/master/ext/eglgles/gsteglglessink.c

/*
 * GStreamer EGL/GLES Sink
 * Copyright (C) 2012 Collabora Ltd.
 *   @author: Reynaldo H. Verdejo Pinochet <reynaldo@collabora.com>
 *   @author: Sebastian Dröge <sebastian.droege@collabora.co.uk>
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
 * copy of this software and associated documentation files (the "Software"),
 * to deal in the Software without restriction, including without limitation
 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
 * and/or sell copies of the Software, and to permit persons to whom the
 * Software is furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER

This file has been truncated. show original

We have optimized it on TK1 into a new package, so there is no redundant memcpy.

nvivafilter is not available on TK1

linuxdev · May 4, 2017, 12:36pm

Also keep in mind that you need “/usr/local/lib” and “/usr/local/lib/gstreamer-1.0” in your linker path if those are to be found automatically. See “ldconfig -p” for a listing of what the linker sees (it may already be in your path). If you add a new lib which is in the linker path, then you need to either reboot or run “sudo ldconfig” for the linker to refresh and see new libs.