Displaying the CUDA consumer output over HDMI

Hi,
I am new to Driver PX2 platform and want to use nvmedia sample app ipp_bayerdgpu. Currently in the sample app the output of CUDA consumer is being saved in a file. But I want to render the output to the HDMI monitor instead of saving in the file.
Please suggest me how to do it.

Thanks,
Ashutosh

Dear ashutosh,
One way is to send the output of CUDA to OpenGL for rendering using EGLStream.

Hi Siva,
Thanks. If there is any sample app which is doing OPENGL rendering with CUDA output then can you please point me to that?

Thanks,
Ashutosh

Hi Ashutosh,
Please check CUDA_EGLStreams_Interop sample in CUDA. It uses CUDA as both producer and consumer. You need to replace CUDA consumer calls with openGL-EGL API calls for rendering. CUDA buffers are wrapped as openGL Texture to render it on screen.

Hi Siva,
Thanks a lot for your inputs. As I am new to this thing, just want your help to validate my approach.

In my usecase, I will get a RAW image from an image sensor and then I need to do some processing on this RAW image for which the code will be written in CUDA and then I need to render the processed image onto a display.

Given this is my usecase, I am thinking I should use NvMediaImageProducer and connect it to CUDA consumer using interop and then render the CUDA processed buffer on to a display over HDMI.

I found nvmipp_bayerdgpu sample closest to my usecase. In the cuda_consumer.c file, I need to place my code in cudaprocThreadFunc and as per your suggestion change the output currently being written to the file and use OpenGL-EGL API for rendering. I will study more about this rendering part.

Let me know if my thinking is ok for the usecase.

Thanks,
Ashutosh

Dear Ashutosh,
Did you get a chance to see nvm_eglStream sample. It can take CUDA as producer and GL as consumer. When CUDA is producer you can give .rgba file as input. So you can combine ipp_bayerdgpu and nvm_eglstream sample apps into a single application for your use case. You may have to set EGLStream attribute EGL_SUPPORT_REUSE_NV to false to allow return of last frame from EGLStream.

Hi Siva,

Thanks for your suggestion. Let me explore this a bit. If I have any doubts I will get back to you for help.

Thanks,
Ashutosh

Hi Ashutosh,
Please check image_streamer_multi sample also if it suits your requirement.
External Media
You can use dwImageNVMedia instead of dwImageCPU

Hi Shiva,
Thanks a lot. I will definitely look into that. It looks helpful.

Thanks,
Ashutosh

Hi Shiva,

Where can I find the details of this image_streamer_multi example? Iam referring Autonomous Vehicle Development Platforms | NVIDIA Docs for the documentation.

Also where can I see the code of this sample app in vibrante SDK?

Thanks,
Ashutosh

Dear Ashutosh,
Please find the sample source code in Driveworks(/usr/local/driveworks-0.6/samples/src/image/image_streamer_multi)

Hi Siva,

Thanks for pointing me to the source code location. Currently I want to go ahead with VibranteSDK. I have following queries about our earlier suggestion to combine ipp_bayerdgpu and nvm_eglstream applications.

  • If I understand correctly the CUDA consumer in ipp_bayerdgpu app will generate a .rgba file as an output which can given as an input to CUDA producer in nvm_eglstream app and the gl_consumer in this app will render it on the screen. Have I understood correctly? But writing the output file is an overhead whcih I want to avoid. IS there a way to achieve it without really writing into a file?

  • Is it possible to have two producer-consumer relations in a single app and how can we connect the two?

  • Can I attach GL_consumer code (not as a consumer but more as a renderer) with CUDA consumer of ipp_bayerdgpu to achieve my goal? Do you think there is any better approach?

  • I could not find much information about EGLStream attribute EGL_SUPPORT_REUSE_NV. What does it do and how will it help incase we combile ipp_bayerdpgu and nvm_egl_stream app?

Please help me with these queries.

Regards,
Ashutosh

Hi Siva,

One more thing I tried the nvm_eglstream sample app with CUDA producer and GL_Consumer but I always got an error saying “Invalid EGL Pipeline, please use CUDA consumer with CUDA producer”.

So it seems we cannot use any other consumer with CUDA producer.

Is this intended behavior or am I missing something?

Thanks,
Ashutosh

Dear ashutosh,
The first strategy is a bit complicated as you have to deal with EGL streams directly. So I suggested you to look into image_streamer_multi sample in driveworks. It does similar to what you want and reduces your efforts. Instead of CPUImage you can use NvmediaImage in the sample.
Is there any specific reason you want to use NVmedia API only?

Hi Siva,

Thanks. You are right even I feel the first strategy is bit complicated.

I used DriveInstall (V4.1) to build and flash the image into the DrivePx2. When I checked /usr/local/ on my ubuntu machine I found driveworks-0.2.1 version and inside that I could not find image_streamer_multi sample app.

I think I need to upgrade so can you tell me which DriveInstall version I should use to get the Driverworks-0.6 version? Also can you just give me brief idea about the differences in VibranteSDK and Driveworks SDK.

Regards,
Ashutosh

Dear Ashutosh,
It is a new sample available in driveworks-0.6. Can you upgrade to latest Drive version. Please check at https://developer.nvidia.com/nvidia-drive-downloads.

Drivworks SDK provides a set of API to build autonomous applications, object detection and path planning algorithms etc. It also has tools to acquire the data and calibration. Moore information about driveworks is available at /usr/local/driveworks-0.6/doc.

Hi Siva,

I could create my application. I am getting NvMediaImage from sensor then streaming it to CUDA and then sending it to GL for display. Currently I am using ar0231-rccb sensor as it is supported. I need your help in following issues

  1. When I run my application, I get the following error “Driveworks exception thrown: DW_SAL_CANNOT_CREATE_SENSOR :CameraGMSL: cannot connect to camera” . But if I run the app agaian then sometimes in the nest try or sometimes after couple of tries my app works. Even I tried the sample gmsl camera app and there also I saw this issue. How can I resolve it?

  2. Currently I am using ar0231 sensor as it is the one which is supporte on this platform. Is there any reference docuemnt to show how to write and integrate own iamge sensor driver so that my sensor is also available and can list as supported sensors when I run sample_sensor_info app?

Regards,
Ashutosh

Dear Ashutosh,
Can you please file a bug with source code and relevant information to reproduce it on our side.
Please login to https://developer.nvidia.com/drive with your credentials. Please check MyAccount->MyBugs->Submit a new bug to file bug.
Please share ID here to follow up

Hi Siva,

I was busy with something so could not file the issue earlier. I have files the bug and the bug id is 2108298.

Also please help me with any doc which explains how to write a new image sensor driver for DrivePX2?

Regards,
Ashutosh

Dear Ashutosh,
Could you please share the command you have used to run sample_camera_gmsl_raw app and also the camera details.

Thanks
Sivaramakrishna