Convert NvBuffer to vx_image

Hello,

I’m trying to acquire frames with the library Argus and to use them with Visionworks.
I tried to use methods found in several samples of the L4T Multimedia API.

Thus, I’m using Argus and EGLStream to capture the frames and I manage to retrieve a pointer to the data.
Then I create a vx_image that I fill with the frame data. To check if it works I also render it.

std::unique_ptr<nvxio::Render> render(nvxio::createDefaultRender(m_context, "Camera Capture", STREAM_SIZE.width, STREAM_SIZE.height));

I first tried to display a raw (IYUV) image but I can’t create such a render.
That’s why I convert my vx_image to a RGBX format.

EGLStream::Image *image = iFrame->getImage();
EGLStream::IImage* iImage = interface_cast<EGLStream::IImage>(image);
EGLStream::IImage2D* iImage2D = interface_cast<EGLStream::IImage2D>(image);
Argus::Status status;
uint8_t *data[3];
for (uint32_t i = 0; i < iImage->getBufferCount(); i++){
	const uint8_t *d = static_cast<const uint8_t*>(iImage->mapBuffer(i));
	if(!d)
		printf("\tFailed to map Buffer !\n");
	Argus::Size size = iImage2D->getSize(i);
	printf("\tIImage(2D): buffer %u (%ux%u, %u stride)\n", i, size.width, size.height, iImage2D->getStride(i));
	data[i] = (uint8_t *)d;
}

//Create Image
vx_image img = vxCreateImage(m_context, STREAM_SIZE.width, STREAM_SIZE.height, VX_DF_IMAGE_IYUV);
int j = 1;
for (int ch = 0; ch < 3; ch++){
	if (ch > 0)
		j = 2;
	void *ptr;
	vx_map_id map_id;
	vx_imagepatch_addressing_t addr;
	vx_rectangle_t rect = {0u, 0u, STREAM_SIZE.width/j, STREAM_SIZE.height/j};
	vxMapImagePatch(img, &rect, ch, &map_id, &addr, &ptr, VX_WRITE_ONLY, VX_MEMORY_TYPE_HOST, 1);
	uint8_t *row_ptr = (uint8_t*) ptr;
	memcpy(row_ptr, data[ch], addr.dim_x * addr.dim_y);
	vxUnmapImagePatch(img, map_id);
}
//void *ptrs[] = {	data, &data[1], &data[2]	};
//vx_image img = vxCreateImageFromHandle(m_context, VX_DF_IMAGE_IYUV, &addr, ptrs, VX_MEMORY_TYPE_HOST);
vx_image img_rgb = vxCreateImage(m_context, STREAM_SIZE.width, STREAM_SIZE.height, VX_DF_IMAGE_RGBX);
vxuColorConvert(m_context, img, img_rgb);
printf("Image YUV: %d\n", vxGetStatus((vx_reference) img));
printf("Image RGB: %d\n", vxGetStatus((vx_reference) img_rgb));

// Display the frame acquired
m_render->putImage(img_rgb);
m_render->flush();

vxReleaseImage(&img);
vxReleaseImage(&img_rgb);

}

I’ve also tried to use the vxCreateImageFromHandle function without success:

//Creation de l'image
vx_imagepatch_addressing_t addr;
addr.dim_x = STREAM_SIZE.width;
addr.dim_y = STREAM_SIZE.height;
addr.stride_x = 3 * sizeof(vx_uint8);
addr.stride_y = static_cast<vx_int32>(addr.dim_x * addr.stride_x);
void *ptrs[] = {	data, &data[1], &data[2]	};
vx_image img = vxCreateImageFromHandle(m_context, VX_DF_IMAGE_IYUV, &addr, ptrs, VX_MEMORY_TYPE_HOST);
vx_image img_rgb = vxCreateImage(m_context, STREAM_SIZE.width, STREAM_SIZE.height, VX_DF_IMAGE_RGBX);
vxuColorConvert(m_context, img, img_rgb);

It always make a “Segmentation Fault” at the m_renderer->putImage(img_rgb) method.
I think I’m doing something bad but I can’t find what.

I’d really appreciate some help.
Thank you :)

Edit:
I’ve just found that there is a mistake during the conversion:

[NVX LOG] context=(nil),ref=0x1cba390,status=-1002 : vxuColorConvert: Internal CUDA error

Hi YassineH,
You should be able to get camera frames via Visionworks APIs. Any reason you use Argus → Visionworks?

Hi Dane,

I’m using Argus because it seems to use less the GPU (and CPU) than VisionWorks. I measured with tegrastats.
In fact, I’m comparing several methods available on the TX1 to capture a frame and display it.

So I tried:

  • Argus(Acquisition) + EGL(Display)
  • VisionWorks (Acquisition + Display)
  • Argus(Acquisition) + VisionWorks(Display)

Thus I observed that Argus is more efficient for the acquisition (even if I didn’t manage to capture at 120fps).

Hi YassineH,
Please share logs of tegrastats which makes you see the different usage of GPU/CPU.

And we have complete functionality in VisionWorks, we now do not have plan for Argus+VisionWorks.

Hi Dane,

Here are the logs:
(Acquisition = 1920x1080 at 30fps)

///////////////////////////////////////////
///////////	VISIONWORKS	///////////
///////////////////////////////////////////


[1920x1080]
***	NO RENDER 30fps	***

RAM 2253/3995MB (lfb 181x4MB) cpu [0%,0%,0%,0%]@1734 GR3D 35%@76 EDP limit 0
RAM 2294/3995MB (lfb 170x4MB) cpu [46%,25%,21%,35%]@1734 GR3D 0%@76 EDP limit 0
RAM 2294/3995MB (lfb 170x4MB) cpu [81%,5%,11%,11%]@1734 GR3D 2%@76 EDP limit 0
RAM 2294/3995MB (lfb 170x4MB) cpu [83%,5%,7%,10%]@1734 GR3D 7%@76 EDP limit 0
RAM 2294/3995MB (lfb 170x4MB) cpu [83%,8%,7%,5%]@1734 GR3D 12%@76 EDP limit 0
RAM 2295/3995MB (lfb 170x4MB) cpu [84%,15%,2%,15%]@1734 GR3D 17%@76 EDP limit 0
RAM 2295/3995MB (lfb 170x4MB) cpu [85%,9%,3%,5%]@1734 GR3D 23%@76 EDP limit 0
RAM 2295/3995MB (lfb 170x4MB) cpu [84%,15%,3%,3%]@1734 GR3D 27%@76 EDP limit 0
RAM 2295/3995MB (lfb 170x4MB) cpu [84%,7%,4%,11%]@1734 GR3D 29%@76 EDP limit 0
RAM 2295/3995MB (lfb 170x4MB) cpu [83%,6%,8%,4%]@1734 GR3D 29%@76 EDP limit 0
RAM 2296/3995MB (lfb 170x4MB) cpu [69%,10%,8%,9%]@204 GR3D 0%@76 EDP limit 0


***	RENDER 30fps	***

RAM 2073/3995MB (lfb 178x4MB) cpu [39%,26%,51%,30%]@1555 GR3D 74%@614 EDP limit 0
RAM 2073/3995MB (lfb 178x4MB) cpu [58%,27%,32%,36%]@1428 GR3D 75%@998 EDP limit 0
RAM 2073/3995MB (lfb 178x4MB) cpu [55%,29%,36%,30%]@1555 GR3D 77%@998 EDP limit 0
RAM 2073/3995MB (lfb 178x4MB) cpu [56%,20%,35%,32%]@1428 GR3D 85%@998 EDP limit 0
RAM 2073/3995MB (lfb 178x4MB) cpu [36%,33%,50%,40%]@1326 GR3D 71%@998 EDP limit 0
RAM 2074/3995MB (lfb 178x4MB) cpu [53%,12%,37%,39%]@1555 GR3D 72%@998 EDP limit 0
RAM 2074/3995MB (lfb 178x4MB) cpu [41%,22%,35%,43%]@1428 GR3D 87%@998 EDP limit 0
RAM 2074/3995MB (lfb 178x4MB) cpu [37%,28%,40%,31%]@1555 GR3D 79%@998 EDP limit 0
RAM 2074/3995MB (lfb 178x4MB) cpu [35%,36%,40%,22%]@1036 GR3D 79%@998 EDP limit 0
RAM 2075/3995MB (lfb 178x4MB) cpu [23%,41%,39%,43%]@1555 GR3D 75%@998 EDP limit 0
///////////////////////////////////
///////////	ARGUS	///////////
///////////////////////////////////

[1920x1080]

***	RENDER 30fps	***

RAM 2147/3995MB (lfb 179x4MB) cpu [20%,89%,24%,24%]@1734 GR3D 22%@76 EDP limit 0
RAM 2148/3995MB (lfb 179x4MB) cpu [10%,58%,37%,40%]@1734 GR3D 24%@76 EDP limit 0
RAM 2148/3995MB (lfb 179x4MB) cpu [21%,89%,22%,9%]@1734 GR3D 21%@76 EDP limit 0
RAM 2149/3995MB (lfb 179x4MB) cpu [23%,76%,25%,15%]@1734 GR3D 15%@76 EDP limit 0
RAM 2150/3995MB (lfb 179x4MB) cpu [16%,15%,22%,94%]@1734 GR3D 8%@76 EDP limit 0
RAM 2150/3995MB (lfb 179x4MB) cpu [11%,19%,18%,97%]@1734 GR3D 5%@76 EDP limit 0
RAM 2150/3995MB (lfb 179x4MB) cpu [100%,20%,16%,10%]@1734 GR3D 5%@76 EDP limit 0
RAM 2151/3995MB (lfb 179x4MB) cpu [100%,17%,18%,11%]@1734 GR3D 5%@76 EDP limit 0
RAM 2159/3995MB (lfb 179x4MB) cpu [100%,17%,12%,14%]@1734 GR3D 5%@76 EDP limit 0
RAM 2205/3995MB (lfb 171x4MB) cpu [100%,19%,14%,7%]@1734 GR3D 5%@76 EDP limit 0


***	NO RENDER 30fps	***

RAM 2242/3995MB (lfb 110x4MB) cpu [17%,14%,21%,81%]@1734 GR3D 0%@76 EDP limit 0
RAM 2242/3995MB (lfb 110x4MB) cpu [15%,15%,9%,100%]@1734 GR3D 0%@76 EDP limit 0
RAM 2243/3995MB (lfb 110x4MB) cpu [11%,12%,12%,100%]@1734 GR3D 0%@76 EDP limit 0
RAM 2244/3995MB (lfb 110x4MB) cpu [14%,19%,10%,100%]@1734 GR3D 0%@76 EDP limit 0
RAM 2282/3995MB (lfb 110x4MB) cpu [14%,14%,11%,100%]@1734 GR3D 0%@76 EDP limit 0
RAM 2373/3995MB (lfb 110x4MB) cpu [15%,14%,11%,100%]@1734 GR3D 0%@76 EDP limit 0
RAM 2463/3995MB (lfb 110x4MB) cpu [15%,13%,8%,100%]@1734 GR3D 0%@76 EDP limit 0
RAM 2554/3995MB (lfb 101x4MB) cpu [17%,10%,12%,100%]@1734 GR3D 0%@76 EDP limit 0
RAM 2645/3995MB (lfb 78x4MB) cpu [20%,7%,11%,100%]@1734 GR3D 0%@76 EDP limit 0
RAM 2739/3995MB (lfb 55x4MB) cpu [13%,23%,12%,100%]@1734 GR3D 0%@76 EDP limit 0
RAM 2186/3995MB (lfb 85x4MB) cpu [33%,23%,9%,31%]@102 GR3D 0%@76 EDP limit 0

I tried with and without rendering.
I note that Argus doesn’t uses GPU to capture frames while VisionWorks does.
Please, correct me if I told a mistake.

Hi YassineH,
Here is our test result of NO_RENDER cases:
[gstreamer]
gst-launch-1.0 nvcamerasrc num-buffers=300 ! ‘video/x-raw(memory:NVMM),width=1920,height=1080’ ! fakesink
RAM 1486/3995MB (lfb 355x4MB) cpu [6%,0%,9%,6%]@1734 EMC 4%@1600 AVP 66%@12 NVDEC 192 MSENC 192 GR3D 0%@998 EDP limit 1734
RAM 1486/3995MB (lfb 355x4MB) cpu [0%,3%,9%,6%]@1734 EMC 4%@1600 AVP 71%@12 NVDEC 192 MSENC 192 GR3D 0%@998 EDP limit 1734
RAM 1486/3995MB (lfb 355x4MB) cpu [1%,9%,10%,0%]@1734 EMC 4%@1600 AVP 71%@12 NVDEC 192 MSENC 192 GR3D 0%@998 EDP limit 1734
RAM 1486/3995MB (lfb 355x4MB) cpu [9%,10%,1%,1%]@1734 EMC 4%@1600 AVP 71%@12 NVDEC 192 MSENC 192 GR3D 0%@998 EDP limit 1734

[VisionWorks] with attached main_nvgstcamera_capture.cpp
~/VisionWorks-1.6-Samples/bin/aarch64/linux/release/nvx_sample_nvgstcamera_capture -r “1920x1080” -f 30
RAM 1526/3995MB (lfb 349x4MB) cpu [5%,18%,10%,92%]@1734 EMC 9%@1600 AVP 27%@12 NVDEC 192 MSENC 192 GR3D 14%@998 EDP limit 1734
RAM 1526/3995MB (lfb 349x4MB) cpu [91%,13%,15%,9%]@1734 EMC 9%@1600 AVP 33%@12 NVDEC 192 MSENC 192 GR3D 13%@998 EDP limit 1734
RAM 1526/3995MB (lfb 349x4MB) cpu [93%,1%,13%,16%]@1734 EMC 9%@1600 AVP 39%@12 NVDEC 192 MSENC 192 GR3D 9%@998 EDP limit 1734

[Argus] with attached main.cpp
~/tegra_multimedia_api/samples/09_camera_jpeg_capture/camera_jpeg_capture
RAM 1510/3995MB (lfb 350x4MB) cpu [7%,10%,10%,11%]@1734 EMC 11%@1600 AVP 85%@20 NVDEC 192 MSENC 192 GR3D 0%@998 EDP limit 1734
RAM 1510/3995MB (lfb 350x4MB) cpu [10%,5%,10%,9%]@1734 EMC 11%@1600 AVP 85%@20 NVDEC 192 MSENC 192 GR3D 0%@998 EDP limit 1734
RAM 1510/3995MB (lfb 350x4MB) cpu [8%,4%,17%,11%]@1734 EMC 11%@1600 AVP 85%@20 NVDEC 192 MSENC 192 GR3D 0%@998 EDP limit 1734

The VisionWorks case is gstreamer+convert_NvBuffer_to_vx_image. So if Argus+convert_NvBuffer_to_vx_image was implemented, the performance would be similar to the VisionWorks case.
main.cpp (15 KB)
main_nvgstcamera_capture.cpp (6.78 KB)

It seems ludicrous that NVidia would provide a camera driver tool that can NOT supply their image processing pipeline tool. Especially, since in the published documentation, Visionworks is shown connected to the camera front-end via EGLStreams, not gstreamer. Could the moderator explain the comment from #4: “we do not now have plans for Argus + Visionworks”.

In reply #6, the moderator makes the statement “so if Argus + convert_NvBuffer_to_vx_image was implemented”. Does that statement mean that: there is no equivalent to
“convert_NVBuffer_to_vx_image” for EGLStream images because NVidia has not implemented the software to make the conversion?

Thanks.

Hi CArcher,
Please refer to the source in VisionWorks-1.6-samples installed via Jetpack-3.0
~/VisionWorks-1.6-Samples/nvxio/src/NVX/FrameSource/ConvertFrame.cpp
~/VisionWorks-1.6-Samples/nvxio/src/NVX/FrameSource/GStreamer/GStreamerNvCameraFrameSourceImpl.cpp
~/VisionWorks-1.6-Samples/nvxio/src/NVX/FrameSource/GStreamer/GStreamerEGLStreamSinkFrameSourceImpl.cpp

Thanks - I will take a look when that download becomes available later in the week. Examples are great - the more the better :)

For now, can you answer a question for me? How do we stream two cameras (through the two ISPs) simultaneously with nvstream and Visionworks? I have not been able to find any documentation on how you specify the CSI channel (0,1, or 2) to FrameSource.

Hi Archer, this should be possible with a patch on VisionWorks-1.6. Will update once Jetpack3.0 is available.

Hi Archer,
Please download VisionWorks-1.6 and refer to the patch:

nvxio/src/NVX/FrameSource/FrameSource.cpp
@@ -301,7 +301,8 @@ std::unique_ptr<FrameSource> createDefaultFrameSource(const std::string& uri)
         else if (path == "nvcamera")
         {
 #ifdef USE_NVGSTCAMERA
-            return makeUP<GStreamerNvCameraFrameSourceImpl>(0);
+            int idx = resolveIntegerValue(keyValues, "index");
+            return makeUP<GStreamerNvCameraFrameSourceImpl>(static_cast<uint>(idx));
 #else
             NVXIO_PRINT("NvCamera source is not available on this platform");
             return nullptr;
samples/nvgstcamera_capture/main_nvgstcamera_capture.cpp
@@ -91,7 +91,7 @@ int main(int argc, char** argv)
     // Parse command line arguments
     //
 
-    std::string resolution = "1280x720", input = "device:///nvcamera";
+    std::string resolution = "1280x720", input = "device:///nvcamera?index=0";
 
     app.setDescription("This sample captures frames from NVIDIA GStreamer camera");
     app.addOption('r', "resolution", "Input frame resolution", nvxio::OptionHandler::oneOf(&resolution,