Nvarguscamerasrc Buffer Metadata is missing

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hi,

We have been using the enable-meta property in nvcamerasrc in order to get the timestamp coming from VI. We used the nvgstcapture application as a based to do something like this for each buffer:

  • Metadata Definition
/* MetaData structure returned by nvcamerasrc */
typedef struct AuxBufferData {
  gint64 frame_num;
  gint64 timestamp;
  void * sensor_data;
} AuxData;
  • Extract meta from buffer
gst_buffer_metadata_quark = g_quark_from_static_string ("GstBufferMetaData");
    meta = (AuxData *) gst_mini_object_get_qdata (GST_MINI_OBJECT_CAST (buffer),
							   gst_buffer_metadata_quark);

   printf(">>> Gstreamer:Frame #%lu : Timestamp: %lu\n", meta->frame_num, meta->timestamp);

This code works very good, and we can extract the timestamp from the buffer without problems.

However, it seems like in the newest Jetpack, with nvarguscamerasrc, this property doesn’t exist anymore, so the metadata is never obtained properly from the buffer.

Do you know why this property is not part of nvarguscamerasrc?
Is there a way to activate the metadata in nvarguscamerasrc?

Thank you.

  • Carlos R.
2

Note: I have the same exact problem (both with nvcamerasrc and nvarguscamerasrc). I assumed the enable-metadata at least on nvcamerasrc would attach a GstMeta to each buffer but I don’t see any (GstBuf.foreach_meta etc.).

What is NVidia’s intention with metadata both within libargus and gstreamer?

3

Hi @alex.sack,

I am able to use the metadata with LibArgus without problems (JP 4.2 works properly), and nvcamerasrc works as well (at least in JP 3.3).

My problem is with nvarguscamerasrc (since JP 4.2), which seems it missed the enable-meta property, so buffers don’t contain the metadata.

4

Carlos, neat.

Question for you: I’m on L4T 28.2.1 (last version our camera provider supports for now) and I don’t see any meta data or timestamps associated with nvcamerasrc in my Python based gstreamer application. Where are the timestamps and meta data stored in Gst land? (I thought it would be GstMeta attached to each GstBuf?).

5

@alex.sack, did you enable enable-meta=true in nvcamerasrc? This would be the first thing to check.

Then, the metadata is not attached to the buffer as a simple GstMeta. NVIDIA seems to use the GstMiniObject belonging to the GstBuffer to store it as a pointer to the metadata structure.

So, in C, you should do something like this:

GQuark gst_buffer_metadata_quark = 0;
    gst_buffer_metadata_quark = g_quark_from_static_string ("GstBufferMetaData");
    meta = (AuxData *) gst_mini_object_get_qdata (GST_MINI_OBJECT_CAST (buffer),
							   gst_buffer_metadata_quark);

Nvcamerasrc internally sets the qdata (using gst_mini_object_set_qdata), with a pointer to the metadata structure. Then you can retrieve that pointer using exactly the same GQuark, in this case the GQuark name is GstBufferMetaData. The quark id in this case is 2581.

For instance, using python, you should do something like:

miniobject = GstBuf.mini_object
meta = miniobject.getqdata (2581)
timestamp = meta.timestamp

I hope it helps.

References:
https://lazka.github.io/pgi-docs/Gst-1.0/classes/Buffer.html
https://lazka.github.io/pgi-docs/Gst-1.0/classes/MiniObject.html#Gst.MiniObject.get_qdata

6

Carlos, this is great! Is this documented ANYWHERE?

7

Did it work?

And no, I was not able to find any documentation about it in the NVIDIA docs! I analyzed the nvgstcapture app and noticed NVIDIA is doing it in that way.

However, it seems like they remove that mechanism in nvarguscamerasrc. And in order to upgrade to the latest Jetpack 4.2 I would need some information about how to get the timestamps for each buffer.

8

No it did not. My pipeline starts with:

nvcamerasrc do-timestamp=true enable-meta=true fpsRange=“30.0 30.0” auto-exposure=1 aeLock=true sensor_id=0 name=nvcam0 ! identity name=tap ! …

Then on the “handoff” callback for each buffer received by the “identity” plugin I do:

mobject = buf.mini_object
meta = mobject.get_qdata(2581)
print(meta)

It’s all zeros.

The buf.pts timetamps are valid but lack precision (they fluctuate a lot but NVidia claims this has been fixed in the laetst JP?).

9

Note that if I do this:

quark = GLib.quark_from_static_string("GstBufferMetaData")

EDIT: I dumped the strings and their quark IDs and here is what I get:

4242 GstBufferMetaData 547138806488
4243 GstBufferUseCase 19607948
…

???

Nothing looks like a timestamp though and UseCase is static and BufferMetaData just rotates. Any clue?

10

Oh ok, that’s actually the correct way to do it! It seems like you are making progress.

However, please recall that the get_qdata function is giving you a pointer to the metadata structure, that is likely why you keep getting the same numbers, because it is a pointer to an structure like:

/* MetaData structure returned by nvcamerasrc */
typedef struct AuxBufferData {
  gint64 frame_num;
  gint64 timestamp;
  void * sensor_data;
} AuxData;

So you would need to grab that pointer and extract the fields from the structure.

11

But this is Python not C! I should get back an object not a raw pointer value (which would kinda be useless right?). Maybe I’m not understanding you.

Or are you saying the quark returned I need to try to deference within Python using ctype? I mean holy cow…

12

Okay, lets see step by step:

  1. NVcamerasrc internally adds the metadata using the gst_mini_object_set_qdata function[1]. This function attaches a gpointer to the mini object. As you might know, gpointers are only pointers pointing out to whatever you want. In this case, I guess they create a structure with the metadata (structure from previous reply), and then they create a gpointer pointing to this structure. This gpointer is attached to the mini object using the set_qdata function.

  2. Using gst_mini_object_get_qdata function [2], you can retrieve the gpointer previously attached corresponding to an specific GQuark. That’s what I’m doing in my application.

  3. When you use GstMiniObject.get_qdata function [3] in Python, you are retrieving an object, but what is your object? your object is a gpointer, and that pointer is pointing out to the metadata structure.

Your main problem here is that you are using Python, and you are trying to get the structure (chunk of data) from a gpointer, and I’m not sure if that is possible, or at least there is not an easy way to do it. You can read some information about it here [4]

[1][url]gst/gstminiobject.c · 1.14.4 · GStreamer / gstreamer · GitLab
[2][url]gst/gstminiobject.c · 1.14.4 · GStreamer / gstreamer · GitLab
[3][url]Gst.MiniObject - Structures - Gst 1.0
[4][url]https://developer.gnome.org/pygobject/stable/class-gobjectgpointer.html[/url]

13

I understand all that actually. But thanks.

I am given a reference in Python to a C pointer to struct. I am looking into ctypes to see if I can perhaps deference it directly using it’s id(). But so far no luck.

There has GOT to be a better way than this though, right? Otherwise NVIDIA is just not implementing this correctly (I have no idea why these aren’t GstMetas which seems like a natural fit to me).

14

Yes, I agree! GstMetas would be so much easier to handle. But this is how they implemented it, maybe NVIDIA could change the approach for following releases.

This takes me back to the original questions for NVIDIA folks:

  1. Do you know why the enable-meta property is not part of nvarguscamerasrc?

  2. Is there a way to activate the metadata in nvarguscamerasrc (not using silent=false, that is not usable from software perspective)?

Thanks,

15

Alright, I found it but now I see frame drops:

from ctypes import *

class GSTBUFMETADATA(Structure):
    _fields_ = [("frame_num", c_int64), ("timestamp", c_int64), ("sensor_data", POINTER(c_int))]

Defines the ctype then for each buffer:

mobject = buf.mini_object
quark = GLib.quark_from_static_string("GstBufferMetaData") 
gpointer = mobject.get_qdata(quark)  
gst = cast(gpointer, POINTER(GSTBUFMETADATA)) 
print(gst.contents.frame_num, gst.contents.timestamp)


29 537157864934000
31 537157898253000
33 537157931604000
35 537157964929000
37 537157998249000
39 537158031575000
41 537158064917000
43 537158098250000
45 537158131579000
47 537158164902000
49 537158198245000
51 537158231582000
53 537158264901000
55 537158298244000
57 537158331582000
59 537158364891000
62 537158398244000
63 537158431571000
65 537158464889000
67 537158498225000
69 537158531544000

Do you see something similar? Why am I dropping frames?

16

Timestamp deltas look excellent now!!! But I am missing about 20 frames from my 10 second clip. There seems to be some frame drop which I am not sure why?

Does “identity” drop frames if the callback is too slow? Kinda seems unlikely to me (I am literally just storing a few values and returning).

Also, where did you get that structure definition above with timestamp and frame_num?

17

Well, I take it back:

259
300
302
299
299
300
298

Other than the initial file, it looks like I see about 300 frames every 10 seconds at 30fps which is right. But why do the frame_numbers skip? (why aren’t they just 0,1,2,3,4… etc.).

18

Hi CarlosR92,
We will review to revive enable-meta property in nvarguscamerasrc. Thanks for reporting it.

For a quick solution on r32.1, you may check
[url]CLOSED. Gst encoding pipeline with frame processing using CUDA and libargus - Jetson TX1 - NVIDIA Developer Forums

19

Can you explain if the reported frame numbers are continuous (1,2,3,4,5) or can there be gaps between them?

Also, what is stored in sensor_data above?

20

@alex.sack, yes, frame numbers are continuous for me! Not sure why they are not for you.

@DaneLLL, yes, in LibArgus it is quite straight forward to query the Metadata information, thanks!

Do you have an expected release date for a new version of nvarguscamerasrc including the enable-meta property?