H264 encoder latency

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hi,

I am attempting to encode a video stream with minimal latency. Using the encoding sample in the tegra multimedia api there is a latency of 1 or 2 frame periods + ~5ms, is there any way of lowering this down to sub 20ms? i.e forcing every frame to be decoded straight away without waiting for the next?

(I have also tried using gstreamer with similar results)

Cheers

2

Hi will.froom,
Please share how you profile the latency. It would be great if you can attach the test code. Thanks.

3

Follow this previous question, I’ve run the camera recording sample from the tegra multimedia api for several times and keep getting different encoding profiling results:

Sometimes it will be:
----------- Element = enc0 -----------
Total Profiling time = 0.836413
Average FPS = 32.2807
Total units processed = 28
Average latency(usec) = 2481
Minimum latency(usec) = 1296
Maximum latency(usec) = 4922

These numbers of latency really make sense, however, sometimes it might also be:

----------- Element = enc0 -----------
Total Profiling time = 0.836353
Average FPS = 31.0874
Total units processed = 27
Average latency(usec) = 33478
Minimum latency(usec) = 2970
Maximum latency(usec) = 71303

So which is the correct result? Two results are not coming from different codes, but keep running the same executable. My guess is similar to @will.froom. In the second case, the frames are waiting for one or two frame periods and then get decoded, but this seems very random. So can we really force every frame to be decoded straight away without waiting for the next?

4

To understand this problem better, I add another log in the encoderCapturePlaneDqCallback function, which should be called when a frame is finished encoding and waits inside a capture plane. We got the following result (I put some orange comments to help us understand):

CONSUMER: Acquired Frame. 1828717843//1st frame acquired from the camera
CONSUMER: Acquired Frame. 1828717844//2nd frame acquired from the camera
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 22.//This is probably some meta data bscause it is only 22 bytes.
CONSUMER: Acquired Frame. 1828717850//3rd frame acquired from the camera
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 26762.//1st frame finishes encoding, which happens after 3rd frame from camera is ready for encoding.
CONSUMER: Acquired Frame. 1828717853//4th frame acquired from the camera
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 24245.//2nd frame finishes encoding, which happens after 4th frame from camera is ready for encoding.
CONSUMER: Acquired Frame. 1828717868
CONSUMER: Released frame. 1828717843
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 23706.
CONSUMER: Acquired Frame. 1828717843
CONSUMER: Released frame. 1828717844
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 24028.
CONSUMER: Acquired Frame. 1828717844
CONSUMER: Released frame. 1828717850
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 7422.
CONSUMER: Acquired Frame. 1828717850
CONSUMER: Released frame. 1828717853
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 13760.
CONSUMER: Acquired Frame. 1828717853
CONSUMER: Released frame. 1828717868
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 26142.
CONSUMER: Acquired Frame. 1828717868
CONSUMER: Released frame. 1828717843
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 35054.
CONSUMER: Acquired Frame. 1828717843
CONSUMER: Released frame. 1828717844
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 30708.
CONSUMER: Acquired Frame. 1828717844
CONSUMER: Released frame. 1828717850
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 32713.
CONSUMER: Acquired Frame. 1828717850
CONSUMER: Released frame. 1828717853
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 30744.
CONSUMER: Acquired Frame. 1828717853
CONSUMER: Released frame. 1828717868
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 33689.
CONSUMER: Acquired Frame. 1828717868
CONSUMER: Released frame. 1828717843
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 34431.
CONSUMER: Acquired Frame. 1828717843
CONSUMER: Released frame. 1828717844
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 30678.
CONSUMER: Acquired Frame. 1828717844
CONSUMER: Released frame. 1828717850
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 31665.
CONSUMER: Acquired Frame. 1828717850
CONSUMER: Released frame. 1828717853
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 29837.
CONSUMER: Acquired Frame. 1828717853
CONSUMER: Released frame. 1828717868
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 30142.
CONSUMER: Acquired Frame. 1828717868
CONSUMER: Released frame. 1828717843
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 31853.
CONSUMER: Acquired Frame. 1828717843
CONSUMER: Released frame. 1828717844
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 25620.
CONSUMER: Acquired Frame. 1828717844
CONSUMER: Released frame. 1828717850
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 26251.
CONSUMER: Acquired Frame. 1828717850
CONSUMER: Released frame. 1828717853
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 23094.
CONSUMER: Acquired Frame. 1828717853
CONSUMER: Released frame. 1828717868
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 23194.
CONSUMER: Acquired Frame. 1828717868//the last frame acquired from the camera
CONSUMER: Released frame. 1828717843
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 24220.//the last frame minus 2 finishes encoding
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 25075.//the last frame minus 1 finishes encoding
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 19293.//the last frame finishes encoding
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 0.
CONSUMER: Got EOS, exiting…
CONSUMER: Done.

From this result, we verify the argument that each frame will finish encoding after 2 frame periods + some time. Since I’m capturing 30fps video with 640x480 resolution, the time of 2 frame periods is 66.7ms. The same question: So can we really force every frame to be decoded straight away without waiting for the next frame?

Current high-end smartphones can encode a 640*480 frame within 10ms, it will be embarrassing if Jetson requires more than 66ms to do so…

5

Also please run with ‘sudo ./jetson_clocks.sh’

6

Is there another comment before this one?

7

I tried your suggestion to run sudo ./jetson_clock.sh before running the sample. I still get two different encoding profiling results:

The first one:
----------- Element = enc0 -----------
Total Profiling time = 0.865153
Average FPS = 31.2084
Total units processed = 28
Average latency(usec) = 1703
Minimum latency(usec) = 110
Maximum latency(usec) = 3464

The result seems pretty nice. However, based on the log, it seems that the encoding latency is still larger than a frame period (33 ms).
The log is:

CONSUMER: Acquired Frame. 1828717843//1st frame.
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 22.//This is probably some meta data bscause it is only 22 bytes.
CONSUMER: Acquired Frame. 1828717844//2nd frame.
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 22020.//1st frame get encoded.
CONSUMER: Acquired Frame. 1828717864
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 25105.
CONSUMER: Acquired Frame. 1828717866
…
CONSUMER: Acquired Frame. 1828717844
CONSUMER: Released frame. 1828717864
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 19179.
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 19732.
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 0.
CONSUMER: Got EOS, exiting…
CONSUMER: Done.

The second result:
----------- Element = enc0 -----------
Total Profiling time = 0.861473
Average FPS = 31.3417
Total units processed = 28
Average latency(usec) = 32612
Minimum latency(usec) = 260
Maximum latency(usec) = 35255

The average encoding latency is 32.6ms, which is around a frame period, but the log shows the latency is actually larger than 2 frame periods (66.7ms).

CONSUMER: Acquired Frame. 1828717843//1st frame.
CONSUMER: Acquired Frame. 1828717844//2nd frame.
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 22.//This is probably some meta data bscause it is only 22 bytes.
CONSUMER: Acquired Frame. 1828717864//3rd frame.
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 22020.//1st frame get encoded.
CONSUMER: Acquired Frame. 1828717866//4th frame.
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 25008.//2nd frame get encoded.
…
CONSUMER: Acquired Frame. 1828717844
CONSUMER: Released frame. 1828717864
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 24758.
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 25853.
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 18977.
CONSUMER: encoderCapturePlaneDqCallback, encoded frame size: 0.
CONSUMER: Got EOS, exiting…
CONSUMER: Done.
PRODUCER: Done – exiting.

So the results are really wired, here are two main questions:

  1. Why does the encoding profiling result keep changing?
  2. Why does the log show longer encoding latency than the profiling result?
8

Please help!

9

Th is an enhancement and as of now we have no plan for this.
[url]https://devtalk.nvidia.com/default/topic/1005063/jetson-tx1/mmapi-dqbuffer-blocking-problem/post/5262049/#5262049[/url]