Hello Everyone,

I am facing an encoder/decoder delay in my video call application running on Tegra K1.

Please read below observation and use cases.

1. TK1 is doing video encoding and decoding simultaneously (both CIF format video)

2. When the encoding is H.264 and Decoding is H.263 format my total latency for my encoding video is less than 400 milliseconds.

3. Whereas when both the encoding and decoding video is H.264 the latency is increased up-to 1000 milliseconds.

So can anyone explain why I am getting higher delay when both encoding and decoding videos are H.264

Regards,
Gopinath S

Hi Gopinath,
1 Can you try the case of maxing out CPU?
echo performance > /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
2 Do you run it with gst command? If yes, please share the command also.

Hi,

1. Yes I tired with maxing both CPU and GPU performance. But it seems there is no improvement.

2. I am not using GST command. I am checking with nVIDIA’s offical Android L for Jetson. Below is the source.

https://devtalk.nvidia.com/default/topic/878820/jetson-tk1/android-l-update-to-jetson-tk1/

Hi Gopinath,
Are there sample app/apk or running steps to simulate your usecase?

Hi Gopinath,
Are there sample app/apk or running steps to simulate your usecase?

Hi DaneLLL,

I cannot give you the sample application/apk for now, because it contains confidential information.Hope you understand. By the mean time I will try to simulate this use cases by excluding those information.

Here is an update. When I tested with skype video call from Jetson device to PC, the delay is less than 80 milliseconds. I am not sure what encoder they are using.

Here are few other questions I have.

1. what are all the hardware and software video encoders supported by TK1?
2. what are all the hardware and software video decoders supported by TK1?

Thanks
Gopinath S

Hi DaneLLL,

Another update. When I checked with Shied tablet K1, the same issue is happening.

Thanks
Gopinath S

Hi Gopinath,
The TK1 hardware codecs are listed in
device/nvidia/platform/ardbeg/media_codecs.xml

Android software codecs are in
frameworks/av/media/libstagefright/data/media_codecs_google_video.xml

Can you remove “OMX.Nvidia.h264.encoder” in media_codecs.xml and try again? It should use “OMX.google.h264.encoder” and let’s see how it works.

So far as I know, skype has its own codecs in apk.

Hi DaneLLL,

Sure I will check it and update.

Thanks
Gopinath S

Hi DaneLLL,

When I checked with OMX.google.h264.encoder, the latency is reduced very much. It is approximately 300 milliseconds.

So here is summery.

1. Decoding in H264 hardware decoder
   Encoding in H264 hardware encoder - latency delay in Encoder is ~800 milliseconds.

2. Decoding in H264 hardware decoder
   Encoding in H264 software google encoder - latency delay in Encoder is ~300 milliseconds.

3. Decoding in H263 hardware decoder
   Encoding in H264 hardware encoder - latency delay in Encoder is ~350 milliseconds.

I have another question. Is it H264 hardware encoder and decoder is separate unit? or it is combined one? I mean, If both are in use, then delay seems increasing…Is that be a reason??

Thanks,
Gopinath S

Hi Gopinath,
No, it should not be the reason. H264 hw encoder and decoder are separate units and work independently on tk1. However, I am not able to explain where the delay comes from as I don’t have the detail of the video call application. But it looks like you can run h264 hw decoder and h264 sw google encoder to get better performance in the usecase.

Hi DaneLLL,

Thanks for the reply.

But we prefer to use HW encoder.

We just started to home in on where the issue is occurring.

Whn I adjusted the timeout value of MediaCodec.dequeueOutputBuffer()method in our application, inconsistently I can see there is no latency.

(at first the value is (1000000 / 30) microseconds , ie 33ms tuned to 30 frames per second and by reducing(to 7 ms) this will results in avoiding latency, but not consistent.)

It seems timing/signalling flaw in the way that the Synchronous MediaCodec API works.

My understanding is the same application code works fine with software encoder where as in case of hw encoder I am getting this inconsistent and synchronous issue. Do I need to handle any thing specific in my application to make hw encoder works fine?

Hi Gopinath, let me try if I can simulate your usecase via native sample app
frameworks\av\cmds\stagefright\codec.cpp

Please check mediacodec.tar.gz. I get ~130 ms in my simultaneous encoding/decoding test. There can be mismatch between our tests. Please specify it so that I can simulate your usecase

Hi DaneLLL,

Thanks for these test results. To be more specific,my results includes network delay since our application is transmitting the encoded video over internet. I guess that may be the case. I will check that too and tell you the results.

Here is an important update. It seems we have found out the root cause of this issue as below.

Jetson hw encoder is outputting a LevelID of 50 (5.0), where as other devices I have tested uses encoder with LevelID 13 (1.3). To determine if this was causing the delay, the LevelID in the Sequence Header was altered in a test sequence from 50 to 13, the delay disappeared.

So my question now is, where this LevelID of encoders are mentioned in Jetson. I have checked with media_profiles.xml in Jeston and it is something like below.

<!--
      The VideoEditor Export codec profile and level values
      correspond to the values in OMX_Video.h.
      E.g. for h264, profile value 1 means OMX_VIDEO_AVCProfileBaseline
      and  level 512 means OMX_VIDEO_AVCLevel31.
      Please note that the values are in decimal.
      These values are for video encoder.
  -->
  <!--
      Codec = h.264, Baseline profile, level 4
  -->
  <ExportVideoProfile name="h264" profile="1"
   level="2048" />

In the above code what is level=2048 means? But I didn’t get much information about changing this level ID. Can you help me on this?

Regrads,
Gopinath S

Hi Gopinath,
Please check if configuring the following setting to hw h264 encoder helps:
int width = 352;
int height = 288;
mOutputFormat->setInt32(“bitrate”, 768000);
mOutputFormat->setInt32(“profile”, OMX_VIDEO_AVCProfileBaseline);
mOutputFormat->setInt32(“level”, OMX_VIDEO_AVCLevel13);

Hi DaneLLL,

Thanks for the reply.These settings(resolutions, bitrate etc.) will vary not only call to call but also dynamically during a call as well.

I will give it a try to test this.

Is it possible to change the level ID in BSP side rather than application? Also can we have dynamic level ID for hw encoder support in BSP?

Hi DaneLLL,

Update

When I used h264 hw encoder, LevelID is 50 (5.0)

(0011 0010 = Level_id:50 [Level 5.0 135 Mb/s]

When I used google software Level ID of encoder is 1.3.

(0000 1101) = Level_id:13 (Level 1.3 768kb/s)

(the above values are verified using TCPdump of my encoded packets)

Regards,
Gopinath S

Hi DaneLLL,

When I tested the above source code, I am getting the latency nearly same ~100ms.

encode & codec -v sdcard/The_Bourne_Ultimate.mp4             
[1] 1432
enc latency = 100869 usNvRmPrivGetChipPlatform: Could not read platform information 
Expected on kernels without fuse support, using silicon
Inside NvxLiteH264DecoderLowLatencyInitNvxLiteH264DecoderLowLatencyInit set DPB and Mjstreaming

303 frames encoded, 57.70 fps. 2765657 bytes received. 514.30 KB/sec
helloHello
track 0: 2137 frames decoded, 40.18 fps. 5019770880 bytes received. 92167.19 KB/sec
[1] + Done                 encode

Also getting few warning messages. But in my actual application I am getting more delay. Meanwhile I look If I am missing something…

Hi Gopinath,
LevelID is configurable. Please make #15 to encoder.cpp and you will get h264 stream with levelID 1.3

Hi DaneLLL,

Thanks for your support. I have done the #15 changes.

Is there any way to verify the levelID1.3 in your encoder.cpp application?

By the mean time, I will make this changes in my application and there I will be able to verify this in TCP dump, since it is transmitted over internet.

Regards,
Gopinath S