transfert learning toolkit-> export model

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Hi,
I’ve set up the docker for the transfer learning. I’ve trained, pruned and retrained my model.
Then, I exported my model to the .etlt format.

This format seems to be usable only by DeepStream SDK.
Then, I set up the Deep Stream SDK and I have the : (deepstream-test-1-app, deepstream-test-2-app etc.) working.
In order to use the trained model with pruning(=transfert learning toolkit), I understand that i have to use thedeepstream-app.

I followed the readme files (got the.prototxt file, the iplugin.prototxt file & the VGG16faster_rcnn_final.caffemodel)
Things seems fine BUT i managed to have everything working thanks to this caffe model.

Where can i find a tutorial or some detailed steps where it is described how to use a CUSTOM .ETLT file:
a) what document to modify (config_infer_primary_fasterRCNN.txt ?)
b) what modifications to do?

Thanks in advance.

AS

2

Hi assansanogo,
See Integrating TAO Models into DeepStream — TAO Toolkit 3.22.05 documentation
To deploy a model trained by TLT to DeepStream you can choose one of below ways:

  1. Generate a device specific optimized TensorRT engine, using tlt-converter which may then be ingested by DeepStream
  2. Integrate the model directly in the DeepStream environment using the exported model file(i.e etlt file) generated by tlt-export.

For your question, you already generate an etlt model. Please see the part “Integrating a FasterRCNN model” of the TLT started guide.
To run a FasterRCNN model in DeepStream, you need a label file and a DeepStream configure file. In addition, you need to compile FasterRCNN DeepStream plugin and sample app. A DeepStream sample with documentation on how to run inference using the trained FasterRCNN models from TLT is provided on github at: GitHub - NVIDIA-AI-IOT/deepstream_4.x_apps: deepstream 4.x samples to deploy TLT training models.

Then, modify pgie_frcnn_uff_config.txt inside it. For example, below etlt model path and your ngc key.

tlt-encoded-model=</path/to/etlt/model>
tlt-model-key=<key to decode the model>

If you use TRT engine to do inference, please disable above two lines and add below line into pgie_frcnn_uff_config.txt

model-engine-file= <TRT engine>

To do inference
$ ./deepstream-custom-uff-frcnn pgie_frcnn_uff_config.txt

3

Hi Thanks for your answer… I’ve tried for the last few days to follow the recommendation you gave me. - no success so far.
Just to put things in context: I was able to prune the model on a x86 Computer via the tLT, and I want to use Deepstream to do inference on the Jetson Xavier.

I followed the 2 steps you suggested.

I’ve tried with the *.etlt files : and I have a set of errors as displayed below.
I’ve tried to convert the files into TRT format (with tlt-converter) and try to use them on the Jetson. there is a minor mismatch between the converter and what Deepstream is expecting. (minor versions)

Based on the link u provided before:

a) can you confirm that .etlt file alone + the ngc key (= api key ?) is enough to “load” the model via the config file?
( is it possible to see a working example somewhere ? (i’ve been using the data supplied/shipped on that website and it is not working on my end)
b) in the folder where the shipped .etlt file is, I see a .pb file. Should we do something with that file? (convert it to uff?)
c) can you confirm that one can export the model via the TLT toolkit docker and use it later on for inference on a Jetson Xavier ?

here is the snapshot of my screen:
https://photos.app.goo.gl/mtU1X9dt7WqmGeXdA

Thanks for your help. I’m stuck, but I think u can enlight me with some pieces of advice.

4

Hi assansanogo,
Yes, I can confirm that one can export the model via the TLT toolkit docker and use it later on for inference on a Jetson Xavier.
For Faster-rcnn, I provide below steps for your reference.

  1. As tlt doc mentioned, please follow GitHub - NVIDIA-AI-IOT/deepstream_4.x_apps: deepstream 4.x samples to deploy TLT training models along with TRT OSS https://github.com/NVIDIA/TensorRT/tree/release/5.1 to compile FasterRCNN DeepStream plugin and sample app.
    2.Copy etlt model generated by tlt.
    FP32 mode: the etlt model is frcnn_kitti_retrain.etlt, generated by tlt-export tool. (Input is tlt format model.)
    Int8 mode: the etlt model is frcnn_kitti_retrain.int8.etlt, generated by tlt-export tool. Another output file is calibration cache file (cal.bin)
  2. If use exported model directly as above, ignore this step.
    If want to generate TRT engine directly, in Jetson devices instead of x86, please download tlt-converter from dev-zone(https://developer.nvidia.com/tlt-converter) in order to generate a TRT engine. For example,run below command in xavier to generate an int8 trt engine. Input is “frcnn_kitti_retrain.int8.etlt” and "cal.bin ", the output is “frcnn_int8.engine”
    $ ./tlt-converter frcnn_kitti_retrain.int8.etlt -e frcnn_int8.engine -k -c cal.bin -o dense_regress/BiasAdd,dense_class/Softmax,proposal -d 3,384,1280 -b 8 -m 4 -t int8 -i nchw
    4.Generate label file
    $ cat frcnn_labels.txt
    Car
    Pedestrian
    Cyclist
    background
    5.Vim config file
    $ cat pgie_frcnn_uff_config.txt
[property]
gpu-id=0
net-scale-factor=1.0
offsets=103.939;116.779;123.68
model-color-format=1
labelfile-path=./frcnn_labels.txt
tlt-encoded-model=./frcnn_kitti_retrain.etlt
tlt-model-key=<your ngc key>
uff-input-dims=3;384;1280;0
uff-input-blob-name=input_1
batch-size=1
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=0
num-detected-classes=4
interval=0
gie-unique-id=1
is-classifier=0
#network-type=0
output-blob-names=dense_regress/BiasAdd;dense_class/Softmax;proposal
parse-bbox-func-name=NvDsInferParseCustomFrcnnUff
custom-lib-path=./nvdsinfer_customparser_frcnn_uff/libnvds_infercustomparser_frcnn_uff.so

[class-attrs-all]
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0

6.If config TRT int8 engine directly, the config is as below.
$ cat pgie_frcnn_uff_config.txt

[property]
gpu-id=0
net-scale-factor=1.0
offsets=103.939;116.779;123.68
model-color-format=1
labelfile-path=./frcnn_labels.txt
model-engine-file=./frcnn_int8.engine
uff-input-dims=3;384;1280;0
uff-input-blob-name=input_1
batch-size=1
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=1
num-detected-classes=4
interval=0
gie-unique-id=1
is-classifier=0
#network-type=0
output-blob-names=dense_regress/BiasAdd;dense_class/Softmax;proposal
parse-bbox-func-name=NvDsInferParseCustomFrcnnUff
custom-lib-path=./nvdsinfer_customparser_frcnn_uff/libnvds_infercustomparser_frcnn_uff.so

[class-attrs-all]
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0

7.Run inference
$ ./deepstream-custom-uff-frcnn pgie_frcnn_uff_config.txt sample_720p.h264

5

Hi,
i’m going to try your approach but just to understand:
all this tensor RT installation happens on the JETSON XAVIER right ? (TRT + TRT OSS)

"
Yes, I can confirm that one can export the model via the TLT toolkit docker and use it later on for inference on a Jetson Xavier.
For Faster-rcnn, I provide below steps for your reference.
1. As tlt doc mentioned, please follow https://github.com/NVIDIA-AI-IOT/deepstream_4.x_apps along with TRT OSS https://github.com/NVIDIA/TensorRT/tree/release/5.1 to compile FasterRCNN DeepStream plugin and sample app.
"

The TENSOR RT version available at ([…] …tree/release) is for amd right?
The Jetson Xavier is ARM based. What version should I download then? Should I not use the one (TRT 5.1) shipped with Jetpack?

Thanks for the clarifications.
AS

6

Hi assansanogo,
1.To run inference using the trained FasterRCNN models from TLT, please git clone GitHub - NVIDIA-AI-IOT/deepstream_4.x_apps: deepstream 4.x samples to deploy TLT training models into your Xavier firstly.
2.As GitHub - NVIDIA-AI-IOT/deepstream_4.x_apps: deepstream 4.x samples to deploy TLT training models mentioned, Deepstream SDK 4.0 and TensorRT 5.1 GA are needed. So please install them into your Xavier with Jetpack.
3.As GitHub - NVIDIA-AI-IOT/deepstream_4.x_apps: deepstream 4.x samples to deploy TLT training models mentioned, it depends on the TensorRT OSS plugins. So please folow https://github.com/NVIDIA/TensorRT/tree/release/5.1 to build a new libnvinfer_plugin.so* to replace your Xavier’s libnvinfer_plugin.so*. See details as below.
4.For https://github.com/NVIDIA/TensorRT/tree/release/5.1, please run at your host linux PC.
1) Download TensorRT OSS sources.
2) Download the TensorRT binary release.
3) Build a docker, for exmaple,

docker build -f docker/ubuntu-18.04.Dockerfile --build-arg CUDA_VERSION=10.1 --tag=tensorrt .

4) Building The TensorRT OSS Components
5) Under out folder,copy the libnvinfer_plugin.so* to your Xaiver.
7

Hi again,
Thanks for the previous hints : I’m getting closer to success, but so far : still 1 error.

a) I managed to have it working ( seeing bounding boxes) WITHOUT the custom parser activated ( in the pgie_frcnn_uff_config.txt )
As a result the bonding boxes are confinned at the top of the image.

b) I then realized that there are no layers called : dense_regress/BiasAdd or dense_SoftMax nor even proposal in the models supplied :
when one issue this command:

!tlt-pull -k $API_KEY -lm -o nvtltea -t iva

I have a collection of models available, but they don’t share the characteristics depicted in your previous answer. Where can we find the correct parser when the outputs are : “output_bbox/BiasAdd” and “output_cov/sigmoid”.
Does it mean I have to rewrite the customparser (difficult when we don’t know the logic behind the network) ?

c) I tried to look for specifically described “frcnn” with this command : !/workspace/ngc registry model list
I downloaded models but they are saved as .h5 or .hdf5. When i replace the path with these files. The tlt-training process fails.
What do I do wrong?

d) Is there somewhere : a downloadable version of the frcnn_kitti.tlt in order to recreate the example u described?
When I use the frcnn.etlt model supplied with https://github.com/NVIDIA-AI-IOT/deepstream_4.x_apps
it does not seem to work with the pgie.txt file
Am I doing something wrong?

please advise,

Thanks again.
AS

8

Please find enclosed a couple of snapshots:

https://photos.app.goo.gl/VUSgqS4FyYnCFj5M8

Thanks !
AS

9

Hi assansanogo,
Firstly,may I know if you are using TLT GA version? According to https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/index.html#downloading_models ,it shows below command to get a list of models that are hosted in the NGC model registry:

ngc registry model list <model_glob_string>

instead of

tlt-pull -k $API_KEY -lm -o nvtltea -t iva

Secondly, I assume you are deploying a Faster-rcnn network at the first time. So its output blob name should be “dense_class/Softmax,dense_regress/BiasAdd, proposal” . All the output blob names are listed as below.

  • For classification use: predictions/Softmax.
  • For detection: output_bbox/BiasAdd,output_cov/Sigmoid
  • For FasterRCNN: dense_class/Softmax,dense_regress/BiasAdd, proposal
  • For SSD: NMS

So, please double check you ran “tlt-export” correctly.See https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/index.html#exporting_models for more detailed info.
And also please make sure you ran “tlt-converter” correctly if you generate a TRT engine instead of etlt model.
See “Generating an engine using tlt-converter” section of https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/index.html#exporting_models

10

Hi assansanogo,
Please have a look at jupyter notebooks for reference too.
Trigger notebook, please see more details at
[url]Integrating TAO Models into DeepStream — TAO Toolkit 3.22.05 documentation

11

Hi,
I was using the early access version T_T’, things should be alright now!!!
Keep you post once i’ve experimented with it!
Everything makes more sense (the custom parsers mismatch… etc.)
until next time.
Thanks for the hints.
AS