[Solved] TensorRT3 - Tensorflow implementation in PX2

Autonomous Vehicles DRIVE PX 2 DriveWorks
1

Hello,

I have a Tensorflow frozen (.pb) file and am trying to convert it to UFF according to

https://devblogs.nvidia.com/tensorrt-3-faster-tensorflow-inference/

This is my code

import uff
import tensorflow as tf
import tensorrt as trt
from tensorrt.parsers import uffparser

G_LOGGER = trt.infer.ConsoleLogger(trt.infer.LogSeverity.INFO)

print(tf.__version__)

I get the following error.

Traceback (most recent call last):
  File "/home/px2host/Serin/PycharmProjects/tensorflow/tensorrt/tensorrt.py", line 3, in <module>
    import tensorrt as trt
  File "/home/px2host/Serin/PycharmProjects/tensorflow/tensorrt/tensorrt.py", line 4, in <module>
    from tensorrt.parsers import uffparser
ImportError: No module named parsers

I have verified if the folder parsers exists

/usr/local/lib/python2.7/dist-packages/tensorrt/parsers
/usr/local/lib/python2.7/dist-packages/tensorrt/parsers/__init__.py
/usr/local/lib/python2.7/dist-packages/tensorrt/parsers/__init__.pyc
/usr/local/lib/python2.7/dist-packages/tensorrt/parsers/caffeparser
/usr/local/lib/python2.7/dist-packages/tensorrt/parsers/uffparser
/usr/local/lib/python2.7/dist-packages/tensorrt/parsers/caffeparser/__init__.py
/usr/local/lib/python2.7/dist-packages/tensorrt/parsers/caffeparser/__init__.pyc
/usr/local/lib/python2.7/dist-packages/tensorrt/parsers/caffeparser/_nv_caffe_parser_bindings.so
/usr/local/lib/python2.7/dist-packages/tensorrt/parsers/uffparser/__init__.py
/usr/local/lib/python2.7/dist-packages/tensorrt/parsers/uffparser/__init__.pyc
/usr/local/lib/python2.7/dist-packages/tensorrt/parsers/uffparser/_nv_uff_parser_bindings.so

$ echo $LD_LIBRARY_PATH returns the following

/usr/local/lib:/usr/local/nvidia/tensorrt/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda-9.0/lib64:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda-9.0/lib64::/usr/local/nvidia/tensorrt/lib

What is the cause of this error?

What is the workflow to implement a Tensorflow model in the DrivePX2?

Regards,
SerinVarghese

2

Dear serinvarghese,

Did you install TensorRT?
TensorRT python API doesn’t support DPX2 platform.

Do you meet this symptom on a x86 Linux system? Thanks.

3

Hello Steve,

  1. TensorRT is installed.
px2host@px2host1:~$ dpkg -l | grep TensorRT
ii  libnvinfer-dev                                           4.0.4-1+cuda9.0                              amd64        TensorRT development libraries and headers
ii  libnvinfer-samples                                       4.0.4-1+cuda9.0                              amd64        TensorRT samples and documentation
ii  libnvinfer4                                              4.0.4-1+cuda9.0                              amd64        TensorRT runtime libraries
ii  python-libnvinfer                                        4.0.4-1+cuda9.0                              amd64        Python bindings for TensorRT
ii  python-libnvinfer-dev                                    4.0.4-1+cuda9.0                              amd64        Python development package for TensorRT
ii  python-libnvinfer-doc                                    4.0.4-1+cuda9.0                              amd64        Documention and samples of python bindings for TensorRT
ii  tensorrt                                                 3.0.4-1+cuda9.0                              amd64        Meta package of TensorRT
ii  uff-converter-tf                                         4.0.4-1+cuda9.0                              amd64        UFF converter for TensorRT package

  1. So for deployment on a DPX2, native(C++) TensorRT3 has to be used?

  2. Yes, this is met on a x86 Linux machine.

px2host@px2host1:~$ uname -a
Linux px2host1 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  1. I am using Tensorflow for modelling and training a network and am using a DXP2 for deployment. Is there an available workflow for the same?

Thanks,
Serin

4

Hi serinvarghese,
You can use the following workflow to deploy Tensorflow model on DPX2.

  1. Tensorflow model → UFF model(on Host)
  2. Copy UFF file to PX2
  3. UFF model → tensorRT engine(on DPX2) (Refer sampleUFFMnist sample)
  4. Load tensorRT engine using C++ API on DPX2 for inferencing (on DPX2)
5

Hello SivaRamaKrishna,

This simplifies a lot of my queries. I have now successfully completed the first 3 steps.

Which is the sample code/documentation for the 4th step?

Thanks,
Serin

6

For the earlier error, it was caused due to the multiple locations of TensorRT installations. I uninstalled all the versions of TensorRT, reinstalled the latest one and changed the LD_LIBRARY_PATH in ~/.bashrc to point to the new installation path. I no longer face the errors.

Hopefully should help someone!

Thanks,
Serin

7

I want to convert my tensorflow faster-rcnn model to TensorRT, but the graph of tensorflow model has some ops:switch,assert…,they can’t be converted by the function:convert_tf2numpy_const_node(cls, tf_node),because invalid dtype or non data value.
so , what should I do for the task?

8

I found the reseaon:some tensorflow types can’t be converted to numpy dtype
def convert_tf2numpy_dtype(cls, dtype):
dt = [‘invalid’, ‘f4’, ‘f8’, ‘i4’, ‘u1’, ‘i2’, ‘i1’, ‘S’,
‘c8’, ‘i8’, ‘b’, ‘qi1’, ‘qu1’, ‘qi4’, ‘bf2’,
‘qi2’, ‘qu2’, ‘u2’, ‘c16’, ‘f2’, ‘r’]
return np.dtype(dt[dtype])
such as: DT_RESOURCE(as ‘r’),how to resolve it?

9

Hi,

Question:

  1. Shouldn’t step 2 be done on the DPX2 using C++ API? As the TensorRT optimizations will be specific to the device that runs it.

  2. I read in some Jetson Forums that the .UFF file has to be copied to the device. I am assuming that it is similar for the DPX2 as well. If no, is there an example like sampleMNIST.cpp for importing the .engine file?

Thanks

10

Hi serinvarghese,
My apologies for the confusion. Please follow the below steps

  1. Tensorflow model → UFF model on Host (Refer Developer Guide :: NVIDIA Deep Learning TensorRT Documentation)
  2. Copy UFF file to PX2
  3. UFF model → tensorRT engine on DPX2 (Refer sampleUFFMnist TensorRT sample)
  4. Load tensorRT engine using C++ API on DPX2 for inferencing
11

It worked for me. :)

12

Hi,
has the tensorRT engine creation for .uff models (TensorFlow) also the ability to do INT8 calibration? I have a TF model with custom layers and want to deploy it to the PX2 by using IN8 precision. If it is possible, how would the work-flow look like?
Thanks

13

Dear maximillian,
Can you check the above mentioned work flow?

14

Thanks for your reply. I understand the workflow and what needs to be done. But I miss the details.

Are plugin layers supported by the .uff model parser?
Is INT8 calibration supported for the engine of a .uff model on PX2?
Which steps can be done in Python, which have to be implemented in C++?

I’m using TensorRT 4 RC and DriveWorks 5.0.5.0b

Thanks.

15

@sivaramakrishan Step 1 states to convert .pb model to .uff model, but the documentation you mentioned doesn’t tell how to serialize this model to put it onto PX2. Can i just pickle/unpickle the file between host and device?

16

Dear atui.dhingraeunh8,
You can copy the uff file to PX2 and generate tensort model file on PX2.

17

@sivaramakrishna
Yes, but from what the documentation has, it generates the uff_model within Python environment, I want to export the .uff file. I found a solution here,

https://devtalk.nvidia.com/default/topic/1025246/jetson-tx2/where-is-convert-to-uff/post/5222473/#5222473

But for some reason, it doesn’t generate the ‘model.uff’ file

18

Dear dhingratul,
Could you please share network details, error log and the steps you have followed.

19

This is resolved.

https://devtalk.nvidia.com/default/topic/1025246/jetson-tx2/where-is-convert-to-uff/post/5253241/#5253241

20

@sivaRamaKrishna Following your steps from #4, I created a .uff file for a tensorflow model(not leNet) on MNIST. When I use sampleUFFMnist with the Input, Output(Line 266, 267) from my network, with the newly generated .uff file, it throws me an error.

ERROR: BiasAdd: kernel weights has count 800 but 22400 was expected
ERROR: Add: kernel weights has count 3211264 but 458752 was expected
ERROR: sample_uff_mnist: Unable to create engine
ERROR: sample_uff_mnist: Model load failed

Following is the .pbtxt file generated with uff conversion

version: 1
descriptor_core_version: 1
descriptors {
  id: "tensorflow_extension"
  version: 1
}
graphs {
  id: "main"
  nodes {
    id: "Output/output"
    inputs: "Output/MatMul"
    inputs: "bc4"
    operation: "Binary"
    fields {
      key: "func"
      value {
        s: "add"
      }
    }
  }
  nodes {
    id: "bc4"
    operation: "Const"
    fields {
      key: "dtype"
      value {
        dtype: DT_FLOAT32
      }
    }
    fields {
      key: "shape"
      value {
        i_list {
          l: 10
        }
      }
    }
    fields {
      key: "values"
      value {
        ref: "weights_bc4"
      }
    }
  }
  nodes {
    id: "Output/MatMul"
    inputs: "Relu_2"
    inputs: "out"
    operation: "FullyConnected"
    fields {
      key: "inputs_orders"
      value {
        ref: "orders_NC_CK"
      }
    }
  }
  nodes {
    id: "out"
    operation: "Const"
    fields {
      key: "dtype"
      value {
        dtype: DT_FLOAT32
      }
    }
    fields {
      key: "shape"
      value {
        i_list {
          l: 1024
          l: 10
        }
      }
    }
    fields {
      key: "values"
      value {
        ref: "weights_out"
      }
    }
  }
  nodes {
    id: "Relu_2"
    inputs: "Add"
    operation: "Activation"
    fields {
      key: "func"
      value {
        s: "relu"
      }
    }
  }
  nodes {
    id: "Add"
    inputs: "MatMul"
    inputs: "bc3"
    operation: "Binary"
    fields {
      key: "func"
      value {
        s: "add"
      }
    }
  }
  nodes {
    id: "bc3"
    operation: "Const"
    fields {
      key: "dtype"
      value {
        dtype: DT_FLOAT32
      }
    }
    fields {
      key: "shape"
      value {
        i_list {
          l: 1024
        }
      }
    }
    fields {
      key: "values"
      value {
        ref: "weights_bc3"
      }
    }
  }
  nodes {
    id: "MatMul"
    inputs: "Reshape_1"
    inputs: "wd1"
    operation: "FullyConnected"
    fields {
      key: "inputs_orders"
      value {
        ref: "orders_NC_CK"
      }
    }
  }
  nodes {
    id: "wd1"
    operation: "Const"
    fields {
      key: "dtype"
      value {
        dtype: DT_FLOAT32
      }
    }
    fields {
      key: "shape"
      value {
        i_list {
          l: 3136
          l: 1024
        }
      }
    }
    fields {
      key: "values"
      value {
        ref: "weights_wd1"
      }
    }
  }
  nodes {
    id: "Reshape_1"
    inputs: "MaxPool_1"
    inputs: "Reshape_1/shape"
    operation: "Reshape"
  }
  nodes {
    id: "Reshape_1/shape"
    operation: "Const"
    fields {
      key: "dtype"
      value {
        dtype: DT_INT32
      }
    }
    fields {
      key: "shape"
      value {
        i_list {
          l: 2
        }
      }
    }
    fields {
      key: "values"
      value {
        ref: "weights_Reshape_1/shape"
      }
    }
  }
  nodes {
    id: "MaxPool_1"
    inputs: "Relu_1"
    operation: "Pool"
    fields {
      key: "func"
      value {
        s: "max"
      }
    }
    fields {
      key: "implicit_padding"
      value {
        s: "same"
      }
    }
    fields {
      key: "inputs_orders"
      value {
        ref: "orders_N+C"
      }
    }
    fields {
      key: "kernel"
      value {
        i_list {
          l: 2
          l: 2
        }
      }
    }
    fields {
      key: "strides"
      value {
        i_list {
          l: 2
          l: 2
        }
      }
    }
  }
  nodes {
    id: "Relu_1"
    inputs: "BiasAdd_1"
    operation: "Activation"
    fields {
      key: "func"
      value {
        s: "relu"
      }
    }
  }
  nodes {
    id: "bc2"
    operation: "Const"
    fields {
      key: "dtype"
      value {
        dtype: DT_FLOAT32
      }
    }
    fields {
      key: "shape"
      value {
        i_list {
          l: 64
        }
      }
    }
    fields {
      key: "values"
      value {
        ref: "weights_bc2"
      }
    }
  }
  nodes {
    id: "BiasAdd_1"
    inputs: "Conv2D_1"
    inputs: "bc2"
    operation: "Binary"
    fields {
      key: "func"
      value {
        s: "add"
      }
    }
  }
  nodes {
    id: "Conv2D_1"
    inputs: "MaxPool"
    inputs: "wc2"
    operation: "Conv"
    fields {
      key: "implicit_padding"
      value {
        s: "same"
      }
    }
    fields {
      key: "inputs_orders"
      value {
        ref: "orders_N+C_+CK"
      }
    }
    fields {
      key: "strides"
      value {
        i_list {
          l: 1
          l: 1
        }
      }
    }
  }
  nodes {
    id: "wc2"
    operation: "Const"
    fields {
      key: "dtype"
      value {
        dtype: DT_FLOAT32
      }
    }
    fields {
      key: "shape"
      value {
        i_list {
          l: 5
          l: 5
          l: 32
          l: 64
        }
      }
    }
    fields {
      key: "values"
      value {
        ref: "weights_wc2"
      }
    }
  }
  nodes {
    id: "MaxPool"
    inputs: "Relu"
    operation: "Pool"
    fields {
      key: "func"
      value {
        s: "max"
      }
    }
    fields {
      key: "implicit_padding"
      value {
        s: "same"
      }
    }
    fields {
      key: "inputs_orders"
      value {
        ref: "orders_N+C"
      }
    }
    fields {
      key: "kernel"
      value {
        i_list {
          l: 2
          l: 2
        }
      }
    }
    fields {
      key: "strides"
      value {
        i_list {
          l: 2
          l: 2
        }
      }
    }
  }
  nodes {
    id: "Relu"
    inputs: "BiasAdd"
    operation: "Activation"
    fields {
      key: "func"
      value {
        s: "relu"
      }
    }
  }
  nodes {
    id: "bc1"
    operation: "Const"
    fields {
      key: "dtype"
      value {
        dtype: DT_FLOAT32
      }
    }
    fields {
      key: "shape"
      value {
        i_list {
          l: 32
        }
      }
    }
    fields {
      key: "values"
      value {
        ref: "weights_bc1"
      }
    }
  }
  nodes {
    id: "BiasAdd"
    inputs: "Conv2D"
    inputs: "bc1"
    operation: "Binary"
    fields {
      key: "func"
      value {
        s: "add"
      }
    }
  }
  nodes {
    id: "Conv2D"
    inputs: "Reshape"
    inputs: "wc1"
    operation: "Conv"
    fields {
      key: "implicit_padding"
      value {
        s: "same"
      }
    }
    fields {
      key: "inputs_orders"
      value {
        ref: "orders_N+C_+CK"
      }
    }
    fields {
      key: "strides"
      value {
        i_list {
          l: 1
          l: 1
        }
      }
    }
  }
  nodes {
    id: "wc1"
    operation: "Const"
    fields {
      key: "dtype"
      value {
        dtype: DT_FLOAT32
      }
    }
    fields {
      key: "shape"
      value {
        i_list {
          l: 5
          l: 5
          l: 1
          l: 32
        }
      }
    }
    fields {
      key: "values"
      value {
        ref: "weights_wc1"
      }
    }
  }
  nodes {
    id: "Reshape"
    inputs: "Input/Placeholder"
    inputs: "Reshape/shape"
    operation: "Reshape"
  }
  nodes {
    id: "Reshape/shape"
    operation: "Const"
    fields {
      key: "dtype"
      value {
        dtype: DT_INT32
      }
    }
    fields {
      key: "shape"
      value {
        i_list {
          l: 4
        }
      }
    }
    fields {
      key: "values"
      value {
        ref: "weights_Reshape/shape"
      }
    }
  }
  nodes {
    id: "Input/Placeholder"
    operation: "Input"
    fields {
      key: "dtype"
      value {
        dtype: DT_FLOAT32
      }
    }
    fields {
      key: "shape"
      value {
        i_list {
        }
      }
    }
  }
  nodes {
    id: "MarkOutput_0"
    inputs: "Output/output"
    operation: "MarkOutput"
  }
}
referenced_data {
  key: "orders_N+C"
  value {
    dim_orders_list {
      l {
        orders {
          key: -1
          value {
            l: 0
            l: 2
            l: 2147483647
            l: 1
          }
        }
      }
    }
  }
}
referenced_data {
  key: "orders_N+C_+CK"
  value {
    dim_orders_list {
      l {
        orders {
          key: -1
          value {
            l: 0
            l: 2
            l: 2147483647
            l: 1
          }
        }
      }
      l {
        orders {
          key: -1
          value {
            l: 2
            l: 2147483647
            l: 1
            l: 0
          }
        }
      }
    }
  }
}
referenced_data {
  key: "orders_NC_CK"
  value {
    dim_orders_list {
      l {
        orders {
          key: -1
          value {
            l: 0
            l: 1
          }
        }
      }
      l {
        orders {
          key: -1
          value {
            l: 1
            l: 0
          }
        }
      }
    }
  }
}
referenced_data {
  key: "weights_Reshape/shape"
  value {
    blob: "\377\377\377\377\034\000\000\000\034\000\000\000\001\000\000\000"
  }
}
referenced_data {
  key: "weights_Reshape_1/shape"
  value {
    blob: "\377\377\377\377@\014\000\000"
  }
}
referenced_data {
  key: "weights_bc1"
  value {
    blob: "(...128 bytes skipped...)"
  }
}
referenced_data {
  key: "weights_bc2"
  value {
    blob: "(...256 bytes skipped...)"
  }
}
referenced_data {
  key: "weights_bc3"
  value {
    blob: "(...4096 bytes skipped...)"
  }
}
referenced_data {
  key: "weights_bc4"
  value {
    blob: "(...40 bytes skipped...)"
  }
}
referenced_data {
  key: "weights_out"
  value {
    blob: "(...40960 bytes skipped...)"
  }
}
referenced_data {
  key: "weights_wc1"
  value {
    blob: "(...3200 bytes skipped...)"
  }
}
referenced_data {
  key: "weights_wc2"
  value {
    blob: "(...204800 bytes skipped...)"
  }
}
referenced_data {
  key: "weights_wd1"
  value {
    blob: "(...12845056 bytes skipped...)"
  }
}