Converting TF 2.0 saved model for TensorRT on Jetson Nano

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

I am trying to convert a TF 2.0 saved_model to tensorRT on the Jetson Nano.

The model was saved in TF 2.0.0. The nano has Jetpack 4.2.2 w/ TensorRT __ and Tensorflow 1.14 (that is the latest Tensorflow release for Jetson).

I have been following the instructions from here which describe how to convert a TF 2.0.0 saved_model into TensorRT.

Below is my code:

import tensorflow as tf
from tensorflow.python.compiler.tensorrt import trt_convert as trt
tf.enable_eager_execution()

converter = trt.TrtGraphConverterV2(input_saved_model_dir=input_saved_model_dir)
converter.convert()
converter.save(output_saved_model_dir)

saved_model_loaded = tf.saved_model.load(
    output_saved_model_dir, tags=[tag_constants.SERVING])
graph_func = saved_model_loaded.signatures[
    signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY]
frozen_func = convert_to_constants.convert_variables_to_constants_v2(
    graph_func)
def wrap_func(*args, **kwargs):
    # Assumes frozen_func has one output tensor
    return frozen_func(*args, **kwargs)[0]
output = wrap_func(input_data).numpy()

It seems to start converting successfully. However I get an KeyError: 'serving_default' error when it reaches the convert_to_tensor line. My complete printout is below found here, but the python traceback appears below. How can I fix this?

Thanks!

printout summary (complete printout here):

Traceback (most recent call last):
  File "tst.py", line 38, in <module>
    convert_savedmodel()
  File "tst.py", line 24, in convert_savedmodel
    converter.convert()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/compiler/tensorrt/trt_convert.py", line 956, in convert
    func = self._saved_model.signatures[self._input_saved_model_signature_key]
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/saved_model/signature_serialization.py", line 196, in __getitem__
    return self._signatures[key]
KeyError: 'serving_default'
2

Ok, I was able to fix this error by using the get_concrete_function to covert the model default call to a concrete call (based on the google tutorial for saved_model here: با استفاده از قالب SavedModel  |  TensorFlow Core(.

Here is my code:

# Concretize Functions
call = model.__call__.get_concrete_function(tf.TensorSpec(IN_SHAPE, tf.float32))

# convert to savedmodel
tf.saved_model.save(model, 'tst', signatures=call)
1 Like
3 
Also, for anyone trying this (especially for those working in NVIDIA), the example provided in the tf.tensorrt <a target='_blank' rel='noopener noreferrer' href='https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html#worflow-with-savedmodel'>documentation</a> is incorrect and missing important scoping statements (e.g. tag_constants should be tf.saved_model.tag_constants). It should be:

```
import tensorflow as tf
from tensorflow.python.compiler.tensorrt import trt_convert as trt
tf.enable_eager_execution()

converter = trt.TrtGraphConverterV2(input_saved_model_dir=input_saved_model_dir)
converter.convert()
converter.save(output_saved_model_dir)

saved_model_loaded = tf.saved_model.load(
    output_saved_model_dir, tags=[tf.saved_model.tag_constants.SERVING])
graph_func = saved_model_loaded.signatures[
    tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY]
frozen_func = trt.convert_to_constants.convert_variables_to_constants_v2(
    graph_func)
def wrap_func(*args, **kwargs):
    # Assumes frozen_func has one output tensor
    return frozen_func(*args, **kwargs)[0]
output = wrap_func(input_data).numpy()
```
1 Like
4

Thanks for posting the corrected scoping requirements rsandler00. Note that for me (in Tensorflow 2.0) I needed to use tf.saved_model.SERVING (not tf.saved_model.tag_constants.SERVING) and (tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY).

I now get the following error in my call output = wrap_func(input_data).numpy()

ValueError: All inputs to ConcreteFunctions must be Tensors; on invocation of pruned, the 0-th input ([[ 6.0191255e+00 … 2.6921434e-02 2.6448354e-02]]) was not a Tensor.

My input_data is a 26 column numpy array. How can I successfully load this in for inference? (and get a numpy array as output).

Thanks

1 Like