I exported a model to ONNX from pytorch 1.0, and tried to load it to tensorRT using:
def build_engine_onnx(model_file):
with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.OnnxParser(network, TRT_LOGGER) as parser:
builder.max_workspace_size = common.GiB(1)
# Load the Onnx model and parse it in order to populate the TensorRT network.
with open(model_file, 'rb') as model:
parser.parse(model.read())
return builder.build_cuda_engine(network)
build_engine_onnx(model_path)
I have received the following error:
ERROR: Network must have at least one output.
When I tried to print the model, I get:
output {
name: "output1"
type {
tensor_type {
elem_type: FLOAT
shape {
dim {
dim_value: 1
}
dim {
dim_value: 17
}
dim {
dim_value: 56
}
dim {
dim_value: 64
}
}
}
}
}
}
opset_import {
version: 9
}
So it seems there is an output layer, but the opset is 9.
How can I resolve this?
Can you share the onnx file with us? You can DM me if you’d like to keep the file off of the forum.
Also, can you provide details on the platforms you are using?
Linux distro and version
GPU type
nvidia driver version
CUDA version
CUDNN version
Python version [if using python]
Tensorflow version
TensorRT version
I’m using Ubuntu 18.04, geforce 1070 (mobile), nvidia driver 410, cuda 10, cudnn 7.4.1, python 3.6, tensorRT 5.0.2.6.
I’ve sent the ONNX by a private message.
I have created the file using the ONNX using pyTorch 1.0 (also tried 0.4.1, same errors).
I have imported the file using tensorRT ONNX parser (the build-in one).
It seems that there is an issue in the convertor process that’s not properly flagging the output tensor. Engineering is still investigating.
In the meantime, a work-around is if you manually flag it with the API, i.e mark the output tensor of the last layer of the network as the output of the network with:
Thanks. I have modified the function build_engine_onnx to be:
def build_engine_onnx(model_file):
with trt.Builder(TRT_LOGGER) as builder, builder.create_network() as network, trt.OnnxParser(network, TRT_LOGGER) as parser:
builder.max_workspace_size = common.GiB(1)
# Load the Onnx model and parse it in order to populate the TensorRT network.
with open(model_file, 'rb') as model:
parser.parse(model.read())
network.mark_output(network.get_layer(network.num_layers - 1).get_output(0))
return builder.build_cuda_engine(network)
The number of detected network layers is only 12 (network.num_layers=12), however, the ONNX file contains ~1000 entries.
Does that make sense?
@bloodymeli
Seems like your ONNX model is currently not supported within TensorRT.
Specifically:
(trt) kevinchen@kevinch-desktop:~/Downloads/TensorRT-5.0.2.6/trt/work$ python test_onnx.py
Unsupported ONNX data type: INT64 (7)
[TensorRT] ERROR: Parameter check failed at: ../builder/Layers.cpp::ConstantLayer::1570, condition: weights.type == DataType::kFLOAT || weights.type == DataType::kHALF || weights.type == DataType::kINT32
[TensorRT] ERROR: (Unnamed Layer* 13) [Constant]: invalid dimensions
Unsupported ONNX data type: INT64 (7)
[TensorRT] ERROR: Parameter check failed at: ../builder/Layers.cpp::ConstantLayer::1570, condition: weights.type == DataType::kFLOAT || weights.type == DataType::kHALF || weights.type == DataType::kINT32
[TensorRT] ERROR: (Unnamed Layer* 14) [Gather]: indices input must be Int32
[TensorRT] ERROR: (Unnamed Layer* 16) [Constant]: invalid dimensions
[TensorRT] ERROR: Network must have at least one output
Layer 13 is unsupported, which causes the network to only be built up to the 12th layer. Looks like you are using INT64 type, which TRT does not support. If you can recreate your model using a supported datatype (float32, float 16, int32) you may be able to go further in the parsing step.
I have removed all the int64 layers, but I’m still unable to go past layer 13. I’ve sent you an updated ONNX file with no int64 layers. The total number of layers is 1117, conversation breaks at layer 13.
Also, is there a way to closely debug the conversation process? In other words, is parser implemented in python? I can’t find this library.
Is there any official examples for transferring the pytorch model to tensorrt? With the workflow, pytorch–>(torch.onnx.export)–>onnx–>tensorrt?
I encountered this problem also, ans i have made a try to export onnx style files with pytorch torch.onnx.export function, while many operations(such as lstm, view and unfold…) encountered this problem: Network must have at least one output.
Just wonder if the problem is solved on your side? I had the same problem. The parser failed to parse the whole network, it breaks at a certain layer. Thanks.
Same problem. Why not TensorRT supports parser for turning PyTorch model to TensorRT model directly. There are so many problems with the intermediary ONXX
Oh My God. Seems I cannot avoid using int64 in my PyTorch Model.
I am using PyTorch, and I want to use TensorRT for speeding up the inference of model.
This is my workflow: PyTorch model → torch.onnx.export the onnx model → tensorrt onnx_parser load the model.
Finally I get the same problem: INT64 is not supported.
This is my PyTorch Model definition: