Custom layer (PlugIn) - TensorRTC++ nvuffparser::IUffParser Vs. TensorRT C++ nvonnxparser::IParser

I have a problem to register a pluginV2 to the Onnx parser as opposed to the uff parser which successfully can be registered to it.

This is my configuration:
Windows 10
Pyton - 3.6.8
VS2015 x64
torch-1.1.0.dist-info
torchsummary-1.5.1.dist-info
torchvision-0.3.0.dist-info
Tensorflow Python\C++ (TF)- 1.9 (C++ version was built from sources)
TensorRT C++ (TRT) - 6.0.1.5
CuDNN - 7.6.3
CUDA - 9.0

I have two models:

• YoloV3 - Implemeted and trained via TF Python, Intended to be inferenced via TRT C++
• SegNet- Implemeted and trained via PyTorch, Intended to be inferenced via TRT C++

YoloV3 model includes one unsupported TF Upsample command which implemented as a TRT plugin by two ways (for good practice, in actually only the first method is used):

• nvinfer1::IPluginCreator, nvinfer1::IPluginV2 and REGISTER_TENSORRT_PLUGIN
• nvinfer1::IPluginFactory, public nvuffparser::IPluginFactoryExt, nvinfer1::IPluginExt and nvuffparser::IUffParser setPluginFactoryExt service

In both ways the PlugIn registry to the uff parser is working well.

I also edited the model with the graphsurgeon in order to replace the upsample name and remove some additional unnecessary nodes.
Additionally, just to make sure that it isn’t must to use graphsurgeon in order to declare on a layer as a plugin layer, I intentionally declared an additional pluging layer without any graphsurgeon manipulation.
I used the first method to do that and it works well, my plugin creator call (inherited from the nvinfer1::IPluginCreator) constructor and createPlugin methods are activated during the nvuffparser::IUffParser parse operation.

The PlugIn creator can be identified by the parser and its methods activated during the parsering process and the CUDA engine build process and finally inference is working well as expected with the plugin methods (enqueue …).

Now, I’m trying to do the same thing for the SegNet model.

The segNet PyTorch was converted to an Onnx format using the following commands

dummy_input = torch.randn(1, 32, 400, 400, device='cuda')
input_names = ["Input"]
output_names = ["Output"]
torch.onnx.export(model, dummy_input, "segNet.onnx", verbose=True, input_names=input_names,  output_names=output_names)

At the beginning the command torch.onnx.export was failed due to unsupported max_unpool2d command.
I updated the file symbolic.py inside the onnx directory of the torch package:

def max_unpool2d(g, self, indices, output_size):
     return g.op("max_unpool2d", self, indices, output_size)

After this update the operation torch.onnx.export starts to work properly without any errors and a segNet.onnx file was successfully generated.

Now I already know that I will need to implement a plugin for the unsupported max_unpool2d PyTorch operation (as I did for the upsample TF operation described above).
Also when I activated this operation:

auto parsed = m_onnxParser->parseFromFile(
     fileName.string().c_str(), static_cast<int>(nvinfer1::ILogger::Severity::kINFO));

I got the following report:
Input filename: …/…/…/…/Data/SegNet/Semantic_Segmentation/segNet.onnx
ONNX IR version: 0.0.4
Opset version: 9
Producer name: pytorch
Producer version: 1.1
Domain:
Model version: 0
Doc string:

WARNING: ONNX model has a newer ir_version (0.0.4) than this parser was built against (0.0.3).
……. Some good reports…….
WARNING: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
Successfully casted down to INT32.
While parsing node number 8 [Gather]:
ERROR: onnx2trt_utils.hpp:347 In function convert_axis:
[8] Assertion failed: axis >= 0 && axis < nbDims
Error code - 8
Error description - Assertion failed: axis >= 0 && axis < nbDims
Error source file - onnx2trt_utils.hpp
Error source line - 347
Error function - convert_axis
Error node – 8

I checked the Onnx parser code from the TensorRT OSS package and I realized that I need to implement my Gather operation as a plugin too.

I implemented the Gather plugin exactly as I implemented for the upsample plugin above, based on the first method (described above) and when I started to run, the constructor of my Gather plugin creator (inherited from the nvinfer1::IPluginCreator) was activated due to the REGISTER_TENSORRT_PLUGIN macro but the createPlugin method wasn’t activated during the nvonnxparser::IParser parseFromFile operation and still I got the same error report (described above)

Please advise how plugin shall be registered to the onnx parser?

Regard,

if you activate your Gather plugin through REGISTER_TENSORRT_PLUGIN macro, then it already register to tensorrt. can you post your gather plugin implementation? I suggest you check getFieldNames(), and getPluginName() see if they are return exactly right.

Hello,
Thank you for your quick response.

Actually, I thought that possibly this is my problem, that something with my plugin name isn’t correct.
I’m not sure that i know what is the right name of the Gather as it represented in the Onnx model.
When I print the Onnx model i can see this:

%192 : Long() = onnx::Gather[axis=0](%191, %190), scope: segnet/segnetDown2[down1]

So I thought that the layer name is “Gather” and this is the value that the getPluginName() shall return.

Is it correct?

I also tried to return “onnx::Gather”, but still it didn’t work well - the createPlugin() of my GatherPluginCreator class isn’t called during the nvonnxparser::IParser parseFromFile() operation.

How can I know what is the real Onnx layer name in order to select the right name for my plugin?
With Tensorflow\uff model it is easy to know that… due to the .pbtxt file that I can generate while creating the uff file.

Can you explain please how the method getFieldNames() shall be synchronized with the getPluginName()?

I cannot find how the getFieldNames() related to the plugin name, as I can see it provides only the plugin fields parameters which include their names but not the plugin name.

For now, my Gather plugin implementation isn’t full it include just the methods structure without the real implementation because at the beginning I just wanted to be sure that I can register it to the Onnx Parser operation.

This is my Gather plugin code:

#ifdef GATHER_PLUGINV2
				namespace Gather
				{					
					namespace
					{
						const char* GATHER_PLUGIN_VERSION{ "1" };
						const char* GATHER_PLUGIN_NAME{ "Gather"};
					} // namespace

					class GatherPlugin : public nvinfer1::IPluginV2
					{
					public:
						GatherPlugin()
						{}
						GatherPlugin(const int &indices, const int &axis) :
							m_indices(indices),
							m_axis(axis)
						{
							std::cout << "************************ GatherPlugin Constructor ***********************" << std::endl;
						}
						GatherPlugin(const void* data, const size_t &length)
						{
							std::cout << "************************ GatherPlugin Constructor ***********************" << std::endl;
						}
						~GatherPlugin()
						{
							std::cout << "************************ GatherPlugin Destructor ***********************" << std::endl;
						}

						/*Method of IPluginV2 class*/
						int getNbOutputs() const override
						{
							std::cout << "************************ GatherPlugin getNbOutputs ***********************" << std::endl;
							return 1;
						}

						/*Method of IPluginV2 class*/
						nvinfer1::Dims getOutputDimensions(int index, const nvinfer1::Dims* inputs, int nbInputDims) override
						{
							std::cout << "************************ GatherPlugin getOutputDimensions ***********************" << std::endl;

							nvinfer1::Dims outputDims;

							return outputDims;
						}

						/*Method of IPluginV2 class*/
						int initialize() override
						{
							std::cout << "************************ GatherPlugin initialize ***********************" << std::endl;

							return 0;
						}

						/*Method of IPluginV2 class*/
						void terminate() override
						{
							std::cout << "************************ GatherPlugin terminate ***********************" << std::endl;
						}
						size_t getWorkspaceSize(int maxBatchSize) const override
						{
							std::cout << "************************ GatherPlugin getWorkspaceSize ***********************" << std::endl;
							return 0;
						}
						int enqueue(
							int batchSize, const void* const* inputs, void** outputs, void* workspace, cudaStream_t stream) override
						{
							std::cout << "************************ GatherPlugin enqueue ***********************" << std::endl;

							return 0;
						}
						size_t getSerializationSize() const override
						{
							std::cout << "************************ GatherPlugin getSerializationSize ***********************" << std::endl;
							
							size_t totalSize = 2 * sizeof(int);

							return totalSize;
						}
						void serialize(void* buffer) const override
						{
							std::cout << "************************ GatherPlugin serialize ***********************" << std::endl;
						}
						bool supportsFormat(nvinfer1::DataType type, nvinfer1::PluginFormat format) const override
						{
							std::cout << "************************ GatherPlugin supportsFormat ***********************" << std::endl;
							return (type == nvinfer1::DataType::kINT32 && format == nvinfer1::PluginFormat::kNCHW);
						}

						void configureWithFormat(const nvinfer1::Dims* inputDims,
							int nbInputs,
							const nvinfer1::Dims* outputDims,
							int nbOutputs,
							nvinfer1::DataType type,
							nvinfer1::PluginFormat format,
							int maxBatchSize) override
						{
							std::cout << "************************ GatherPlugin configureWithFormat ***********************" << std::endl;
							assert(type == nvinfer1::DataType::kINT32 && format == nvinfer1::PluginFormat::kNCHW);
							m_dataType = type;
						}

						const char* getPluginType() const override
						{
							std::cout << "************************ GatherPlugin getPluginType ***********************" << std::endl;

							auto name = GATHER_PLUGIN_NAME;							
							return name;
						}
						const char* getPluginVersion() const override
						{
							std::cout << "************************ GatherPlugin getPluginVersion ***********************" << std::endl;

							auto version = GATHER_PLUGIN_VERSION;
							return version;
						}
						void destroy() override
						{
							std::cout << "************************ GatherPlugin destroy ***********************" << std::endl;
							delete this;
						}

						nvinfer1::IPluginV2* clone() const override
						{
							std::cout << "************************ GatherPlugin clone ***********************" << std::endl;
							return new GatherPlugin(m_indices, m_axis);
						}
						void setPluginNamespace(const char* pluginNamespace) override
						{
							std::cout << "************************ GatherPlugin setPluginNamespace ***********************" << std::endl;
							m_nameSpace = pluginNamespace;
						}
						const char* getPluginNamespace() const override
						{
							std::cout << "************************ GatherPlugin getPluginNamespace ***********************" << std::endl;
							return m_nameSpace.c_str();
						}

					private:
						std::string m_nameSpace;
						int m_indices;
						int m_axis;

						nvinfer1::DataType m_dataType;
					};

                                class BaseCreator : public nvinfer1::IPluginCreator
				{
				public:
					void setPluginNamespace(const char* libNamespace) override
                                        {
					     mNamespace = libNamespace;				
                                        }
					const char* getPluginNamespace() const override
                                        {
                                             return mNamespace.c_str();
                                        }

				protected:
					std::string mNamespace;
				};

					class GatherPluginCreator : public BaseCreator
					{
					public:
						GatherPluginCreator()
						{
							std::cout << "************************ 
 GatherPluginCreator Constructor ***********************" << std::endl;
							m_pluginAttributes.emplace_back(
								nvinfer1::PluginField("indices", nullptr, nvinfer1::PluginFieldType::kINT32, 1));
							m_pluginAttributes.emplace_back(
								nvinfer1::PluginField("axis", nullptr, nvinfer1::PluginFieldType::kINT32, 1));

							m_fieldsCollection.nbFields = m_pluginAttributes.size();
							m_fieldsCollection.fields = m_pluginAttributes.data();
						}

						~GatherPluginCreator()
						{
							std::cout << "************************ GatherPluginCreator Destructor ***********************" << std::endl;
						};

						const char* getPluginName() const override
						{
							std::cout << "************************ GatherPluginCreator getPluginName ***********************" << std::endl;

							auto name = GATHER_PLUGIN_NAME;
							return name;
						};

						const char* getPluginVersion() const override
						{
							std::cout << "************************ GatherPluginCreator getPluginVersion ***********************" << std::endl;
							
							auto version = GATHER_PLUGIN_VERSION;
							return version;
						};
						const nvinfer1::PluginFieldCollection* getFieldNames() override
						{
							std::cout << "************************ GatherPluginCreator getFieldNames ***********************" << std::endl;
							return &m_fieldsCollection;
						}
						nvinfer1::IPluginV2* createPlugin(const char* name, const nvinfer1::PluginFieldCollection* fc) override
						{
							std::cout << "************************ GatherPluginCreator createPlugin ***********************" << std::endl;
							
							std::string layerName(name);

							if (!(std::string::npos == layerName.find("Gather")))
							{
								const nvinfer1::PluginField* fields = fc->fields;

								for (int i = 0; i < fc->nbFields; ++i)
								{
									std::string fieldName(fields[i].name);

									/*In case the field name include "indices" it means that current field is the Gather indices field*/
									if (!(std::string::npos == fieldName.find("indices")))
									{
										assert(fields[i].type == nvinfer1::PluginFieldType::kINT32);
										m_indices = *(static_cast<const int*>(fields[i].data));
									}
									/*In case the field name include "axis" it means that current field is the Gather axis field*/
									if (!(std::string::npos == fieldName.find("axis")))
									{
										assert(fields[i].type == nvinfer1::PluginFieldType::kINT32);
										m_axis = *(static_cast<const int*>(fields[i].data));
									}
								}
							}

							return new GatherPlugin(m_indices, m_axis);
						};

						nvinfer1::IPluginV2* deserializePlugin(const char* name, const void* data, size_t length) override
						{
							std::cout << "************************ GatherPluginCreator deserializePlugin ***********************" << std::endl;
							return new GatherPlugin(data, length);
						};

					private:
						int m_indices;
						int m_axis;
						static nvinfer1::PluginFieldCollection m_fieldsCollection;
						static std::vector<nvinfer1::PluginField> m_pluginAttributes;
					};
                                        
                                        nvinfer1::PluginFieldCollection GatherPluginCreator::m_fieldsCollection{};
					std::vector<nvinfer1::PluginField> GatherPluginCreator::m_pluginAttributes;
					
					REGISTER_TENSORRT_PLUGIN(GatherPluginCreator);					
				}
				#endif /*GATHER_PLUGINV2*/

Thank you for your help

Hi orong13,

I haven’t done this myself, but from what I can tell, the REGISTER_TENSORRT_PLUGIN macro should be the way to go: https://github.com/NVIDIA/TensorRT/issues/6#issuecomment-503801263

Try to use one of the OSS plugins as a reference, such as: https://github.com/NVIDIA/TensorRT/blob/master/plugin/fcPlugin/fcPlugin.cu

Also maybe there is a name collision between your plugin “Gather” and ONNX’s built-in “Gather” op, I’m not sure if that’s possible, but you could try renaming your plugin to “CustomGather” or something to see if you notice anything different.

Hope this helps.

Hello,
As I described above I’m trying to use the REGISTER_TENSORRT_PLUGIN macro.
I know it was registered ok because my plugin constructor is activated by the Onnx parser operation but that’s all…the createPlugin method isn’t activeted so the plugin registration isn’t done correctly.

The OSS plugins examples are great but not enough…
Can you provide a full Onnx (not Uff) example which includes the logic that activate the parser and the plugin object methods as well?

I also asked this here:
https://github.com/NVIDIA/TensorRT/issues/6#issuecomment-570128426

regard,

Same problems for me as well.
Since I find class IPluginFactory in the namespace of both nvuffparser (NvUffParser.h) and nvcaffeparser1 (NvCaffeParser.h), while can’t find in NvOnnxParser.h , I just wonder whether OnnxParser support the pluginfactory in the newest TensorRT?

Why was PluginFactory removed from onnx-tensorrt for full dims support? (https://github.com/onnx/onnx-tensorrt/commit/2db2ae9ca6fb2387abb48efbd4d15d7c847e3bf9)

Hi,
Perhaps as a temporary workaround for onnx tensorrt 6 plugin issues,
A flow of pytorch->onnx->tensorflow->uff->tensorrt could work better.

Thanks

Regarding the removal of PluginFactory from onnx-tensorrt, documentation in https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html#example1_add_custom_layer_python

says :
“In previous versions of TensorRT, you had to implement the nvinfer1::IPluginFactory class to call the createPlugin method during deserialization. This is no longer necessary for plugins registered with TensorRT and added using addPluginV2.”
+
In the documentation , rather in uff context but relevant to onnx too:
"In previous versions of TensorRT, you had to implement the nvuffparser::IPluginFactoryExt and manually pass the plugin parameters to the createPlugin(…) function. Although this flow can still be exercised, it is no longer necessary with the new additions to the Plugin API. For more information, see:
IPluginV2Ext and IPluginCreator in the C++ API

IPluginV2Ext and IPluginCreator in the Python API"

Thanks