Why Assertion result failed in builder::supportedFormats when import BVLC caffe model?

We trained a model in BVLC caffe environment, and try to import to TensorRT 3.0 with hacked sampleGoogleNet code. but assert failed occure:

Begin parsing model...
End parsing model...
Begin building engine...
tftech_CaffeImp_debug: cudnnBuilder2.cpp:165: nvinfer1::builder::Format::Type nvinfer1::builder::supportedFormats(const nvinfer1::builder::Node&, nvinfer1::builder::Format::Type, const nvinfer1::cudnn::HardwareContext&): Assertion `result' failed.
Aborted (core dumped)

Any advice to what the Format::Type support problem?
hmdb51_tsn.zip (36.7 MB)

Hi haifengli, with BN-Inception prototxt, is the modified prototxt using batch normalization? Have you tried disabling/reverting certain features of the prototxt as to narrow down the source of the issue?

Also, which precision are you using - FP32, FP16, INT8 - and does changing precision effect the behavior?

We have reverted batch normalization, we using FP32 now, and we are sorry forgot to change the network name in .prototxt file, but it shouldn’t the cause? Is there any tool to dump the .caffemodel file content to text so we can verify the Format::Type information? we hope there is a work flow we can follow step by step to debug the TensorRT deploy problem. I think the CODE knows the reason as its assertion is failed for this topic.

I have changed the network name to “hello world”, and still has “supportedFormats” problem.
hmdb51_tsn_prototxt.zip (3.23 KB)

Hi,

Sorry for the late reply.

We have checked your model with tensorRT default engine.
The model can be inferenced correctly without error.

This error may come from user imeplementation.
Could you recheck your implementation?

nvidia@tegra-ubuntu:~/tensorrt/bin$ ./giexec --deploy=/home/nvidia/hmdb51_tsn_flow_inference.prototxt --output=fc-action --half2
deploy: /home/nvidia/hmdb51_tsn_flow_inference.prototxt
output: fc-action
half2
Input "data": 10x224x224
Output "fc-action": 51x1x1
name=data, bindingIndex=0, buffers.size()=2
name=fc-action, bindingIndex=1, buffers.size()=2
Average over 10 runs is 6.77557 ms.
Average over 10 runs is 6.75351 ms.
Average over 10 runs is 6.74139 ms.
Average over 10 runs is 6.76657 ms.
Average over 10 runs is 6.80655 ms.
Average over 10 runs is 6.75393 ms.
Average over 10 runs is 6.75662 ms.
Average over 10 runs is 6.7556 ms.
Average over 10 runs is 6.76429 ms.
Average over 10 runs is 6.76983 ms.

Thanks.

Hi, I type your command to my TensorRT 3.0 + CUDA 9.0 + cuDNN 7.0 + Tesla P4 + Driver 384.90 enviroment, it report:

admin@install-server:~/src/tensorrt$ ./bin/giexec --deploy=./data/detectnet/hmdb51_tsn_flow_inference.prototxt --output=fc-action --half2
deploy: ./data/detectnet/hmdb51_tsn_flow_inference.prototxt
output: fc-action
half2
Input "data": 10x224x224
Output "fc-action": 51x1x1
Half2 support requested on hardware without native FP16 support, performance will be negatively affected.
giexec: cudnnBuilder2.cpp:165: nvinfer1::builder::Format::Type nvinfer1::builder::supportedFormats(const nvinfer1::builder::Node&, nvinfer1::builder::Format::Type, const nvinfer1::cudnn::HardwareContext&): Assertion `result' failed.
Aborted (core dumped)

Then I removed --half2 from the command line and got this:

admin@install-server:~/src/tensorrt$ ./bin/giexec --deploy=./data/detectnet/hmdb51_tsn_flow_inference.prototxt --output=fc-action
deploy: ./data/detectnet/hmdb51_tsn_flow_inference.prototxt
output: fc-action
Input "data": 10x224x224
Output "fc-action": 51x1x1
giexec: cudnnBuilder2.cpp:165: nvinfer1::builder::Format::Type nvinfer1::builder::supportedFormats(const nvinfer1::builder::Node&, nvinfer1::builder::Format::Type, const nvinfer1::cudnn::HardwareContext&): Assertion `result' failed.
Aborted (core dumped)

It is still there. The driver 384.90 is install from launchpad, any more advice?

Hi,

Could you try to execute this model with TensorRT2.1?
We want to clarify this issue is from the environment or TensorRT only.

Thanks.

It worked when I run the giexec under TensorRT-2.1.2, below is the output:

output: fc-action
Input "data": 10x224x224
Output "fc-action": 51x1x1
name=data, bindingIndex=0, buffers.size()=2
name=fc-action, bindingIndex=1, buffers.size()=2
Average over 10 runs is 3.34805 ms.
Average over 10 runs is 3.35038 ms.
Average over 10 runs is 3.34715 ms.
Average over 10 runs is 3.34777 ms.
Average over 10 runs is 3.34344 ms.
Average over 10 runs is 3.33759 ms.
Average over 10 runs is 3.33077 ms.
Average over 10 runs is 3.3282 ms.
Average over 10 runs is 3.33197 ms.
Average over 10 runs is 3.33668 ms.

Thank you so much for debug this question with us. we can working on TensorRT 2.x for some time. Would you like to notice us if the TensorRT is ready to parse this .prototxt file? I will receive a email from NVIDIA DevTalk if there is a new reply.

Hi,

Thanks for your feedback.
We have reported this issue to the internal team. Will update information with you later.

Thanks.

Hi,

This issue is fixed in our next release(TensorRT 3 GA).
Please pay attention to our announcement.

Thanks.