update:
the issue is not concat. it’s the padding params used in following pooling layer. remove the padding parameter works can pass this layer.
prototext is from NVcaffe’s inception_v3 model.
layer {
name: “3A/p4_pool”
type: “Pooling”
bottom: “conv6/bn”
top: “3A/p4_pool”
pooling_param {
pool: AVE
kernel_size: 3
pad: 1
}
}
input shape (1, 288, 35 ,35)
I tested with python API and giexec.
#################
run tensorrt with ngc docker 17.12.
model from NVCaffe’s inception_v3.
Issue
can’t handle the concat layer:
to reproduce:
giexec --deploy=deploy.prototxt --batch=1 --output=3A/concat
deploy: deploy.prototxt
batch: 1
output: 3A/concat
Input “data”: 3x299x299
Output “3A/concat”: 288x35x35
giexec: cudnnBuilder2.cpp:165: nvinfer1::builder::Format::Type nvinfer1::builder::supportedFormats(const nvinfer1::builder::Node&, nvinfer1::builder::Format::Type, const nvinfer1::cudnn::HardwareContext&): Assertion `result’ failed.
Aborted (core dumped)
Hi,
From your error, you may meet a known regression issue.
Please try TensorRT2.1 first to check if this error still occurs.
Thanks.
Hi,
This issue is fixed in our next release(TensorRT 3 GA).
Please pay attention to our announcement.
Thanks.
Hi AasstaLLL, thanks for your prompt reply.
Have you tested the tensorflow=>TensorRT path as well?
-Andy
Hi Andy,
Have you also met this issue with TensorFlow model?
This error occurs in the Caffe parser and should be independent of the TensorFlow-based use case.
Thanks.
Hi AastaLLL,
Yes. I tested the models shipping with NGC’s 17.12 TF images. looks like a pooling’s kernel size related issue.
Nvidia provides the great giexec. So I modified nvcnn.py to dump all models from nvcnn.py to prototxt to reuse accelerate the debug. I met following pooling related issue on nvcnn.py’s googlenet:
giexec --deploy=googlenet.prototxt --batch=1 --output=max0
the error messsage is: giexec: caffe/caffeParser.cpp:94: virtual nvinfer1::DimsHW CaffeParserPoolingDimsCallback::compute(nvinfer1::DimsHW, nvinfer1::DimsHW, nvinfer1::DimsHW, nvinfer1::DimsHW, nvinfer1::DimsHW, const char*): Assertion `input.h() + 2 * padding.h() >= kernel.h()’ failed.
the example input is:
input: “data”
input_shape {
dim:128
dim:3
dim:224
dim:224
}
layer {
name: “conv0”
type: “Convolution”
bottom: “data”
top: “conv0”
convolution_param {
num_output: 64
kernel_h: 7
kernel_w: 7
pad_h: 2
pad_w: 2
stride_h: 2
stride_w: 2
weight_filler {
type: “msra”
}
bias_filler {
type: “constant”
value: 0.0001
}
}
}
layer {
name: “conv0/RELU”
type: “ReLU”
bottom: “conv0”
top: “conv0”
}
layer {
name: “max0”
type: “Pooling”
bottom: “conv0”
top: “max0”
pooling_param {
pool: MAX
kernel_size: 3
stride: 2
}
}
The example input size to pooling layer here is 112x112 and should not trigger such assert failure.
if I change the kernel_size from 3 to 2. It can give the correct output. Please kindly review it.
Thanks!
-Andy
Hi,
From your log, you are using Caffe frameworks.
This issue is from Caffe to TensorRT parser and is fixed in TensorRT 3 GA.
So if your image is using older TensorRT version(including TensorRT 3 RC), it’s still possible to meet this issue.
Check TensorRT version via this command:
dpkg -l | grep TensorRT
TensorRT 3 GA should be:
ii libnvinfer-dev <b>4.0.1</b>-1+cuda9.0 amd64 TensorRT development libraries and headers
Thanks.
Hi AastaLLL,
My version should be 3GA. I’m using NCG docker images. following is the output from my working environment.
root@c8bc4228b1cc:/workspace# dpkg -l | grep TensorRT
ii libnvinfer-dev 4.0.1-1+cuda9.0 amd64 TensorRT development libraries and headers
ii libnvinfer-samples 4.0.1-1+cuda9.0 amd64 TensorRT samples and documentation
ii libnvinfer4 4.0.1-1+cuda9.0 amd64 TensorRT runtime libraries
ii onnx2trt 0.1.0 all ONNX parser for TensorRT 3.0
ii python-libnvinfer 4.0.1-1+cuda9.0 amd64 Python bindings for TensorRT
ii python-libnvinfer-dev 4.0.1-1+cuda9.0 amd64 Python development package for TensorRT
ii python-libnvinfer-doc 4.0.1-1+cuda9.0 amd64 Documention and samples of python bindings for TensorRT
ii tensorrt 3.0.1-1+cuda9.0 amd64 Meta package of TensorRT
root@c8bc4228b1cc:/workspace# gi
giexec giexec_debug gifmaker.py git git-receive-pack git-shell git-upload-archive git-upload-pack
root@c8bc4228b1cc:/workspace# whereis giexec
giexec: /opt/tensorrt/bin/giexec
root@c8bc4228b1cc:/workspace# ldd /opt/tensorrt/bin/giexec
linux-vdso.so.1 => (0x00007fff00b9d000)
libnvinfer.so.4 => /usr/lib/x86_64-linux-gnu/libnvinfer.so.4 (0x00007fdf3a66c000)
libnvparsers.so.4.0.1 => /usr/lib/x86_64-linux-gnu/libnvparsers.so.4.0.1 (0x00007fdf39f78000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fdf39d6f000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fdf39b6b000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fdf3994e000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007fdf395cb000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fdf392c2000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fdf390ac000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fdf38ce1000)
libcudnn.so.7 => /usr/lib/x86_64-linux-gnu/libcudnn.so.7 (0x00007fdf2784a000)
libcublas.so.9.0 => /usr/local/cuda-9.0/targets/x86_64-linux/lib/libcublas.so.9.0 (0x00007fdf24294000)
libcudart.so.9.0 => /usr/local/cuda-9.0/targets/x86_64-linux/lib/libcudart.so.9.0 (0x00007fdf24026000)
/lib64/ld-linux-x86-64.so.2 (0x00007fdf461dd000)
root@c8bc4228b1cc:/workspace#
Hi,
Thanks for this important feedback.
Let us check this issue internally and update information with you later.
Thanks and sorry for the inconvenience.
Hi Andy,
There is some miscommunication.
This fix will be available in our next TensorRT release for Jetson. (TensorRT-3 GA)
For the x86-based package, the fix will be available in our next release. (Not current TensorRT-3 GA release)
Please pay attention to our announcement for the udpate.
Thanks and sorry for the inconvenience.