Is zero-padding not supporting in conv with cuDNN v2??? I got error CUDNN_STATUS_BAD_PARAM

I already posted the same issue in caffe git repository.


Hi!

I got an error when I use following conditions;

  1. use cuDNN v2
  2. use conv layer with
    2.1) bottom size < kernel size (either width or height)
    2.2) bottom size + pad > kernel size (either width or height)

F0525 17:36:32.711757 4106 cudnn_conv_layer.cu:53] Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM
from

CUDNN_CHECK(cudnnGetConvolutionForwardWorkspaceSize(handle_[g],
bottom_descs_[i],
filter_desc_,
conv_descs_[i],
top_descs_[i],
algo,
&workspaceSizeInBytes_temp));

Some layers have to be defined with above layer conditions, especially when I want to use inception layer in googlenet.

I confirmed the same error condition with CIFAR-10 example by changing conv3 from “bottom size: 8 / pad: 2, kernel: 5” to “bottom size: 8 / pad: 5, kernel: 10”.

If I compile without cuDNN options, i.e. use conv_layer.cu, it works.

It seems like a bug in cuDNN v2.

One of the failure case I tested is as follows, (it has been worked with cuDNN v1 successfully)

workspace_limit_bytes: 5761
bottom_desc → n: 96, c: 160, h: 2, w: 2, nStride: 640, cStride: 4, hStride: 2, wStride: 1, dataType: 0
top_desc → n: 96, c: 320, h: 2, w: 2, nStride: 1280, nStride: 4, hStride: 2, wStride: 1, dataType: 0
conv_desc → pad_h: 1, pad_w: 1, u: 1, v: 1, upscale_x: 1, upscale_y: 1, mode: 1
filter_desc → k: 320, c: 160, kernel_h: 3, kernel_w: 3, dataType: 0

Even if I manually set workspaceSizeInBytes with large value and directly run cudnnConvolutionForward, I got the same error.

CUDNN_CHECK(cudnnConvolutionForward(handle_[g],
cudnn::dataType::one,
bottom_descs_[i], bottom_data + bottom_offset_ * g,
filter_desc_, weight + weight_offset_ * g,
conv_descs_[i],
algo, workspace, workspaceSizeInBytes,
cudnn::dataType::zero,
top_descs_[i], top_data + top_offset_ * g));
(I set workspaceSizeInBytes = 11522 with above sample case)

Is there anyone can help me to fix this issue?

I have the same issue try to run deepdream dreamify.py script
with cudnn v2 on Tegra4Linux (based on ubuntu 14.04).

I tried to play a bit with the kernel size and other paramters…
But if I for example change kernel_size to 8 instead of 7 I got
" net.cpp:860] Cannot copy param 0 weights from layer ‘conv1/7x7_s2’; shape mismatch. Source param shape is 64 3 7 7 (9408); target param shape is 64 3 8 8 (12288)"

It also says
“To learn this layer’s parameters from scratch rather than copying from a saved net, rename the layer.”

But dont how to do this… How to change the layer, and do I really need to do this?
Anybody solved this issue in the meanwhile?

Without cudnn all works fine, gpu & cpu computing…
I tried the same thing a few weeks ago, and hoped it would be fixed, but it don`t seem like that. And now is cudnn v3 released, so I dont think this will be fixed inside cudnn v2 anymore…

I was not able to solve this problem for the last week…

In the meanwhile I implemented a fallback logic via a script. If bvlc_googlenet is choosen, a caffe version without cuda just using gpu is used. If another dataset (for ex. bvlc_alexnet which works with the most included layers) is selected, a caffe version compiled with USE_CUDNN enabled will be taken.

I will try to improve that with the use of cudnn v1 later. People seems to work successfully with that version…

Have you tried cudnn v3? It has been fixed