My system parameters:
OS: Windows 10 64bit
TensorFlow GPU: 1.5.0 (installed with pip)
Python Version: 3.6.4
IDE : Spyder 3.2.6
CUDA: v9.0
CuDNN: v7.0.5 for CUDA 9.0
GPU: GeForce GTX 960M 4GB
NVIDIA drivers: 397.64
So, I was trying to run some CNN (and also CapsNet where the first layer is just a convolution layer). And it keeps crushing with the error below. From what I found on the internet I decided to reinstall cuda and cudnn. And the code with CNN worked. Though capsnet still didn’t. I googled more and found out that cuda reset your drivers after installation, so, I needed to update my drivers. I had something like 380 or 384. Don’t remember exactly. I installed 397. And now they both don’t work again. I’m desperate. I don’t know what to do anymore.
The CNN code works fine on CPU though.
Someone also suggested to delete .nv folder in home directory. I couldn’t find that .nv folder.
Error given by code with CNN:
2018 15:50:40.136802: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2018 15:50:40.925004: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1105] Found device 0 with properties:
name: GeForce GTX 960M major: 5 minor: 0 memoryClockRate(GHz): 1.176
pciBusID: 0000:01:00.0
totalMemory: 4.00GiB freeMemory: 3.34GiB
2018 15:50:40.930269: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) ‑> (device: 0, name: GeForce GTX 960M, pci bus id: 0000:01:00.0, compute capability: 5.0)
2018 15:50:47.834229: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:50:47.866263: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:50:47.905931: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:50:47.907669: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:50:47.909351: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:50:47.911059: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:50:48.585303: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2018 15:50:48.586924: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
2018 15:50:48.588290: F C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\kernels\conv_ops.cc:717] Check failed: stream‑>parent()‑>GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &algorithms)
2018 15:53:45.749902: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2018 15:53:46.246273: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1105] Found device 0 with properties:
name: GeForce GTX 960M major: 5 minor: 0 memoryClockRate(GHz): 1.176
pciBusID: 0000:01:00.0
totalMemory: 4.00GiB freeMemory: 3.34GiB
2018 15:53:46.248497: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) ‑> (device: 0, name: GeForce GTX 960M, pci bus id: 0000:01:00.0, compute capability: 5.0)
2018 15:53:49.372376: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:53:49.373772: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:53:49.377767: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:53:49.379147: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:53:49.380568: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:53:49.382145: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_blas.cc:444] failed to create cublas handle: CUBLAS_STATUS_ALLOC_FAILED
2018 15:53:49.646183: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2018 15:53:49.647436: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
2018 15:53:49.648615: F C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\kernels\conv_ops.cc:717] Check failed: stream‑>parent()‑>GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &algorithms)
Error given by capsnet code:
2018 15:49:02.623151: W C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\common_runtime\bfc_allocator.cc:277] *****************************************___________________________________________________________
2018 15:49:02.624700: W C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\framework\op_kernel.cc:1198] Resource exhausted: OOM when allocating tensor with shape[24,6,6,3,3,32,32,4,4,4] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
2018 16:22:12.617066: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2018 16:22:13.091828: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1105] Found device 0 with properties:
name: GeForce GTX 960M major: 5 minor: 0 memoryClockRate(GHz): 1.176
pciBusID: 0000:01:00.0
totalMemory: 4.00GiB freeMemory: 3.34GiB
2018 16:22:13.094135: I C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\common_runtime\gpu\gpu_device.cc:1195] Creating TensorFlow device (/device:GPU:0) ‑> (device: 0, name: GeForce GTX 960M, pci bus id: 0000:01:00.0, compute capability: 5.0)
2018 16:23:36.501624: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2018 16:23:36.504510: E C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\stream_executor\cuda\cuda_dnn.cc:352] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
2018 16:23:36.507233: F C:\tf_jenkins\workspace\rel‑win\M\windows‑gpu\PY\36\tensorflow\core\kernels\conv_ops.cc:717] Check failed: stream‑>parent()‑>GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &algorithms)