The problem is any Cuda call returns with error code 30 (unknown error). This happens after suspend/wake and fixes after reboot. I tried driver versions 375.66 and 384.47, but the problem persists.
<b>$ uname -a</b>
Linux WS1005 4.8.0-58-generic #63~16.04.1-Ubuntu SMP Mon Jun 26 18:08:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
<b>$nvidia-smi</b>
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.47 Driver Version: 384.47 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 1070 Off | 00000000:01:00.0 On | N/A |
| 28% 31C P8 8W / 151W | 530MiB / 8113MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 3424 G /usr/lib/xorg/Xorg 354MiB |
| 0 3987 G compiz 93MiB |
| 0 4777 G ...el-token=1C9EB7F783F4F988F1752CC22A98C44A 79MiB |
+-----------------------------------------------------------------------------+
<b>$deviceQuery</b>
./deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
cudaGetDeviceCount returned 30
-> unknown error
Result = FAIL
<b>$python</b>
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import caffe
>>> caffe.set_device(0)
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0719 11:13:07.302003 5523 common.cpp:151] Check failed: error == cudaSuccess (30 vs. 0) unknown error
*** Check failure stack trace: ***