Hi All,
I have developed and I am using regularly a CUDA/Thrust library for finite element simulations, for which performance is critical. The library has a testing framework that monitors execution time of most operations on a known dataset, ensuring that performance is consistent. In the last 2 years, through several CUDA and driver updates, performance of the library has been consistent (i.e. timing of operations never increased).
About 3 weeks ago I have upgraded to CUDA 8.0 and the testing framework passed all the tests, confirming that performance of the library was at least as before.
Suddenly, and without having recompiled the binary, I am experiencing significant drops in performance. Certain operations based on a mix of custom kernels / thrust that run in 2ms run now in 50ms, and others that run in 60ms run now in 170ms.
I cannot explain what has caused this degradation, as I have not recompiled the code, the configuration of the PC has not changed, and not other application is running which could interfere. I might have upgraded the CUDA driver, but I cannot tell if the upgrade was before or after I run the last testing. I have tried therefore to use an older driver version than the latest version (i.e. I have reinstalled ver 372.70 of August 30th), but to no avail, performance continues to be poor.
Has anyone ever experienced anything similar? Does anyone have a hint at what might be the cause?
The configuration I am using is as follows:
GPU: GTX Titan Black
OS: Windows 10 64bit Professional
CUDA: 8.0 64bit
Any comment is highly appreciated.
Thank you and Best Regards,
Andrea