System hangs when running any cuda code with kworker blocked message

I am using an Ubuntu Server 16.04 and have installed cuda toolkit 9.0, driver 384.98 and cuDNN 7.0. The devicequery program in NVIDIA samples returns with ‘passed’. However, when I run the mnistcudnn from cuDNN samples, my system hangs after a certain point in the code. The same is the case when running any other code which requires GPU. No keyboard interrupt works. I have to open another session from a remote machine and find the error message. I ran ‘dmesg’ and got the following:

[drm] Initialized drm 1.1.0 20060810
[174562.182545] nvidia-nvlink: Nvlink Core is being initialized, major device number 244
[174562.183033] vgaarb: device changed decodes: PCI:0000:03:00.0,olddecodes=none,decodes=none:owns=io+mem
[174562.183259] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 384.98 Thu Oct 26 15:16:01 PDT 2017 (using threaded interrupts)
[174562.185201] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 384.98 Thu Oct 26 14:41:13 PDT 2017
[174562.186203] [drm] [nvidia-drm] [GPU ID 0x00000300] Loading driver
[174598.215585] NVRM: Your system is not currently configured to drive a VGA console
on the primary VGA device. The NVIDIA Linux graphics driver
requires the use of a text-mode VGA console. Use of other console
drivers including, but not limited to, vesafb, may result in
corruption and stability problems, and is not supported.
[174648.998010] nvidia-uvm: Loaded the UVM driver in 8 mode, major device number 243
[175080.288092] INFO: task kworker/10:1:140 blocked for more than 120 seconds.

[175080.288149] Tainted: P OE 4.4.0-98-generic #121-Ubuntu
[175080.288194] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[175080.288244] kworker/10:1 D ffff880ff48a7b68 0 140 2 0x00000000
[175080.288586] Workqueue: events os_execute_work_item [nvidia]
[175080.288590] ffff880ff48a7b68 0000000000000000 ffff880ff8f56200 ffff880ff52ce200
[175080.288594] ffff880ff48a8000 ffff881ff68309e8 ffff880ff52ce200 ffff880ff48a7e10
[175080.288598] ffff880ff8382ec8 ffff880ff48a7b80 ffffffff81840585 7fffffffffffffff
[175080.288602] Call Trace:
[175080.288612] [] schedule+0x35/0x80
[175080.288619] [] schedule_timeout+0x1b5/0x270
[175080.288623] [] __down+0x7f/0xd0
[175080.288632] [] ? do_gettimeofday+0x29/0x90
[175080.288637] [] down+0x41/0x50
[175080.288838] [] os_acquire_semaphore+0x37/0x40 [nvidia]
[175080.289033] [] os_acquire_mutex+0xe/0x10 [nvidia]
[175080.289359] [] _nv031416rm+0x5c/0x120 [nvidia]
[175080.289824] [] ? _nv007008rm+0x38/0x120 [nvidia]
[175080.290121] [] ? _nv001138rm+0x65/0xd0 [nvidia]
[175080.290417] [] ? rm_execute_work_item+0x49/0xc0 [nvidia]
[175080.290615] [] ? os_alloc_mem+0x71/0xf0 [nvidia]
[175080.290815] [] ? os_execute_work_item+0x46/0x70 [nvidia]
[175080.290822] [] ? process_one_work+0x165/0x480
[175080.290826] [] ? worker_thread+0x4b/0x4c0
[175080.290830] [] ? process_one_work+0x480/0x480
[175080.290833] [] ? kthread+0xe5/0x100
[175080.290836] [] ? kthread_create_on_node+0x1e0/0x1e0
[175080.290842] [] ? ret_from_fork+0x3f/0x70
[175080.290845] [] ? kthread_create_on_node+0x1e0/0x1e0
[175080.290856] INFO: task kworker/10:2:297 blocked for more than 120 seconds.
[175080.290902] Tainted: P OE 4.4.0-98-generic #121-Ubuntu
[175080.290944] “echo 0 > /proc/sys/kernel/hung_task_timeout_secs” disables this message.
[175080.290993] kworker/10:2 D ffff880ff3fcfb68 0 297 2 0x00000000

and so on. The main issue I feel is the message in green, which I also see in the original terminal.

I cannot even reboot my system. I have to force restart the system.

Can anyone please help me with this issue.