nvidia-persistenced: Failed to query NVIDIA devices.

Hi. I notice a nvidia failure during boot up in system log. OS is Ubuntu 16.04.

May 6 08:20:41 Eliot nvidia-persistenced: Started (677)
May 6 08:20:41 Eliot nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 121 has read and write permissions for those files.
May 6 08:20:41 Eliot nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
May 6 08:20:41 Eliot nvidia-persistenced: Shutdown (677)

I have checked that /dev/nvidia* files exist.

crw-rw-rw- 1 root root 195, 0 May 6 08:20 nvidia0
crw-rw-rw- 1 root root 195, 255 May 6 08:20 nvidiactl
crw-rw-rw- 1 root root 195, 254 May 6 08:20 nvidia-modeset
crw-rw-rw- 1 root root 242, 0 May 6 08:20 nvidia-uvm

The permission levels set allows read and write permission to all. No execution rights assigned.

Can someone advice me what and why the failure occurred and how to solve it?

Thanks.

I confirm the same error message exists on Ubuntu 16.04 with 367.18 beta driver. I suppose the problem could be in /lib/udev/rules.d/71-nvidia.rules: /usr/bin/start-nvidia-persistenced is probably started before nvidia device files exist? Is this the correct order of starting it before any nvidia modules are loaded? Actually was investigating the issue of nvidia module options(from /etc/modprobe.d) not being set when module is loaded at boot and discovered this error. BTW module options issue is also not clear: doing modprobe manually works(options are applied), but at boot options are ignored.

I noticed the same error with Ubuntu 16:10 and the 367.18 beta driver, when I was looking into my 367.18 failure this morning:

May 21 16:04:55 sager nvidia-persistenced: Started (619)
May 21 16:04:55 sager nvidia-persistenced: Failed to query NVIDIA devices. Please ensure that the NVIDIA device files (/dev/nvidia*) exist, and that user 123 has read and write permissions for those files.
May 21 16:04:55 sager nvidia-persistenced: The daemon no longer has permission to remove its runtime data directory /var/run/nvidia-persistenced
May 21 16:04:55 sager nvidia-persistenced: Shutdown (619)
May 21 16:04:55 sager nvidia-persistenced: Started (980)
May 21 16:04:55 sager nvidia-persistenced: Shutdown (980)
May 21 16:04:56 sager nvidia-persistenced: Started (2595)

Please attach nvidia bug report. Is the any reproduction steps to repro this issue? We are tracking this issue under 200202604

It seems I do not have access to existing issues, even using my nvidia developer account(created it recently). I can only submit new bugs. So should I attach nvidia-bug-report.sh output here? On reproducing steps: I get these messages on every boot when udev gets its rules processed. Not sure if this could be reproduced after boot process is complete.