Hey
I have an issues with slow nvidia-smi, which is 20 min to set persistence mode on all cards, 100% CPU when running nvidia-smi -x -pl 1.
There was solution posted in another thread, to set persistence with nvidia-persistenced (/usr/bin/nvidia-persistenced --user nvidia-persistenced --persistence-mode --verbose), unfortunately, this was keeping updated GPUs settings only when x11 was running. When I was closing x11 server all cards went to performance level 8 even under load. Opening x11 server bumps them back to P2, but it’s not a robust solution. For some reason, closing x11 server ignores previously set configuration.
My setup is 13 x Nvidia 1070 under headless ubuntu 16 system. CPU is a cheap Celeron.