There are a number of possibilities. It is difficult to say anything specific. If you have a serial console you should be able to see what is going on, and perhaps fix without flashing. If you do end up with a failure and cannot get access, then perhaps cloning could be performed to see what logs show.
If you start this server manually on command line without sudo, does it look like the virtual server is working?
If you start this server manually on command line with sudo, does it look like the virtual server is working?
When you start X manually on command line there is no login screen. The way the automatic startup scripts work (this isn’t part of X, it is part of init scripts) is to run X with X itself running a login/display manager (X does not provide a desktop environment…X can only run one application…the window manager is a special application since this can run other applications…but so far as X is concerned it is only running one application). Login results in X respawning with the window manager as the argument (and sudo as a different user…whatever the login manager sees for login credentials) instead of the login/display manager.
A typical failure where the screen goes black would not cause failure of ssh or physical console. A typical black screen failure might still illustrate part of what goes on…
If an X server is purely in software, and not hardware accelerated, then it is unlikely to ever bring down the system even if misconfigured. When direct hardware access is involved, then a failure would show up within the driver, and the driver is usually running in kernel space. NVIDIA provides an OpenGL library which pairs with its X driver. If you were to replace the NVIDIA version of the OpenGL library with the Mesa version, then X will crash and burn since the NVIDIA driver was not linked against the Mesa OpenGL library. Any attempt to start X this way would either result in a crash or an infinite cycle of starting X, X failing, and then trying again to start X. One can run “sha1sum -c /etc/nv_tegra_release” to verify if the correct NVIDIA version of various files are valid.
The NVIDIA driver must be used if GPU acceleration is required. The NVIDIA driver is linked against only a specific X ABI version. Using a newer or older X server with a different ABI would cause a crash even if the OpenGL library is the NVIDIA-provided version.
If your Xvfb is not the same ABI as the installed X server, then crash would occur.
If your Xvfb works with the NVIDIA driver, but references a Mesa version of the OpenGL library, then crash would occur.
Some init scripts can be considered “optional” due to the way they are configured. For example, one can make a valid ethernet connection optional…if not, then lack of networking would prevent boot. An init script can make a boot configuration mandatory, e.g., accessing the hard disk and mounting a particular partition as “/” would never be optional (well…there may be odd cases of this being optional, but it is not “common”). If your Xvfb configuration was mandatory, and then Xvfb failed for some reason, then further boot would halt.
You might ask which virtual desktops people have successfully used, and how they were added or configured. I don’t have a specific recommendation, but perhaps there are other virtual desktop servers (other than Xvfb) which are known to work.