Massive FPS drop down on GTX660 with 367.* drivers

Greetings,

I’ve been experiencing massive FPS drops (e.g. from 50 down to 10-15 fps !) in Second Life after installing the 367.27 drivers.
The drops happen half an hour or so after the session was started. Switching the Second Life viewer from non-deferred to deferred rendering mode (which would normally cause a lowering of FPS rates by approximately 30%) seems to temporarily “cure” the issue (i.e. the FPS rate raises again), but it’s only temporary.
Sometimes, also, minimizing the viewer and re-maximizing it a minute or so later seems to temporarily cure the issue, but the drop will keep happening later on.

After reverting to the 364.19 driver, things work properly again (no more drop: stable FPS rate).

A wild guess (but one from a seasoned programmer) would be that something got screwed up in v367’s memory management (to me, the “cures” seem related with some memory garbage collection in NVIDIA’s driver, nvidia-smi systematically reporting a lower memory consumption in the VRAM after them).

I’m attaching the traditional nvidia-bug-report.log.gz file here after.
nvidia-bug-report.log.gz (191 KB)

Are you using KDE desktop env and playing game via steam ? Is the issue hit if you disable the kde desktop effects ?

Nope. Using MATE, here, and no compositing manager. The windows manager in use is Sawfish. Never had a single issue in many years (Gnome 2 + Sawfish, the MATE + SAwfish) with this combination…

I’m not playing any Steam game either.

I’m sometimes using VLC while SLing, and sometimes suing a 3D Mahjong OpenGL game, but their use doesn’t seem related with the FPS drops (the drops happen even without using them).

I’m seeing the same issue with the latest drivers (367.35).

I reverted again to v364.19, which works just fine.

It looks like 364.19 is affected as well, although the occurrences are much rarer than with 367.x: with 364.19, the FPS drop only happens, sometimes, after I switched to another desktop in another application and come back later at the OpenGL application; minimizing the OpenGL app and un-minimizing it often (but not always) cures the issue.

I’m now back-pedaling to v361.45.18…

So far I could not reproduce the same issues with v361.45.18 which is working like a charm. So, it looks like the bug was introduced in v364 and got only worst with v367.

I’ll keep watching my FPS rates with v361 and will keep you informed if I see anything weird.

Same problem here with Quadro M1000M. Anything after v361.45.18 is affected, including v370.23.

Hi dinosaur_, Are you using any app/tool to check FPS?

No, the FPS rate is directly taken from the Second Life viewer which measures it accurately (frame per frame) and provides it as both instantaneous and averaged rates.

v367.44 is also affected. Back to 361.44.11 for me…

I meant back to v361.45.18 (and why is it the Edit post action doesn’t work on this forum ?).

Updates:

  • I saw also the issue happening, after a 4+ hours long session in SL with driver v361.45.18 (so, it happens in all UVM-enabled drivers, apparently, but way more often with newer drivers than with older ones).

  • It also happens with the latest stable driver and beta driver.

  • I upgraded my system and replaced the GTX 660 with a GTX 970: same issue.

  • The slow down happens during the glFinish() call (the Second Life viewer got “fast timers” that allow to accurately time various calls, and the glFinish() call got a fast timer of his own): for example, I just (minutes ago) had the issue happening, and the frame rate dropped from 130fps to 20fps. The glFinish() call fast timer then raised from 0.4ms per frame to over 30ms per frame.

  • Resizing the viewer window seems to be the best work-around for this issue (it usually gets things back in order). The resizing causes the reallocation of all the frame buffers (glGenFramebuffers(), glBindFramebuffer() calls & Co).

It’s of course a wild guess of mine, since I don’t have access to the driver sources, but it furiously looks like a memory fragmentation/garbage collection issue to me: is there a garbage collector call during glFinish() ?

Hello,

I am having the same problems with my fps drop after 15-20 mins of gameplay of Second Life on the Firestorm viewer. I was wondering how can I revert back to the 364.19 driver like you said. I’m new to Nvidia and wish to be able to play SL without any lag after 15 mins of play.

Hi Zalloi and PeteG, Can I get nvidia bug report of your system?

@Sandipt

Since I upgraded my system (GTX 970, kernel 4.7.2, NVIDIA driver 370.23), I’m attaching a new nvidia-installer.log as well.

Please, also see my “Updates” message above: it contains what I believe to be interesting clues for you to find that nasty bug and squash it ! :-D
nvidia-bug-report.log.gz (215 KB)

I have the GTX960M on Version 372.70 but if I knew how to find the bug report log I would! My apologize, all new to this.

Please run script /usr/bin/nvidia-bug-report.sh as root/super user.

We are tracking this issue in bug 200233115

Here’s the bug report with v370.23 drivers. BTW, as far as I can tell this issue is not limited to Second Life. It happens in benchmarks too, e.g. glmark2 or GpuTest FurMark GpuTest - Cross-Platform GPU Stress Test and OpenGL Benchmark for Windows, Linux and OS X | Geeks3D.com
Especially in GpuTest you can clearly see how FPS and GPU utilization drops with newer drivers. FPS is 10-20% lower and GPU utilization is around 70-90%. With v361 drivers utilization is always 99-100%.
nvidia-bug-report.log.gz (102 KB)

Could not repro on config ( ubuntu 16.04 + Driver 367.27 , 367.35 , 370.23 , 361.45.18 , 352.68 + Geforce GTX 660 + HP Z820 WS + Acer predetor mustang N16E-GS ( Geforce GTX 965M ) )

Connected Geforce GTX 660 on WS system.
Nouveau disabled and Installed driver as above config
Installed Firestorm application ( includes Second life ) → Linux for Second Life - Firestorm Viewer - The Phoenix Firestorm Project Inc.
GPUtest application ( includes funmark benchmark script )–> GpuTest 0.7.0 Released (Win / Linux / OS X), New FP64 OpenGL 4 Test and Online GPU Database | Geeks3D
glmark2 application → fixmynix.com

  1. Observation with Firestorm application
    Installed firestorm and launch application by ./firestorm, created login id , selected grid option as second life and logged in
    able to play the game and using short key ctrl+shift+1 could see FPS status.
    I see while playing the game FPS goes upto 60 fps to 65 in " ubuntu unity desktop" , whereas playing application using desktop only in xinit it goes to 150 to 180 FPS.
    if the cursor is clicked outside the firestorm application, automatically FPS goes down to 17 to 18, this happens in all the drivers. if again i clicked in the game, FPS goes up again to 180
    Attempted tried resizing the app during play, FPS does not come down to 15-20
    played game for 1hour 30 minutes

  2. Observation with Gputest application
    played script Gputest , start_furmark_benchmark_fullscreen_1920x1080.sh , start_furmark_windowed_1024x640.sh
    observed test runs with 60 FPS in window mode, when performed resize to full screen FPS goes down to 30 FPS
    tested with all above config driver observed same behavior, tested for 1 hour.

  3. Observation with glmark2
    Installed and configured as per link, and launched glmark2
    tested with repro driver but the score are similar to all, i did not observe any low score as compare to older 352 driver.

Same also tested with Acer predetor mustang N16E-GS ( Geforce GTX 965M ) with ubuntu OS, performed above procedure could not repro for the same.

Query:-

  1. Need to know the exact repro steps with configuration used?
  2. When the FPS hit to 15 to 20, does it appear during playing game or just leaving the game ideal?
  3. How long it takes to reproduce this issue?
  4. If the cursor is clicked outside the firestorm application, automatically FPS goes down to 17 to 18, this happens in all the drivers. if again i clicked in the game, FPS goes up again to 180, does by this you mean to say the issue you are facing?
  5. How do you switched non-deferred to deferred rendering mode in second life game?
  6. Can you please share us the video once you are able to see the FPS drop?
  7. once FPS drops to 15-20 , do it continuous for long state or its happens for a short while?

I’m not using Unity, neither Ubuntu. PCLinuxOS 2016, Mate edition, here. The frame rates are not affected the least by the desktop backend for me (Mate + Sawfish: very lightweight and no measurable overhead on any app) when compared with a pure X11 session.
I’m not using Firestorm either, but the viewer type should not make any difference (they all share the same shaders and renderer, but for the Singularity viewer which had them modified a bit).

This is normal (the viewer yields to the OS when its window looses the focus). You can disable this behaviour (which I did, years ago, because it’s annoying…) via a debug setting. Make sure the “Advanced” menu entry is visible in the menu bar (CTRL ALT D if it’s not), and select the “Debug settings” entry in it (the shortcut for it is usually CTRL ALT S). Then search for the BackgroundYieldTime settings and set it to -1 to disable it entirely.

This normal behaviour is of course not the issue at hand…

You misread my reports… Resizing the window is actually a (usually successful) mean to recover a proper frame rate whenever it drops; the resizing causes the reallocation of all frame buffers by the viewer.

[quote]

played game for 1hour 30 minutes|/quote]
The issue happens at random, but my observations so far are that driver v367.35 was by far the worst and would cause drops in a matter of 15 to 30 minutes.

With v361, I only encountered the issue a couple of times and after 3-4 hours of SLing…

I can’t help you there… Not using it.

For the configuration, I posted a couple of nvidia-bug-report.log.gz… My desktop system is not as powerful as yours however (Core i5 @ 4.6GHz, 32Gb RAM, GTX660 when I posted my first message in this thread and now GTX970).
For the repro steps, please see my previous posts. When comparing with your report, I’d say that I most often see the FPS drops after switching from another virtual desktop (which usually contains a web browser) back to the virtual desktop where the SL viewer is running. But sometimes (especially with v367), the drops happen without any such virtual desktop switch.
Oh, also, I’m not sure Firestorm got it enabled, but I do use the drivers in multi-threaded mode with my viewer; if not already there, you can add “export __GL_THREADED_OPTIMIZATIONS=1” in the wrapper script (“firestorm”, I suppose) of the Firestorm installation directory. If the viewer consumes more than one CPU core when running (and the scene is fully “rezzed”, i.e. all textures are decoded, all mesh objects loaded at full LOD, etc), then chances are the multi-threaded mode is enabled (the viewer rendering loop consumes only one full core by itself (the renderer is not multi-threaded, alas), and the driver threads then eat up a third to half a second core on my system).

I could not infer a specific scheme; sometimes it happens while I’m roaming and teleporting a lot in busy places, sometimes my avatar is just left idling in a low traffic place and the drops happen just as well…

The setting is called “ALM” or “Advanced Lighting Model” or “Deferred Rendering”, depending on the viewer. They all switch the renderer to the deferred mode (with “materials” and optionally shadows support). You can switch these settings on and off from the Preferences floater (CTRL P), “Graphics” tab (or equivalent).
I’m usually using the viewer in non-deferred mode (i.e. ALM off), because I don’t like the slightly blurry, dark and gloomy aspect of the ALM mode in SL viewers…

Hard to do… It happens at random, so I can’t start a video and let it fill up my hard drive till the drop happens. I will however try and make one short video showing the slowed down rate and how I make it raise back to its normal level…

If you do nothing after the drop started, it will only get worst with time: I’m always trying hard to get things back in order. From what I found out so far, your best bet is to resize the viewer window to force the reallocation of all the frame buffers, vertex buffers & Co. If this fails, minimizing the window for a minute or so, switching to another desktop, then un-minimizing it may also work…