Linux PCI-E 3.0 Support
  1 / 2    
Hello, I sent a request to NVIDIA support about this issue a few months back but there has not been a resolution yet to the problem that I know of. I provided bug report logs as was requested at the time. I use my Kepler 680 cards for compute purposes (CUDA/OpenCL) and the PCI-E bandwidth is very beneficial for this use. With driver 295.33, all my slots negotiate at 8 GT/s per Rev 3.0 spec with several different boards that I have which is working fine currently. Any drivers after 295.33 are capped at 5 GT/s as reported by lspci and nvidia-settings. I have tried all drivers up to 310.14 beta and all are limited to 5 GT/s. Are there any plans to fix this issue? I would like to be able to upgrade my driver to a newer version at some point and this will be necessary if I am ever to buy any future released NVIDIA cards. Thanks for your time. Screenshots of nvidia-settings: 295.33: [url]https://docs.google.com/open?id=0B3fM8POXHOrVYkctcXZfMGRJSVk[/url] 304.10: [url]https://docs.google.com/open?id=0B3fM8POXHOrVaVJVSEc0OWNwYU0[/url]
Hello,

I sent a request to NVIDIA support about this issue a few months back but there has not been a resolution yet to the problem that I know of. I provided bug report logs as was requested at the time.

I use my Kepler 680 cards for compute purposes (CUDA/OpenCL) and the PCI-E bandwidth is very beneficial for this use. With driver 295.33, all my slots negotiate at 8 GT/s per Rev 3.0 spec with several different boards that I have which is working fine currently. Any drivers after 295.33 are capped at 5 GT/s as reported by lspci and nvidia-settings. I have tried all drivers up to 310.14 beta and all are limited to 5 GT/s.

Are there any plans to fix this issue? I would like to be able to upgrade my driver to a newer version at some point and this will be necessary if I am ever to buy any future released NVIDIA cards.

Thanks for your time.

Screenshots of nvidia-settings:

295.33: https://docs.google.com/open?id=0B3fM8POXHOrVYkctcXZfMGRJSVk
304.10: https://docs.google.com/open?id=0B3fM8POXHOrVaVJVSEc0OWNwYU0

#1
Posted 11/03/2012 09:06 PM   
And you see a drop in performance too?
And you see a drop in performance too?

#2
Posted 11/04/2012 03:58 PM   
[quote="Licaon_Kter"]And you see a drop in performance too?[/quote] Yes, the performance drop is approximately 15% when going from 8 GT/s to 5 GT/s with the application I primarily run. I recently found a handy tool in the NVIDIA CUDA Toolkit called bandwidthTest. I compiled that tool via CUDA Toolkit 4.2.9. The version I compiled with CUDA Toolkit 5.0.35 required at least driver 304.54 and because of that I used the 4.2.9 build instead. Here are the results between 5 and 8 GT/s using driver 295.33 and drivers that came after that version. 5 GT/s - Size: 33554432 bytes Host to Device Bandwidth: 5990.8 MB/s Device to Host Bandwidth: 6396.6 MB/s 8 GT/s - Size: 33554432 bytes Host to Device Bandwidth: 11712.8 MB/s Device to Host Bandwidth: 12129.4 MB/s The bandwidth measurement is nearly double with driver 295.33.
Licaon_Kter said:And you see a drop in performance too?


Yes, the performance drop is approximately 15% when going from 8 GT/s to 5 GT/s with the application I primarily run.

I recently found a handy tool in the NVIDIA CUDA Toolkit called bandwidthTest. I compiled that tool via CUDA Toolkit 4.2.9. The version I compiled with CUDA Toolkit 5.0.35 required at least driver 304.54 and because of that I used the 4.2.9 build instead.

Here are the results between 5 and 8 GT/s using driver 295.33 and drivers that came after that version.

5 GT/s - Size: 33554432 bytes

Host to Device Bandwidth: 5990.8 MB/s
Device to Host Bandwidth: 6396.6 MB/s

8 GT/s - Size: 33554432 bytes

Host to Device Bandwidth: 11712.8 MB/s
Device to Host Bandwidth: 12129.4 MB/s

The bandwidth measurement is nearly double with driver 295.33.

#3
Posted 11/04/2012 07:14 PM   
You're right that PCIe 3.0 was disabled because of reliability issues with the current generation of PCIe controllers, such as the one in the X79 chipset platform. There exists an NVIDIA tool on Windows to opt-in to unsupported PCIe 3.0 support, but there isn't a Linux equivalent to that tool yet. I filed an enhancement request and will update that thread when there's news. Thanks, - Pierre-Loup
You're right that PCIe 3.0 was disabled because of reliability issues with the current generation of PCIe controllers, such as the one in the X79 chipset platform. There exists an NVIDIA tool on Windows to opt-in to unsupported PCIe 3.0 support, but there isn't a Linux equivalent to that tool yet. I filed an enhancement request and will update that thread when there's news.

Thanks,
- Pierre-Loup

Pierre-Loup A. Griffais
NVIDIA Linux Graphics

#4
Posted 11/04/2012 11:19 PM   
[quote="Plagman"]You're right that PCIe 3.0 was disabled because of reliability issues with the current generation of PCIe controllers, such as the one in the X79 chipset platform. There exists an NVIDIA tool on Windows to opt-in to unsupported PCIe 3.0 support, but there isn't a Linux equivalent to that tool yet. I filed an enhancement request and will update that thread when there's news. Thanks, - Pierre-Loup[/quote] Thanks for the information and for putting in a request. Is there a link to the site where the requests are filed? Otherwise if you could post an update in this thread when available, I would appreciate it. I also wanted to mention that the PCI-E 3.0 support with the one driver that supports it has been running stable via my systems and the Kepler cards with 7-months of full load on the GPUs. I have not run into any reliability issues that I could find.
Plagman said:You're right that PCIe 3.0 was disabled because of reliability issues with the current generation of PCIe controllers, such as the one in the X79 chipset platform. There exists an NVIDIA tool on Windows to opt-in to unsupported PCIe 3.0 support, but there isn't a Linux equivalent to that tool yet. I filed an enhancement request and will update that thread when there's news.

Thanks,
- Pierre-Loup


Thanks for the information and for putting in a request. Is there a link to the site where the requests are filed? Otherwise if you could post an update in this thread when available, I would appreciate it.

I also wanted to mention that the PCI-E 3.0 support with the one driver that supports it has been running stable via my systems and the Kepler cards with 7-months of full load on the GPUs. I have not run into any reliability issues that I could find.

#5
Posted 11/06/2012 03:17 AM   
The request number is 1171212; there is no publicly accessible tracker but I will update this thread when the bug gets resolved.
The request number is 1171212; there is no publicly accessible tracker but I will update this thread when the bug gets resolved.

Pierre-Loup A. Griffais
NVIDIA Linux Graphics

#6
Posted 11/06/2012 03:33 AM   
where can we see the status of this ? I really want pcie 3 enabled
where can we see the status of this ?
I really want pcie 3 enabled

#7
Posted 11/15/2012 06:13 AM   
Hi, My system has the following specs: - Intel Core i7-3820 - Gigabyte GA-X79-UD5 - 2x GeForce GTX 680 (Gigabyte GV-N680OC-2GD) - CentOS 6.3 (64 bits) - Cuda toolkit 4.2 310.14 driver and pinned memory: Host to Device Bandwidth: 5985.9 MB/s Device to Host Bandwidth: 6390.9 MB/s 295.33 driver and pinned memory: Host to Device Bandwidth: 11192.6 MB/s Device to Host Bandwidth: 12178.8 MB/s I will do further tests with my own codes.
Hi,

My system has the following specs:
- Intel Core i7-3820
- Gigabyte GA-X79-UD5
- 2x GeForce GTX 680 (Gigabyte GV-N680OC-2GD)
- CentOS 6.3 (64 bits)
- Cuda toolkit 4.2

310.14 driver and pinned memory:
Host to Device Bandwidth: 5985.9 MB/s
Device to Host Bandwidth: 6390.9 MB/s


295.33 driver and pinned memory:
Host to Device Bandwidth: 11192.6 MB/s
Device to Host Bandwidth: 12178.8 MB/s


I will do further tests with my own codes.

#8
Posted 11/15/2012 07:54 PM   
On a similar note, how is SLI support these days? I remember people reporting that it was very minimal performance gain, or even a performance loss. Is this still the case?
On a similar note, how is SLI support these days? I remember people reporting that it was very minimal performance gain, or even a performance loss. Is this still the case?

#9
Posted 11/16/2012 02:06 PM   
[quote="mlopezp"]Hi, My system has the following specs: - Intel Core i7-3820 - Gigabyte GA-X79-UD5 - 2x GeForce GTX 680 (Gigabyte GV-N680OC-2GD) - CentOS 6.3 (64 bits) - Cuda toolkit 4.2 310.14 driver and pinned memory: Host to Device Bandwidth: 5985.9 MB/s Device to Host Bandwidth: 6390.9 MB/s 295.33 driver and pinned memory: Host to Device Bandwidth: 11192.6 MB/s Device to Host Bandwidth: 12178.8 MB/s I will do further tests with my own codes. [/quote] Thanks for sharing your bandwidth results. Those numbers are very similar to what I have seen on two of my systems. I hope that NVIDIA can resolve this issue and remove the restriction.
mlopezp said:Hi,

My system has the following specs:
- Intel Core i7-3820
- Gigabyte GA-X79-UD5
- 2x GeForce GTX 680 (Gigabyte GV-N680OC-2GD)
- CentOS 6.3 (64 bits)
- Cuda toolkit 4.2

310.14 driver and pinned memory:
Host to Device Bandwidth: 5985.9 MB/s
Device to Host Bandwidth: 6390.9 MB/s


295.33 driver and pinned memory:
Host to Device Bandwidth: 11192.6 MB/s
Device to Host Bandwidth: 12178.8 MB/s


I will do further tests with my own codes.


Thanks for sharing your bandwidth results. Those numbers are very similar to what I have seen on two of my systems. I hope that NVIDIA can resolve this issue and remove the restriction.

#10
Posted 11/19/2012 02:52 AM   
[quote="scix55"]On a similar note, how is SLI support these days? I remember people reporting that it was very minimal performance gain, or even a performance loss. Is this still the case?[/quote] I have not tested this in a long time. I think the last time I had SLI enabled in Linux was with a pair of 6800GT or 8800GT cards. Unfortunately, there were few apps available at the time to take advantage of the SLI. Now that Steam is pushing to port games to Linux, SLI should be of greater benefit for Linux users.
scix55 said:On a similar note, how is SLI support these days? I remember people reporting that it was very minimal performance gain, or even a performance loss. Is this still the case?


I have not tested this in a long time. I think the last time I had SLI enabled in Linux was with a pair of 6800GT or 8800GT cards. Unfortunately, there were few apps available at the time to take advantage of the SLI. Now that Steam is pushing to port games to Linux, SLI should be of greater benefit for Linux users.

#11
Posted 11/19/2012 02:57 AM   
[quote="JeroenLF"]I have not tested this in a long time. I think the last time I had SLI enabled in Linux was with a pair of 6800GT or 8800GT cards. Unfortunately, there were few apps available at the time to take advantage of the SLI. Now that Steam is pushing to port games to Linux, SLI should be of greater benefit for Linux users.[/quote] True. I'm mainly considering this as I am thinking of getting a 1440p monitor, but I don't know if performance will be dropped too severely with a single 660. I'm mainly playing games under wine, and in that case it's quite possible that the CPU bottleneck would be too great, which would mean that SLI wouldn't help much even if it's working correctly. Additionally this would mean that, assuming there is enough video ram, 1440p probably won't have a huge additional performance hit. But it's hard to say, without actually testing such a configuration. I've found the relationship between CPU, GPU, and video settings is often less obvious on linux (with wine) in comparison to windows. I agree that working SLI would be great for native games at least, provided that they are heavy enough on the GPU side to drop below 60fps. I'm not sure that any of the current Source games would do that actually with a 660 or greater system, but perhaps future games will.
JeroenLF said:I have not tested this in a long time. I think the last time I had SLI enabled in Linux was with a pair of 6800GT or 8800GT cards. Unfortunately, there were few apps available at the time to take advantage of the SLI. Now that Steam is pushing to port games to Linux, SLI should be of greater benefit for Linux users.


True. I'm mainly considering this as I am thinking of getting a 1440p monitor, but I don't know if performance will be dropped too severely with a single 660. I'm mainly playing games under wine, and in that case it's quite possible that the CPU bottleneck would be too great, which would mean that SLI wouldn't help much even if it's working correctly. Additionally this would mean that, assuming there is enough video ram, 1440p probably won't have a huge additional performance hit. But it's hard to say, without actually testing such a configuration. I've found the relationship between CPU, GPU, and video settings is often less obvious on linux (with wine) in comparison to windows.

I agree that working SLI would be great for native games at least, provided that they are heavy enough on the GPU side to drop below 60fps. I'm not sure that any of the current Source games would do that actually with a 660 or greater system, but perhaps future games will.

#12
Posted 11/19/2012 03:17 AM   
A future driver release will add a kernel module parameter, NVreg_EnablePCIeGen3=1, that will enable PCIe gen 3 when possible. Please note that this option is experimental and many systems are expected to have stability problems when this option is enabled. Use it at your own risk.
A future driver release will add a kernel module parameter, NVreg_EnablePCIeGen3=1, that will enable PCIe gen 3 when possible. Please note that this option is experimental and many systems are expected to have stability problems when this option is enabled. Use it at your own risk.

Aaron Plattner
NVIDIA Linux Graphics

#13
Posted 01/03/2013 10:05 PM   
Aaron, I loaded driver 310.32 with kernel 3.7.5 and my cards are negotiating at PCI-E 3.0. So far everything is looking good. I ran bandwidthTest and the numbers are similar to driver 295.33. Thank you very much for adding this option to the latest Linux NVIDIA drivers. I very much appreciate this. The CUDA application I am running performs the best with PCI-E 3.0 enabled. I also appreciate that the option can be enabled as a kernel module parameter. I am running a small busybox based image that does not have X and this makes it so that I can still load PCI-E 3.0 support and run my CUDA apps without having to add X.
Aaron,

I loaded driver 310.32 with kernel 3.7.5 and my cards are negotiating at PCI-E 3.0. So far everything is looking good. I ran bandwidthTest and the numbers are similar to driver 295.33.

Thank you very much for adding this option to the latest Linux NVIDIA drivers. I very much appreciate this. The CUDA application I am running performs the best with PCI-E 3.0 enabled. I also appreciate that the option can be enabled as a kernel module parameter. I am running a small busybox based image that does not have X and this makes it so that I can still load PCI-E 3.0 support and run my CUDA apps without having to add X.

#14
Posted 02/01/2013 01:56 PM   
As Aaron stated, the nvidia driver option doesn't work everywhere unfortunately. I just bought a system which was supposed to be a Gen3 GPU compute machine. It has an Intel W2600CR motherboard with dual E5-2620 and 4 x ASUS GTX 670 cards. None of them are recognized as Gen3. I tried the latest Nvidia driver, and applying the "NVreg_EnablePCIeGen3=1", but when I run a simple "deviceQuery", the system crashes and reboots. If I run deviceQuery without this option enabled, everything is fine. I'm running RHEL63. Are there any other server motherboards that would take the dual E5-2620 CPUs and 4 x ASUS GTX 670 and guaranteed run at proper Gen3 speed? It seems that the whole Gen3 thing is a "scam" when it comes to these boards. If you're going to say that your board supports Gen3 (like the Intel board does), then it should, and if it doesn't, it shouldn't be able to say that it does because in the end, it's the users that get caught in between the mess.
As Aaron stated, the nvidia driver option doesn't work everywhere unfortunately.
I just bought a system which was supposed to be a Gen3 GPU compute machine.
It has an Intel W2600CR motherboard with dual E5-2620 and 4 x ASUS GTX 670 cards.
None of them are recognized as Gen3.
I tried the latest Nvidia driver, and applying the "NVreg_EnablePCIeGen3=1", but when I run a simple "deviceQuery", the system crashes and reboots.
If I run deviceQuery without this option enabled, everything is fine.
I'm running RHEL63.
Are there any other server motherboards that would take the dual E5-2620 CPUs and 4 x ASUS GTX 670 and guaranteed run at proper Gen3 speed?
It seems that the whole Gen3 thing is a "scam" when it comes to these boards. If you're going to say that your board supports Gen3 (like the Intel board does), then it should, and if it doesn't, it shouldn't be able to say that it does because in the end, it's the users that get caught in between the mess.

#15
Posted 02/12/2013 09:32 PM   
  1 / 2    
Scroll To Top