A problem running 4x Tesla C2075 on TyanS7025 Any CUDA computation is extremely slow.
Hi,
I've got a problem running 4x Tesla C2075 on motherboard Tyan S7025AGM2NR under Windows 7 64-bit Professional. At the beginning I was unable to successfully boot up the operating system. I found and flashed the very last BIOS v1.08. Now the system boots up correctly, but suffers from poor user action responsiveness (e.g. starting application) and when running any CUDA computation on single GPU it is extremly slow. It is the same for DeviceQuery example from SDK where listing of all device properties takes 13s (i.e. 3s per devices, but when there is only 1x C2075 it is instant). VGA output is provided by onbord VGA AST2050 and all GPUs run in TCC mode.

I've contacted mainboard manufacturer, but his answer was it is usually a driver problem. So I've changed drivers but without any significant effect.

Server configuration: Tyan S7025AGM2NR BIOS v1.08, 2xCPU Xeon E5530 @ 2.4GHz, 12GB DDR3, 4x Tesla C2075, Windows 7 64-bit Professional (clear installation, all updates installed), NVIDIA Driver 285.67, CUDA Toolkit 4.1 RC1, power supply 1350W


Does anyone observed the same behavior?

Thanks for help.

Note: This problem is identical for 3x Tesla C2075.
Hi,

I've got a problem running 4x Tesla C2075 on motherboard Tyan S7025AGM2NR under Windows 7 64-bit Professional. At the beginning I was unable to successfully boot up the operating system. I found and flashed the very last BIOS v1.08. Now the system boots up correctly, but suffers from poor user action responsiveness (e.g. starting application) and when running any CUDA computation on single GPU it is extremly slow. It is the same for DeviceQuery example from SDK where listing of all device properties takes 13s (i.e. 3s per devices, but when there is only 1x C2075 it is instant). VGA output is provided by onbord VGA AST2050 and all GPUs run in TCC mode.



I've contacted mainboard manufacturer, but his answer was it is usually a driver problem. So I've changed drivers but without any significant effect.



Server configuration: Tyan S7025AGM2NR BIOS v1.08, 2xCPU Xeon E5530 @ 2.4GHz, 12GB DDR3, 4x Tesla C2075, Windows 7 64-bit Professional (clear installation, all updates installed), NVIDIA Driver 285.67, CUDA Toolkit 4.1 RC1, power supply 1350W





Does anyone observed the same behavior?



Thanks for help.



Note: This problem is identical for 3x Tesla C2075.

#1
Posted 12/09/2011 01:45 PM   
[quote name='svjezii' date='09 December 2011 - 08:45 AM' timestamp='1323438315' post='1339976']
Hi,
I've got a problem running 4x Tesla C2075 on motherboard Tyan S7025AGM2NR under Windows 7 64-bit Professional. At the beginning I was unable to successfully boot up the operating system. I found and flashed the very last BIOS v1.08. Now the system boots up correctly, but suffers from poor user action responsiveness (e.g. starting application) and when running any CUDA computation on single GPU it is extremly slow. It is the same for DeviceQuery example from SDK where listing of all device properties takes 13s (i.e. 3s per devices, but when there is only 1x C2075 it is instant). VGA output is provided by onbord VGA AST2050 and all GPUs run in TCC mode.

I've contacted mainboard manufacturer, but his answer was it is usually a driver problem. So I've changed drivers but without any significant effect.

Server configuration: Tyan S7025AGM2NR BIOS v1.08, 2xCPU Xeon E5530 @ 2.4GHz, 12GB DDR3, 4x Tesla C2075, Windows 7 64-bit Professional (clear installation, all updates installed), NVIDIA Driver 285.67, CUDA Toolkit 4.1 RC1, power supply 1350W


Does anyone observed the same behavior?

Thanks for help.

Note: This problem is identical for 3x Tesla C2075.
[/quote]


Hi,

I am suffering the same problem. Mine is a Tyan 7015 with 8*Tesla C2075, 8*Xeon E5620 2.4GHZ, 48GB DDR3, Win7-64bit. Extremely slow.

I took down 7 Tesla cards and the response time is almost normal. However, I ran the NVIDIA example and my C2075 is not working correctly. I cannot run some of the examples. When plug in 8*Tesla cards, the GPU driver process crashes down the OS.

Have you solved the problem? I am planning to install Ubuntu instead.
[quote name='svjezii' date='09 December 2011 - 08:45 AM' timestamp='1323438315' post='1339976']

Hi,

I've got a problem running 4x Tesla C2075 on motherboard Tyan S7025AGM2NR under Windows 7 64-bit Professional. At the beginning I was unable to successfully boot up the operating system. I found and flashed the very last BIOS v1.08. Now the system boots up correctly, but suffers from poor user action responsiveness (e.g. starting application) and when running any CUDA computation on single GPU it is extremly slow. It is the same for DeviceQuery example from SDK where listing of all device properties takes 13s (i.e. 3s per devices, but when there is only 1x C2075 it is instant). VGA output is provided by onbord VGA AST2050 and all GPUs run in TCC mode.



I've contacted mainboard manufacturer, but his answer was it is usually a driver problem. So I've changed drivers but without any significant effect.



Server configuration: Tyan S7025AGM2NR BIOS v1.08, 2xCPU Xeon E5530 @ 2.4GHz, 12GB DDR3, 4x Tesla C2075, Windows 7 64-bit Professional (clear installation, all updates installed), NVIDIA Driver 285.67, CUDA Toolkit 4.1 RC1, power supply 1350W





Does anyone observed the same behavior?



Thanks for help.



Note: This problem is identical for 3x Tesla C2075.







Hi,



I am suffering the same problem. Mine is a Tyan 7015 with 8*Tesla C2075, 8*Xeon E5620 2.4GHZ, 48GB DDR3, Win7-64bit. Extremely slow.



I took down 7 Tesla cards and the response time is almost normal. However, I ran the NVIDIA example and my C2075 is not working correctly. I cannot run some of the examples. When plug in 8*Tesla cards, the GPU driver process crashes down the OS.



Have you solved the problem? I am planning to install Ubuntu instead.

#2
Posted 12/16/2011 12:26 AM   
Yes, I still have the problems. I've contacted NVIDIA support and TYAN support and asked them both for a help.
Yes, I still have the problems. I've contacted NVIDIA support and TYAN support and asked them both for a help.

#3
Posted 12/22/2011 12:21 PM   
[quote name='svjezii' date='22 December 2011 - 02:21 PM' timestamp='1324556492' post='1345780']
Yes, I still have the problems. I've contacted NVIDIA support and TYAN support and asked them both for a help.
[/quote]

Hi, pleas try this:
Go in to BIOS and change the following:
Advanced -> PCI Express Configuration -> Active State Power-Management [Disabled]
Chipset -> CPU Bridge Configuration -> QPI L0s and L1 [Disabled]
[quote name='svjezii' date='22 December 2011 - 02:21 PM' timestamp='1324556492' post='1345780']

Yes, I still have the problems. I've contacted NVIDIA support and TYAN support and asked them both for a help.





Hi, pleas try this:

Go in to BIOS and change the following:

Advanced -> PCI Express Configuration -> Active State Power-Management [Disabled]

Chipset -> CPU Bridge Configuration -> QPI L0s and L1 [Disabled]

#4
Posted 12/23/2011 11:50 AM   
[quote name='brano' date='23 December 2011 - 12:50 PM' timestamp='1324641028' post='1346315']
Hi, pleas try this:
Go in to BIOS and change the following:
Advanced -> PCI Express Configuration -> Active State Power-Management [Disabled]
Chipset -> CPU Bridge Configuration -> QPI L0s and L1 [Disabled]
[/quote]

Both are disabled as you suggest. Enabling QPI L1 made the issue worse.

Anyway thanks for help.
[quote name='brano' date='23 December 2011 - 12:50 PM' timestamp='1324641028' post='1346315']

Hi, pleas try this:

Go in to BIOS and change the following:

Advanced -> PCI Express Configuration -> Active State Power-Management [Disabled]

Chipset -> CPU Bridge Configuration -> QPI L0s and L1 [Disabled]





Both are disabled as you suggest. Enabling QPI L1 made the issue worse.



Anyway thanks for help.

#5
Posted 01/02/2012 10:04 AM   
[quote name='svjezii' date='02 January 2012 - 12:04 PM' timestamp='1325498681' post='1350376']
Both are disabled as you suggest. Enabling QPI L1 made the issue worse.

Anyway thanks for help.
[/quote]


Have you tried to run the system with the monitor plugged in to one of the Tesla Cards instead?
[quote name='svjezii' date='02 January 2012 - 12:04 PM' timestamp='1325498681' post='1350376']

Both are disabled as you suggest. Enabling QPI L1 made the issue worse.



Anyway thanks for help.







Have you tried to run the system with the monitor plugged in to one of the Tesla Cards instead?

#6
Posted 01/02/2012 10:17 AM   
[quote name='brano' date='02 January 2012 - 11:17 AM' timestamp='1325499432' post='1350381']
Have you tried to run the system with the monitor plugged in to one of the Tesla Cards instead?
[/quote]

Yes of course, no performance effect. But there was an side effect of screen blinking (irregular) with three blue/black lines in the middle and over the full width of screen. That's probably because of the card providing a video output was also used for cuda computation.
[quote name='brano' date='02 January 2012 - 11:17 AM' timestamp='1325499432' post='1350381']

Have you tried to run the system with the monitor plugged in to one of the Tesla Cards instead?





Yes of course, no performance effect. But there was an side effect of screen blinking (irregular) with three blue/black lines in the middle and over the full width of screen. That's probably because of the card providing a video output was also used for cuda computation.

#7
Posted 01/06/2012 12:15 PM   
[quote name='svjezii' date='06 January 2012 - 02:15 PM' timestamp='1325852158' post='1352355']
Yes of course, no performance effect. But there was an side effect of screen blinking (irregular) with three blue/black lines in the middle and over the full width of screen. That's probably because of the card providing a video output was also used for cuda computation.
[/quote]

Hi,
that sounds like a bad card. Try to remove that card and test another one running the monitor over.
[quote name='svjezii' date='06 January 2012 - 02:15 PM' timestamp='1325852158' post='1352355']

Yes of course, no performance effect. But there was an side effect of screen blinking (irregular) with three blue/black lines in the middle and over the full width of screen. That's probably because of the card providing a video output was also used for cuda computation.





Hi,

that sounds like a bad card. Try to remove that card and test another one running the monitor over.

#8
Posted 01/09/2012 09:06 AM   
svjezii, tianren - could you solve the problem ? My Tyan-based system has almost similar one - extremely slow when 8 GPUs are plugged.
svjezii, tianren - could you solve the problem ? My Tyan-based system has almost similar one - extremely slow when 8 GPUs are plugged.

#9
Posted 02/21/2012 07:07 PM   
[quote name='Romant' date='21 February 2012 - 08:07 PM' timestamp='1329851256' post='1372417']
svjezii, tianren - could you solve the problem ? My Tyan-based system has almost similar one - extremely slow when 8 GPUs are plugged.
[/quote]

Hi,
thanks to TYAN support I've already closed this issue, but didn't post the solution here. I'm sorry for that. The key is to decrease the maximum available memory for Windows to at least 1GB less than it is installed, to the benefit of hardware reserved memory that is reserved for use by the BIOS and some drivers for other peripherals (some info about memory allocations http://support.microsoft.com/kb/978610). You can see this memory organization in Resource monitor.
Step by step tutorial is in attached document.

In my case, it simply worked.

I hope this will help you.
[quote name='Romant' date='21 February 2012 - 08:07 PM' timestamp='1329851256' post='1372417']

svjezii, tianren - could you solve the problem ? My Tyan-based system has almost similar one - extremely slow when 8 GPUs are plugged.





Hi,

thanks to TYAN support I've already closed this issue, but didn't post the solution here. I'm sorry for that. The key is to decrease the maximum available memory for Windows to at least 1GB less than it is installed, to the benefit of hardware reserved memory that is reserved for use by the BIOS and some drivers for other peripherals (some info about memory allocations http://support.microsoft.com/kb/978610). You can see this memory organization in Resource monitor.

Step by step tutorial is in attached document.



In my case, it simply worked.



I hope this will help you.

#10
Posted 02/22/2012 06:51 AM   
Solution is found, thanks to Tyan support. It is a bug in Win7, system has problems related to RAM counting.

Say, there are 72 gigabytes of RAM installed.

1. Run "msconfig"
2. Click "boot" tab
3. Click "advanced" option
4. Check the "max memory" box, the max memory field will automatically put "73728" (72GB)
5. Enter "72704" and apply the changes, restart the system, the system will run at what appears to be normal speed

The key basically is to change the max memory from whatever it shows to 1GB less than what it shows (subtract 1024).
This will insure proper operation of the system regardless of the total amount of memory installed.
Solution is found, thanks to Tyan support. It is a bug in Win7, system has problems related to RAM counting.



Say, there are 72 gigabytes of RAM installed.



1. Run "msconfig"

2. Click "boot" tab

3. Click "advanced" option

4. Check the "max memory" box, the max memory field will automatically put "73728" (72GB)

5. Enter "72704" and apply the changes, restart the system, the system will run at what appears to be normal speed



The key basically is to change the max memory from whatever it shows to 1GB less than what it shows (subtract 1024).

This will insure proper operation of the system regardless of the total amount of memory installed.

#11
Posted 02/22/2012 08:11 AM   
[quote name='svjezii' date='22 February 2012 - 09:51 AM' timestamp='1329893503' post='1372883']
Hi,
thanks to TYAN support I've already closed this issue, but didn't post the solution here. I'm sorry for that. The key is to decrease the maximum available memory for Windows to at least 1GB less than it is installed, to the benefit of hardware reserved memory that is reserved for use by the BIOS and some drivers for other peripherals (some info about memory allocations http://support.microsoft.com/kb/978610). You can see this memory organization in Resource monitor.
Step by step tutorial is in attached document.

In my case, it simply worked.

I hope this will help you.
[/quote]

Yeah, I have also got the answer from Tyan describing the same solution, thank you anyway! You are mentioning some attached file ... attached to what ? Don't see any links or something like that.
[quote name='svjezii' date='22 February 2012 - 09:51 AM' timestamp='1329893503' post='1372883']

Hi,

thanks to TYAN support I've already closed this issue, but didn't post the solution here. I'm sorry for that. The key is to decrease the maximum available memory for Windows to at least 1GB less than it is installed, to the benefit of hardware reserved memory that is reserved for use by the BIOS and some drivers for other peripherals (some info about memory allocations http://support.microsoft.com/kb/978610). You can see this memory organization in Resource monitor.

Step by step tutorial is in attached document.



In my case, it simply worked.



I hope this will help you.





Yeah, I have also got the answer from Tyan describing the same solution, thank you anyway! You are mentioning some attached file ... attached to what ? Don't see any links or something like that.

#12
Posted 02/22/2012 08:20 AM   
[quote name='Romant' date='22 February 2012 - 09:20 AM' timestamp='1329898823' post='1372916']
Yeah, I have also got the answer from Tyan describing the same solution, thank you anyway! You are mentioning some attached file ... attached to what ? Don't see any links or something like that.
[/quote]

Sorry, the file needs to be attached explicitly, not only by its selection. Here it is.
[quote name='Romant' date='22 February 2012 - 09:20 AM' timestamp='1329898823' post='1372916']

Yeah, I have also got the answer from Tyan describing the same solution, thank you anyway! You are mentioning some attached file ... attached to what ? Don't see any links or something like that.





Sorry, the file needs to be attached explicitly, not only by its selection. Here it is.

#13
Posted 02/22/2012 12:42 PM   
Scroll To Top