Consumption of host memory increases abnormally
Hi,

I just found that the consumption of host memory seems to increase abnormally in the new CUDA release.
The following is a very simple demo program.

int main()
{
// set GPU ID
cudaSetDevice( 0 );

// allocate host memory
const long NGB = 9;
const long N = NGB*1024*1024*1024/sizeof(float);
float *Array = new float [N];

// arbitrary work on Array
// ...

delete [] Array;
exit(0);
}

The "top" command shows that the demo program consumes about 18.3 GB, which is about 2 times larger than expected!!
However, if i comment out the line "cudaSetDevice( 0 );", the memory consumption becomes normal (about 9 GB).
The information of the system giving this issue is as follows.

CUDA version: 4.0 RC2
Driver version: 270.41.06
OS: Scientific Linux SL release 5.4 (Boron)

I suspect that this issue is due to the new graphic driver and(or) CUDA since this problem appeared after a system upgrade.
I also tested the demo program in a different system with older versions of CUDA and driver, and everything works fine.
In this case, the memory consumption is normal even when I include any CUDA function.
The information of this system is as follows:

CUDA version: 3.2
Driver version: 260.19.21
OS: CentOS release 5.5 (Final)

Has anyone had a similar problem? Any help is appreciated!
Hi,



I just found that the consumption of host memory seems to increase abnormally in the new CUDA release.

The following is a very simple demo program.



int main()

{

// set GPU ID

cudaSetDevice( 0 );



// allocate host memory

const long NGB = 9;

const long N = NGB*1024*1024*1024/sizeof(float);

float *Array = new float [N];



// arbitrary work on Array

// ...



delete [] Array;

exit(0);

}



The "top" command shows that the demo program consumes about 18.3 GB, which is about 2 times larger than expected!!

However, if i comment out the line "cudaSetDevice( 0 );", the memory consumption becomes normal (about 9 GB).

The information of the system giving this issue is as follows.



CUDA version: 4.0 RC2

Driver version: 270.41.06

OS: Scientific Linux SL release 5.4 (Boron)



I suspect that this issue is due to the new graphic driver and(or) CUDA since this problem appeared after a system upgrade.

I also tested the demo program in a different system with older versions of CUDA and driver, and everything works fine.

In this case, the memory consumption is normal even when I include any CUDA function.

The information of this system is as follows:



CUDA version: 3.2

Driver version: 260.19.21

OS: CentOS release 5.5 (Final)



Has anyone had a similar problem? Any help is appreciated!

#1
Posted 05/12/2011 02:58 PM   
I'm pretty sure you're looking at the total memory column, and this is irrelevant.
The RSS column is what your application really uses, and I don't think this one will change dramatically with just the addition of cudaSetDevice().

You can try "pmap -x <pid>" to see where your memory is.

Same thing if you mmap a 200GB file (which sits on disk). You would see 200GB in top. That does not mean you're using 200GB.

-Guillaume

[quote name='llittlefish' date='12 May 2011 - 09:58 AM' timestamp='1305212325' post='1236339']
Hi,

I just found that the consumption of host memory seems to increase abnormally in the new CUDA release.
The following is a very simple demo program.

int main()
{
// set GPU ID
cudaSetDevice( 0 );

// allocate host memory
const long NGB = 9;
const long N = NGB*1024*1024*1024/sizeof(float);
float *Array = new float [N];

// arbitrary work on Array
// ...

delete [] Array;
exit(0);
}

The "top" command shows that the demo program consumes about 18.3 GB, which is about 2 times larger than expected!!
However, if i comment out the line "cudaSetDevice( 0 );", the memory consumption becomes normal (about 9 GB).
The information of the system giving this issue is as follows.

CUDA version: 4.0 RC2
Driver version: 270.41.06
OS: Scientific Linux SL release 5.4 (Boron)

I suspect that this issue is due to the new graphic driver and(or) CUDA since this problem appeared after a system upgrade.
I also tested the demo program in a different system with older versions of CUDA and driver, and everything works fine.
In this case, the memory consumption is normal even when I include any CUDA function.
The information of this system is as follows:

CUDA version: 3.2
Driver version: 260.19.21
OS: CentOS release 5.5 (Final)

Has anyone had a similar problem? Any help is appreciated!
[/quote]
I'm pretty sure you're looking at the total memory column, and this is irrelevant.

The RSS column is what your application really uses, and I don't think this one will change dramatically with just the addition of cudaSetDevice().



You can try "pmap -x <pid>" to see where your memory is.



Same thing if you mmap a 200GB file (which sits on disk). You would see 200GB in top. That does not mean you're using 200GB.



-Guillaume



[quote name='llittlefish' date='12 May 2011 - 09:58 AM' timestamp='1305212325' post='1236339']

Hi,



I just found that the consumption of host memory seems to increase abnormally in the new CUDA release.

The following is a very simple demo program.



int main()

{

// set GPU ID

cudaSetDevice( 0 );



// allocate host memory

const long NGB = 9;

const long N = NGB*1024*1024*1024/sizeof(float);

float *Array = new float [N];



// arbitrary work on Array

// ...



delete [] Array;

exit(0);

}



The "top" command shows that the demo program consumes about 18.3 GB, which is about 2 times larger than expected!!

However, if i comment out the line "cudaSetDevice( 0 );", the memory consumption becomes normal (about 9 GB).

The information of the system giving this issue is as follows.



CUDA version: 4.0 RC2

Driver version: 270.41.06

OS: Scientific Linux SL release 5.4 (Boron)



I suspect that this issue is due to the new graphic driver and(or) CUDA since this problem appeared after a system upgrade.

I also tested the demo program in a different system with older versions of CUDA and driver, and everything works fine.

In this case, the memory consumption is normal even when I include any CUDA function.

The information of this system is as follows:



CUDA version: 3.2

Driver version: 260.19.21

OS: CentOS release 5.5 (Final)



Has anyone had a similar problem? Any help is appreciated!

#2
Posted 05/26/2011 01:09 PM   
Thomasco, thanks a lot for your reply /no.gif' class='bbc_emoticon' alt=':no:' />
You are right. Here I refer to the "VIRT" memory shown by the top command.
The actual memory consumption (the RSS column shown by "pmap -x") is normal.
However, abnormal virtual memory consumption is still an unpleasant property.
Especially, in the system with a limited amount of virtual memory, the program
can crash due to the enormous consumption of virtual memory.

I have verified that the abnormal virtual memory consumption only occurs in CUDA 4.0
(in both RC2 and the production release).
If I reinstall both driver and CUDA toolkit to version 3.2, this issue no longer appears.

Does anyone has any suggestion?
Thomasco, thanks a lot for your reply /no.gif' class='bbc_emoticon' alt=':no:' />

You are right. Here I refer to the "VIRT" memory shown by the top command.

The actual memory consumption (the RSS column shown by "pmap -x") is normal.

However, abnormal virtual memory consumption is still an unpleasant property.

Especially, in the system with a limited amount of virtual memory, the program

can crash due to the enormous consumption of virtual memory.



I have verified that the abnormal virtual memory consumption only occurs in CUDA 4.0

(in both RC2 and the production release).

If I reinstall both driver and CUDA toolkit to version 3.2, this issue no longer appears.



Does anyone has any suggestion?

#3
Posted 06/01/2011 08:55 AM   
I also noticed this on my seven GPU testbed with CUDA 4.0. Once I open a CUDA context on a device, the process grabs 24 GB of virtual memory. The actual memory usage is fine, so there is no reason to be worried, but it is a little weird.

Is this related to the Unified Virtual Addressing feature of CUDA 4?
I also noticed this on my seven GPU testbed with CUDA 4.0. Once I open a CUDA context on a device, the process grabs 24 GB of virtual memory. The actual memory usage is fine, so there is no reason to be worried, but it is a little weird.



Is this related to the Unified Virtual Addressing feature of CUDA 4?

#4
Posted 06/01/2011 02:19 PM   
this is related to UVA. we have to carve out a chunk of virtual memory equal to the total physical GPU memory, plus the total system memory, plus some small fudge factor for alignment purposes.

we actually throttle back on the UVA region if you run out of virtual memory. this will restrict the amount of memory you can allocate, though.
this is related to UVA. we have to carve out a chunk of virtual memory equal to the total physical GPU memory, plus the total system memory, plus some small fudge factor for alignment purposes.



we actually throttle back on the UVA region if you run out of virtual memory. this will restrict the amount of memory you can allocate, though.

#5
Posted 06/01/2011 09:17 PM   
Tmurray, thanks for the explanation /smile.gif' class='bbc_emoticon' alt=':smile:' />
So, do you have any suggestion for the case that the program is terminated due to
the large amount of virtual memory consumption (which exceeds the upper limit
reported by "ulimit")?
Is there any solution to get rid of this issue?
Otherwise, I will have to try to convince the system administrator to reset the upper limit
of virtual memory to "unlimited".
Tmurray, thanks for the explanation /smile.gif' class='bbc_emoticon' alt=':smile:' />

So, do you have any suggestion for the case that the program is terminated due to

the large amount of virtual memory consumption (which exceeds the upper limit

reported by "ulimit")?

Is there any solution to get rid of this issue?

Otherwise, I will have to try to convince the system administrator to reset the upper limit

of virtual memory to "unlimited".

#6
Posted 06/02/2011 03:40 AM   
Scroll To Top