CUDA limitations
Hi,

I noticed that errors, freeze and system crash happens when launching too much applications simultaneously. Errors begin happening when I launch 80 times the application of the SDK adding 2 float vectors in a third one. Does someone knows what are the causes of that problem please ? There are my two leads :
1 ) If CUDA create a CUDA Stream for each app, it could be a difficulty managing 80 CUDA Streams simultaneously
2 ) If CUDA puts operations from all apps in the same default CUDA Stream, it could be a difficulty scheduling many operations (but as nothing is asynchronous, there are only at most 80 operations simultaneously, that's not a big number).

Best regards
Hi,



I noticed that errors, freeze and system crash happens when launching too much applications simultaneously. Errors begin happening when I launch 80 times the application of the SDK adding 2 float vectors in a third one. Does someone knows what are the causes of that problem please ? There are my two leads :

1 ) If CUDA create a CUDA Stream for each app, it could be a difficulty managing 80 CUDA Streams simultaneously

2 ) If CUDA puts operations from all apps in the same default CUDA Stream, it could be a difficulty scheduling many operations (but as nothing is asynchronous, there are only at most 80 operations simultaneously, that's not a big number).



Best regards

#1
Posted 02/27/2012 10:52 AM   
I'd assume the problem is a shortage of device memory for all the contexts.
Just out of curiosity: Why do you want to run 80 CUDA apps on the same device in parallel?
I'd assume the problem is a shortage of device memory for all the contexts.

Just out of curiosity: Why do you want to run 80 CUDA apps on the same device in parallel?

Always check return codes of CUDA calls for errors. Do not use __syncthreads() in conditional code unless the condition is guaranteed to evaluate identically for all threads of each block. Run your program under cuda-memcheck to detect stray memory accesses. If your kernel dies for larger problem sizes, it might exceed the runtime limit and trigger the watchdog timer.

#2
Posted 02/27/2012 11:15 AM   
Sorry for the late answer, I was sick. Some applications doesn't have enough data to really efficiently use GPU. If they are all submited simultaneously, they take less time to execute. I'm also considering sharing GPU between multiple clients (like virtual machines).
Sorry for the late answer, I was sick. Some applications doesn't have enough data to really efficiently use GPU. If they are all submited simultaneously, they take less time to execute. I'm also considering sharing GPU between multiple clients (like virtual machines).

#3
Posted 03/05/2012 09:12 AM   
If buying a card with a lot of memory doesn't help (there are Geforce cards with 3GB of memory), maybe you can run a server process that does the GPU work on behalf of the other programs using just a single context.

You could also free the context in between, but that is probably too slow to gain any advantage over using just the CPU alone.
If buying a card with a lot of memory doesn't help (there are Geforce cards with 3GB of memory), maybe you can run a server process that does the GPU work on behalf of the other programs using just a single context.



You could also free the context in between, but that is probably too slow to gain any advantage over using just the CPU alone.

Always check return codes of CUDA calls for errors. Do not use __syncthreads() in conditional code unless the condition is guaranteed to evaluate identically for all threads of each block. Run your program under cuda-memcheck to detect stray memory accesses. If your kernel dies for larger problem sizes, it might exceed the runtime limit and trigger the watchdog timer.

#4
Posted 03/06/2012 10:03 AM   
That's what I'm doing, creating a "server" receiving gpu treatment requests and efficiently executing them on available GPUs. I was just trying to understand why running many CUDA applications crashed my system. Thanks for your help.
That's what I'm doing, creating a "server" receiving gpu treatment requests and efficiently executing them on available GPUs. I was just trying to understand why running many CUDA applications crashed my system. Thanks for your help.

#5
Posted 03/08/2012 12:50 PM   
Scroll To Top