Jetson TK1 memory leak?

Hi,

we are using a Jetson Tegra TK1 board to develop a SW using the GPU’s power. We use CUDA 6.5 version on a 14.04 LTS Ubuntu O.S.
During our developing we have discovered a little increasing of the usage memory, although we made no dynamic allocation (we only call cudaMalloc at the start of the program).
We print CUDA memory as follow:

void printMemStats()
{
  size_t free = 0;
  size_t total = 0;
  float free_f = 0;
  float total_f = 0;
  float alloced_f = 0;
  cudaMemGetInfo(&free, &total);
  free_f = free;
  total_f = total;
  alloced_f = total_f - free_f;
  printf("\nMemory stats:\n");
  printf("\tfree %.0f bytes %.0fkB %.0fMB\n", free_f, free_f / 1024.0f,
	 free_f / 1024.0f / 1024.0f);
  printf("\ttotal %.0f bytes %.0fkB %.0fMB\n", total_f, total_f / 1024.0f,
	 total_f / 1024.0f / 1024.0f);
  printf("\tallocated (total - free) %.0f bytes %.0fkB %.0fMB\n",
	 alloced_f, alloced_f / 1024.0f,
	 alloced_f / 1024.0f / 1024.0f);
  printf("\n");
}

So, we wrote a test program that print the memory stats every hour, with no processing at all. We went ahead with test during all the weekend (about 70 hours).
Below you can find the test result’s log.
As you can see, the allocated memory has increased (very small increase but there is) although the test program was the only process running at the moment (except for the O.S processes).
Do you have any clues about this? Is this a know “issue” on the TK1 boards?

Thank you in advance, best regards
Emanuele

Memory stats:
	free 1950515200 bytes 1904800kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162598912 bytes 158788kB 155MB


Memory stats:
	free 1951436800 bytes 1905700kB 1861MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 161677312 bytes 157888kB 154MB


Memory stats:
	free 1951301632 bytes 1905568kB 1861MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 161812480 bytes 158020kB 154MB


Memory stats:
	free 1951166464 bytes 1905436kB 1861MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 161947648 bytes 158152kB 154MB


Memory stats:
	free 1950515200 bytes 1904800kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162598912 bytes 158788kB 155MB


Memory stats:
	free 1950531584 bytes 1904816kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162582528 bytes 158772kB 155MB


Memory stats:
	free 1950277632 bytes 1904568kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162836480 bytes 159020kB 155MB


Memory stats:
	free 1950277632 bytes 1904568kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162836480 bytes 159020kB 155MB


Memory stats:
	free 1950277632 bytes 1904568kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162836480 bytes 159020kB 155MB


Memory stats:
	free 1950277632 bytes 1904568kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162836480 bytes 159020kB 155MB


Memory stats:
	free 1950150656 bytes 1904444kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162963456 bytes 159144kB 155MB


Memory stats:
	free 1950277632 bytes 1904568kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162836480 bytes 159020kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950294016 bytes 1904584kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162820096 bytes 159004kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950167040 bytes 1904460kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 162947072 bytes 159128kB 155MB


Memory stats:
	free 1950040064 bytes 1904336kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 163074048 bytes 159252kB 156MB


Memory stats:
	free 1950040064 bytes 1904336kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 163074048 bytes 159252kB 156MB


Memory stats:
	free 1950040064 bytes 1904336kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 163074048 bytes 159252kB 156MB


Memory stats:
	free 1950040064 bytes 1904336kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 163074048 bytes 159252kB 156MB


Memory stats:
	free 1950040064 bytes 1904336kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 163074048 bytes 159252kB 156MB


Memory stats:
	free 1950040064 bytes 1904336kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 163074048 bytes 159252kB 156MB


Memory stats:
	free 1950040064 bytes 1904336kB 1860MB
	total 2113114112 bytes 2063588kB 2015MB
	allocated (total - free) 163074048 bytes 159252kB 156MB

Hi,

Thanks for your feedback. This problem is a known issue.
We have feedbacked this to the core developer and will be prioritized internally.

Thanks and sorry for the inconvenience.

Hi,

could you suggest us a workaround or a solution that we can apply at the moment?
Could you provide us a plan for the issue’s resolution?

Thanks in advance for your feedback.

Hi,

This issue is fixed in R21.6.
Please reflash tk1 with our latest OS:
https://developer.nvidia.com/linux-tegra-r216

Thanks.

Hi @AastaLLL,

thank you for the info.

We reflashed the TK1 board with the R21.6 OS version.
However we have some trouble while compiling CUDA code in our Ubuntu machine. We updated the cuda arm libs with the ones contained into the R21.6 and we changed the CUDA runtime library as shared (no more static) because the new libraries are shared object. So, we received a linker error:

/usr/arm-linux-gnueabihf/bin/ld: cannot find -lcudadevrt

So, I was wondering, where can I find the libcudadevrt.so?
Or should I compile CUDA code with the previous static libs?

Many thanks,
Emanuele

Hi,

Want to confirm first:
Have you installed CUDA toolkit via JetPack3.1?

Thanks.

Hi AastaLLL,

We’ve installed CUDA toolkit via JetPack3.0.
Do we need to update to 3.1?

Thanks.

Hi,

We will double check the CUDA functionality and update information to you later.
Thanks.

Hi,

We have tested rel-21.6 with CUDA-6.5 containing in JetPack3.1.
We can compile CUDA sample without error.

Could you try if you can compile the vectorAdd sample located at ‘/home/ubuntu/NVIDIA_CUDA-6.5_Samples/0_Simple/vectorAdd’?

Thanks.

Hi AastaLLL,

yes it compiles.
However, there is one important thing which I would like to please clarify with you. My understanding is that the r21.6 just goes to execute somethings on the board (Jetson TK1) i.e. it goes to change, on the

rootfs

folder, new bins, libs and kernel, in case we need it.
So, on the host side, no change happens (as you can see from the attached file, the r21.5, version contained on the JetPack3.1 only appears on target side). Hence, I could build my code on the host machine without changes on the version.
Please, can you confirm my assumption?

Many thanks in advance,
Emanuele

Hi AastaLLL,

please, do you have any update on my previous comment?

Thanks and regards

Hi,

Sorry for the late response.

Rel21.6 is the OS of TK1.
If you cross-compile the TK1 application on x86-machine, you may need to update some libraries to rel-21.6.
May I know your environment? Do you develop app through Nsight?

Thanks.

Hi,

yes, we cross-compile through a x64 Ubuntu 14.04 LTS. And yes, we use Nsight.
So, what lib should I need to update? Because, as I said on the post ID 5, we have some troubles changing the lib path (previously we had static ones, with the new R21.6 we only view shared lib).

Many thanks, regards
Emanuele

Hi,

Could you share a simple source that can reproduce this issue?
We want to check this in detail.

Thanks.

Hi,

attached you can find our test program.
Test.tar.gz (5.25 KB)

Hi,

Could you apply these in Settings->NVCC Linker:->Libraries:

1. Add this path in Library search path
/usr/local/cuda-6.5/targets/x86_64-linux/lib

2. Select ‘Shared’ for CUDA Runtime Library

Please check comment #22.
Thanks.

Hi AastaLLL,

my understanding is that TK1 boards mount an ARM® Cortex™-A15, that is an ARM 32-bit, so it has a different instruction set architecture than x86_64-linux. So, please, why do I have to use libs for x86_64 architecture?

Anyway, I tried it out, but, as expected, I received the following linker error:

/usr/local/cuda-6.5/targets/x86_64-linux/lib/libcudart.so: file not recognized: File format not recognized

Hi,

For CUDA, general libraries is placed at ‘[CUDA_ROOT]/lib’ or ‘[CUDA_ROOT]/lib64’.
‘[CUDA_ROOT]/targets/…’ is used for cross-compiler a device application.
Please check comment #22.

Could you share the list the include/linking path in Nsight?
Thanks.

Hi,

sure, the include path:

/usr/local/cuda-6.5/targets/armv7-linux-gnueabihf/include

the linking path:

/usr/local/cuda-6.5/targets/armv7-linux-gnueabihf/lib

cause as I said, we compile for an ARM® Cortex™-A15 32-bit, that is an armv7-linux.

By the way the problem at building time is that armv7-linux only contains static libs, while in R21.6 we can only find shared libs. So, where can we find armv7-linux .so libs?

Thanks and regards,
Emanuele

Hi,

Could you select ‘None’ for CUDA Runtime Library and recheck it?
Please check comment #22.
Thanks