I’m about to learn TensorFlow and saw an article suggesting that for someone new, the right thing is to learn 2.0:
I’m wondering if it is supported or will be or if anyone has experience/thoughts/suggestions?
thanks!
Hi,
TensorFlow 2.0 prebuilt for Nano is not available yet.
Our latest package is v1.13.1 but we keep updating it.
Please pay attention to this topic for our announcement:
[url]https://devtalk.nvidia.com/default/topic/1048776/jetson-nano/official-tensorflow-for-jetson-nano-/post/5322533/#5322533[/url]
Thanks.
Dear Developers,
TensorFlow 2.0 makes really life much easier, and it would be nice to have it in Nano…
terveisin, Markus
Any progress? Its gone 1/2 a year… Are you guys playing golf?
Markus
Hi,
Unfortunately, our latest TensorFlow package is v1.14.
We will pass your request to our internal team again.
Sorry for the inconvenience.
Hi
I made a post where I detailed how to compile tensorflow 2 for jetson nano
I also provide the python wheel as it’s takes lot of time to compile.
Best,
Wov,thank’s!!!
I tried and failed myself, did not find the right combination of bazel and other libraries.
Great that you made it!
Markus
Hmm, did not work for me, this is what I did in Jetson Nano:
wget https://pythops.s3-eu-west-1.amazonaws.com/builds/tensorflow_gpu-2.0.0-cp36-cp36m-linux_aarch64.whl
pip3 install -U --user pip six numpy wheel setuptools mock
pip3 install -U --user keras_applications keras_preprocessing --no-deps
pip3 install --user --upgrade grpcio==1.24.3
pip3 install --user ./tensorflow_gpu-2.0.0-cp36-cp36m-linux_aarch64.whl
I get wierd errors, cannot run even simple gpu-program (cpu part is fine, gpu fails) like:
import time
import tensorflow as tf
cpu_slot = 0
gpu_slot = 0
# Using CPU at slot 0
with tf.device('/CPU:' + str(cpu_slot)):
# Starting a timer
start = time.time()
# Doing operations on CPU
A = tf.constant([[3, 2], [5, 2]])
print(tf.eye(2,2))
# Printing how long it took with CPU
end = time.time() - start
print(end)
# Using the GPU at slot 0
with tf.device('/GPU:' + str(gpu_slot)):
# Starting a timer
start = time.time()
# Doing operations on CPU
A = tf.constant([[3, 2], [5, 2]])
print(tf.eye(2,2))
# Printing how long it took with CPU
end = time.time() - start
print(end)
OUTPUT:
tf.Tensor(
[[1. 0.]
[0. 1.]], shape=(2, 2), dtype=float32)
0.06385016441345215
2019-12-03 05:20:55.231268: F tensorflow/stream_executor/cuda/cuda_driver.cc:175] Check failed: err == cudaSuccess || err == cudaErrorInvalidValue Unexpected CUDA error: unknown error
Other error with a neural net was (see Non-OK-status: Internal: invalid configuration argument Aborted (core dumped) · Issue #13093 · keras-team/keras · GitHub):
F ./tensorflow/core/kernels/random_op_gpu.h:227] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: unknown error
Which CUDA version you are using? (I have 10.0) According to some posts, this could be cuda-version-dependend.
Terveisin, Markus
Ok, probably got it:
In Deploy custom object detection, designed on tensorflow 1.14 - Jetson Nano - NVIDIA Developer Forums
There are some dependencies between TensorFlow and CUDA.
You will need to use the identical CUDA driver/toolkit which the package is built with.
So it seems I must either build from source and not use your wheel, or install the same cuda (and probably there is something else as well…) as you have.
I try the source first…
Terveisin, Markus
Hi,
I’m using cuda 10. I downloaded it using sdk manager
$ dpkg -l | grep cuda-toolkit
ii cuda-toolkit-10-0 10.0.326-1 arm64 CUDA Toolkit 10.0 meta-package
@markus
Can you run to see it does detect the gpu ?
python3 -c "import tensorflow as tf; print(tf.test.is_gpu_available())"
I think it’s related to cuda version
I got this error too.
>>> import tensorflow as tf
>>> tf.test.is_gpu_available()
2019-12-03 15:14:00.457759: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2019-12-03 15:14:00.458324: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x37d06ee0 executing computations on platform Host. Devices:
2019-12-03 15:14:00.458374: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version
2019-12-03 15:14:00.523929: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-12-03 15:14:00.682957: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 15:14:00.683468: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x37d049a0 executing computations on platform CUDA. Devices:
2019-12-03 15:14:00.683524: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2019-12-03 15:14:00.683935: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 15:14:00.684051: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2019-12-03 15:14:00.696288: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-03 15:14:00.723665: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-03 15:14:00.743580: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-03 15:14:00.790846: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-03 15:14:00.823362: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-03 15:14:00.851777: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-03 15:14:00.888587: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-03 15:14:00.888867: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 15:14:00.889126: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 15:14:00.889233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-03 15:14:00.889429: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-03 15:14:00.890396: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-03 15:14:00.890449: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2019-12-03 15:14:00.890476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2019-12-03 15:14:00.890728: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 15:14:00.891017: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 15:14:00.891159: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/device:GPU:0 with 2270 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
True
>>> tf.debugging.set_log_device_placement(True)
>>>
>>> a = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0]])
2019-12-03 15:14:11.076714: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 15:14:11.077411: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2019-12-03 15:14:11.077849: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-03 15:14:11.078090: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-03 15:14:11.078270: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-03 15:14:11.078453: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-03 15:14:11.078617: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-03 15:14:11.078793: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-03 15:14:11.079122: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-03 15:14:11.079542: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 15:14:11.080004: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 15:14:11.080247: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-03 15:14:11.083934: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 15:14:11.084130: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2019-12-03 15:14:11.084244: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-03 15:14:11.084317: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-03 15:14:11.084378: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-03 15:14:11.084443: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-03 15:14:11.084501: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-03 15:14:11.084567: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-03 15:14:11.084623: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-03 15:14:11.084836: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 15:14:11.085143: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 15:14:11.085284: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-03 15:14:11.085383: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-03 15:14:11.085441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2019-12-03 15:14:11.085484: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2019-12-03 15:14:11.085724: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 15:14:11.086047: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 15:14:11.086225: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2270 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
>>> b = tf.constant([[1.0, 2.0], [3.0, 4.0], [5.0, 6.0]])
>>>
>>>
>>> c = tf.matmul(a, b)
2019-12-03 15:14:16.993486: I tensorflow/core/common_runtime/eager/execute.cc:574] Executing op MatMul in device /job:localhost/replica:0/task:0/device:GPU:0
2019-12-03 15:14:17.047313: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-03 15:18:10.046885: F tensorflow/stream_executor/cuda/cuda_driver.cc:175] Check failed: err == cudaSuccess || err == cudaErrorInvalidValue Unexpected CUDA error: unknown error
Aborted
Yes, To me it looks like detecting GPU (see below):
Also cuda version is the same 10.0,
and both have libcudnn.so.7
Have you tried
I failed with it, but cannot remember what was the problem, giving another try now.
Did I get it wight: your tensorflow2 build cannot do matrix multiplication with GPU ?
Cheers, Markus
dlinano@jetson-nano:~$ python3
Python 3.6.8 (default, Oct 7 2019, 12:59:55)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorflow as tf
>>>
>>> tf.test.is_gpu_available()
2019-12-03 05:23:52.676044: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2019-12-03 05:23:52.677015: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x242bb840 executing computations on platform Host. Devices:
2019-12-03 05:23:52.677080: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): Host, Default Version
2019-12-03 05:23:52.684981: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2019-12-03 05:23:52.771415: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 05:23:52.771709: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x242b1480 executing computations on platform CUDA. Devices:
2019-12-03 05:23:52.771763: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2019-12-03 05:23:52.772241: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 05:23:52.772358: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2019-12-03 05:23:52.773417: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-03 05:23:52.778872: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2019-12-03 05:23:52.781903: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2019-12-03 05:23:52.782734: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2019-12-03 05:23:52.786244: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2019-12-03 05:23:52.789204: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2019-12-03 05:23:52.797546: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2019-12-03 05:23:52.797870: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 05:23:52.798133: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 05:23:52.798216: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2019-12-03 05:23:52.798365: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2019-12-03 05:23:52.799491: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-03 05:23:52.799539: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2019-12-03 05:23:52.799568: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2019-12-03 05:23:52.799819: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 05:23:52.800057: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:973] ARM64 does not support NUMA - returning NUMA node zero
2019-12-03 05:23:52.800200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/device:GPU:0 with 1347 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
True
>>>
It stuck at
2019-12-03 15:14:17.047313: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
and then it return this error
2019-12-03 15:18:10.046885: F tensorflow/stream_executor/cuda/cuda_driver.cc:175] Check failed: err == cudaSuccess || err == cudaErrorInvalidValue Unexpected CUDA error: unknown error
Aborted
Similar bug has been observed with tf2 so it’s most likely a bug in tf/cuda . I submitted an issue in TensorFlow repository, hopefully they can confirm the bug and release a fix/workaround
Ok,
Unfortunately we are stuck with cuda 10.0 as the newest version as far I understand (I have had jetson nano for 1/2 year now and to my experience we can expect zero support from NVIDIA).
Markus
Ok,
Managed tensorflow2 in jetson nano with the help of
the wheel file is in ( it saves you about 80h of compilation time)
To my experience, nvidia support has zero values.
Markus
Nvidia has released tensorflow 2
https://developer.download.nvidia.com/compute/redist/jp/v42/tensorflow-gpu/
Thank’s, that was too late for me, but good for others!
Hi,
TensorFlow 2.0 can be installed with JetPack4.3 now.
https://docs.nvidia.com/deeplearning/frameworks/install-tf-jetson-platform/index.html#install
$ sudo pip3 install --pre --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v43 tensorflow-gpu
Thanks.
Wow, we got Tensorflow 2 for Christmas. I will eagerly evaluate it.