System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
- TensorFlow installed from (source or binary): source
- TensorFlow version (use command below): 1.8.0
- TensorRT version: 4.0.0.3
- Bazel version (if compiling from source): 0.10.1
- CUDA/cuDNN version: 9.0/7.1.3
- GPU model and memory: GTX 1080 8gb
- Exact command to reproduce: ./run_all.sh https://developer.download.nvidia.com/devblogs/tftrt_sample.tar.xz
Source code / logs
Namespace(FP16=True, FP32=True, INT8=True, batch_size=4, dump_diff=False, native=True, num_loops=10, topN=5, update_graphdef=False, with_timeline=False, workspace_size=2048)
Starting at 2018-05-08 21:55:57.996556
2018-05-08 21:55:58.083232: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-05-08 21:55:58.083539: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.86
pciBusID: 0000:01:00.0
totalMemory: 7.93GiB freeMemory: 7.42GiB
2018-05-08 21:55:58.083552: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-05-08 21:55:58.293783: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-05-08 21:55:58.293812: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0
2018-05-08 21:55:58.293817: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N
2018-05-08 21:55:58.293942: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6088 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
INFO:tensorflow:Starting execution
2018-05-08 21:55:58.682060: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-05-08 21:55:58.682096: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-05-08 21:55:58.682101: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0
2018-05-08 21:55:58.682104: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N
2018-05-08 21:55:58.682190: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6088 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
INFO:tensorflow:Starting Warmup cycle
INFO:tensorflow:Warmup done. Starting real timing
iter 0 0.012465400695800781
iter 1 0.012455949783325196
iter 2 0.012448663711547852
iter 3 0.012451510429382324
iter 4 0.012464923858642578
iter 5 0.012446017265319824
iter 6 0.012460331916809082
iter 7 0.012507786750793457
iter 8 0.012440338134765624
iter 9 0.012530536651611328
Comparison= True
INFO:tensorflow:Timing loop done!
images/s : 320.8 +/- 0.7, s/batch: 0.01247 +/- 0.00003
RES, Native, 4, 320.84, 0.71, 0.01247, 0.00003
2018-05-08 21:56:06.104449: I tensorflow/core/grappler/devices.cc:51] Number of eligible GPUs (core count >= 8): 1
2018-05-08 21:56:06.550307: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2660] Max batch size= 4 max workspace size= 2147483648
2018-05-08 21:56:06.550338: I tensorflow/contrib/tensorrt/convert/convert_nodes.cc:2666] starting build engine
munmap_chunk(): invalid pointer
./run_all.sh: line 13: 21719 Aborted (core dumped) python tftrt_sample.py --native --FP32 --FP16 --INT8 --num_loops 10 --topN 5 --batch_size 4 --workspace_size 2048 --log_file log.txt --network resnet_v1_50_frozen.pb --input_node input --output_nodes resnet_v1_50/predictions/Reshape_1 --img_size 224 --img_file grace_hopper.jpg