When training my FasterRCNN on my own custom dataset, I run into the issue of
No positive ROIs
If will then crash, on a random epoch. here is the console output:
.
.
.
TF_reshape_3_regr (TFReshape) (1, 256, 4) 0 dense_regress[0][0]
==================================================================================================
Total params: 42,947,507
Trainable params: 42,516,403
Non-trainable params: 431,104
__________________________________________________________________________________________________
2019-10-28 20:15:43,162 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loading pretrained weights from /workspace/tlt-experiments/pretrained_models/tlt_resnet50_faster_rcnn_v1/resnet50.h5
2019-10-28 20:15:45,221 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Pretrained weights loaded!
2019-10-28 20:15:45,387 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: training example num: 47
2019-10-28 20:15:45,679 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Starting training
2019-10-28 20:15:45,679 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Epoch 1/12
Found 47 examples in training dataset, valid image extension isjpg, jpeg and png(case sensitive)
Compressed_class_mapping: {u'paint': 1, u'pore': 0}
Name mapping:{u'paint': u'paint', u'pore': u'pore'}
Training dataset stats(compressed via class mapping):
{u'paint': 65, u'pore': 73}
No positive ROIs.
WARNING:tensorflow:From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
2019-10-28 20:15:52,185 [WARNING] tensorflow: From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
2019-10-28 20:16:04.393678: I tensorflow/stream_executor/dso_loader.cc:152] successfully opened CUDA library libcublas.so.10.0 locally
2019-10-28 20:16:05.004549: E tensorflow/stream_executor/cuda/cuda_driver.cc:806] failed to allocate 2.49G (2674714112 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2019-10-28 20:16:05.368621: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.54GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-10-28 20:16:05.368663: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.54GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-10-28 20:16:05.456343: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.38GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-10-28 20:16:05.456368: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.38GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-10-28 20:16:05.870316: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.38GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-10-28 20:16:05.870359: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.38GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-10-28 20:16:05.955358: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.35GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-10-28 20:16:05.955379: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.35GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-10-28 20:16:06.740765: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.54GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-10-28 20:16:06.740790: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.54GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
1/47 [..............................] - ETA: 17:20 - rpn_cls: 0.8178 - rpn_regr: 0.0731 - detector_cls: 0.5741 - detector_regr: 0.0000e+00No positive ROIs.
2/47 [>.............................] - ETA: 8:39 - rpn_cls: 0.7998 - rpn_regr: 0.0612 - detector_cls: 0.4603 - detector_regr: 0.0000e+00 No positive ROIs.
3/47 [>.............................] - ETA: 5:45 - rpn_cls: 0.7892 - rpn_regr: 0.0592 - detector_cls: 0.3998 - detector_regr: 0.0000e+00No positive ROIs.
4/47 [=>............................] - ETA: 4:18 - rpn_cls: 0.7805 - rpn_regr: 0.0563 - detector_cls: 0.3576 - detector_regr: 0.0000e+00No positive ROIs.
5/47 [==>...........................] - ETA: 3:25 - rpn_cls: 0.7711 - rpn_regr: 0.0537 - detector_cls: 0.3252 - detector_regr: 0.0000e+00No positive ROIs.
6/47 [==>...........................] - ETA: 2:50 - rpn_cls: 0.7611 - rpn_regr: 0.0515 - detector_cls: 0.3032 - detector_regr: 0.0000e+00No positive ROIs.
7/47 [===>..........................] - ETA: 2:25 - rpn_cls: 0.7518 - rpn_regr: 0.0500 - detector_cls: 0.2849 - detector_regr: 0.0000e+00No positive ROIs.
8/47 [====>.........................] - ETA: 2:06 - rpn_cls: 0.7430 - rpn_regr: 0.0487 - detector_cls: 0.2689 - detector_regr: 0.0000e+00No positive ROIs.
9/47 [====>.........................] - ETA: 1:51 - rpn_cls: 0.7344 - rpn_regr: 0.0484 - detector_cls: 0.2546 - detector_regr: 0.0000e+00No positive ROIs.
10/47 [=====>........................] - ETA: 1:39 - rpn_cls: 0.7260 - rpn_regr: 0.0481 - detector_cls: 0.2419 - detector_regr: 0.0000e+00No positive ROIs.
11/47 [======>.......................] - ETA: 1:29 - rpn_cls: 0.7176 - rpn_regr: 0.0476 - detector_cls: 0.2305 - detector_regr: 0.0000e+00No positive ROIs.
12/47 [======>.......................] - ETA: 1:20 - rpn_cls: 0.7094 - rpn_regr: 0.0471 - detector_cls: 0.2202 - detector_regr: 0.0000e+00No positive ROIs.
13/47 [=======>......................] - ETA: 1:13 - rpn_cls: 0.7015 - rpn_regr: 0.0467 - detector_cls: 0.2108 - detector_regr: 0.0000e+00No positive ROIs.
14/47 [=======>......................] - ETA: 1:07 - rpn_cls: 0.6938 - rpn_regr: 0.0463 - detector_cls: 0.2023 - detector_regr: 0.0000e+00No positive ROIs.
15/47 [========>.....................] - ETA: 1:02 - rpn_cls: 0.6861 - rpn_regr: 0.0459 - detector_cls: 0.1945 - detector_regr: 0.0000e+00No positive ROIs.
16/47 [=========>....................] - ETA: 57s - rpn_cls: 0.6785 - rpn_regr: 0.0455 - detector_cls: 0.1874 - detector_regr: 0.0000e+00 No positive ROIs.
17/47 [=========>....................] - ETA: 52s - rpn_cls: 0.6710 - rpn_regr: 0.0451 - detector_cls: 0.1808 - detector_regr: 0.0000e+00No positive ROIs.
18/47 [==========>...................] - ETA: 49s - rpn_cls: 0.6635 - rpn_regr: 0.0447 - detector_cls: 0.1747 - detector_regr: 0.0000e+00No positive ROIs.
19/47 [===========>..................] - ETA: 45s - rpn_cls: 0.6561 - rpn_regr: 0.0443 - detector_cls: 0.1691 - detector_regr: 0.0000e+00No positive ROIs.
20/47 [===========>..................] - ETA: 42s - rpn_cls: 0.6487 - rpn_regr: 0.0440 - detector_cls: 0.1639 - detector_regr: 0.0000e+00No positive ROIs.
21/47 [============>.................] - ETA: 39s - rpn_cls: 0.6414 - rpn_regr: 0.0437 - detector_cls: 0.1590 - detector_regr: 0.0000e+00No positive ROIs.
22/47 [=============>................] - ETA: 36s - rpn_cls: 0.6342 - rpn_regr: 0.0435 - detector_cls: 0.1544 - detector_regr: 0.0000e+00No positive ROIs.
23/47 [=============>................] - ETA: 34s - rpn_cls: 0.6271 - rpn_regr: 0.0433 - detector_cls: 0.1501 - detector_regr: 0.0000e+00No positive ROIs.
24/47 [==============>...............] - ETA: 31s - rpn_cls: 0.6201 - rpn_regr: 0.0431 - detector_cls: 0.1461 - detector_regr: 0.0000e+00No positive ROIs.
25/47 [==============>...............] - ETA: 29s - rpn_cls: 0.6133 - rpn_regr: 0.0430 - detector_cls: 0.1423 - detector_regr: 0.0000e+00No positive ROIs.
26/47 [===============>..............] - ETA: 27s - rpn_cls: 0.6065 - rpn_regr: 0.0428 - detector_cls: 0.1387 - detector_regr: 0.0000e+00No positive ROIs.
27/47 [================>.............] - ETA: 25s - rpn_cls: 0.5999 - rpn_regr: 0.0426 - detector_cls: 0.1353 - detector_regr: 0.0000e+00No positive ROIs.
28/47 [================>.............] - ETA: 23s - rpn_cls: 0.5933 - rpn_regr: 0.0425 - detector_cls: 0.1322 - detector_regr: 0.0000e+00No positive ROIs.
29/47 [=================>............] - ETA: 22s - rpn_cls: 0.5869 - rpn_regr: 0.0424 - detector_cls: 0.1291 - detector_regr: 0.0000e+00No positive ROIs.
30/47 [==================>...........] - ETA: 20s - rpn_cls: 0.5805 - rpn_regr: 0.0423 - detector_cls: 0.1263 - detector_regr: 0.0000e+00No positive ROIs.
31/47 [==================>...........] - ETA: 18s - rpn_cls: 0.5743 - rpn_regr: 0.0421 - detector_cls: 0.1235 - detector_regr: 0.0000e+00No positive ROIs.
32/47 [===================>..........] - ETA: 17s - rpn_cls: 0.5681 - rpn_regr: 0.0420 - detector_cls: 0.1209 - detector_regr: 0.0000e+00No positive ROIs.
33/47 [====================>.........] - ETA: 15s - rpn_cls: 0.5621 - rpn_regr: 0.0419 - detector_cls: 0.1185 - detector_regr: 0.0000e+00No positive ROIs.
34/47 [====================>.........] - ETA: 14s - rpn_cls: 0.5561 - rpn_regr: 0.0418 - detector_cls: 0.1161 - detector_regr: 0.0000e+00No positive ROIs.
35/47 [=====================>........] - ETA: 13s - rpn_cls: 0.5502 - rpn_regr: 0.0417 - detector_cls: 0.1138 - detector_regr: 0.0000e+00No positive ROIs.
36/47 [=====================>........] - ETA: 11s - rpn_cls: 0.5445 - rpn_regr: 0.0415 - detector_cls: 0.1117 - detector_regr: 0.0000e+00No positive ROIs.
37/47 [======================>.......] - ETA: 10s - rpn_cls: 0.5389 - rpn_regr: 0.0415 - detector_cls: 0.1096 - detector_regr: 0.0000e+00No positive ROIs.
38/47 [=======================>......] - ETA: 9s - rpn_cls: 0.5333 - rpn_regr: 0.0414 - detector_cls: 0.1076 - detector_regr: 0.0000e+00 No positive ROIs.
39/47 [=======================>......] - ETA: 8s - rpn_cls: 0.5279 - rpn_regr: 0.0413 - detector_cls: 0.1057 - detector_regr: 0.0000e+00No positive ROIs.
40/47 [========================>.....] - ETA: 7s - rpn_cls: 0.5226 - rpn_regr: 0.0412 - detector_cls: 0.1039 - detector_regr: 0.0000e+00No positive ROIs.
41/47 [=========================>....] - ETA: 6s - rpn_cls: 0.5174 - rpn_regr: 0.0412 - detector_cls: 0.1021 - detector_regr: 0.0000e+00No positive ROIs.
42/47 [=========================>....] - ETA: 4s - rpn_cls: 0.5123 - rpn_regr: 0.0411 - detector_cls: 0.1004 - detector_regr: 0.0000e+00No positive ROIs.
43/47 [==========================>...] - ETA: 3s - rpn_cls: 0.5073 - rpn_regr: 0.0410 - detector_cls: 0.0988 - detector_regr: 0.0000e+00No positive ROIs.
44/47 [===========================>..] - ETA: 2s - rpn_cls: 0.5024 - rpn_regr: 0.0410 - detector_cls: 0.0972 - detector_regr: 0.0000e+00No positive ROIs.
45/47 [===========================>..] - ETA: 1s - rpn_cls: 0.4976 - rpn_regr: 0.0409 - detector_cls: 0.0957 - detector_regr: 0.0000e+00No positive ROIs.
46/47 [============================>.] - ETA: 0s - rpn_cls: 0.4929 - rpn_regr: 0.0409 - detector_cls: 0.0942 - detector_regr: 0.0000e+00No positive ROIs.
47/47 [==============================] - 44s 936ms/step - rpn_cls: 0.4883 - rpn_regr: 0.0408 - detector_cls: 0.0928 - detector_regr: 0.0000e+00
2019-10-28 20:16:29,661 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Mean number of bounding boxes from RPN overlapping ground truth boxes: 0.0
2019-10-28 20:16:29,662 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Classifier accuracy for bounding boxes from RPN: 1.0
2019-10-28 20:16:29,662 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss RPN classifier: 0.275080920098
2019-10-28 20:16:29,662 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss RPN regression: 0.0379338169748
2019-10-28 20:16:29,662 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss Detector classifier: 0.0275653046019
2019-10-28 20:16:29,662 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss Detector regression: 0.0
2019-10-28 20:16:29,662 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Elapsed time: 43.9829471111
2019-10-28 20:16:29,662 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Total loss changed from inf to 0.340580041675, saving weights
2019-10-28 20:16:50,733 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Epoch 2/12
No positive ROIs.
1/47 [..............................] - ETA: 22s - rpn_cls: 0.0448 - rpn_regr: 0.0442 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
2/47 [>.............................] - ETA: 21s - rpn_cls: 0.0463 - rpn_regr: 0.0441 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
3/47 [>.............................] - ETA: 20s - rpn_cls: 0.0457 - rpn_regr: 0.0409 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
4/47 [=>............................] - ETA: 19s - rpn_cls: 0.0449 - rpn_regr: 0.0394 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
5/47 [==>...........................] - ETA: 19s - rpn_cls: 0.0444 - rpn_regr: 0.0387 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
6/47 [==>...........................] - ETA: 18s - rpn_cls: 0.0442 - rpn_regr: 0.0380 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
7/47 [===>..........................] - ETA: 18s - rpn_cls: 0.0441 - rpn_regr: 0.0372 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
8/47 [====>.........................] - ETA: 18s - rpn_cls: 0.0438 - rpn_regr: 0.0364 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
9/47 [====>.........................] - ETA: 17s - rpn_cls: 0.0437 - rpn_regr: 0.0358 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
10/47 [=====>........................] - ETA: 17s - rpn_cls: 0.0435 - rpn_regr: 0.0355 - detector_cls: 1.3113e-07 - detector_regr: 0.0000e+00No positive ROIs.
11/47 [======>.......................] - ETA: 16s - rpn_cls: 0.0432 - rpn_regr: 0.0351 - detector_cls: 1.3990e-07 - detector_regr: 0.0000e+00No positive ROIs.
12/47 [======>.......................] - ETA: 16s - rpn_cls: 0.0428 - rpn_regr: 0.0348 - detector_cls: 5.5294e-07 - detector_regr: 0.0000e+00No positive ROIs.
13/47 [=======>......................] - ETA: 15s - rpn_cls: 0.0424 - rpn_regr: 0.0346 - detector_cls: 8.7298e-07 - detector_regr: 0.0000e+00No positive ROIs.
14/47 [=======>......................] - ETA: 15s - rpn_cls: 0.0421 - rpn_regr: 0.0344 - detector_cls: 1.1962e-06 - detector_regr: 0.0000e+00No positive ROIs.
15/47 [========>.....................] - ETA: 14s - rpn_cls: 0.0417 - rpn_regr: 0.0341 - detector_cls: 1.4529e-06 - detector_regr: 0.0000e+00No positive ROIs.
16/47 [=========>....................] - ETA: 14s - rpn_cls: 0.0413 - rpn_regr: 0.0338 - detector_cls: 1.6592e-06 - detector_regr: 0.0000e+00No positive ROIs.
17/47 [=========>....................] - ETA: 13s - rpn_cls: 0.0409 - rpn_regr: 0.0335 - detector_cls: 1.8252e-06 - detector_regr: 0.0000e+00No positive ROIs.
18/47 [==========>...................] - ETA: 13s - rpn_cls: 0.0405 - rpn_regr: 0.0333 - detector_cls: 1.9593e-06 - detector_regr: 0.0000e+00No positive ROIs.
19/47 [===========>..................] - ETA: 12s - rpn_cls: 0.0402 - rpn_regr: 0.0330 - detector_cls: 2.0678e-06 - detector_regr: 0.0000e+00No positive ROIs.
20/47 [===========>..................] - ETA: 12s - rpn_cls: 0.0399 - rpn_regr: 0.0328 - detector_cls: 2.1558e-06 - detector_regr: 0.0000e+00No positive ROIs.
21/47 [============>.................] - ETA: 12s - rpn_cls: 0.0396 - rpn_regr: 0.0326 - detector_cls: 2.2269e-06 - detector_regr: 0.0000e+00No positive ROIs.
22/47 [=============>................] - ETA: 11s - rpn_cls: 0.0394 - rpn_regr: 0.0325 - detector_cls: 2.2843e-06 - detector_regr: 0.0000e+00No positive ROIs.
23/47 [=============>................] - ETA: 11s - rpn_cls: 0.0391 - rpn_regr: 0.0323 - detector_cls: 2.3304e-06 - detector_regr: 0.0000e+00No positive ROIs.
24/47 [==============>...............] - ETA: 10s - rpn_cls: 0.0389 - rpn_regr: 0.0321 - detector_cls: 2.3670e-06 - detector_regr: 0.0000e+00No positive ROIs.
25/47 [==============>...............] - ETA: 10s - rpn_cls: 0.0386 - rpn_regr: 0.0319 - detector_cls: 2.3957e-06 - detector_regr: 0.0000e+00No positive ROIs.
26/47 [===============>..............] - ETA: 9s - rpn_cls: 0.0384 - rpn_regr: 0.0317 - detector_cls: 2.4178e-06 - detector_regr: 0.0000e+00 No positive ROIs.
27/47 [================>.............] - ETA: 9s - rpn_cls: 0.0382 - rpn_regr: 0.0316 - detector_cls: 2.4344e-06 - detector_regr: 0.0000e+00No positive ROIs.
28/47 [================>.............] - ETA: 8s - rpn_cls: 0.0380 - rpn_regr: 0.0315 - detector_cls: 2.4463e-06 - detector_regr: 0.0000e+00No positive ROIs.
29/47 [=================>............] - ETA: 8s - rpn_cls: 0.0378 - rpn_regr: 0.0314 - detector_cls: 2.4542e-06 - detector_regr: 0.0000e+00No positive ROIs.
30/47 [==================>...........] - ETA: 7s - rpn_cls: 0.0376 - rpn_regr: 0.0313 - detector_cls: 2.4588e-06 - detector_regr: 0.0000e+00No positive ROIs.
31/47 [==================>...........] - ETA: 7s - rpn_cls: 0.0375 - rpn_regr: 0.0312 - detector_cls: 2.4605e-06 - detector_regr: 0.0000e+00No positive ROIs.
32/47 [===================>..........] - ETA: 6s - rpn_cls: 0.0373 - rpn_regr: 0.0311 - detector_cls: 2.4597e-06 - detector_regr: 0.0000e+00No positive ROIs.
33/47 [====================>.........] - ETA: 6s - rpn_cls: 0.0372 - rpn_regr: 0.0310 - detector_cls: 2.4569e-06 - detector_regr: 0.0000e+00No positive ROIs.
34/47 [====================>.........] - ETA: 6s - rpn_cls: 0.0370 - rpn_regr: 0.0309 - detector_cls: 2.4523e-06 - detector_regr: 0.0000e+00No positive ROIs.
35/47 [=====================>........] - ETA: 5s - rpn_cls: 0.0369 - rpn_regr: 0.0308 - detector_cls: 2.4462e-06 - detector_regr: 0.0000e+00No positive ROIs.
36/47 [=====================>........] - ETA: 5s - rpn_cls: 0.0367 - rpn_regr: 0.0307 - detector_cls: 2.4387e-06 - detector_regr: 0.0000e+00No positive ROIs.
37/47 [======================>.......] - ETA: 4s - rpn_cls: 0.0366 - rpn_regr: 0.0306 - detector_cls: 2.4302e-06 - detector_regr: 0.0000e+00No positive ROIs.
38/47 [=======================>......] - ETA: 4s - rpn_cls: 0.0364 - rpn_regr: 0.0305 - detector_cls: 2.4207e-06 - detector_regr: 0.0000e+00No positive ROIs.
39/47 [=======================>......] - ETA: 3s - rpn_cls: 0.0363 - rpn_regr: 0.0304 - detector_cls: 2.4105e-06 - detector_regr: 0.0000e+00No positive ROIs.
40/47 [========================>.....] - ETA: 3s - rpn_cls: 0.0361 - rpn_regr: 0.0304 - detector_cls: 2.6760e-06 - detector_regr: 0.0000e+00No positive ROIs.
41/47 [=========================>....] - ETA: 2s - rpn_cls: 0.0360 - rpn_regr: 0.0303 - detector_cls: 2.9209e-06 - detector_regr: 0.0000e+00No positive ROIs.
42/47 [=========================>....] - ETA: 2s - rpn_cls: 0.0359 - rpn_regr: 0.0302 - detector_cls: 3.1470e-06 - detector_regr: 0.0000e+00No positive ROIs.
43/47 [==========================>...] - ETA: 1s - rpn_cls: 0.0358 - rpn_regr: 0.0301 - detector_cls: 3.3560e-06 - detector_regr: 0.0000e+00No positive ROIs.
44/47 [===========================>..] - ETA: 1s - rpn_cls: 0.0356 - rpn_regr: 0.0300 - detector_cls: 3.5492e-06 - detector_regr: 0.0000e+00No positive ROIs.
45/47 [===========================>..] - ETA: 0s - rpn_cls: 0.0355 - rpn_regr: 0.0300 - detector_cls: 3.7280e-06 - detector_regr: 0.0000e+00No positive ROIs.
46/47 [============================>.] - ETA: 0s - rpn_cls: 0.0354 - rpn_regr: 0.0299 - detector_cls: 3.8937e-06 - detector_regr: 0.0000e+00No positive ROIs.
47/47 [==============================] - 22s 465ms/step - rpn_cls: 0.0353 - rpn_regr: 0.0298 - detector_cls: 4.0472e-06 - detector_regr: 0.0000e+00
2019-10-28 20:17:12,582 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Mean number of bounding boxes from RPN overlapping ground truth boxes: 0.0
2019-10-28 20:17:12,582 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Classifier accuracy for bounding boxes from RPN: 1.0
2019-10-28 20:17:12,582 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss RPN classifier: 0.029723406908
2019-10-28 20:17:12,583 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss RPN regression: 0.0265477568981
2019-10-28 20:17:12,583 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss Detector classifier: 1.11088755552e-05
2019-10-28 20:17:12,583 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss Detector regression: 0.0
2019-10-28 20:17:12,583 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Elapsed time: 42.9208798409
2019-10-28 20:17:12,583 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Total loss changed from 0.340580041675 to 0.0562822726817, saving weights
2019-10-28 20:17:16,472 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Epoch 3/12
No positive ROIs.
1/47 [..............................] - ETA: 22s - rpn_cls: 0.0206 - rpn_regr: 0.0165 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
2/47 [>.............................] - ETA: 21s - rpn_cls: 0.0227 - rpn_regr: 0.0174 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
3/47 [>.............................] - ETA: 20s - rpn_cls: 0.0218 - rpn_regr: 0.0184 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
4/47 [=>............................] - ETA: 20s - rpn_cls: 0.0219 - rpn_regr: 0.0192 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
5/47 [==>...........................] - ETA: 19s - rpn_cls: 0.0220 - rpn_regr: 0.0194 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
6/47 [==>...........................] - ETA: 19s - rpn_cls: 0.0220 - rpn_regr: 0.0194 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
7/47 [===>..........................] - ETA: 18s - rpn_cls: 0.0219 - rpn_regr: 0.0192 - detector_cls: 1.0401e-06 - detector_regr: 0.0000e+00No positive ROIs.
8/47 [====>.........................] - ETA: 18s - rpn_cls: 0.0218 - rpn_regr: 0.0190 - detector_cls: 1.6300e-06 - detector_regr: 0.0000e+00No positive ROIs.
9/47 [====>.........................] - ETA: 17s - rpn_cls: 0.0216 - rpn_regr: 0.0187 - detector_cls: 2.0192e-06 - detector_regr: 0.0000e+00No positive ROIs.
10/47 [=====>........................] - ETA: 17s - rpn_cls: 0.0214 - rpn_regr: 0.0187 - detector_cls: 2.2804e-06 - detector_regr: 0.0000e+00No positive ROIs.
11/47 [======>.......................] - ETA: 16s - rpn_cls: 0.0211 - rpn_regr: 0.0186 - detector_cls: 2.4568e-06 - detector_regr: 0.0000e+00No positive ROIs.
12/47 [======>.......................] - ETA: 16s - rpn_cls: 0.0209 - rpn_regr: 0.0184 - detector_cls: 2.5754e-06 - detector_regr: 0.0000e+00No positive ROIs.
13/47 [=======>......................] - ETA: 15s - rpn_cls: 0.0207 - rpn_regr: 0.0183 - detector_cls: 2.8114e-06 - detector_regr: 0.0000e+00No positive ROIs.
14/47 [=======>......................] - ETA: 15s - rpn_cls: 0.0206 - rpn_regr: 0.0181 - detector_cls: 2.9856e-06 - detector_regr: 0.0000e+00No positive ROIs.
15/47 [========>.....................] - ETA: 15s - rpn_cls: 0.0205 - rpn_regr: 0.0180 - detector_cls: 3.1137e-06 - detector_regr: 0.0000e+00No positive ROIs.
16/47 [=========>....................] - ETA: 14s - rpn_cls: 0.0205 - rpn_regr: 0.0179 - detector_cls: 3.2071e-06 - detector_regr: 0.0000e+00No positive ROIs.
17/47 [=========>....................] - ETA: 14s - rpn_cls: 0.0205 - rpn_regr: 0.0178 - detector_cls: 3.2740e-06 - detector_regr: 0.0000e+00No positive ROIs.
18/47 [==========>...................] - ETA: 13s - rpn_cls: 0.0205 - rpn_regr: 0.0178 - detector_cls: 3.3204e-06 - detector_regr: 0.0000e+00No positive ROIs.
19/47 [===========>..................] - ETA: 13s - rpn_cls: 0.0205 - rpn_regr: 0.0177 - detector_cls: 3.3509e-06 - detector_regr: 0.0000e+00No positive ROIs.
20/47 [===========>..................] - ETA: 12s - rpn_cls: 0.0204 - rpn_regr: 0.0176 - detector_cls: 3.3689e-06 - detector_regr: 0.0000e+00No positive ROIs.
21/47 [============>.................] - ETA: 12s - rpn_cls: 0.0204 - rpn_regr: 0.0176 - detector_cls: 3.3770e-06 - detector_regr: 0.0000e+00No positive ROIs.
22/47 [=============>................] - ETA: 11s - rpn_cls: 0.0203 - rpn_regr: 0.0176 - detector_cls: 3.3773e-06 - detector_regr: 0.0000e+00No positive ROIs.
23/47 [=============>................] - ETA: 11s - rpn_cls: 0.0202 - rpn_regr: 0.0175 - detector_cls: 3.3714e-06 - detector_regr: 0.0000e+00No positive ROIs.
24/47 [==============>...............] - ETA: 10s - rpn_cls: 0.0201 - rpn_regr: 0.0175 - detector_cls: 3.3606e-06 - detector_regr: 0.0000e+00No positive ROIs.
25/47 [==============>...............] - ETA: 10s - rpn_cls: 0.0201 - rpn_regr: 0.0175 - detector_cls: 3.3459e-06 - detector_regr: 0.0000e+00No positive ROIs.
26/47 [===============>..............] - ETA: 10s - rpn_cls: 0.0200 - rpn_regr: 0.0175 - detector_cls: 3.3280e-06 - detector_regr: 0.0000e+00No positive ROIs.
27/47 [================>.............] - ETA: 9s - rpn_cls: 0.0200 - rpn_regr: 0.0175 - detector_cls: 3.3077e-06 - detector_regr: 0.0000e+00 No positive ROIs.
28/47 [================>.............] - ETA: 9s - rpn_cls: 0.0200 - rpn_regr: 0.0175 - detector_cls: 3.2861e-06 - detector_regr: 0.0000e+00No positive ROIs.
29/47 [=================>............] - ETA: 8s - rpn_cls: 0.0199 - rpn_regr: 0.0175 - detector_cls: 3.2628e-06 - detector_regr: 0.0000e+00No positive ROIs.
30/47 [==================>...........] - ETA: 8s - rpn_cls: 0.0199 - rpn_regr: 0.0175 - detector_cls: 3.2384e-06 - detector_regr: 0.0000e+00No positive ROIs.
31/47 [==================>...........] - ETA: 7s - rpn_cls: 0.0198 - rpn_regr: 0.0176 - detector_cls: 3.2130e-06 - detector_regr: 0.0000e+00No positive ROIs.
32/47 [===================>..........] - ETA: 7s - rpn_cls: 0.0198 - rpn_regr: 0.0176 - detector_cls: 3.1869e-06 - detector_regr: 0.0000e+00No positive ROIs.
33/47 [====================>.........] - ETA: 6s - rpn_cls: 0.0197 - rpn_regr: 0.0176 - detector_cls: 3.1604e-06 - detector_regr: 0.0000e+00No positive ROIs.
34/47 [====================>.........] - ETA: 6s - rpn_cls: 0.0197 - rpn_regr: 0.0176 - detector_cls: 3.1335e-06 - detector_regr: 0.0000e+00No positive ROIs.
35/47 [=====================>........] - ETA: 5s - rpn_cls: 0.0197 - rpn_regr: 0.0176 - detector_cls: 3.1064e-06 - detector_regr: 0.0000e+00No positive ROIs.
36/47 [=====================>........] - ETA: 5s - rpn_cls: 0.0197 - rpn_regr: 0.0176 - detector_cls: 3.0792e-06 - detector_regr: 0.0000e+00No positive ROIs.
37/47 [======================>.......] - ETA: 4s - rpn_cls: 0.0196 - rpn_regr: 0.0176 - detector_cls: 3.0520e-06 - detector_regr: 0.0000e+00No positive ROIs.
38/47 [=======================>......] - ETA: 4s - rpn_cls: 0.0196 - rpn_regr: 0.0176 - detector_cls: 3.3027e-06 - detector_regr: 0.0000e+00No positive ROIs.
39/47 [=======================>......] - ETA: 3s - rpn_cls: 0.0196 - rpn_regr: 0.0177 - detector_cls: 3.5323e-06 - detector_regr: 0.0000e+00No positive ROIs.
40/47 [========================>.....] - ETA: 3s - rpn_cls: 0.0195 - rpn_regr: 0.0177 - detector_cls: 3.7429e-06 - detector_regr: 0.0000e+00No positive ROIs.
41/47 [=========================>....] - ETA: 2s - rpn_cls: 0.0195 - rpn_regr: 0.0177 - detector_cls: 3.9362e-06 - detector_regr: 0.0000e+00No positive ROIs.
42/47 [=========================>....] - ETA: 2s - rpn_cls: 0.0195 - rpn_regr: 0.0177 - detector_cls: 4.1137e-06 - detector_regr: 0.0000e+00No positive ROIs.
43/47 [==========================>...] - ETA: 1s - rpn_cls: 0.0194 - rpn_regr: 0.0177 - detector_cls: 4.2769e-06 - detector_regr: 0.0000e+00No positive ROIs.
44/47 [===========================>..] - ETA: 1s - rpn_cls: 0.0194 - rpn_regr: 0.0177 - detector_cls: 4.4270e-06 - detector_regr: 0.0000e+00No positive ROIs.
45/47 [===========================>..] - ETA: 0s - rpn_cls: 0.0194 - rpn_regr: 0.0177 - detector_cls: 4.5650e-06 - detector_regr: 0.0000e+00No positive ROIs.
46/47 [============================>.] - ETA: 0s - rpn_cls: 0.0193 - rpn_regr: 0.0178 - detector_cls: 4.6921e-06 - detector_regr: 0.0000e+00No positive ROIs.
47/47 [==============================] - 23s 480ms/step - rpn_cls: 0.0193 - rpn_regr: 0.0178 - detector_cls: 4.8092e-06 - detector_regr: 0.0000e+00
2019-10-28 20:17:39,033 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Mean number of bounding boxes from RPN overlapping ground truth boxes: 0.0
2019-10-28 20:17:39,033 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Classifier accuracy for bounding boxes from RPN: 1.0
2019-10-28 20:17:39,033 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss RPN classifier: 0.0184327360698
2019-10-28 20:17:39,033 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss RPN regression: 0.0185074357098
2019-10-28 20:17:39,034 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss Detector classifier: 1.01928689521e-05
2019-10-28 20:17:39,034 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Loss Detector regression: 0.0
2019-10-28 20:17:39,034 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Elapsed time: 26.4510099888
2019-10-28 20:17:39,034 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Total loss changed from 0.0562822726817 to 0.0369503646486, saving weights
2019-10-28 20:17:42,882 [INFO] /usr/local/lib/python2.7/dist-packages/iva/faster_rcnn/scripts/train.pyc: Epoch 4/12
No positive ROIs.
1/47 [..............................] - ETA: 21s - rpn_cls: 0.0166 - rpn_regr: 0.0110 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
2/47 [>.............................] - ETA: 20s - rpn_cls: 0.0146 - rpn_regr: 0.0104 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
3/47 [>.............................] - ETA: 20s - rpn_cls: 0.0138 - rpn_regr: 0.0115 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
4/47 [=>............................] - ETA: 20s - rpn_cls: 0.0142 - rpn_regr: 0.0134 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
5/47 [==>...........................] - ETA: 19s - rpn_cls: 0.0143 - rpn_regr: 0.0142 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
6/47 [==>...........................] - ETA: 19s - rpn_cls: 0.0145 - rpn_regr: 0.0144 - detector_cls: 1.1921e-07 - detector_regr: 0.0000e+00No positive ROIs.
7/47 [===>..........................] - ETA: 18s - rpn_cls: 0.0146 - rpn_regr: 0.0144 - detector_cls: 1.6520e-06 - detector_regr: 0.0000e+00No positive ROIs.
8/47 [====>.........................] - ETA: 18s - rpn_cls: 0.0149 - rpn_regr: 0.0147 - detector_cls: 2.6339e-06 - detector_regr: 0.0000e+00No positive ROIs.
9/47 [====>.........................] - ETA: 17s - rpn_cls: 0.0151 - rpn_regr: 0.0149 - detector_cls: 3.2817e-06 - detector_regr: 0.0000e+00No positive ROIs.
10/47 [=====>........................] - ETA: 17s - rpn_cls: 0.0152 - rpn_regr: 0.0152 - detector_cls: 3.7165e-06 - detector_regr: 0.0000e+00No positive ROIs.
11/47 [======>.......................] - ETA: 16s - rpn_cls: 0.0152 - rpn_regr: 0.0153 - detector_cls: 4.0102e-06 - detector_regr: 0.0000e+00No positive ROIs.
12/47 [======>.......................] - ETA: 16s - rpn_cls: 0.0152 - rpn_regr: 0.0154 - detector_cls: 4.2075e-06 - detector_regr: 0.0000e+00No positive ROIs.
13/47 [=======>......................] - ETA: 16s - rpn_cls: 0.0154 - rpn_regr: 0.0156 - detector_cls: 4.3374e-06 - detector_regr: 0.0000e+00No positive ROIs.
14/47 [=======>......................] - ETA: 15s - rpn_cls: 0.0154 - rpn_regr: 0.0157 - detector_cls: 4.4193e-06 - detector_regr: 0.0000e+00No positive ROIs.
15/47 [========>.....................] - ETA: 15s - rpn_cls: 0.0154 - rpn_regr: 0.0158 - detector_cls: 4.4664e-06 - detector_regr: 0.0000e+00No positive ROIs.
16/47 [=========>....................] - ETA: 14s - rpn_cls: 0.0153 - rpn_regr: 0.0159 - detector_cls: 4.4881e-06 - detector_regr: 0.0000e+00No positive ROIs.
17/47 [=========>....................] - ETA: 14s - rpn_cls: 0.0152 - rpn_regr: 0.0160 - detector_cls: 4.4910e-06 - detector_regr: 0.0000e+00No positive ROIs.
18/47 [==========>...................] - ETA: 13s - rpn_cls: 0.0152 - rpn_regr: 0.0160 - detector_cls: 4.4799e-06 - detector_regr: 0.0000e+00No positive ROIs.
19/47 [===========>..................] - ETA: 13s - rpn_cls: 0.0151 - rpn_regr: 0.0160 - detector_cls: 4.4584e-06 - detector_regr: 0.0000e+00No positive ROIs.
20/47 [===========>..................] - ETA: 12s - rpn_cls: 0.0150 - rpn_regr: 0.0160 - detector_cls: 4.4292e-06 - detector_regr: 0.0000e+00No positive ROIs.
21/47 [============>.................] - ETA: 12s - rpn_cls: 0.0150 - rpn_regr: 0.0160 - detector_cls: 4.3943e-06 - detector_regr: 0.0000e+00No positive ROIs.
22/47 [=============>................] - ETA: 11s - rpn_cls: 0.0149 - rpn_regr: 0.0160 - detector_cls: 4.3552e-06 - detector_regr: 0.0000e+00No positive ROIs.
23/47 [=============>................] - ETA: 11s - rpn_cls: 0.0148 - rpn_regr: 0.0160 - detector_cls: 4.3130e-06 - detector_regr: 0.0000e+00No positive ROIs.
24/47 [==============>...............] - ETA: 10s - rpn_cls: 0.0148 - rpn_regr: 0.0159 - detector_cls: 4.2686e-06 - detector_regr: 0.0000e+00No positive ROIs.
25/47 [==============>...............] - ETA: 10s - rpn_cls: 0.0147 - rpn_regr: 0.0159 - detector_cls: 4.2228e-06 - detector_regr: 0.0000e+00No positive ROIs.
26/47 [===============>..............] - ETA: 9s - rpn_cls: 0.0147 - rpn_regr: 0.0159 - detector_cls: 4.1761e-06 - detector_regr: 0.0000e+00 No positive ROIs.
27/47 [================>.............] - ETA: 9s - rpn_cls: 0.0147 - rpn_regr: 0.0159 - detector_cls: 4.1288e-06 - detector_regr: 0.0000e+00Traceback (most recent call last):
File "/usr/local/bin/tlt-train-g1", line 10, in <module>
sys.exit(main())
File "./common/magnet_train.py", line 30, in main
File "./faster_rcnn/scripts/train.py", line 287, in main
File "./faster_rcnn/utils/roi_helpers.py", line 76, in calc_iou_np
IndexError: index 1 is out of bounds for axis 1 with size 1
Please let me know if this is a new issue or something I have not configured properly.
Here is my spec:
random_seed: 42
enc_key: 'API'
verbose: True
network_config {
input_image_config {
image_type: RGB
image_channel_order: 'bgr'
size_min {
min:700
}
image_channel_mean {
key: 'b'
value: 103.939
}
image_channel_mean {
key: 'g'
value: 116.779
}
image_channel_mean {
key: 'r'
value: 123.68
}
image_scaling_factor: 1.0
}
feature_extractor: "resnet:50"
anchor_box_config {
scale: 64.0
scale: 128.0
scale: 256.0
ratio: 1.0
ratio: 0.5
ratio: 2.0
}
freeze_bn: True
freeze_blocks: 0
freeze_blocks: 1
roi_mini_batch: 256
rpn_stride: 16
conv_bn_share_bias: True
roi_pooling_config {
pool_size: 7
pool_size_2x: False
}
all_projections: True
use_pooling:False
}
training_config {
kitti_data_config {
images_dir : '/workspace/tlt-experiments/data/cam/training_images'
labels_dir: '/workspace/tlt-experiments/data/cam/training_labels'
}
training_data_parser: 'raw_kitti'
data_augmentation {
use_augmentation: True
spatial_augmentation {
hflip_probability: 0.5
vflip_probability: 0.0
zoom_min: 1.0
zoom_max: 1.0
translate_max_x: 0
translate_max_y: 0
}
color_augmentation {
color_shift_stddev: 0.0
hue_rotation_max: 0.0
saturation_shift_max: 0.0
contrast_scale_max: 0.0
contrast_center: 0.5
}
}
num_epochs: 12
class_mapping {
key: 'pore'
value: 0
}
class_mapping {
key: "paint"
value: 1
}
pretrained_weights: "/workspace/tlt-experiments/pretrained_models/tlt_resnet50_faster_rcnn_v1/resnet50.h5"
pretrained_model: ""
output_weights: "/workspace/tlt-experiments/faster_rcnn/frcnn_kitti.tltw"
output_model: "/workspace/tlt-experiments/faster_rcnn/frcnn_kitti.tlt"
rpn_min_overlap: 0.3
rpn_max_overlap: 0.7
classifier_min_overlap: 0.0
classifier_max_overlap: 0.5
gt_as_roi: False
std_scaling: 1.0
classifier_regr_std {
key: 'x'
value: 10.0
}
classifier_regr_std {
key: 'y'
value: 10.0
}
classifier_regr_std {
key: 'w'
value: 5.0
}
classifier_regr_std {
key: 'h'
value: 5.0
}
rpn_mini_batch: 256
rpn_pre_nms_top_N: 12000
rpn_nms_overlap_threshold: 0.7
reg_config {
reg_type: 'L2'
weight_decay: 1e-4
}
optimizer {
adam {
lr: 0.00001
beta_1: 0.9
beta_2: 0.999
decay: 0.0
}
}
lr_scheduler {
step {
base_lr: 0.00001
gamma: 1.0
step_size: 30
}
}
lambda_rpn_regr: 1.0
lambda_rpn_class: 1.0
lambda_cls_regr: 1.0
lambda_cls_class: 1.0
inference_config {
images_dir: '/workspace/tlt-experiments/data/cam/inference'
model: '/workspace/tlt-experiments/faster_rcnn/frcnn_kitti.epoch12.tlt'
detection_image_output_dir: '/workspace/tlt-experiments/data/cam/inference_results_imgs'
labels_dump_dir: '/workspace/tlt-experiments/data/cam/inference_results_imgs'
rpn_pre_nms_top_N: 6000
rpn_nms_max_boxes: 300
rpn_nms_overlap_threshold: 0.7
bbox_visualize_threshold: 0.6
classifier_nms_max_boxes: 300
classifier_nms_overlap_threshold: 0.3
}
evaluation_config {
dataset {
images_dir : '/workspace/tlt-experiments/data/cam/validation_images'
labels_dir: '/workspace/tlt-experiments/data/cam/validation_labels'
}
data_parser: 'raw_kitti'
model: '/workspace/tlt-experiments/faster_rcnn/frcnn_kitti.epoch12.tlt'
labels_dump_dir: '/workspace/tlt-experiments/data/cam/validation_images'
rpn_pre_nms_top_N: 6000
rpn_nms_max_boxes: 300
rpn_nms_overlap_threshold: 0.7
classifier_nms_max_boxes: 300
classifier_nms_overlap_threshold: 0.3
object_confidence_thres: 0.0001
use_voc07_11point_metric:False
}
}
Here is a sample annotation:
pore 0.00 0 0.00 326.00 479.00 338.00 490.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
paint 0.00 0 0.00 267.00 465.00 312.00 544.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Thank you in advance for your help!