Worse COCO mAP for YOLOv3 on TensorRT

Hi all,

I’m using the sample code for converting yolov3 for use in TensorRT. Sample code documentation can be found at [url]https://docs.nvidia.com/deeplearning/sdk/tensorrt-sample-support-guide/index.html#yolov3_onnx[/url]

Did a mAP benchmarking test and realised that the mAP values are FAR lower compared to the paper’s mAP values.

TensorRT YOLOv3
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.251
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.415
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.272
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.121
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.271
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.339
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.222
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.290
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.292
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.133
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.307
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.399

Paper’s mAP value at IoU 50% is 57.9%

There is a discrepancy of ~ 16%

Is there anything that I missed out?

Thank you.

Hi hengchenkim,

Please fill out our issue template below to help better debug this issue:

If you open a TensorRT issue, here is our policy.

TensorRT developers respond to issues. We want to focus on fixing bugs and adding features.

Provide details on the platforms you are using:
Linux distro and version
GPU type
Nvidia driver version
CUDA version
CUDNN version
Python version [if using python]
Tensorflow version
TensorRT version
If Jetson, OS, hw versions

Describe the problem

Issues encountered

Files

Include any logs, source, models (.uff, .pb, etc.) that would be helpful to diagnose the problem.

If relevant, please include the full traceback.

Reproducibility

Please provide a minimal test case that reproduces your error.

Thanks,
NVIDIA Enterprise Support

Provide details on the platforms you are using:
Linux distro and version: Ubuntu 16.04.6 LTS
GPU type: T4
Nvidia driver version: 418.74
CUDA version: 10.1
CUDNN version: V10.0.130
Python version: 2.7
Tensorflow version: -
TensorRT version: 5.1.5-1
If Jetson, OS, hw versions: -

Describe the problem

• I benchmarked the mAP results of the sample code documented in Sample Support Guide :: NVIDIA Deep Learning TensorRT Documentation. The mAP performance is worse off compared to the original algorithm by 16% for IoU of 50%

Issues encountered

Files

Include any logs, source, models (.uff, .pb, etc.) that would be helpful to diagnose the problem.

If relevant, please include the full traceback.

Reproducibility

The test set used is the COCO validation 5000 image split provided in YOLO: Real-Time Object Detection

Hi hengchenkim,

Sorry for the delay. As per engineering, can you see if your problem still persists with TensorRT 6.0.1? (just released)

If you have nvidia-docker setup, you can just pull the nvcr.io/nvidia/tensorrt:19.09-py3 image from NGC and run this in the container: TensorRT | NVIDIA NGC

To build from source, see the docs: Installation Guide :: NVIDIA Deep Learning TensorRT Documentation

Thanks,
NVIDIA Enterprise Support

Hi,

For the accuracy, I’ve downloaded the default weights from [url]https://pjreddie.com/media/files/yolov3.weights[/url] and tested on the original YoloV3-608 network and verified the mAP results.

However, when I used the same default weights on the sample code referenced at [url]https://docs.nvidia.com/deeplearning/sdk/tensorrt-sample-support-guide/index.html#yolov3_onnx[/url], there was a significant mAP drop.

Thank you.

Hi,

Just to clarify, you still experienced this using TensorRT 6?

Thanks,
NVIDIA Enterprise Support

Hi,

Can you confirm that you still experience this issue using TensorRT 6.0?

Thanks,
NVIDIA Enterprise Support

Hi,

I have the same issue with Jetson tx2. the accuracy of my trained model drops significantly (I tried with FP16 and also FP32).
Ubuntu 18
tensorRT version: 5.1.6
cuda: 10.0

now I’m trying to upgrade to tensorRT 6 following the steps here Installation Guide :: NVIDIA Deep Learning TensorRT Documentation but I couldn’t

1-I’m new user of tensorRT, could you please tell me why the accuracy drops when converting a model into tensorRT (in my case, i got good results using darknet on GTX1080 , but low accuracy when i used it with deepstream 4.0)
2-could you please tell me the steps how to upgrade tensorrt on jetson tx2

I also have accurate issue using tensorRT, did you solve this problem already?

I’ve verified COCO mAP of TensorRT optimized YOLOv3 models on Jetson Nano. Details could be found in my GitHub code and blog post.

mAP of the YOLOv3-608x608 is OK. However, the YOLOv3-Tiny models are bad.

tensorrt-yolov3-map1154×649 46.6 KB

it’s ok for yolov3 tiny on tx2,but when i changed the yolov3 network frame ,the accurate is very bad. I used deepstream.