Hi,
I’ve been able to successfully convert the stock mscoco inceptionV2 SSD tensorflow model to uff format and run the f32 sample_uff_ssd example. However, when providing it 500 randomly selected images from the mscoco dataset the calibration fails. It appears to be consuming all the images but when it’s building the histogram, it has a issue. Here is the output I’m getting:
ERROR: Tensor FeatureExtractor/InceptionV2/InceptionV2/Mixed_5c/Branch_3/Conv2d_0b_1x1/Relu6/relu2 is uniformly zero; network calibration failed.
At first I thought the images I supplied may not have a wide enough range, but even changing them or the number of images provided all result in the same failure.
Any suggestions on what I’m doing wrong? The f32 mode can do detection on the images I’m feeding for calibration so I’m believe they are formatted correctly.
Also the sampleInt8 example IS working with the mnist data on this machine…
Here is my system information:
From deviceQuery:
16 GB RAM
dpkg -l | grep TensorRT
Metadata-Version: 2.1
I also try to run the trt sampleUffSSD. which is failed in float32 and float16.
Result in float32:
./bin/sample_uff_ssd
data/ssd/sample_ssd.uff
Begin parsing model...
End parsing model...
Begin building engine...
End building engine...
Num batches 2
Data Size 540000
*** deserializing
Cuda failure: 74[1] 20102 abort (core dumped) ./bin/sample_uff_ssd
int8 is like this:
./bin/sample_uff_ssd --int8
data/ssd/sample_ssd.uff
Begin parsing model...
End parsing model...
Begin building engine...
ERROR: runtime.cpp (16) - Cuda Error in allocate: 2
ERROR: runtime.cpp (16) - Cuda Error in allocate: 2
ERROR: sample_uff_ssd: Unable to create engine
sample_uff_ssd: sampleUffSSD.cpp:667: int main(int, char**): Assertion `tmpEngine != nullptr' failed.
[1] 12311 abort (core dumped) ./bin/sample_uff_ssd --int8
@liuyoungshop
Error 74:https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html#group__CUDART__TYPES_1g3f51e3575c2178246db0a94a430e0038
cudaErrorMisalignedAddress = 74
Not sure why the int8 mode can’t find your list.txt file, but be sure to have all your file names the same length and to strip the extension off the end. See my discussion here: https://devtalk.nvidia.com/default/topic/1036911/tensorrt/cannot-run-sampleuffssd-for-int8-inference-calibration-failed-/post/5267689/#5267689
Hi,
I got the similar problem.
My error code is: ERROR: Tensor FeatureExtractor/InceptionV2/InceptionV2/Mixed_4d/Branch_3/Conv2d_0b_1x1/Relu6/relu2 is uniformly zero; network calibration failed.
Let me know if you solved it. Thanks!
@joe-dev , Thank you for your response.
I’m new in cuda and not so sure about the error message you provided. I searched the relative question in the google but some people said that there is not enough gpu memory. But I assign 2GB GPU memory, while the other code is as same as the original sampleUffSsd code. Is that the code problem or my cuda or cudnn or tensorrt version problem?
My cuda version is 9.0 and cudnn version is 7.1.4, tensorrt version is 4.0.1.6.
Besides, I updated the problem of int8.
Please let me know if you have already run the code successfully!
Thanks so much!
Best,
Yang
@liuyoungshop
I’m using CUDA 9.0 and cudnn 7.0.5 and TensorRT 4.0.1.6. So I don’t think the version is the issue.
As you can see above, I have a GTX 1050 TI graphics card with 4 GB of memory, what card are you running on?
Something you can try is, in a separate terminal run:
watch -n 1 “nvidia-smi”
This will update every second with the current memory allocation and percentage utilization of the GPU and may show if you’re running out of memory and if so what’s using it.
Also I’d recommend adding some debug code to track down exactly which line is causing your failure.
I am still having the problem I originally posted where the calibration is failing.
Hope this helps and good luck!
@joe-dev
I used the same GPU as you used and definitely did not run out of memory.
Based on your suggestion, it seems that the error is happened in this line in the doInference func:
CHECK(cudaMemcpyAsync(buffers[inputIndex], inputData, batchSize * INPUT_C * INPUT_H * INPUT_W * sizeof(float), cudaMemcpyHostToDevice, stream));
Let me check whether other tensorrt sample application works fine in copying the data from host to devices.
Thanks a lot!
I’ll post it here if I solve the problem.
If you have any suggestion, please let me know about that!
I sense there may be an inconsistency to the frozen_inference_graph.pb used because I literally can’t see an input_tensor nodedownload.tensorflow.org/models/object_detection/ssd_inception_v2_coco_2018_01_28.tar.gz
@joe-dev , @nikolasmarkou
you are right, my problem is also caused by the model.
I’ve been exploring the issue on how to use the ssd detection plugin using recent version of tensorflow (1.8) since friday, it seems that there is an extra input to previously there before.
would you mind to paste the previous layer in your frozengraph and indicate which is the extra input tensor?
I’d like to discuss with you to solve this problem.
@nikolasmarkou ,
we need to consider whether it is an important node. if it is not an important node, we can simplely replace it as ‘Input’ Tensor.
@liuyoungshop , I believe a lot of people are here for the same problem as I’m having, that is to get their own custom models or later version tf models to work with tensorrt
Older model
Loading ssd_inception_v2_coco_2017_frozen_inference_graph.pb
Using output node NMS
Converting to UFF graph
Warning: No conversion function registered for layer: NMS yet.
Converting as custom op NMS NMS
name: "NMS"
op: "NMS"
input: "concat_box_loc"
input: "concat_priorbox"
input: "concat_box_conf"
attr {
key: "iouThreshold_u_float"
value {
f: 0.6000000238418579
}
}
Newer model
Loading ssd_inception_v2_coco_2018_01_28_frozen_inference_graph.pb
Using output node NMS
Converting to UFF graph
Warning: No conversion function registered for layer: NMS yet.
Converting as custom op NMS NMS
name: "NMS"
op: "NMS"
input: "Input"
input: "Squeeze"
input: "concat_priorbox"
input: "concat_box_conf"
attr {
key: "iouThreshold_u_float"
value {
f: 0.6000000238418579
}
}
Now from the graph i can see that the Squeeze layer is a concat of box locations (i think)
When i try to use this with sample SSD code i get this
~/Downloads/trt/trt$ cp conversion/ssd_inception_v2_coco_2018_01_28_frozen_inference_graph.pb.uff data/ssd/sample_ssd.uff && ./bin/sample_uff_ssd
data/ssd/sample_ssd.uff
Begin parsing model...
End parsing model...
Begin building engine...
sample_uff_ssd: NvPluginSSD.cu:713: virtual nvinfer1::Dims nvinfer1::plugin::DetectionOutput::getOutputDimensions(int, const nvinfer1::Dims*, int): Assertion `nbInputDims == 3' failed.
Aborted (core dumped)
@nikolasmarkou ,
I got what you mean, it should be the problem of plugin_map. In this case, maybe the name in the original frozen graph is changed. maybe the concat_1 cannot be found in the new graph.
PostProcessor/ToFloat should be wrong. This is caused by “ToFloat” is mapping to Input.
So, One thing we can do is just show all the node names by something like this
[print(n.name) for n in tf.get_default_graph().as_graph_def().node]
Would you mind print it now?
This is a listing of nodes using the convert-to-uff -l
[code]Loading ssd_inception_v2_coco_2018_01_28_frozen_inference_graph.pb
754 RealDiv: “Postprocessor/Decode/div_1”
@nikolasmarkou ,
I got what you mean, it should be the problem of plugin_map. In this case, maybe the name in the original frozen graph is changed. maybe the concat_1 cannot be found in the new graph.
PostProcessor/ToFloat should be wrong. This is caused by “ToFloat” is mapping to Input.
So, One thing we can do is just show all the node names by something like this
[print(n.name) for n in tf.get_default_graph().as_graph_def().node]
Would you mind print it now?
The nodes are pasted in file below
Loading ssd_inception_v2_coco_2018_01_28_frozen_inference_graph.pbAutomatically deduced output nodes: Preprocessor/map/while/Switch, Postprocessor/BatchMultiClassNonMaxSuppression/map/while/Switch,...
@nikolasmarkou ,
I solved problem. Please change your config like this:
namespace_plugin_map = {
"MultipleGridAnchorGenerator": PriorBox,
"Postprocessor": NMS,
"Preprocessor": Input,
# "ToFloat": Input,
# "image_tensor": Input,
"MultipleGridAnchorGenerator/Concatenate": concat_priorbox,
"concat": concat_box_loc,
"concat_1": concat_box_conf,
}
namespace_remove = {
"ToFloat",
"image_tensor",
"Preprocessor/map/TensorArrayStack_1/TensorArrayGatherV3",
}
def preprocess(dynamic_graph):
# remove the unrelated or error layers
dynamic_graph.remove(dynamic_graph.find_nodes_by_path(namespace_remove), remove_exclusive_dependencies=False)
# Now create a new graph by collapsing namespaces
dynamic_graph.collapse_namespaces(namespace_plugin_map)
# Remove the outputs, so we just have a single output node (NMS).
dynamic_graph.remove(dynamic_graph.graph_outputs, remove_exclusive_dependencies=False)
# Remove the Squeeze to avoid "Assertion `isPlugin(layerName)' failed"
Squeeze = dynamic_graph.find_node_inputs_by_name(dynamic_graph.graph_outputs[0], 'Squeeze')
dynamic_graph.forward_inputs(Squeeze)
@nikolasmarkou ,
I solved problem. Please change your config like this:
namespace_plugin_map = {
"MultipleGridAnchorGenerator": PriorBox,
"Postprocessor": NMS,
"Preprocessor": Input,
# "ToFloat": Input,
# "image_tensor": Input,
"MultipleGridAnchorGenerator/Concatenate": concat_priorbox,
"concat": concat_box_loc,
"concat_1": concat_box_conf,
}
namespace_remove = {
"ToFloat",
"image_tensor",
"Preprocessor/map/TensorArrayStack_1/TensorArrayGatherV3",
}
def preprocess(dynamic_graph):
# remove the unrelated or error layers
dynamic_graph.remove(dynamic_graph.find_nodes_by_path(namespace_remove), remove_exclusive_dependencies=False)
# Now create a new graph by collapsing namespaces
dynamic_graph.collapse_namespaces(namespace_plugin_map)
# Remove the outputs, so we just have a single output node (NMS).
dynamic_graph.remove(dynamic_graph.graph_outputs, remove_exclusive_dependencies=False)
# Remove the Squeeze to avoid "Assertion `isPlugin(layerName)' failed"
Squeeze = dynamic_graph.find_node_inputs_by_name(dynamic_graph.graph_outputs[0], 'Squeeze')
dynamic_graph.forward_inputs(Squeeze)