Tensorflow run with C++ API cause XAVIER crash

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1 
Tensoflow version: '1.12.0-rc2'
Device: Xavier
NV PORWER Mode: MAXN
CUDA: 10.0.117
CUDNN : 7.3.1

##############KERNEL CODE#####################
void semanticSegPro::getImgData(uchar* data)
{

```
auto input_tensor_mapped = input_tensor.tensor<uchar, 4>();

for (int y = 0; y < height; ++y)
{
    const uchar* source_row = data + (y * width * 3);
    for (int x = 0; x < width; ++x)
    {
        const uchar* source_pixel = source_row + (x * 3);
        for (int c = 0; c < 3; ++c)
        {
            const uchar* source_value = source_pixel + c;
            input_tensor_mapped(0, y, x, c) = *source_value;
        }
    }
}

QString s=QDateTime::currentDateTime().toString("yyyy-MM-dd hh:mm:ss.zzz");
qDebug()<<"BF = "<<s;

Status status_run =
session->Run({{inputTensorName.toStdString(), input_tensor}},
{outputTensorName.toStdString()}, {}, &outputs);

if (!status_run.ok()) {
    qDebug()<<"ERROR: RUN failed..."<<QString::fromStdString(status_run.ToString());
}

s=QDateTime::currentDateTime().toString("yyyy-MM-dd hh:mm:ss.zzz");
qDebug()<<"AFT = "<<s;

output_tensor = outputs[0];
auto tmap = output_tensor.tensor<int64, 3>();

for(int i = 0; i < height; i++)
{
    for(int j =0; j< width; j++)
    {
        outData[i*width + j] = tmap(0,i,j);
    }
}

emit segData((uchar*)outData);
```

}
##############KERNEL CODE#####################

    Run the c++ API  'session->Run()'
The program abnormal termination or Xavier crash(Mouse can't move).

I used the same program with PC(GTX 1070), it runs well.
I also used python API with Xavier platform, it runs well also.
Why the program crash with Xavier and C++ API???
2

Hi,

How do you install the TensorFlow package on Xavier?
Do you install the C++ library?

Thanks.

3

Hi,

I install the C++ library, and I have found the reason why Xavier crash.
And the deeplab work well in Xavier with C++ library.

By the way, I have another problem with TX2, as
https://devtalk.nvidia.com/default/topic/1043608/jetson-tx2/image-segmentation-with-deeplab-model-gives-wrong-result-with-jetson-tx2/post/5318574/#5318574

It seems that deeplab with C++ API in TX2 only works with Jetpack 3.1 and CUDA8.0.
And it works in Xavier with Jetpack 4.11 and CUDA10.

Now I can’t re-flash the device with jetpack3.1 for warning “Error:downloading update lock”, while jetpack3.3 use well.
Coud you help me?
@AastaLLL

4

Hi,

Could you share what causes the crash on Xavier?
For TX2, please set the device into recovery mode to reflash a new image.

Thanks.

5

Hi,

I think the crash on Xavier is caused by memory shared total 8G with gpu memory and cpu memory.
May be the new session greedily demand all 8G memory, and caused Xavier crash when I passed input tensor to session from
cpu memory to gpu memory.

Status status_run =
session->Run({{inputTensorName.toStdString(), input_tensor}},
{outputTensorName.toStdString()}, {}, &outputs);

So I set the SessionOptions as:

ConfigProto config;
tensorflow::GPUOptions* GPUOptions = new ::tensorflow::GPUOptions();
GPUOptions->set_per_process_gpu_memory_fraction(0.4);
config.set_allocated_gpu_options(GPUOptions);
SessionOptions Sconfig;
Sconfig.config = config;

For TX2, I try to reflash TX2 with Jetpack. When I reflash the device with Jetpack3.1, it warns “Error:downloading update lock”. How can I reflash a new image and drives to TX2 without Jetpack?
Or why Jetpack 3.3 with cuda9.0 caused wrong result with deeplab?
Thank U.

@AastaLLL

6

Hi,

Could you try to execute Jetpack Installer in a new folder?
Sometime the installer lock the download folder if it doesn’t finish normally.

Thanks.