Digits 6 AWS EC2 Instance, fails to create dataset.....Help please

So This is my second time using digits, my first time attempting to create my own dataset. The first time I trained a model, was using the DIGITS example guide. DIGITS/examples/object-detection at master · NVIDIA/DIGITS · GitHub

I’m having an issue with creating a dataset, DIGITS seems to fail. I’m getting a zero “Entry Count” for both the “train_db DB” and the “val_db DB”. Here are the db.log reports that I’m getting.

create_test_db_db.log
       libdc1394 error: Failed to initialize libdc1394
       2018-04-18 12:34:45 [INFO ] Generic DB creation Done

create_val_db_db.log
       libdc1394 error: Failed to initialize libdc1394
       2018-04-18 12:34:44 [INFO ] Created features db for stage val_db in val_db/features
       2018-04-18 12:34:56 [INFO ] Created labels db for stage val_db in val_db/labels
       2018-04-18 12:34:56 [INFO ] Processed 1/3
       2018-04-18 12:35:02 [INFO ] Feature shape for stage val_db: (3, 1290, 1290)
       2018-04-18 12:35:02 [INFO ] Label shape for stage val_db: (1, 6, 16)
       2018-04-18 12:35:02 [INFO ] Processed 2/3
       2018-04-18 12:35:02 [INFO ] Processed 3/3
       2018-04-18 12:35:02 [INFO ] Created mean file for stage val_db in val_db/mean.binaryproto
       2018-04-18 12:35:02 [INFO ] Found 83 entries for stage val_db
       2018-04-18 12:35:02 [INFO ] Generic DB creation Done

create_train_db_db.log
       libdc1394 error: Failed to initialize libdc1394
       2018-04-18 12:34:44 [INFO ] Created features db for stage train_db in train_db/features
       2018-04-18 12:34:56 [INFO ] Created labels db for stage train_db in train_db/labels
       2018-04-18 12:34:56 [INFO ] Processed 1/4
       2018-04-18 12:35:05 [INFO ] Feature shape for stage train_db: (3, 1290, 1290)
       2018-04-18 12:35:05 [INFO ] Label shape for stage train_db: (1, 6, 16)
       2018-04-18 12:35:05 [INFO ] Processed 2/4
       2018-04-18 12:35:06 [INFO ] Processed 3/4
       2018-04-18 12:35:06 [INFO ] Created mean file for stage train_db in train_db/mean.binaryproto
       2018-04-18 12:35:06 [INFO ] Processed 4/4
       2018-04-18 12:35:07 [INFO ] Found 114 entries for stage train_db
       2018-04-18 12:35:07 [INFO ] Generic DB creation Done

I have around 600 images to train and 100 for validation. They are all vehicles, ie cars, trucks, vans, etc, but the images are from above and angled down. So the vehicle model I created already doesn’t work very well. Which is why I’m training a new model. I attempted this several times, with no success. So I tried to create another dataset of the example images found in the DIGITS example guide and it worked perfectly fine. So the issue has something to do with my images/labels. I’m now using a smaller count of images to create the dataset, just to get this to work. about 100 training images and 80 validation. This is not the quantity I will be using in the end.

I’m using images with a resolution of “1280 x 720” and “720 x 1280”. So the camera was set landscape and portrait to capture images. I’ve attempted to create the dataset with .jpg and .png.
Labeling I used labelimg, once the .xml file was created, I used a python script that converted the .xml to .txt with the KITTI format. I used the default class mappings that DIGITS uses, ie car, truck, van, pick up, etc. Which I found here https://github.com/NVIDIA/DIGITS/blob/digits-5.0/digits/extensions/data/objectDetection/README.md#label-format

Here are the settings I’m using to create the dataset.

Training image folder
       /data/path/to/images/
Training label folder
       /data/path/to/labels/
Validation image folder
       /data/path/to/images
Validation label folder
       /data/path/to/labels

Pad image (Width x Height)
       1290 x 1290               
Resize image (Width x Height)
      
Channel conversion
       RGB
Minimum box size (in pixels) for validation set
       25
Custom classes

Feature Encoding
       PNG (lossless)
Label Encoding
       None
Encoder batch size
       32
Number of encoder threads
       4
DB backend
       LMDB

Any insights? I would be very grateful!!!

I guess your issue was related to the format of dataset you created. Can you share some sample label files?

tcyang,

Thank you for getting back to me. I appreciate it.

Here are three of the label files that I’m using.

car 0.00 0 0.00 326.00 4.00 870.00 526.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

pickup 0.00 0 0.00 1088.00 37.00 1277.00 267.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
pickup 1.00 0 0.00 1.00 137.00 434.00 426.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
van 0.00 0 0.00 204.00 4.00 586.00 117.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
car 1.00 0 0.00 647.00 1.00 772.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

pickup 0.00 0 0.00 1092.00 36.00 1277.00 260.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
van 0.00 0 0.00 202.00 2.00 582.00 121.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
car 1.00 0 0.00 604.00 283.00 918.00 720.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

Let me try to duplicate your issue.

That would be very much appreciated.

I haven’t been able to duplicate your issue with the label data you provided. My DIGITS can create the datasets successfully.

After you created the dataset, can you click the ‘Datasets’ tab and the dataset you just created to see the results? It should be similar to

940×848 106 KB

On that page, the correct ‘Entry Count’ is displayed.

However, there is another page that appears immediately after DIGITS finishes creating the dataset. On that page, you will see the sections ‘Job Information’, ‘Create train_db DB’, ‘Create val_db DB’ and 'Create test_db DB. The ‘Entry Count’ information on the last three sections are not updated automatically and are displayed as zeros. Please ignore those results.

So it appears that the issues that I’ve been having with training my network have had nothing to do with the dataset creation. Well, I can’t completely rule it out.

So I went back into the dataset section and clicked on one of the previous attempts, and yep there was the entry counts. I even clicked on Explore the DB and saw all the images.

I’ve been having issues training, all the analytics: accuracy, map, etc, would always be at zero when I trained with any of the datasets I created. So I assumed it was the fault of the dataset, and when I attempted to train again and it came back as zero entries I just assummed dataset.

I’ll move on and start trouble shooting the training aspects now.

Thank you for the insight.

Michael

A video walkthrough of natively installing NVIDIA DIGITS on Ubuntu 18.04 LTS is available here:

-Cuda Education