Hi,
I have an instance of Ubuntu 16.04 running DIGITS 6.
I have downloaded and successfully expanded the MS-COCO dataset as detailed in https://github.com/dusty-nv/jetson-inference.
When I run DIGITS and enter the paths to reach the training and validation data folders, I get the error in DIGITS "Folder does not exist or is not reachable".
1. I have checked the paths are correct and tried variants ( from root, home etc).
2. I have followed instructions to make sure that www-data has read access. Guidance from : https://github.com/NVIDIA/DIGITS/issues/1026 and also from https://askubuntu.com/questions/244406/how-do-i-give-www-data-user-to-a-folder-in-my-home-folder.
All help most welcome! Thanks!
Mike
When I run DIGITS and enter the paths to reach the training and validation data folders, I get the error in DIGITS "Folder does not exist or is not reachable".
It's the AWS AMI for DIGITS 6 - so I suspect the first of the two options above. It is selected in AWS from EC2 -> AWS Marketplace -> NVIDIA DIGITS 6 on Ubuntu 16.04
Is there a better place I can be doing this?
I am trying to get Object Detection working on my TX2, but do not have a suitable machine for building the models - hence working in the cloud to get DIGITS operational.
Thanks for the response!
It's the AWS AMI for DIGITS 6 - so I suspect the first of the two options above. It is selected in AWS from EC2 -> AWS Marketplace -> NVIDIA DIGITS 6 on Ubuntu 16.04
Is there a better place I can be doing this?
I am trying to get Object Detection working on my TX2, but do not have a suitable machine for building the models - hence working in the cloud to get DIGITS operational.
I have now also looked again at the NGC docker containers at https://ngc.nvidia.com/registry/nvidia-digits and I'm afraid I find the instructions completely baffling.
For example, mention is made of an 'Actions' column, but I don't think there is one.
It also says use 'docker pull', without explaining how to do that.
Apologies for being slow!
In fact, it was orignally from this page that I opted to follow the links to AWS...
If I can be doing this a better way, please let me know, as I am spending a great deal of time trying to get a simple build environment working.
Cheers, Mike
Hi Cliff,
I am working through instructions for the NGC container now...
However, I'm still hoping to be able to get the AWS AMI image working - no resaon it shouldn't?
Thanks for your help.
Cheers
Mike
Ah, thanks for pointing out about the "actions" - looks like they renamed that column in the NGC UI to "Pull" after we wrote the text of the README. We'll get that updated.
Beyond that, the instructions in the NGC README - and this forum, generally - assume you're using the NGC AMI, not the DIGITS AMI. (I was working on finding you an alternative place to ask the question in the context of the DIGITS AMI before suggesting you start over in a different AMI.) The fact that we have two paths to getting DIGITS on AWS is a bit of a historical artifact we're looking to reconcile in the future.
From first principles, the main thing will be to know what paths are involved, where and how those volumes are mounted, and what user account DIGITS is running as. I'm seeking answers to at least the latter of those questions internally, but any extra details you could offer as well would be helpful.
Ah, thanks for pointing out about the "actions" - looks like they renamed that column in the NGC UI to "Pull" after we wrote the text of the README. We'll get that updated.
Beyond that, the instructions in the NGC README - and this forum, generally - assume you're using the NGC AMI, not the DIGITS AMI. (I was working on finding you an alternative place to ask the question in the context of the DIGITS AMI before suggesting you start over in a different AMI.) The fact that we have two paths to getting DIGITS on AWS is a bit of a historical artifact we're looking to reconcile in the future.
From first principles, the main thing will be to know what paths are involved, where and how those volumes are mounted, and what user account DIGITS is running as. I'm seeking answers to at least the latter of those questions internally, but any extra details you could offer as well would be helpful.
Hi Cliff,
Thanks for getting back.
I think the 'penny had dropped' on the NGC container approach. However it seems that the Volta instance is 5 times more expensive per hour, which is significant for a self-funded learner when getting started.
So as I ssh in, the prompt is $ ubuntu@ip-privateipaddress.
Following Dusty's instructions, a simple ls from here shows the directory 'coco' seems correctly listed and drilling down, all the subdirectories and data apper to be present. i.e.
Training image folder: coco/train/images/dog
Training label folder: coco/train/labels/dog
Validation image folder: coco/val/images/dog
Validation label folder: coco/val/labels/dog
ubuntu@ip:~$ cd coco
ubuntu@ip-:~/coco$ ls
coco2kitti.py train val
ubuntu@ip-:~/coco$ cd train
ubuntu@ip-:~/coco/train$ ls
images labels
ubuntu@ip-:~/coco/train$ cd images
ubuntu@ip-:~/coco/train/images$ ls
airplane bottle chair dog
ubuntu@ip-:~/coco/train/images$ cd dog
ubuntu@ip-:~/coco/train/images/dog$
Permissions are as follows ( I have reloaded the datasets so permissions have not been modified in any way)
ubuntu@ip-:~/coco$ ls -l
total 12
-rw-rw-r-- 1 ubuntu ubuntu 2512 May 1 2017 coco2kitti.py
drwxrwxr-x 4 ubuntu ubuntu 4096 May 3 2017 train
drwxrwxr-x 4 ubuntu ubuntu 4096 May 2 2017 val
By way of accounts, does this help?
/home$ ls
digits ubuntu
Thanks again,
Mike
I think the 'penny had dropped' on the NGC container approach. However it seems that the Volta instance is 5 times more expensive per hour, which is significant for a self-funded learner when getting started.
So as I ssh in, the prompt is $ ubuntu@ip-privateipaddress.
Following Dusty's instructions, a simple ls from here shows the directory 'coco' seems correctly listed and drilling down, all the subdirectories and data apper to be present. i.e.
Training image folder: coco/train/images/dog
Training label folder: coco/train/labels/dog
Validation image folder: coco/val/images/dog
Validation label folder: coco/val/labels/dog
ubuntu@ip:~$ cd coco
ubuntu@ip-:~/coco$ ls
coco2kitti.py train val
ubuntu@ip-:~/coco$ cd train
ubuntu@ip-:~/coco/train$ ls
images labels
ubuntu@ip-:~/coco/train$ cd images
ubuntu@ip-:~/coco/train/images$ ls
airplane bottle chair dog
ubuntu@ip-:~/coco/train/images$ cd dog
ubuntu@ip-:~/coco/train/images/dog$
Permissions are as follows ( I have reloaded the datasets so permissions have not been modified in any way)
ubuntu@ip-:~/coco$ ls -l
total 12
-rw-rw-r-- 1 ubuntu ubuntu 2512 May 1 2017 coco2kitti.py
drwxrwxr-x 4 ubuntu ubuntu 4096 May 3 2017 train
drwxrwxr-x 4 ubuntu ubuntu 4096 May 2 2017 val
By way of accounts, does this help?
/home$ ls
digits ubuntu
So one thought is that I am logged in as ubuntu and installing data owned by that account and in that path. However, maybe the server is accessing using the digits account? This might mean:
1. the server is looking in the wrong account path.
2. It doesn't have permission to see the folders.
Could that be a possibility?
So one thought is that I am logged in as ubuntu and installing data owned by that account and in that path. However, maybe the server is accessing using the digits account? This might mean:
1. the server is looking in the wrong account path.
2. It doesn't have permission to see the folders.
There are two possible issues at play here. One or both might be at play depending on which AMI you're using.
(1) You've got your dataset in /home/ubuntu/coco . However, /home/ubuntu (and therefore subdirectories of the same) would normally only be accessible to the 'ubuntu' user, and DIGITS -- whether containerized or not -- usually runs as a different user, leading to possible permissions problems accessing the dataset. The solution is to move the dataset to some shared directory that all users on the system can access.
(2) When running in a containerized environment, the dataset is probably mounted at different paths as viewed from outside the container (say it's /host/path/to/coco ) versus inside the container (normally just /data for DIGITS containers). It's the [b]-v[/b] argument to [b]docker run[/b] that configures this; for example, if you're doing something like [b]docker run ... -v /host/path/to/coco:/data ...[/b], then when you go into the DIGITS UI and set up the dataset, the path you use to tell DIGITS where to find the dataset needs to be /data, not /host/path/to/coco.
Hope this helps,
Cliff
There are two possible issues at play here. One or both might be at play depending on which AMI you're using.
(1) You've got your dataset in /home/ubuntu/coco . However, /home/ubuntu (and therefore subdirectories of the same) would normally only be accessible to the 'ubuntu' user, and DIGITS -- whether containerized or not -- usually runs as a different user, leading to possible permissions problems accessing the dataset. The solution is to move the dataset to some shared directory that all users on the system can access.
(2) When running in a containerized environment, the dataset is probably mounted at different paths as viewed from outside the container (say it's /host/path/to/coco ) versus inside the container (normally just /data for DIGITS containers). It's the -v argument to docker run that configures this; for example, if you're doing something like docker run ... -v /host/path/to/coco:/data ..., then when you go into the DIGITS UI and set up the dataset, the path you use to tell DIGITS where to find the dataset needs to be /data, not /host/path/to/coco.
Hi! My question is almost exactly the same as the one above.
- I am using the AMI with NVIDIA DIGITS 6 on Ubuntu 16.04: https://aws.amazon.com/marketplace/pp/B076DHKCZJ
- I have downloaded and successfully expanded the MS-COCO dataset as detailed in https://github.com/dusty-nv/jetson-inference.
- I keep getting the error in DIGITS that says "Folder does not exist or is not reachable".
I think the problem probably has to do with giving www-data read access, but I'm having trouble figuring out the appropriate combination of commands to do so with my directory structure. I am logged in as ubuntu, and I am trying to put the 'coco' directory into /home/digits/data. Here are the current permissions:
ubuntu@ip:/home/digits$ ls -la
total 28
drwxr-xr-x 4 digits digits 4096 Oct 2 22:34 .
drwxr-xr-x 4 root root 4096 Oct 2 22:34 ..
-rw-r--r-- 1 digits digits 220 Oct 2 22:34 .bash_logout
-rw-r--r-- 1 digits digits 3771 Oct 2 22:34 .bashrc
drwxrwx--- 3 digits digits 4096 Nov 24 19:43 data
drwxr-xr-x 2 digits digits 4096 Nov 8 03:33 jobs
-rw-r--r-- 1 digits digits 655 Oct 2 22:34 .profile
Should I be trying to put the 'coco' directory into /data instead? Any help would be much appreciated!
- I keep getting the error in DIGITS that says "Folder does not exist or is not reachable".
I think the problem probably has to do with giving www-data read access, but I'm having trouble figuring out the appropriate combination of commands to do so with my directory structure. I am logged in as ubuntu, and I am trying to put the 'coco' directory into /home/digits/data. Here are the current permissions:
ubuntu@ip:/home/digits$ ls -la
total 28
drwxr-xr-x 4 digits digits 4096 Oct 2 22:34 .
drwxr-xr-x 4 root root 4096 Oct 2 22:34 ..
-rw-r--r-- 1 digits digits 220 Oct 2 22:34 .bash_logout
-rw-r--r-- 1 digits digits 3771 Oct 2 22:34 .bashrc
drwxrwx--- 3 digits digits 4096 Nov 24 19:43 data
drwxr-xr-x 2 digits digits 4096 Nov 8 03:33 jobs
-rw-r--r-- 1 digits digits 655 Oct 2 22:34 .profile
Should I be trying to put the 'coco' directory into /data instead? Any help would be much appreciated!
Hi pineapple_flora,
So firstly move all the data into the digits/data folder. home/digits/data/coco/...
In DIGITS, simply specify this path as /data/coco...
Secondly set permissions on the entire data folder (recursively) to be readable for all. This will also allow www.data access.
If you are following the dusty-nv instructions, you may also find the network in prototext will not work with the specified dataset. I learnt that after 14 hours of build time, only to find the model hadn't been learning anything. That's what I'm working on now...
So firstly move all the data into the digits/data folder. home/digits/data/coco/...
In DIGITS, simply specify this path as /data/coco...
Secondly set permissions on the entire data folder (recursively) to be readable for all. This will also allow www.data access.
If you are following the dusty-nv instructions, you may also find the network in prototext will not work with the specified dataset. I learnt that after 14 hours of build time, only to find the model hadn't been learning anything. That's what I'm working on now...
Hi mikeisted,
Thanks for the suggestions! I followed them and am not seeing a "Folder does not exist or is not reachable" error any more, hooray!
I was able to get a little further in the dusty-nv tutorial, but I ran into another roadblock when trying to launch the DetectNet-COCO-Dog model. At a high level, the error that I'm seeing is: "ERROR: Check failed: error == cudaSuccess (8 vs. 0) invalid device function." It might make sense for me to start another thread about this topic or to search through the issues at dusty-nv/jetson-inference -- I was just curious if you had experienced anything similar with your DIGITS 6 AMI.
Thanks for the suggestions! I followed them and am not seeing a "Folder does not exist or is not reachable" error any more, hooray!
I was able to get a little further in the dusty-nv tutorial, but I ran into another roadblock when trying to launch the DetectNet-COCO-Dog model. At a high level, the error that I'm seeing is: "ERROR: Check failed: error == cudaSuccess (8 vs. 0) invalid device function." It might make sense for me to start another thread about this topic or to search through the issues at dusty-nv/jetson-inference -- I was just curious if you had experienced anything similar with your DIGITS 6 AMI.
invalid device function usually means that the architecture a CUDA code was compiled for does not match the architecture of the GPU you are running on.
What sort of AWS instance type did you launch the DIGITS AMI on? And for clarity are we talking about NGC or not NGC?
NGC: https://ngc.nvidia.com/registry/nvidia-digits
Not NGC: AWS Marketplace -> NVIDIA DIGITS 6
(I asssume this is "not NGC" here)
invalid device function usually means that the architecture a CUDA code was compiled for does not match the architecture of the GPU you are running on.
What sort of AWS instance type did you launch the DIGITS AMI on? And for clarity are we talking about NGC or not NGC?
I have an instance of Ubuntu 16.04 running DIGITS 6.
I have downloaded and successfully expanded the MS-COCO dataset as detailed in https://github.com/dusty-nv/jetson-inference.
When I run DIGITS and enter the paths to reach the training and validation data folders, I get the error in DIGITS "Folder does not exist or is not reachable".
1. I have checked the paths are correct and tried variants ( from root, home etc).
2. I have followed instructions to make sure that www-data has read access. Guidance from : https://github.com/NVIDIA/DIGITS/issues/1026 and also from https://askubuntu.com/questions/244406/how-do-i-give-www-data-user-to-a-folder-in-my-home-folder.
All help most welcome! Thanks!
Mike
Is there a better place I can be doing this?
I am trying to get Object Detection working on my TX2, but do not have a suitable machine for building the models - hence working in the cloud to get DIGITS operational.
Thanks for the response!
For example, mention is made of an 'Actions' column, but I don't think there is one.
It also says use 'docker pull', without explaining how to do that.
Apologies for being slow!
In fact, it was orignally from this page that I opted to follow the links to AWS...
If I can be doing this a better way, please let me know, as I am spending a great deal of time trying to get a simple build environment working.
Cheers, Mike
I am working through instructions for the NGC container now...
However, I'm still hoping to be able to get the AWS AMI image working - no resaon it shouldn't?
Thanks for your help.
Cheers
Mike
Beyond that, the instructions in the NGC README - and this forum, generally - assume you're using the NGC AMI, not the DIGITS AMI. (I was working on finding you an alternative place to ask the question in the context of the DIGITS AMI before suggesting you start over in a different AMI.) The fact that we have two paths to getting DIGITS on AWS is a bit of a historical artifact we're looking to reconcile in the future.
From first principles, the main thing will be to know what paths are involved, where and how those volumes are mounted, and what user account DIGITS is running as. I'm seeking answers to at least the latter of those questions internally, but any extra details you could offer as well would be helpful.
Thanks for getting back.
I think the 'penny had dropped' on the NGC container approach. However it seems that the Volta instance is 5 times more expensive per hour, which is significant for a self-funded learner when getting started.
So as I ssh in, the prompt is $ ubuntu@ip-privateipaddress.
Following Dusty's instructions, a simple ls from here shows the directory 'coco' seems correctly listed and drilling down, all the subdirectories and data apper to be present. i.e.
Training image folder: coco/train/images/dog
Training label folder: coco/train/labels/dog
Validation image folder: coco/val/images/dog
Validation label folder: coco/val/labels/dog
ubuntu@ip:~$ cd coco
ubuntu@ip-:~/coco$ ls
coco2kitti.py train val
ubuntu@ip-:~/coco$ cd train
ubuntu@ip-:~/coco/train$ ls
images labels
ubuntu@ip-:~/coco/train$ cd images
ubuntu@ip-:~/coco/train/images$ ls
airplane bottle chair dog
ubuntu@ip-:~/coco/train/images$ cd dog
ubuntu@ip-:~/coco/train/images/dog$
Permissions are as follows ( I have reloaded the datasets so permissions have not been modified in any way)
ubuntu@ip-:~/coco$ ls -l
total 12
-rw-rw-r-- 1 ubuntu ubuntu 2512 May 1 2017 coco2kitti.py
drwxrwxr-x 4 ubuntu ubuntu 4096 May 3 2017 train
drwxrwxr-x 4 ubuntu ubuntu 4096 May 2 2017 val
By way of accounts, does this help?
/home$ ls
digits ubuntu
Thanks again,
Mike
1. the server is looking in the wrong account path.
2. It doesn't have permission to see the folders.
Could that be a possibility?
(1) You've got your dataset in /home/ubuntu/coco . However, /home/ubuntu (and therefore subdirectories of the same) would normally only be accessible to the 'ubuntu' user, and DIGITS -- whether containerized or not -- usually runs as a different user, leading to possible permissions problems accessing the dataset. The solution is to move the dataset to some shared directory that all users on the system can access.
(2) When running in a containerized environment, the dataset is probably mounted at different paths as viewed from outside the container (say it's /host/path/to/coco ) versus inside the container (normally just /data for DIGITS containers). It's the -v argument to docker run that configures this; for example, if you're doing something like docker run ... -v /host/path/to/coco:/data ..., then when you go into the DIGITS UI and set up the dataset, the path you use to tell DIGITS where to find the dataset needs to be /data, not /host/path/to/coco.
Hope this helps,
Cliff
I have managed to get it running through the combination of actions as suggested.
Thanks for your help!
Mike
- I am using the AMI with NVIDIA DIGITS 6 on Ubuntu 16.04: https://aws.amazon.com/marketplace/pp/B076DHKCZJ
- I have downloaded and successfully expanded the MS-COCO dataset as detailed in https://github.com/dusty-nv/jetson-inference.
- I keep getting the error in DIGITS that says "Folder does not exist or is not reachable".
I think the problem probably has to do with giving www-data read access, but I'm having trouble figuring out the appropriate combination of commands to do so with my directory structure. I am logged in as ubuntu, and I am trying to put the 'coco' directory into /home/digits/data. Here are the current permissions:
ubuntu@ip:/home/digits$ ls -la
total 28
drwxr-xr-x 4 digits digits 4096 Oct 2 22:34 .
drwxr-xr-x 4 root root 4096 Oct 2 22:34 ..
-rw-r--r-- 1 digits digits 220 Oct 2 22:34 .bash_logout
-rw-r--r-- 1 digits digits 3771 Oct 2 22:34 .bashrc
drwxrwx--- 3 digits digits 4096 Nov 24 19:43 data
drwxr-xr-x 2 digits digits 4096 Nov 8 03:33 jobs
-rw-r--r-- 1 digits digits 655 Oct 2 22:34 .profile
Should I be trying to put the 'coco' directory into /data instead? Any help would be much appreciated!
So firstly move all the data into the digits/data folder. home/digits/data/coco/...
In DIGITS, simply specify this path as /data/coco...
Secondly set permissions on the entire data folder (recursively) to be readable for all. This will also allow www.data access.
If you are following the dusty-nv instructions, you may also find the network in prototext will not work with the specified dataset. I learnt that after 14 hours of build time, only to find the model hadn't been learning anything. That's what I'm working on now...
Thanks for the suggestions! I followed them and am not seeing a "Folder does not exist or is not reachable" error any more, hooray!
I was able to get a little further in the dusty-nv tutorial, but I ran into another roadblock when trying to launch the DetectNet-COCO-Dog model. At a high level, the error that I'm seeing is: "ERROR: Check failed: error == cudaSuccess (8 vs. 0) invalid device function." It might make sense for me to start another thread about this topic or to search through the issues at dusty-nv/jetson-inference -- I was just curious if you had experienced anything similar with your DIGITS 6 AMI.
What sort of AWS instance type did you launch the DIGITS AMI on? And for clarity are we talking about NGC or not NGC?
NGC: https://ngc.nvidia.com/registry/nvidia-digits
Not NGC: AWS Marketplace -> NVIDIA DIGITS 6
(I asssume this is "not NGC" here)