"Folder does not exist or is not reachable" on DIGITS AMI
Hello txbob, Thanks for the quick reply! I am using "Not NGC": AWS Marketplace -> NVIDIA DIGITS 6. The AWS instance type is g2.2xlarge. Here are the first and last several lines of output from caffe_output.log for the job: [snip] libdc1394 error: Failed to initialize libdc1394 I1126 11:55:07.177713 579 upgrade_proto.cpp:1044] Attempting to upgrade input file specified using deprecated 'solver_type' field (enum)': /jobs/20171126-115505-4756/solver.prototxt I1126 11:55:07.177882 579 upgrade_proto.cpp:1051] Successfully upgraded file specified using deprecated 'solver_type' field (enum) to 'type' field (string). W1126 11:55:07.177889 579 upgrade_proto.cpp:1053] Note that future Caffe releases will only support 'type' field (string) for a solver's type. I1126 11:55:07.234048 579 caffe.cpp:197] Using GPUs 0 I1126 11:55:07.234225 579 caffe.cpp:202] GPU 0: GRID K520 I1126 11:55:07.730933 579 solver.cpp:48] Initializing solver from parameters: [/snip] [snip] F1126 11:55:09.247619 579 pooling_layer.cu:212] Check failed: error == cudaSuccess (8 vs. 0) invalid device function *** Check failure stack trace: *** @ 0x7f205acaddaa (unknown) @ 0x7f205acadce4 (unknown) @ 0x7f205acad6e6 (unknown) @ 0x7f205acb0687 (unknown) @ 0x7f205b4166a0 caffe::PoolingLayer<>::Forward_gpu() @ 0x7f205b223358 caffe::Net<>::ForwardFromTo() @ 0x7f205b2236d7 caffe::Net<>::Forward() @ 0x7f205b2a4f8a caffe::Solver<>::Test() @ 0x7f205b2a580e caffe::Solver<>::TestAll() @ 0x7f205b2a7ddd caffe::Solver<>::Step() @ 0x7f205b2a879e caffe::Solver<>::Solve() @ 0x40af75 train() @ 0x4086bc main @ 0x7f20597a4f45 (unknown) @ 0x408e8d (unknown) @ (nil) (unknown) [/snip] Maybe somewhere I have to change a setting to accommodate the GPU architecture; just not sure where to look for it on the AWS DIGITS 6 instance?
Hello txbob,

Thanks for the quick reply! I am using "Not NGC": AWS Marketplace -> NVIDIA DIGITS 6. The AWS instance type is g2.2xlarge.

Here are the first and last several lines of output from caffe_output.log for the job:

[snip]
libdc1394 error: Failed to initialize libdc1394
I1126 11:55:07.177713 579 upgrade_proto.cpp:1044] Attempting to upgrade input file specified using deprecated 'solver_type' field (enum)': /jobs/20171126-115505-4756/solver.prototxt
I1126 11:55:07.177882 579 upgrade_proto.cpp:1051] Successfully upgraded file specified using deprecated 'solver_type' field (enum) to 'type' field (string).
W1126 11:55:07.177889 579 upgrade_proto.cpp:1053] Note that future Caffe releases will only support 'type' field (string) for a solver's type.
I1126 11:55:07.234048 579 caffe.cpp:197] Using GPUs 0
I1126 11:55:07.234225 579 caffe.cpp:202] GPU 0: GRID K520
I1126 11:55:07.730933 579 solver.cpp:48] Initializing solver from parameters:
[/snip]

[snip]
F1126 11:55:09.247619 579 pooling_layer.cu:212] Check failed: error == cudaSuccess (8 vs. 0) invalid device function
*** Check failure stack trace: ***
@ 0x7f205acaddaa (unknown)
@ 0x7f205acadce4 (unknown)
@ 0x7f205acad6e6 (unknown)
@ 0x7f205acb0687 (unknown)
@ 0x7f205b4166a0 caffe::PoolingLayer<>::Forward_gpu()
@ 0x7f205b223358 caffe::Net<>::ForwardFromTo()
@ 0x7f205b2236d7 caffe::Net<>::Forward()
@ 0x7f205b2a4f8a caffe::Solver<>::Test()
@ 0x7f205b2a580e caffe::Solver<>::TestAll()
@ 0x7f205b2a7ddd caffe::Solver<>::Step()
@ 0x7f205b2a879e caffe::Solver<>::Solve()
@ 0x40af75 train()
@ 0x4086bc main
@ 0x7f20597a4f45 (unknown)
@ 0x408e8d (unknown)
@ (nil) (unknown)
[/snip]

Maybe somewhere I have to change a setting to accommodate the GPU architecture; just not sure where to look for it on the AWS DIGITS 6 instance?

#16
Posted 11/27/2017 12:46 AM   
Scroll To Top

Add Reply