Torch7 on TX1

raak · May 21, 2016, 8:35pm

Anyone tried torch7 on Jetson TX1 ?

raak · May 22, 2016, 3:33am

Received. Segmentation errors when running test.
also while running tensor multiplications anyone had similar issues ?

dusty_nv · May 22, 2016, 3:44pm

I use Torch7 frequently on TX1 without issue. Here’s the install script I use: [url]https://github.com/dusty-nv/jetson-reinforcement/blob/master/CMakePreBuild.sh[/url]

raak · May 23, 2016, 2:57am

Fine, i am in using lua 5.2
This a sample script.

#!/usr/bin/env lua
require ‘torch’;
a = torch.Tensor(5,3)
a = torch.rand(5,3)
print(a)
b=torch.rand(3,4)
c=torch.Tensor(5,4)
c:mm(a,b)

0.2683 0.4384 0.4055
0.6545 0.9442 0.7435
0.4213 0.0596 0.2069
0.0922 0.7576 0.9385
0.7945 0.9670 0.1425
[torch.DoubleTensor of size 5x3]

Segmentation fault

I have also tried with LUAJIT, installed via your script, same issue

raak · May 24, 2016, 6:31pm

NVM fixed it

raak · August 4, 2016, 10:56am

the script is not working anymore ? there was issue with OpenBlas, then torch directory issues

raak · August 9, 2016, 12:03pm

Hi Dusty is this still working ?

regards

dusty_nv · August 9, 2016, 5:53pm

Recently cutorch/cudnn updated to using half FP16 when available. It introduced a couple bugs, which the Torch maintainers are investigating.

[url]https://github.com/torch/cutorch/issues/446[/url]
[url]https://github.com/soumith/cudnn.torch/issues/236[/url]

After updating the script above, it is working again (albeit with FP32 at the moment). Try downloading the script again.

raak · August 10, 2016, 4:30am

Thanks man.

dusty_nv · August 17, 2016, 2:05pm

Compiling recent Torch updates onboard, some FP16 CUDA templates in cutorch may run the board out of memory and OOM killer may kill the script.

Here is an updated script containing the workaround (reducing -J4): [url]https://github.com/dusty-nv/jetson-reinforcement/blob/master/CMakePreBuild.sh[/url]

raak · August 17, 2016, 2:09pm

Thanks I will confirm you tomorrow, can you DM me your email ?

raak · August 18, 2016, 12:14pm

nvm resolved

jkjung · September 1, 2016, 1:42am

I installed Torch7 on a Jetson TX1 with a clean installation of JetPack 2.2.1 (L4T R24.1) 64-bit. But when I ran self-testing of individual modules of Torch 7, I saw some errors/failures and wondered whether those errors/failures are real issues or not…

The way I installed Torch7 is to just follow instructions on the official web site with the following tweaks.

Official Torch7 installation instructions: [url]http://torch.ch/docs/getting-started.html[/url]
I’d modify the install-deps script to call install_openblas() instead of apt-get install the blas packages
I’d also modify torch/extra/cutorch/rocks/cutorch-scm-1.rockspec by replacing “-j$(getconf _NPROCESSORS_ONLN)” with “-j1”. This is to avoid nvcc error during compilation of cutorch due to running out of resources (memory?).

I’ll describe the test errors/failures in the next post.

jkjung · September 1, 2016, 1:52am

When I test Torch 7 on Jetson TX1, both torch.test() and nn.test() run fine without any issues. However, I see errors/failures with nnx, cutorch and cunn. I wonder whether those errors/failures are mostly due to “running out of memory”…

nnx.test() showed at least 4 errors and got killed before the whole test is completed.

th> require 'nnx'
true	
                                                                      [0.0188s]
th> nnx.test()
Running 39 tests
 1/39 SpatialGraph_3 .................................................... [WAIT]
 1/39 SpatialGraph_3 .................................................... [PASS]
 2/39 SpatialReSampling_2 ............................................... [PASS]
 3/39 SpatialPadding .................................................... [ERROR]
 4/39 PushPullTable ..................................................... [PASS]
 5/39 SpatialMatching_4 ................................................. [PASS]
 6/39 CTCCriterion ...................................................... [ERROR]
 7/39 SpatialPyramid_focused ............................................ [PASS]
 8/39 MultiSoftMax ...................................................... [PASS]
                                                                      [+warning]
 9/39 SpatialReSamplingEx2 .............................................. [PASS]
10/39 SpatialFovea_focused .............................................. [PASS]
11/39 SpatialMatching_1 ................................................. [PASS]
12/39 SpatialGraph_2 .................................................... [PASS]
13/39 SpatialReSampling_1 ............................................... [PASS]
14/39 HardShrink ........................................................ [PASS]
15/39 SoftMaxTree ....................................................... [PASS]
16/39 Abs ............................................................... [PASS]
17/39 SpatialMatching_3 ................................................. [PASS]
18/39 SpatialNormalization_Gaussian2D ................................... [WAIT]<SpatialNormalization> WARNING: this module has been deprecated,	
 please use SpatialContrastiveNormalization instead	
18/39 SpatialNormalization_Gaussian2D ................................... [PASS]
19/39 SpatialGraph_1 .................................................... [PASS]
20/39 SpatialFovea_bilinear ............................................. [PASS]
21/39 SpatialMatching_2 ................................................. [PASS]
22/39 SpatialPyramid_unfocused .......................................... [PASS]
23/39 SpatialConvolution ................................................ [PASS]
24/39 SpatialGraph_4 .................................................... [PASS]
25/39 SpatialMaxPooling ................................................. [PASS]
26/39 SpatialUpSampling ................................................. [PASS]
27/39 TreeNLLCriterion .................................................. [ERROR]
28/39 SpatialReSamplingEx5 .............................................. [PASS]
29/39 Balance ........................................................... [PASS]
30/39 SpatialReSamplingEx3 .............................................. [PASS]
31/39 SpatialReSamplingEx1 .............................................. [ERROR]
32/39 SpatialGraph_5 .................................................... [PASS]
33/39 SpatialFovea_unfocused ............................................ [PASS]
34/39 SpatialNormalization_io ........................................... [WAIT]<SpatialNormalization> WARNING: this module has been deprecated,	
 please use SpatialContrastiveNormalization instead	
34/39 SpatialNormalization_io ........................................... [PASS]
35/39 SpatialMatching_5 ................................................. [PASS]
36/39 SpatialLinear ..................................................... [PASS]
37/39 SpatialReSamplingEx4 .............................................. [WAIT]
Killed
ubuntu@tegra-ubuntu:~$

cutorch.test() showed at least 1 failure (#38) and also got killed in the middle of the test.

th> require 'cutorch'
{
  getPeerToPeerAccess : function: 0x7f853aed60
  getStream : function: 0x7f85314740
  getDeviceCount : function: 0x7f853aecf0
  setHeapTracking : function: 0x7f853ae950
  manualSeedAll : function: 0x7f85339b60
  getRNGState : function: 0x7f85339b98
  setKernelPeerToPeerAccess : function: 0x7f853aee48
  reserveBlasHandles : function: 0x7f8b567688
  setDefaultStream : function: 0x7f8b5676c0
  getMemoryUsage : function: 0x7f853ae7c0
  streamBarrier : function: 0x7f853144d8
  manualSeed : function: 0x7f85339af8
  driverVersion : 7050
  synchronize : function: 0x7f853b9ed8
  reserveStreams : function: 0x7f85314608
  getDevice : function: 0x7f853ae918
  seed : function: 0x7f853ae830
  getBlasHandle : function: 0x7f853145a0
  withDevice : function: 0x7f7e38d250
  deviceReset : function: 0x7f853aec88
  Event : {...}
  test : function: 0x7f7e38d160
  _stategc : userdata: 0x7f8b5a2670
  getNumStreams : function: 0x7f85314670
  streamWaitFor : function: 0x7f8b566fc0
  synchronizeAll : function: 0x7f8b567618
  initialSeed : function: 0x7f85339ac0
  getDeviceProperties : function: 0x7f853ae758
  CudaHostAllocator : torch.Allocator
  getNumBlasHandles : function: 0x7f8b566ff8
  getState : function: 0x7f8533a240
  setStream : function: 0x7f853146d8
  getKernelPeerToPeerAccess : function: 0x7f853ae6e8
  setRNGState : function: 0x7f85339bd0
  createCudaHostTensor : function: 0x7f7e38d2c0
  streamWaitForMultiDevice : function: 0x7f8b5670d8
  streamSynchronize : function: 0x7f853ae8e0
  seedAll : function: 0x7f85339a88
  setDevice : function: 0x7f853ae7f8
  setPeerToPeerAccess : function: 0x7f853aedd0
  hasHalf : false
  streamBarrierMultiDevice : function: 0x7f853ae870
  setBlasHandle : function: 0x7f8b567060
  _state : userdata: 0x006960f0
}
                                                                      [9.2228s]
th> cutorch.test()
seed: 	1472709470	
Running 157 tests
  1/157 tan2 ............................................................ [PASS]
  2/157 cat ............................................................. [PASS]
  3/157 neg1 ............................................................ [PASS]
  4/157 atan2 ........................................................... [PASS]
  5/157 acos1 ........................................................... [PASS]
  6/157 split ........................................................... [PASS]
  7/157 streamWaitFor ................................................... [PASS]
  8/157 acos2 ........................................................... [PASS]
  9/157 zero ............................................................ [PASS]
 10/157 logNormal ....................................................... [PASS]
 11/157 sqrt2 ........................................................... [PASS]
 12/157 pow1 ............................................................ [PASS]
 13/157 permute ......................................................... [PASS]
 14/157 cross ........................................................... [PASS]
 15/157 tensorToTable ................................................... [PASS]
 16/157 inverse ......................................................... [PASS]
 17/157 sinh1 ........................................................... [PASS]
 18/157 sign1 ........................................................... [PASS]
 19/157 min ............................................................. [PASS]
 20/157 prod ............................................................ [PASS]
 21/157 ger ............................................................. [PASS]
 22/157 addmm ........................................................... [PASS]
 23/157 addmv ........................................................... [PASS]
 24/157 random_seed ..................................................... [PASS]
 25/157 lerp ............................................................ [PASS]
 26/157 sigmoid2 ........................................................ [PASS]
 27/157 log1p1 .......................................................... [PASS]
 28/157 sin2 ............................................................ [PASS]
 29/157 bernoulli ....................................................... [PASS]
 30/157 geometric ....................................................... [PASS]
 31/157 maskedCopy ...................................................... [PASS]
 32/157 maskedFill ...................................................... [PASS]
 33/157 frac2 ........................................................... [PASS]
 34/157 trace ........................................................... [PASS]
 35/157 floor2 .......................................................... [PASS]
 36/157 sum ............................................................. [PASS]
 37/157 indexCopy2 ...................................................... [PASS]
 38/157 cudaTypeCopy .................................................... [FAIL]
 39/157 reshape ......................................................... [PASS]
 40/157 streamBarrier ................................................... [PASS]
 41/157 cos1 ............................................................ [PASS]
 42/157 indexAdd ........................................................ [PASS]
 43/157 isSetTo ......................................................... [PASS]
 44/157 mean ............................................................ [PASS]
 45/157 multinomial_without_replacement_gets_all ........................ [PASS]
 46/157 logicalTensor ................................................... [PASS]
 47/157 cmin ............................................................ [PASS]
 48/157 sin1 ............................................................ [PASS]
 49/157 mm .............................................................. [PASS]
 50/157 ceil2 ........................................................... [PASS]
 51/157 cinv1 ........................................................... [PASS]
 52/157 multinomial_without_replacement ................................. [PASS]
 53/157 repeatTensor .................................................... [PASS]
 54/157 abs1 ............................................................ [PASS]
 55/157 elementSize ..................................................... [PASS]
 56/157 log2 ............................................................ [PASS]
 57/157 cosh1 ........................................................... [PASS]
 58/157 csub ............................................................ [PASS]
 59/157 index ........................................................... [PASS]
 60/157 log1p2 .......................................................... [PASS]
 61/157 viewAs .......................................................... [PASS]
 62/157 diag ............................................................ [PASS]
 63/157 std ............................................................. [PASS]
 64/157 powExponentTensor ............................................... [PASS]
 65/157 kernelP2PAccess ................................................. [PASS]
 66/157 storageToTable .................................................. [PASS]
 67/157 uniform ......................................................... [PASS]
 68/157 renorm .......................................................... [PASS]
 69/157 rsqrt ........................................................... [PASS]
 70/157 scatter ......................................................... [PASS]
 71/157 addbmm .......................................................... [PASS]
 72/157 indexSelect2 .................................................... [PASS]
 73/157 cinv2 ........................................................... [PASS]
 74/157 cudaHostTensor .................................................. [PASS]
 75/157 tanh1 ........................................................... [PASS]
 76/157 cmax ............................................................ [PASS]
 77/157 indexFill2 ...................................................... [PASS]
 78/157 copyNoncontiguous ............................................... [PASS]
 79/157 copyRandomizedTest .............................................. [PASS]
 80/157 streamWaitForMultiDevice ........................................ [PASS]
 81/157 exp1 ............................................................ [PASS]
 82/157 max ............................................................. [PASS]
 83/157 trunc2 .......................................................... [PASS]
 84/157 clamp2 .......................................................... [PASS]
 85/157 atan1 ........................................................... [PASS]
 86/157 multinomial_vector .............................................. [PASS]
 87/157 cudaStorageTypeCopy ............................................. [PASS]
 88/157 cdiv ............................................................ [PASS]
 89/157 trunc1 .......................................................... [PASS]
 90/157 sqrt1 ........................................................... [PASS]
 91/157 clamp1 .......................................................... [PASS]
 92/157 cosh2 ........................................................... [PASS]
 93/157 multi_gpu_copy_noncontig ........................................ [WAIT]
Killed
ubuntu@tegra-ubuntu:~$

cunn.test() finished with 1 failure and 6 errors.

th> require 'cunn'
true	
                                                                      [0.0741s]
th> cunn.test()
seed: 	365418063	
Running 158 tests
  1/158 SpatialDilatedConvolution_backward_batch ........................ [PASS]
  2/158 SpatialUpSamplingBilinear_backward_batch ........................ [PASS]
  3/158 getParameters ................................................... [PASS]
  4/158 Threshold_forward ............................................... [PASS]
  5/158 Dropout_forward ................................................. [PASS]
  6/158 Tanh_transposed ................................................. [PASS]
  7/158 LogSoftMax_backward_batch ....................................... [PASS]
  8/158 Sigmoid_transposed .............................................. [PASS]
  9/158 SoftShrink_backward ............................................. [PASS]
 10/158 ClassNLLCriterionSingleTarget ................................... [PASS]
 11/158 SpatialFullConvolution_backward_batch ........................... [PASS]
 12/158 SpatialConvolutionLocal_forward_single .......................... [PASS]
 13/158 Sqrt_backward ................................................... [PASS]
 14/158 LogSigmoid_transposed ........................................... [PASS]
 15/158 ClassNLLCriterionMultipleTarget ................................. [PASS]
 16/158 SpatialClassNLLCriterion ........................................ [PASS]
 17/158 SoftMax_forward ................................................. [PASS]
 18/158 LogSoftMax_forward .............................................. [PASS]
 19/158 SpatialDilatedConvolution_forward_single ........................ [PASS]
 20/158 Threshold_backward .............................................. [PASS]
 21/158 SparseLinear_forward ............................................ [PASS]
 22/158 SpatialAveragePooling_backward_batch ............................ [WAIT]THCudaCheck FAIL file=/home/ubuntu/torch/extra/cunn/lib/THCUNN/SpatialAveragePooling.cu line=218 error=7 : too many resources requested for launch
 22/158 SpatialAveragePooling_backward_batch ............................ [ERROR]
 23/158 VolumetricConvolution_forward_single ............................ [PASS]
 24/158 SoftPlus_backward ............................................... [PASS]
 25/158 TemporalConvolution_forward ..................................... [PASS]
 26/158 SpatialAveragePooling_includepad ................................ [PASS]
 27/158 Square_backward ................................................. [PASS]
 28/158 Sigmoid_backward ................................................ [PASS]
 29/158 BatchNormalization .............................................. [PASS]
 30/158 VolumetricFullConvolution_pair_test ............................. [ERROR]
 31/158 LogSigmoid_backward ............................................. [PASS]
 32/158 VolumetricConvolution_backward_single ........................... [PASS]
 33/158 Sqrt_transposed ................................................. [PASS]
 34/158 MarginCriterion_forward ......................................... [PASS]
 35/158 SparseLinear_backward ........................................... [PASS]
 36/158 TemporalConvolution_backward_batch .............................. [PASS]
 37/158 SpatialSubSampling_forward ...................................... [PASS]
 38/158 SpatialAveragePooling_backward .................................. [ERROR]
 39/158 ELU_transposed .................................................. [PASS]
 40/158 SpatialBatchNormalization ....................................... [PASS]
 41/158 PReLU_backward .................................................. [PASS]
 42/158 SpatialFractionalMaxPooling_backward ............................ [PASS]
 43/158 VolumetricAveragePooling_forward ................................ [PASS]
 44/158 LeakyReLU_transposed ............................................ [PASS]
 45/158 MultiLabelMarginCriterion_backward .............................. [PASS]
 46/158 SpatialSubSampling_forward_batch ................................ [PASS]
 47/158 SpatialMaxUnpooling_forward_batch ............................... [PASS]
 48/158 LogSoftMax_backward ............................................. [PASS]
 49/158 SoftShrink_forward .............................................. [PASS]
 50/158 SpatialUpSamplingBilinear_forward ............................... [PASS]
 51/158 Square_forward .................................................. [PASS]
 52/158 SpatialDilatedMaxPooling_forward ................................ [PASS]
 53/158 LeakyReLU_backward .............................................. [PASS]
 54/158 SpatialMaxUnpooling_backward_batch .............................. [PASS]
 55/158 SpatialConvolutionMM_backward_single ............................ [PASS]
 56/158 SpatialCrossMapLRN_backward_batch ............................... [PASS]
 57/158 TemporalConvolution_forward_batch ............................... [PASS]
 58/158 VolumetricMaxPooling_forward .................................... [PASS]
 59/158 MarginCriterion_backward ........................................ [PASS]
 60/158 LookupTable_backward ............................................ [PASS]
 61/158 ReLU6_backward .................................................. [PASS]
 62/158 SpatialConvolutionMM_forward_single ............................. [PASS]
 63/158 Square_transposed ............................................... [PASS]
 64/158 SpatialMaxPooling_forward ....................................... [PASS]
 65/158 VolumetricReplicationPadding_backward ........................... [PASS]
 66/158 Sigmoid_forward ................................................. [PASS]
 67/158 ELU_backward .................................................... [PASS]
 68/158 SpatialAdaptiveMaxPooling_backward .............................. [PASS]
 69/158 SpatialDilatedMaxPooling_backward_batch ......................... [PASS]
 70/158 SpatialUpSamplingNearest_backward_batch ......................... [PASS]
 71/158 VolumetricFullConvolution ....................................... [ERROR]
 72/158 VolumetricMaxUnpooling_backward_batch ........................... [PASS]
 73/158 Tanh_forward .................................................... [PASS]
 74/158 CMul_forward_batch .............................................. [PASS]
 75/158 SpatialAveragePooling_forward_batch ............................. [PASS]
 76/158 l1cost .......................................................... [PASS]
 77/158 SpatialAveragePooling_forward ................................... [PASS]
 78/158 SpatialReplicationPadding_backward .............................. [PASS]
 79/158 ELU_forward ..................................................... [PASS]
 80/158 LeakyReLU_forward ............................................... [PASS]
 81/158 VolumetricBatchNormalization .................................... [PASS]
 82/158 SoftMarginCriterion ............................................. [PASS]
 83/158 Sqrt_zero ....................................................... [PASS]
 84/158 SpatialSubSampling_backward ..................................... [PASS]
 85/158 SpatialConvolutionLocal_backward_batch .......................... [ERROR]
 86/158 Dropout ......................................................... [PASS]
 87/158 VolumetricMaxPooling_backward ................................... [PASS]
 88/158 VolumetricMaxUnpooling_forward_batch ............................ [PASS]
 89/158 mse ............................................................. [PASS]
 90/158 PReLU_forward ................................................... [PASS]
 91/158 ReLU6_forward ................................................... [PASS]
 92/158 SpatialAdaptiveMaxPooling_forward ............................... [PASS]
 93/158 MultiLabelMarginCriterion_forward ............................... [PASS]
 94/158 SpatialUpSamplingBilinear_forward_batch ......................... [PASS]
 95/158 SpatialAdaptiveMaxPooling_backward_noncontig .................... [PASS]
 96/158 SpatialAdaptiveMaxPooling_backward_batch ........................ [PASS]
 97/158 HardTanh_forward ................................................ [PASS]
 98/158 SoftMax_backward ................................................ [PASS]
 99/158 WeightedEuclidean_backward_batch ................................ [PASS]
100/158 CMul_backward_batch ............................................. [PASS]
101/158 SpatialConvolutionLocal_backward_single ......................... [PASS]
102/158 SpatialDilatedConvolution_backward_single ....................... [FAIL]
103/158 HardTanh_transposed ............................................. [PASS]
104/158 SpatialSubSampling_backward_batch ............................... [PASS]
105/158 SpatialAdaptiveMaxPooling_forward_noncontig ..................... [PASS]
106/158 LookupTable_forward ............................................. [PASS]
107/158 SpatialFullConvolution_backward_single .......................... [PASS]
108/158 SpatialConvolutionLocal_forward_batch ........................... [PASS]
109/158 WeightedEuclidean_forward_batch ................................. [PASS]
110/158 SpatialAdaptiveMaxPooling_forward_batch ......................... [PASS]
111/158 Euclidean_forward_batch ......................................... [PASS]
112/158 SpatialLPPooling_forward ........................................ [PASS]
113/158 SoftShrink_transposed ........................................... [PASS]
114/158 Abs_backward .................................................... [PASS]
115/158 VolumetricConvolution_forward_batch ............................. [PASS]
116/158 SmoothL1 ........................................................ [PASS]
117/158 SpatialUpSamplingBilinear_backward .............................. [PASS]
118/158 ClassNLLCriterionSingleTargetWeights ............................ [PASS]
119/158 SoftPlus_forward ................................................ [PASS]
120/158 SpatialUpSamplingNearest_backward ............................... [PASS]
121/158 SpatialMaxPooling_backward_batch ................................ [PASS]
122/158 SpatialFullConvolution_forward_single ........................... [PASS]
123/158 VolumetricReplicationPadding_forward ............................ [PASS]
124/158 SpatialMaxPooling_forward_batch ................................. [PASS]
125/158 SpatialUpSamplingNearest_forward ................................ [PASS]
126/158 Euclidean_backward_batch ........................................ [PASS]
127/158 TemporalMaxPooling .............................................. [PASS]
128/158 SpatialDilatedMaxPooling_forward_batch .......................... [PASS]
129/158 Abs_forward ..................................................... [PASS]
130/158 Abs_transposed .................................................. [PASS]
131/158 SpatialLPPooling_backward ....................................... [PASS]
132/158 Tanh_backward ................................................... [PASS]
133/158 SpatialSoftMax .................................................. [PASS]
134/158 LogSigmoid_forward .............................................. [PASS]
135/158 SpatialMaxPooling_backward ...................................... [PASS]
136/158 SpatialFullConvolution_forward_batch ............................ [PASS]
137/158 HardTanh_backward ............................................... [PASS]
138/158 GPU ............................................................. [PASS]
139/158 SpatialDilatedMaxPooling_backward ............................... [PASS]
140/158 SpatialCrossMapLRN_forward_batch ................................ [PASS]
141/158 VolumetricDilatedConvolution .................................... [ERROR]
142/158 SpatialReflectionPadding_backward ............................... [PASS]
143/158 SpatialReflectionPadding_forward ................................ [PASS]
144/158 VolumetricAveragePooling_backward ............................... [PASS]
145/158 RReLU_backward .................................................. [PASS]
146/158 distkldiv ....................................................... [PASS]
147/158 SpatialDilatedConvolution_forward_batch ......................... [PASS]
148/158 VolumetricConvolution_backward_batch ............................ [PASS]
149/158 SpatialConvolutionMM_forward_batch .............................. [PASS]
150/158 SpatialReplicationPadding_forward ............................... [PASS]
151/158 SpatialFractionalMaxPooling_forward ............................. [PASS]
152/158 SpatialConvolutionMM_backward_batch ............................. [PASS]
153/158 RReLU_forward ................................................... [PASS]
154/158 ClassNLLCriterionMultipleTargetWeights .......................... [PASS]
155/158 LogSoftMax_forward_batch ........................................ [PASS]
156/158 Sqrt_forward .................................................... [PASS]
157/158 SpatialUpSamplingNearest_forward_batch .......................... [PASS]
158/158 TemporalConvolution_backward .................................... [PASS]
Completed 320 asserts in 158 tests with 1 failure and 6 errors
--------------------------------------------------------------------------------
SpatialAveragePooling_backward_batch
 Function call failed
/home/ubuntu/torch/install/share/lua/5.1/nn/THNN.lua:109: cuda runtime error (7) : too many resources requested for launch at /home/ubuntu/torch/extra/cunn/lib/THCUNN/SpatialAveragePooling.cu:218
stack traceback:
	[C]: in function 'v'
	/home/ubuntu/torch/install/share/lua/5.1/nn/THNN.lua:109: in function 'SpatialAveragePooling_updateGradInput'
	...torch/install/share/lua/5.1/nn/SpatialAveragePooling.lua:67: in function 'updateGradInput'
	/home/ubuntu/torch/install/share/lua/5.1/nn/Module.lua:31: in function 'backward'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:2932: in function 'v'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6029: in function </home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6027>
	[C]: in function 'xpcall'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:477: in function '_pcall'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:436: in function '_run'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:355: in function 'run'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6049: in function 'test'
	[string "_RESULT={cunn.test()}"]:1: in main chunk
	[C]: in function 'xpcall'
	/home/ubuntu/torch/install/share/lua/5.1/trepl/init.lua:652: in function 'repl'
	...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:199: in main chunk
	[C]: at 0x00406878

--------------------------------------------------------------------------------
VolumetricFullConvolution_pair_test
 Function call failed
/home/ubuntu/torch/install/share/lua/5.1/nn/THNN.lua:109: cuda runtime error (7) : too many resources requested for launch at /home/ubuntu/torch/extra/cunn/lib/THCUNN/vol2col.h:139
stack traceback:
	[C]: in function 'v'
	/home/ubuntu/torch/install/share/lua/5.1/nn/THNN.lua:109: in function 'VolumetricFullConvolution_updateOutput'
	...h/install/share/lua/5.1/nn/VolumetricFullConvolution.lua:117: in function 'forward'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:5261: in function 'v'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6029: in function </home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6027>
	[C]: in function 'xpcall'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:477: in function '_pcall'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:436: in function '_run'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:355: in function 'run'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6049: in function 'test'
	[string "_RESULT={cunn.test()}"]:1: in main chunk
	[C]: in function 'xpcall'
	/home/ubuntu/torch/install/share/lua/5.1/trepl/init.lua:652: in function 'repl'
	...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:199: in main chunk
	[C]: at 0x00406878

--------------------------------------------------------------------------------
SpatialAveragePooling_backward
 Function call failed
/home/ubuntu/torch/install/share/lua/5.1/nn/THNN.lua:109: cuda runtime error (7) : too many resources requested for launch at /home/ubuntu/torch/extra/cunn/lib/THCUNN/SpatialAveragePooling.cu:218
stack traceback:
	[C]: in function 'v'
	/home/ubuntu/torch/install/share/lua/5.1/nn/THNN.lua:109: in function 'SpatialAveragePooling_updateGradInput'
	...torch/install/share/lua/5.1/nn/SpatialAveragePooling.lua:67: in function 'updateGradInput'
	/home/ubuntu/torch/install/share/lua/5.1/nn/Module.lua:31: in function 'backward'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:2874: in function 'v'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6029: in function </home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6027>
	[C]: in function 'xpcall'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:477: in function '_pcall'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:436: in function '_run'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:355: in function 'run'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6049: in function 'test'
	[string "_RESULT={cunn.test()}"]:1: in main chunk
	[C]: in function 'xpcall'
	/home/ubuntu/torch/install/share/lua/5.1/trepl/init.lua:652: in function 'repl'
	...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:199: in main chunk
	[C]: at 0x00406878

--------------------------------------------------------------------------------
VolumetricFullConvolution
 Function call failed
/home/ubuntu/torch/install/share/lua/5.1/nn/THNN.lua:109: cuda runtime error (7) : too many resources requested for launch at /home/ubuntu/torch/extra/cunn/lib/THCUNN/vol2col.h:139
stack traceback:
	[C]: in function 'v'
	/home/ubuntu/torch/install/share/lua/5.1/nn/THNN.lua:109: in function 'VolumetricFullConvolution_updateOutput'
	...h/install/share/lua/5.1/nn/VolumetricFullConvolution.lua:117: in function 'forward'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:5283: in function 'v'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6029: in function </home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6027>
	[C]: in function 'xpcall'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:477: in function '_pcall'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:436: in function '_run'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:355: in function 'run'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6049: in function 'test'
	[string "_RESULT={cunn.test()}"]:1: in main chunk
	[C]: in function 'xpcall'
	/home/ubuntu/torch/install/share/lua/5.1/trepl/init.lua:652: in function 'repl'
	...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:199: in main chunk
	[C]: at 0x00406878

--------------------------------------------------------------------------------
SpatialConvolutionLocal_backward_batch
 Function call failed
$ Torch: not enough memory: you tried to allocate 0GB. Buy new RAM! at /home/ubuntu/torch/pkg/torch/lib/TH/THGeneral.c:226
stack traceback:
	[C]: at 0x7f9abf4664
	[C]: in function '__sub'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:1319: in function 'v'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6029: in function </home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6027>
	[C]: in function 'xpcall'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:477: in function '_pcall'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:436: in function '_run'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:355: in function 'run'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6049: in function 'test'
	[string "_RESULT={cunn.test()}"]:1: in main chunk
	[C]: in function 'xpcall'
	/home/ubuntu/torch/install/share/lua/5.1/trepl/init.lua:652: in function 'repl'
	...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:199: in main chunk
	[C]: at 0x00406878

--------------------------------------------------------------------------------
SpatialDilatedConvolution_backward_single
error on state (backward) 
LT failed: 2.7460753917694 >= 0.01
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:1763: in function 'v'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6029: in function </home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6027>
--------------------------------------------------------------------------------
VolumetricDilatedConvolution
 Function call failed
/home/ubuntu/torch/install/share/lua/5.1/nn/THNN.lua:109: cuda runtime error (7) : too many resources requested for launch at /home/ubuntu/torch/extra/cunn/lib/THCUNN/vol2col.h:139
stack traceback:
	[C]: in function 'v'
	/home/ubuntu/torch/install/share/lua/5.1/nn/THNN.lua:109: in function 'VolumetricDilatedConvolution_updateGradInput'
	...nstall/share/lua/5.1/nn/VolumetricDilatedConvolution.lua:51: in function 'updateGradInput'
	/home/ubuntu/torch/install/share/lua/5.1/nn/Module.lua:31: in function 'backward'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:5360: in function 'v'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6029: in function </home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6027>
	[C]: in function 'xpcall'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:477: in function '_pcall'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:436: in function '_run'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:355: in function 'run'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6049: in function 'test'
	[string "_RESULT={cunn.test()}"]:1: in main chunk
	[C]: in function 'xpcall'
	/home/ubuntu/torch/install/share/lua/5.1/trepl/init.lua:652: in function 'repl'
	...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:199: in main chunk
	[C]: at 0x00406878

--------------------------------------------------------------------------------
/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:363: An error was found while running tests!
stack traceback:
	[C]: in function 'assert'
	/home/ubuntu/torch/install/share/lua/5.1/torch/Tester.lua:363: in function 'run'
	/home/ubuntu/torch/install/share/lua/5.1/cunn/test.lua:6049: in function 'test'
	[string "_RESULT={cunn.test()}"]:1: in main chunk
	[C]: in function 'xpcall'
	/home/ubuntu/torch/install/share/lua/5.1/trepl/init.lua:652: in function 'repl'
	...untu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:199: in main chunk
	[C]: at 0x00406878	
                                                                      [255.4545s]	
th>

raak · September 1, 2016, 5:46am

Hey Jung,

use this to install https://github.com/dusty-nv/jetson-reinforcement/blob/master/CMakePreBuild.sh

jkjung · September 1, 2016, 6:22am

As far as I can tell, dusty’s script shouldn’t make any difference in the compiled nnx, cutorch and cunn modules. But anyway I’ll probably give it a try when I got the time.

cpadwick11 · September 1, 2016, 4:43pm

check out this thread:

https://devtalk.nvidia.com/default/topic/959945/jetson-tx1/torch-7-cuda-tests-quot-killed-quot-on-the-tx1/

I think torch is working fine, it is just running out of resources when you try to run the tests and the OS kills the process.

raak · September 5, 2016, 5:47am

ah :\ new issues with torch

Killed
CMake Error at THC_generated_THCTensorMathCompareT.cu.o.cmake:264 (message):
Error generating file
/tmp/luarocks_cutorch-scm-1-8224/cutorch/build/lib/THC/CMakeFiles/THC.dir//./THC_generated_THCTensorMathCompareT.cu.o

make[2]: *** [lib/THC/CMakeFiles/THC.dir/./THC_generated_THCTensorMathCompareT.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs…
make[1]: *** [lib/THC/CMakeFiles/THC.dir/all] Error 2
make: *** [all] Error 2

Error: Failed installing dependency: https://raw.githubusercontent.com/torch/rocks/master/cutorch-scm-1.rockspec - Build error: Failed building.

jkjung · September 5, 2016, 5:55am

@raak, what you’re reporting seems to be a known issue. Please refer to my earlier comment or dusty’s build script for solution.

I’d also modify torch/extra/cutorch/rocks/cutorch-scm-1.rockspec by replacing “-j$(getconf _NPROCESSORS_ONLN)” with “-j1”. This is to avoid nvcc error during compilation of cutorch due to running out of resources (memory?).

raak · September 5, 2016, 6:04am

yes i used the new script