libml_module.so panic error

When I try to initialize the TCP viewer or Carter Navigation Stack operations ( bazel run packages/navsim/apps:navsim_viewer_tcp) and ( bazel run //apps/carter/navsim:navsim_navigate – --more packages/navsim/maps/warehouse.json,packages/navsim/robots/carter.json) respectively, I get the error:

2019-11-19 11:24:36.555 ERROR engine/alice/backend/modules.cpp@307: ml: /home/USER/.cache/bazel/_bazel_USER/078620953386c8eebc0e244bf318a091/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/carter/navsim/navsim_navigate.runfiles/com_nvidia_isaac//external/com_nvidia_isaac/packages/ml/libml_module.so: cannot open shared object file: No such file or directory
2019-11-19 11:24:36.555 PANIC engine/alice/backend/modules.cpp@309: Could not load all required modules for application

| Isaac application terminated unexpectedly |

#01 engine/alice/tools/main(+0x8046a) [0x5633ac82b46a]
#02 engine/alice/tools/main(+0xe86d0) [0x5633ac8936d0]
#03 engine/alice/tools/main(+0xe8a40) [0x5633ac893a40]
#04 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f8fae273890]
#05 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f8fad90de97]
#06 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f8fad90f801]
#07 engine/alice/tools/main(+0x9d1f7) [0x5633ac8481f7]
#08 engine/alice/tools/main(+0x34bdc) [0x5633ac7dfbdc]
#09 engine/alice/tools/main(+0x350f3) [0x5633ac7e00f3]
#10 engine/alice/tools/main(+0x352e4) [0x5633ac7e02e4]
#11 engine/alice/tools/main(+0x15c23) [0x5633ac7c0c23]
#12 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f8fad8f0b97]
#13 engine/alice/tools/main(+0x1af7a) [0x5633ac7c5f7a]

Minidump written to: /tmp/6900a552-281c-4e31-16cfe697-3d3320a8.dmp

I have installed TensorFlow and CUDA 10.0 as well.

I need help to view the training data when training Isaac with respective metrics as soon as possible for a research assignment. I would appreciate help… thanks!

Any update on this issue?
I have exactly the same error.

NB: if I do

locate libml_module

I’m able to find multiple libml_module.so in my system, e.g.

/home/user1/.cache/bazel/_bazel_user1/720d1ce4d0169d46a807b418dc594c19/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/packages/ml/libml_module.so

but none of them is found in the expected path described in the error message:

2019-11-26 12:12:17.892 ERROR   engine/alice/backend/modules.cpp@307: ml: /home/user1/.cache/bazel/_bazel_user1/720d1ce4d0169d46a807b418dc594c19/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/carter/navsim/navsim_navigate.runfiles/com_nvidia_isaac//external/com_nvidia_isaac/packages/ml/libml_module.so: cannot open shared object file: No such file or directory

Isaac SDK would try to load the same module from several paths before giving the error message. For libml_module.so, very likely it is failing to load the module for some missing dependency shared libraries such as cuda, cudnn and tensorrt. Please check if cuda 10.0 is symlinked in /usr/local/cuda and included in LD_LIBRARY_PATH env.

I have the very similar issue, and still could not find any solution for this …
2020-02-25 19:16:19.025 INFO engine/alice/backend/modules.cpp@226: Loaded module ‘packages/viewers/libviewers_module.so’: Now has 149 components total
2020-02-25 19:16:19.025 ERROR engine/alice/backend/modules.cpp@250: packages/ml/libml_module.so: libcurand.so.10.0: cannot open shared object file: No such file or directory
2020-02-25 19:16:19.025 PANIC engine/alice/backend/modules.cpp@252: Could not load all required modules for application

| Isaac application terminated unexpectedly |

#01 engine/alice/tools/main(+0x880fa) [0x55c55e6a60fa]
#02 engine/alice/tools/main(+0xf97d0) [0x55c55e7177d0]
#03 engine/alice/tools/main(+0xf9b40) [0x55c55e717b40]
#04 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f7a1bde9890]
#05 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f7a1b483e97]
#06 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f7a1b485801]
#07 engine/alice/tools/main(+0xa33c7) [0x55c55e6c13c7]
#08 engine/alice/tools/main(+0x3b91c) [0x55c55e65991c]
#09 engine/alice/tools/main(+0x3cc39) [0x55c55e65ac39]
#10 engine/alice/tools/main(+0x3cfdf) [0x55c55e65afdf]
#11 engine/alice/tools/main(+0x17433) [0x55c55e635433]
#12 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f7a1b466b97]
#13 engine/alice/tools/main(+0x1d4ca) [0x55c55e63b4ca]

Minidump written to: /tmp/ed9a4028-27a7-4e8a-17fc07ac-d03974fa.dmp
/home/ccv-main/.cache/bazel/_bazel_ccv-main/1ad872625d63393b99e3875bb9ef9171/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/navsim/navsim_navigate: line 38: 10395 Aborted (core dumped) engine/alice/tools/main --app apps/navsim/navsim_navigate.app.json --more “packages/navsim/robots/$ROBOT_ID.json,packages/navsim/maps/$MAP.json” $@

@gkobas
Please install cuda 10.0 locally and add /usr/local/cuda-10.0/lib64 to LD_LIBRARY_PATH env

1 Like

Thank you so much , now it works :)

I used the following commands for those who seeks a solution :

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.0/lib64
echo ‘export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.0/lib64’ >> ~/.bashrc

then I could be able to run
bazel run //apps/navsim:navsim_navigate

4 Likes