Developing PyCodelets to be used by other applications

Hi all,

Probably a set of dumb questions here but I am in the process of learning about the Isaac SDK and have been running some toy experiments to learn more about what I can do with Isaac.

My first test was to create a simple C++ codelet that could be used within carter_sim to save images to file and I managed to get this to work well enough but most of my coding is in Python so I was attempting to replicate this with python but have run into a few snags in my understanding I hope you can help me with.

  1. Am I correct that PyCodelets can only be run by Application objects within python as py_binary files or can they be used to create isaac_cc_modules like C++ codelets? Do they never get used within association with a app.json files?

  2. With a python Application object, is it safe to load multiple graphs as follows or will that cause issues?

app.load_graph("apps/carter_sim/carter.graph.json")
app.load_graph("apps/carter_sim/navigation.graph.json")

I am hoping the answer to these questions can help me debug the errors that I am currently getting with my code (ERROR and PANIC output shown below) and increase my understanding of how I can properly integrate PyCodelets into my applications with Isaac. Any advice as to what I am likely missing/doing wrong based on these errors would also be appreciated.

2019-05-23 11:49:13.317 ERROR   engine/alice/backend/modules.cpp@212: Module with name 'packages/sight/libsight_module.so' already loaded
2019-05-23 11:49:13.332 ERROR   engine/alice/components/deprecated/Behavior.cpp@41: Could not find child node with alias 'random'
2019-05-23 11:49:13.332 ERROR   engine/alice/components/deprecated/Behavior.cpp@60: Could not start behavior with alias 'random'
2019-05-23 11:49:13.332 PANIC   engine/alice/application.cpp@115: No node with name 'map'
2019-05-23 11:49:13.333 ERROR   packages/navigation/PoseAsGoal.cpp@47: Could not read pose 'world'_T_'pose_as_goal'.
2019-05-23 11:49:13.392 ERROR   engine/alice/components/deprecated/Behavior.cpp@41: Could not find child node with alias 'random'
2019-05-23 11:49:13.392 ERROR   engine/alice/components/deprecated/Behavior.cpp@60: Could not start behavior with alias 'random'
2019-05-23 11:49:13.392 PANIC   engine/alice/application.cpp@115: No node with name 'map'
2019-05-23 11:49:13.392 ERROR   engine/alice/components/deprecated/Behavior.cpp@41: Could not find child node with alias 'global_loc'
2019-05-23 11:49:13.392 ERROR   engine/alice/components/deprecated/Behavior.cpp@60: Could not start behavior with alias 'global_loc'

1- As far as my understanding of the SDK goes you are correct. You can only use python codeletes within py_binary rules. isaac_cc_modules use cc_binary rules under the hood which is only for c++ thus if you have any pycodelet in your application you have to use graph.json and config.json instead of app.json.

2- I am not sure if you can load multiple graphs this way but concerning the errors shown above it would be great if you can share your graph.json and config.json so that I can get better insight as to what is happening.

Thanks very much for your help and I think you are right about my first issue.

Regarding my second issue since I put this up I have actually (seemingly) managed to load multiple graph.json and config.json files and stopped the errors with some changes.

Error 1. can seemingly be ignored as even when it pops up the application still seems to run fine. I was able to remove this error message by commenting out line 26 in Application.py seemingly without consequence.

The rest of the errors which were actually stopping my code from running apparently had to do with the order I was reading in graph and config files. It would appear that you need to load your config files before your graph files. The following seems to now work fine for me.

app.load_config("apps/carter_sim/carter.config.json")
app.load_config("apps/carter_sim/navigation.config.json")
app.load_config("apps/carter_sim/python_image_save/carter_py_img_saver.config.json")
app.load_config("apps/assets/maps/carter_warehouse_p.config.json")
app.load_graph("apps/carter_sim/carter.graph.json")
app.load_graph("apps/carter_sim/navigation.graph.json")
app.load_graph("apps/carter_sim/python_image_save/carter_py_img_saver.graph.json")
app.load_graph("apps/assets/maps/carter_warehouse_p.graph.json")

I actually have a new third problem now.

  1. My new problem is how to actually read the ImageProto messages I am receiving. I was able to simply convert code from a c++ example for my previous implementation but it unclear to me how I should interpret ImageProto messages in Python. Does anyone know how I can transfrom the ImageProtoReader object to a numpy array for housing images? I am able to read the number of rows, column, channels and elementType easily, however I don’t know how to use this information in combination with the dataBufferIndex (which is currently always zero) to actually turn this ImageProto into something I can manipulate (numpy array).

As far as my understanding goes, the proto messages you receive from the channels are made up from 3 parts:

  1. A header where you have for example timestamps
  2. The actual proto message where you have the different attributes presented in the documentation*(eg. dataBufferIndex, Rows, Columns...)
  3. The data buffer that contains the image data (pixel values)

I use this piece of code to access image data published from the realsense component:

#include "engine/gems/image/color.hpp"
#include "engine/gems/image/io.hpp"
#include "engine/gems/image/utils.hpp"
#include "messages/camera.hpp"
#include "messages/messages.hpp"

ImageConstView3ub color_image;
FromProto(rx_color().getProto().getImage(), rx_color().buffers(), color_image);

Thanks for your information. It helped set me along the right track.

As you mentioned the key is understanding how the proto messages are formatted, with a header, the proto of different attributes and the data buffer of image data. For my old C++ implementation I used the same basic function as you did. I was copying from the example in the opencv_edge_detection tutorial.

I have now managed to work out a Python implementation shown below to read an image. I have spaced it out a bit to show all the components to try and make it clear for others. Brackets show expected values.

# Colour camera
# Extract the proto reader of interest
rgb_image_proto = self.rgb_rx.get_proto().image     # Note that image is only for colour cameras

# Extract proto attributes
rgb_rows = rgb_image_proto.rows     # number of rows (carter_sim - 540)
rgb_cols = rgb_image_proto.cols     # number of columns (carter_sim - 960)
rgb_channels = rgb_image_proto.channels     # number of channels (colour camera - 3)
rgb_buffer_idx = rgb_image_proto.dataBufferIndex    # buffer index for get_buffer_content

# Check data type is correct (colour camera - uint8)
if str(rgb_image_proto.elementType) == 'uint8':
  rgb_data_type = np.uint8
else:
  print("ERROR! Colour camera expects uint8 data type. Got {} data type".format(rgb_image_proto.elementType))
  return

# Read the image buffer
rgb_image_buffer = self.rgb_rx.get_buffer_content(rgb_buffer_idx)

# Transform into image
rgb_image = np.frombuffer(rgb_image_buffer, dtype=rgb_data_type)
rgb_image = rgb_image.reshape((rgb_rows, rgb_cols, rgb_channels))
1 Like