UDP multicast issues

Has anyone ever used the Jetson in a UDP multicast situation or with Lightweight Communications and Marshalling (LCM)? I’m in a robotics club at school, and we are running into some problems trying to pick up a UDP multicast.

We have a lower level system that includes PID loops and motor controls ran by a TS7800 embedded machine. This system will broadcast sensor data over wifi or ethernet using Lightweight Communications and Marshaling. We need to pick up the sensor data up using the Jetson. LCM uses UDP multicast messages to transmit information between processes, as well as computers within the same multicast group.

-We are using “The Grinch” kernel
-We have already had the low system transmitting and getting picked up on a Windows laptop, so we know that much is working.
-Using iptables shows there isn’t any firewall blocking the connection.
-We connected the Jetson with the low computer using a TCP connection with netcat, and that worked fine.
-Tried the same thing with the -u flag for UDP and had success talking back and forth

We’re trying to subscribe to ip 239.255.76.67:7667

By definition UDP gets dropped without any attempt at reliability if anything at all isn’t perfect. You can’t use it with wifi and expect good results…not a function of Jetson, just a function of UDP doing what it is supposed to do. Wired local networks tend to work for this, wifi not so much.

Have you tested with tcpdump (or wireshark) on Jetson if it sees the incoming packets?

We are connected using an ethernet cable. We also tried wifi and swapping out the cable.

I’ve only used wireshark once and I’m fairly new to Linux. Please bear with me. I’ll try my best to describe what I’m seeing. I’ll to post a screenshot later if needed.

I set the filter to UDP on wireshark, I would assume the UDP packets would be flooding the screen as the TS7800 (low level controller computer) is outputting many per second. I get a lot of multicast listener reports. The TS7800’s IP is using IGMPv2 protocol for “membership report group 239.*yada”

After running netstat -g I got back:
no support for AF INET (igmp) on this system. I looked in the proc/net/ folder for igmp, but only found igmpv6.
I’m not sure if this only covers IPv6 or both IPv4&IPv6.

I hope this helps.

I get the same netstat error on one of my computers without multicast support and on another it doesn’t print any error (multicast is supported by the kernel).

What does this say on your Jetson:

zgrep MULTICAST /proc/config.gz

I don’t know if that option relates to your UDP multicast issue but at least sounds similar.

CONFIG_IP_MULTICAST is not set

Kulve got it! This is new information, the packets aren’t being lost…the kernel just wasn’t set up to understand or deal with multicast. Add the right modules and when “no support” disappears, the rest should start falling in place.

I haven’t been using Linux very long, and I’m not sure how to do what you’re saying.

dmesg | grep ‘Ethernet driver’ returns: r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded

lsmod doesn’t show that driver. I don’t know how to unload it and stick another one in its place.

The driver for the network card is fine as is. Different drivers deal with different protocols or hardware…in this case it is a protocol. Drivers will only show up in lsmod if they are in module format, many are integrated directly in the default kernels.

The directory /proc is really a reflection of information in the running kernel…nothing there is a “real” file, and typically writing to one (when possible) will directly change the kernel’s state. The file /proc/config.gz is just a compressed reflection of the running kernel’s configuration. One can take the original kernel source, use that config, and set localversion…and you have an exact duplicate of what is currently running.

This makes adding individual drivers and protocols much easier. Somewhere in any of the available methods for configuring and compiling the kernel (easiest directly on Jetson) it should be possible to find something in networking for multicast as related to your requirements…add that as a “module” if possible, then you can just load the module and suddenly your feature will work. There is no need for “Jetson-specific” information to build and install modules…get localversion set and any module build information should be correct from any admin type manual, e.g., Ubuntu. It’s easiest to just start by attempting compile an exact match kernel…then ask questions on whatever issue you run into.

I downloaded the kernel source from Github and unpacked it into /usr/src/. I then ran “make menuconfig” and switched on Multicasting (I opened up .config and saw the CONFIG_IP_MULTICAST=y). The help option said this was to control the CONFIG_IP_MULTICAST. I compiled the kernel using make, make modules, make modules_install, then make install.

After a reboot I checked the zgrep MULTICAST /proc/config.gz and it said it was still not set. I’m going to guess after the make install it didn’t set up my new kernel to boot. I was going to try to force it to use it, but I couldn’t seem to find the boot/grub folder or even determine what type of bootloader the Jetson has.

We just now found Santyago’s kernel building tutorial. We’re going to give that a go before we set this thing on fire. Or switch to a TCP socket or something instead.

Thank you by the way for the help. I feel like we’re closing in on the problem. [standby]

If you built as a module, make install is not needed…just module_install. Many people make a very simple mistake when building for a kernel…localversion must match. If you look in /lib/modules/, you will see a single subdirectory in the original Jetson as it first arrives. That subdirectory is named after the kernel version plus the localversion. For an R19.x release it is 3.10.24, and use of “uname -r” while booted to that kernel will reveal the rest, something roughtly of the format “-abcdefg”. This is how a kernel finds its modules. If you start with a kernel matching the running kernel, new modules will be compatible; if you fail to set localversion it won’t see it. Install without this will lead to a new subdirectory with just the “3.10.24” and no “-abcdefg” extension.

localversion is roughly 2 lines down from the top of menuconfig…I don’t have it running right now so I’m just going from memory.

The other thing is that the boot loader needs to be able to point to the kernel if you are doing more than a module install. This in turn requires a flash and not just a make install when using fastboot; in the case of u-boot you need the zImage copied into /boot, preferably to an alternate name and then edit extlinux.conf for the new entry.

What is your “uname -r” output? What is the content of “/lib/modules/”? Which boot loader are you using?

Just tried this and in make menuconfig it does not allow of setting CONFIG_IP_MULTICAST=m to modules option. only Y or empty is accepted. After I recompiled and reflashed the kernel I am also seeing the “CONFIG_IP_MULTICAST is not set” in the running kernel. Does anyone know why the kernel is rejecting this setting? We need to get multicast working desperately!

If you add the config feature as directly integrated in the kernel, then the kernel itself must be installed. The CONFIG_LOCALVERSION should be changed when doing this, as not all of the old module features are guaranteed to still be compatible when integrated features are changed (they might be, but might not be). So if the original version shows as something like “3.10.40-gdacac96”, you should consider changing the “-gdacac96” to something like “-gdacac96_2”.

FYI, non-module features do not show up in “/proc/config.gz”, only module features are listed there.

Note that when you change CONFIG_LOCALVERSION from something like “-gdacac96” to “-gdacac96_2” you must also install modules as this new “uname -r” implies modules will not be searched for in the original location. Within the original module directory is a subdirectory “extra”…these are not installed via module install, you will have to recursively copy these from the origial module directory to the new module directory. If you need more information on complete kernel and module install just ask.

I gave the same issue a shot, without success so far, but I am also a newbie to these kind of tweaks. It would be amazing if some of you have some hints:

I got this kernel src http://developer.download.nvidia.com/embedded/L4T/r21_Release_v4.0/source/kernel_src.tbz2, copied the current kernel configuration with zcat /proc/config.gz > .config. With make menuconfig I changed the .config according to https://delog.wordpress.com/2014/11/26/enable-ip-multicast-routing-in-linux-kernel/ and made sure that the CONFIG_LOCALVERSION matches the pattern “…-acbde” of the (only) folder I see in /lib/modules.
I then make, make modules, make modules_install and reboot.

zgrep MULTICAST /proc/config.gz still shows CONFIG_IP_MULTICAST not set :(
(/lib/modules does not show a new additional folder)

Any ideas?

If you look at the “make menuconfig” check box for features you’ll notice some of them are ‘', some ‘M’, and some empty. Empty boxes are not set, both ‘M’ and '’ are enabled. However, ‘M’ is enabled in the form of a module, whereas ‘*’ is enabled by integration directly into the kernel.

If a feature is enabled as a module, a module is built and modules can be loaded or unloaded on an existing kernel at run time. An integrated feature does not build a module, and modules cannot be loaded or unloaded for that feature. The kernel itself must be updated (which means there is a strong chance of invalidating modules which depended on certain features and traits of the kernel they were compiled against…think of it as a mix of ABI version and features).

The usual method of making sure your modules and kernel were designed to work together is the module loading scheme which you already know about (requiring the CONFIG_LOCALVERSION you have set up). Should you upgrade the kernel itself (not the source code, but the non-module configuration) expect that it is best to start with a new CONFIG_LOCALVERSION. If that new configuration extends an existing configuration you’ll probably just want a slight change to the existing CONFIG_LOCALVERRSION. For example, if you’ve been using “-gdacac96”, then you might alter this to something like “-gdacac96_1” (or something informative like “-gdacac96_1_mcast”.

In the case where you must install a kernel with a new CONFIG_LOCALVERSION (using “-gdacac96_1”) the make modules_install would put modules in “/lib/modules/3.10.40-gdacac96_1”. It’s up to you to copy the new kernel (not modules) in “/boot” and update “/boot/extlinux/extlinux.conf” (I’m assuming L4T R21.4…earlier R19.x L4T used fastboot or “/boot/extlinux.conf”).

The name of the kernel file will depend upon format, e.g., the “zImage” is popular and is compressed, the “Image” file is the same thing with no compression. Other formats do exist as well, e.g., currently u-boot uses zImage, but at one time had its own format. For a Jetson TK1, the zImage is produced in kernel source at 32-bit ARMv7 architecture-dependent location “arch/arm/boot/”. You do not normally have to worry about updating firmware for minor version changes and can ignore that part unless you change a major version or the hardware itself. I’d advise incrementing the file name in such a way that the non-module change is obvious, e.g., “-mcast” instead of including the “-gdacac96”, and then placing a zImage in “/boot” with a name like “zImage-mcast”.

Notice in the “/boot/extlinux/extlinux.conf” file that there is a prologue followed by a section for an individual kernel boot. You can duplicate one of those and alter file names to create a second bootable entry naming the new kernel. Serial console can then be used at boot time to select the alternate entry; once things work as you want, you can make the new entry default. When copying an entry note that the “APPEND” key/value pair is one very long line and must remain that way…don’t let line wrap fool you. Here’s a contrived example extlinux.conf:

TIMEOUT 30
DEFAULT primary

MENU TITLE Jetson-TK1 eMMC boot options

LABEL primary
      MENU LABEL primary kernel
      LINUX /boot/zImage
      FDT /boot/tegra124-jetson_tk1-pm375-000-c00-00.dtb
      APPEND console=ttyS0,115200n8 console=tty1 no_console_suspend=1 lp0_vec=2064@0xf46ff000 mem=2015M@2048M memtype=255 ddr_die=2048M@2048M section=256M pmuboard=0x0177:0x0000:0x02:0x43:0x00 tsec=32M@3913M otf_key=c75e5bb91eb3bd947560357b64422f85 usbcore.old_scheme_first=1 core_edp_mv=1150 core_edp_ma=4000 tegraid=40.1.1.0.0 debug_uartport=lsport,3 power_supply=Adapter audio_codec=rt5640 modem_id=0 android.kerneltype=normal fbcon=map:1 commchip_id=0 usb_port_owner_info=0 lane_owner_info=6 emc_max_dvfs=0 touch_id=0@0 board_info=0x0177:0x0000:0x02:0x43:0x00 root=/dev/mmcblk0p1 rw rootwait tegraboot=sdmmc gpt

LABEL <b>mcast</b>
      MENU LABEL <b>multicast</b>
      LINUX /boot/<b>zImage-mcast</b>
      FDT /boot/tegra124-jetson_tk1-pm375-000-c00-00.dtb
      APPEND console=ttyS0,115200n8 console=tty1 no_console_suspend=1 lp0_vec=2064@0xf46ff000 mem=2015M@2048M memtype=255 ddr_die=2048M@2048M section=256M pmuboard=0x0177:0x0000:0x02:0x43:0x00 tsec=32M@3913M otf_key=c75e5bb91eb3bd947560357b64422f85 usbcore.old_scheme_first=1 core_edp_mv=1150 core_edp_ma=4000 tegraid=40.1.1.0.0 debug_uartport=lsport,3 power_supply=Adapter audio_codec=rt5640 modem_id=0 android.kerneltype=normal fbcon=map:1 commchip_id=0 usb_port_owner_info=0 lane_owner_info=6 emc_max_dvfs=0 touch_id=0@0 board_info=0x0177:0x0000:0x02:0x43:0x00 root=/dev/mmcblk0p1 rw rootwait tegraboot=sdmmc gpt

Note that the original module directory would still exist and be bootable alongside the new module directory via serial console. Changing “DEFAULT” from “primary” to “mcast” would make the “mcast” label the default entry (best done after testing). If very quickly after boot starts hit a key on the keyboard of serial console, you’ll get a prompt for the u-boot environment. You actually need to wait a moment instead and hit a key when it gets to kernel selection menu…then you can select something like “2” for the second entry. If you hit a key too early and end up on u-boot environment command line just type “boot” to continue and hit a keyboard key again after that.

Within the old module directory (“/lib/modules/3.10.40-gdacac96”) you will see a subdirectory “extra”. A normal “make modules_install” has no knowledge of this directory, it is for nVidia-specific files. You would want to recursively copy this to the new module directory, e.g.:

cd /lib/modules/3.10.40-mcast
sudo cp -adR ../3.10.40-gdacac96/extra .

Incidentally, the “extra” subdirectory is a big part of why back-porting to 3.10.40 works better than going mainline. Those files are not loadable in a 4.x.x kernel (additionally, the video driver is bound to the Xorg ABI version…this version changes frequently).

Wow @linuxdev,thanks for sharing your knowledge in such great detail! I’ll dig into this during the next days!