Close

[SOLVED] docker, udev, USB naming

wjcarpenterWJCarpenter wrote 09/02/2022 at 17:18 • 5 min read • Like

I'm surely not the first person to figure this out, but I didn't come across this solution while searching around. They are probably out there and I just didn't find them.

The problem, in a nutshell, is how to reliably map USB devices into a docker container. The more specific problem, for me, is that I run the Home Assistant docker container (with docker-compose) and want to map in the USB ports for my Zigbee and Z-Wave interfaces. (I use the popular HUSBZB-1 combo dongle, which slightly complicates things, but that's incidental.) Things were pretty easy when I could always count on them being /dev/ttyUSB0`and /dev/ttyUSB1 on the host machine. When I started using an unrelated USB device on the same host machine, things turned into a little dance of plugging, unplugging, restarting containers, pulling out hair, etc.

It's easy to find solutions for creating reliable, persistent names for USB devices on Linux. The answer is to use something in udev rules. udev even has a really easy way to create a symlink with whatever fixed name you choose. It's very simple, and you will find zillions of internet articles describing how to do it. It doesn't take long to discover, however, that docker can't cope with mapping those host machine symlink names to a container device name. When you start looking for solutions to that problem, you find that it's been a docker sore spot for years and years.

The solution is to use hard links instead of symlinks. It's not particularly difficult, but it's also not particularly obvious if your only exposure to udev is from reading canned recipes on various web pages.

Here's how I did it. I have 3 USB devices of interest: the Zigbee and Z-Wave devices mentioned, and a 3D printer that I use with Octoprint in an unrelated docker container. There are many articles around to tell you how to find the internal details (vendor, serial number), etc of USB devices for use with udev, so I won't go into that here. However, I will direct your attention to the very useful pseudo-directory /dev/serial/by-id/. (There is also /dev/serial/by-path/. You don't want that one.) Without you doing anything, it will contain symlinks to your serial USB devices. Here's part of mine:

lrwxrwxrwx 1 root root 13 Sep  1 18:18 usb-Silicon_Labs_HubZ_Smart_Home_Controller_813004BE-if00-port0 -> ../../ttyUSB2
lrwxrwxrwx 1 root root 13 Sep  1 18:18 usb-Silicon_Labs_HubZ_Smart_Home_Controller_813004BE-if01-port0 -> ../../ttyUSB3

For a given device, the name of the symlink is always going to be the same (at least as far as I can tell). If docker could map symlinks to devices, that would be all you would need for reliable names. But, alas. There are various Linux utilities for dereferencing symlinks, and the one we want to use here is realpath. It provides an absolute path to the real file. (A commonly-suggested alternative, readlink, would only give you the relative path mentioned in the symlink, which is not directly helpful.)

$ realpath /dev/serial/by-id/usb-Silicon_Labs_HubZ_Smart_Home_Controller_813004BE-if00-port0
/dev/ttyUSB2

Armed with that, we can now stitch it into udev rules. If you want to know more about udev rules, there are plenty of explanations available, so this is going to be just a distilled description of how to apply them. The persistent names I use are /dev/ttyUSB-ender3v2, /dev/ttyUSB-zigbee, and /dev/ttyUSB-zwave. Those are the hard links I want to end up with. Here is a tiny shell script that maps one of the above symlinks to the applicable hard link, depending on its single command line argument:

#!/bin/bash

p=""
if [ "$1" == "ender3v2" ]
then
  p=`realpath /dev/serial/by-id/usb-1a86_USB_Serial-if00-port0`
elif [ "$1" == "zwave" ]
then
  p=`realpath /dev/serial/by-id/usb-Silicon_Labs_HubZ_Smart_Home_Controller_813004BE-if00-port0`
elif [ "$1" == "zigbee" ]
then
  p=`realpath /dev/serial/by-id/usb-Silicon_Labs_HubZ_Smart_Home_Controller_813004BE-if01-port0`
fi

if  [ "x$p" != "x" ]
then
    rm -f /dev/ttyUSB-$1
    ln $p /dev/ttyUSB-$1
fi

No rocket science there. Your devices will be different, and your naming choices will be different, but it should be pretty obvious how to change the script to suit. If you have multiple devices of the same type, you will have to do some additional logic to disambiguate them, but I leave that to you. Put the script in a file with a handy name in a handy location and make it executable (at least executable by root). Mine is /root/bin/udev-usb-tweak.sh. You probably want it to be on your boot or root filesystem since your USB devices might be recognized during boot up before any filesystems are mounted. In any case, experiment to verify that it works.

To run the script when the host machine boots or one of the USB devices is plugged in, use the udev RUN action. My /etc/udev/rules.d/99-usb-persistent.rules contains these rules:

# These devices get mapped into docker containers. docker containers can't deal properly with symlinks mounted from
# outside, so we can't use the udev SYMLINK action. Instead run a script that finds the path to the
# dynamically allocated USB device and makes a hard link to it with a fixed name. Naturally, we remove
# that hard link when the USB device is removed.

# This is the Crealtiy Ender 3v2 printer mapped into the Octoprint container
SUBSYSTEM=="tty", ATTRS{idVendor}=="1a86", ATTRS{idProduct}=="7523", ACTION=="add",    RUN+="/root/bin/udev-usb-tweak.sh ender3v2"
SUBSYSTEM=="tty", ATTRS{idVendor}=="1a86", ATTRS{idProduct}=="7523", ACTION=="remove", RUN+="/bin/rm -f /dev/ttyUSB-ender3v2"

# This is the combined z-wave/zigbee dongle mapped into the Home Assistant container.
SUBSYSTEM=="tty", ATTRS{idVendor}=="10c4", ATTRS{idProduct}=="8a2a", ATTRS{serial}=="813004BE", ACTION=="add",    RUN+="/root/bin/udev-usb-tweak.sh zwave"
SUBSYSTEM=="tty", ATTRS{idVendor}=="10c4", ATTRS{idProduct}=="8a2a", ATTRS{serial}=="813004BE", ACTION=="add",    RUN+="/root/bin/udev-usb-tweak.sh zigbee"
SUBSYSTEM=="tty", ATTRS{idVendor}=="10c4", ATTRS{idProduct}=="8a2a", ATTRS{serial}=="813004BE", ACTION=="remove", RUN+="/bin/rm -f /dev/ttyUSB-zwave /dev/ttyUSB-zigbee"

For each of the ACTION=="add" lines, the script is called with the appropriate command line argument. There are two separate invocations for the Zigbee/Z-Wave dongle because it's really two devices in one. For tidiness, there are also rules to remove my hard links on the ACTION=="remove" lines. Notice that running things via `RUN` requires giving the full absolute path to the executable. 

In theory, you could do everything from the script directly in the udev rules file, but I have better things to do with my time than to fight through multiple layers of competing escaping, encoding, and arcane corner cases. OK, OK, you talked me into it. Here is the same thing, but all in the udev rules file. It doesn't use the separate shell script. Are you happy now?

# These devices get mapped into docker containers. docker containers can't deal properly with symlinks mounted from
# outside, so we can't use the udev SYMLINK action. Instead run a script that finds the path to the
# dynamically allocated USB device and makes a hard link to it with a fixed name. Naturally, we remove
# that hard link when the USB device is removed.

# This is the Crealtiy Ender 3v2 printer mapped into the Octoprint container
SUBSYSTEM=="tty", ATTRS{idVendor}=="1a86", ATTRS{idProduct}=="7523", ACTION=="add",    RUN+="/bin/rm -f /dev/ttyUSB-ender3v2"
SUBSYSTEM=="tty", ATTRS{idVendor}=="1a86", ATTRS{idProduct}=="7523", ACTION=="add",    RUN+="/bin/bash -c '/bin/ln `/usr/bin/realpath /dev/serial/by-id/usb-1a86_USB_Serial-if00-port0` /dev/ttyUSB-ender3v2'"

SUBSYSTEM=="tty", ATTRS{idVendor}=="1a86", ATTRS{idProduct}=="7523", ACTION=="remove", RUN+="/bin/rm -f /dev/ttyUSB-ender3v2"

# This is the combined z-wave/zigbee dongle mapped into the Home Assistant container.
SUBSYSTEM=="tty", ATTRS{idVendor}=="10c4", ATTRS{idProduct}=="8a2a", ATTRS{serial}=="813004BE", ACTION=="add",    RUN+="/bin/rm -f /dev/ttyUSB-zwave /dev/ttyUSB-zigbee"
SUBSYSTEM=="tty", ATTRS{idVendor}=="10c4", ATTRS{idProduct}=="8a2a", ATTRS{serial}=="813004BE", ACTION=="add",    RUN+="/bin/bash -c '/bin/ln `/usr/bin/realpath /dev/serial/by-id/usb-Silicon_Labs_HubZ_Smart_Home_Controller_813004BE-if00-port0` /dev/ttyUSB-zwave'"
SUBSYSTEM=="tty", ATTRS{idVendor}=="10c4", ATTRS{idProduct}=="8a2a", ATTRS{serial}=="813004BE", ACTION=="add",    RUN+="/bin/bash -c '/bin/ln `/usr/bin/realpath /dev/serial/by-id/usb-Silicon_Labs_HubZ_Smart_Home_Controller_813004BE-if01-port0` /dev/ttyUSB-zigbee'"

SUBSYSTEM=="tty", ATTRS{idVendor}=="10c4", ATTRS{idProduct}=="8a2a", ATTRS{serial}=="813004BE", ACTION=="remove", RUN+="/bin/rm -f /dev/ttyUSB-zwave /dev/ttyUSB-zigbee"

All that's left is to change your docker device mapping to use the persistent names that you have chosen. It's pretty sweet once you see it working.

Like

Discussions

Amparo Dios wrote 02/15/2023 at 21:04 point

@WJCarpenter , are you sure, that it worked for you?

I tried this approach with hardlinks and it seems not to work. At least my cups container refuses to find my printer.

@triplepoint , did you ever succeeded here? What was your solution? Privileged mode isn't actually a good solution because in privileged mode you even don't need the hardlink.

  Are you sure? yes | no

WJCarpenter wrote 02/16/2023 at 02:53 point

It's definitely working for me for the 3 devices I mentioned. I haven't changed it since I finished the original posting. FWIW, the machine where all this runs for me is Ubuntu 20.04.5 LTS.

  Are you sure? yes | no

Amparo Dios wrote 02/16/2023 at 16:59 point

Thank you for the quick reply! I run Debian. I am trying to connect my usb MFP to cups and sane containers. If I start container with the option "--device /dev/bus/usb/001/024" then everything works and the sane can identify my scanner inside the container.

But if I do:

ln /dev/bus/usb/001/024 /dev/usb/scanner

and then use the option "--device /dev/usb/scanner" it doesn't find the scanner, although both links are totally the same with same permissions, groups, ids, etc.

I just don't get it. Do you have any idea how can I research and try to find the core issue for this problem?

  Are you sure? yes | no

WJCarpenter wrote 02/17/2023 at 01:20 point

(This reply will probably show up out of order. There was no "reply" button for me on your latest post where you mention "n /dev/bus/usb/001/024 /dev/usb/scanner")

It certainly seems like what you did should work, though I'm not really an expert on this area. I wonder if maybe the CUPS/SANE code is looking specifically for devices under /dev/bus/usb for some reason. Are you able to see your mapped /dev/usb/scanner from inside the container. If yes, maybe try linking it *inside the container* back to some /dev/bus/usb path.

  Are you sure? yes | no

Amparo Dios wrote 02/17/2023 at 13:24 point

Thank you for your answer. I thought about that and I even tried to create a hard link on host system like that:

ln /dev/bus/usb/001/024 /dev/bus/usb/001/031

and then pass "--device /dev/bus/usb/001/031" and made sure, that this device appeared in container. And even that didn't work.

ChatGPT thinks, that docker doesn't support hard links because the filesystem in container is on some other layer and will not work as expected.

Can you share the command with which you create your container? Do you use privileged mode?

  Are you sure? yes | no

WJCarpenter wrote 02/18/2023 at 04:52 point

I start my containers with docker-compose. The relevant section of the config files are

    devices:
     - /dev/ttyUSB-ender3v2:/dev/ttyACM0

and

    devices:
      - /dev/ttyUSB-zwave:/dev/ttyUSB0
      - /dev/ttyUSB-zigbee:/dev/ttyUSB1

Naturally, the application software inside the containers is able to find the devices using the mapped-to names (ttyACM0, ttyUSB0, ttyUSB1).

  Are you sure? yes | no

triplepoint wrote 09/11/2022 at 03:02 point

Aha, that last problem was resolved by running the container in privileged mode.  I suspect there may be a more elegant solution, but I'm willing to accept this for now.

  Are you sure? yes | no

triplepoint wrote 09/11/2022 at 02:51 point

This is pretty neat, and inspired me to give it ago.  Back story, I have an RTL-SDR dongle plugged into a server running RTL-433 in a docker container, and I was getting tired of editing the docker-compose file with the new hub and device ids liike `/dev/bus/usb/001/002` when the machine rebooted.

So here's what I came up with:

```

SUBSYSTEM=="usb", ATTR{idVendor}=="0bda", ATTR{idProduct}=="2838", ACTION=="add", RUN="/bin/bash -c '/bin/rm -f /dev/rtlsdr_%s{serial}; /bin/ln $env{DEVNAME} /dev/rtlsdr_%s{serial}'"

ENV{ID_MODEL}=="RTL2838UHIDIR", ACTION=="remove", RUN="/bin/rm -f /dev/rtlsdr_%E{ID_SERIAL_SHORT}"

```

I'm using the $env and $attr replacement variables detailed in https://man7.org/linux/man-pages/man7/udev.7.html to set up the symlink.  For instance the `$s{serial}` element inserts the device's serial number into the link name, so it ends up being `/dev/rtlsdr_000000001` (I guess the factory sets the serials to 1 for every part :| ).  For the link source path, I'm using $env{DEVNAME}, which is the `/dev/bus/usb/001/002`-looking path.

The remove action was interesting, in that t the $s{serial} value wasn't in context there.  I got it to work by switching to the udev environment variable %E{ID_SERIAL_SHORT} instead of $s{serial}, but they're the same value.

You can get a list of those device properties with:

```

udevadm info --query=property --name /dev/bus/usb/001/041
BUSNUM=001
DEVNAME=/dev/bus/usb/001/041
DEVNUM=041
DEVPATH=/devices/pci0000:00/0000:00:1d.0/usb1/1-1/1-1.1/1-1.1.1
DEVTYPE=usb_device
DRIVER=usb
ID_BUS=usb
ID_MODEL=RTL2838UHIDIR
ID_MODEL_ENC=RTL2838UHIDIR
ID_MODEL_FROM_DATABASE=RTL2838 DVB-T
ID_MODEL_ID=2838
ID_REVISION=0100
ID_SERIAL=Realtek_RTL2838UHIDIR_00000001
ID_SERIAL_SHORT=00000001
ID_USB_INTERFACES=:ffffff:
ID_VENDOR=Realtek
ID_VENDOR_ENC=Realtek
ID_VENDOR_FROM_DATABASE=Realtek Semiconductor Corp.
ID_VENDOR_ID=0bda
MAJOR=189
MINOR=40
PRODUCT=bda/2838/100
SUBSYSTEM=usb
TYPE=0/0/0
USEC_INITIALIZED=413970332955

```

After all that, it _still_ seems that Docker doesn't want to read from a hard linked device.  I'm still troubleshooting that, but I hope this helps someone out that makes it this far.

  Are you sure? yes | no