Close
0%
0%

Linux-ESPNOW

An attempt at implementing a direct link between a linux station and an ESP module using ESPNOW protocol for real time robot control

Similar projects worth following
For a robotic application, I would like to use a direct wireless link between a control computer and a distant robot. With minimum latency, and delay.

Expressif proposed ESPNOW, a way to link two ESP8266 or ESP32 using vendor specific action frame of 802.11. This is way faster than UDP on regular WIFI. My first test show less than 300us to transmit 200Bytes of data !

A solution is to use two of this module, but then we need to use another link between the computer and one of the modules. We don't want to use USB because of the possible jitter and delay it introduces.

Since some wireless card support monitoring and injection of raw packets, it should be possible to use directly the WiFi card of the control PC !! But how??

This project will keep track of my investigation, and any advise is warmly welcome.

Ideally, at the end, we could have a ESP-NOW driver for linux that allow to efficiently connect our favorite ESP chip to a control PC.

Some ressources to be update on the go

ESP NOW protocol:

https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/wifi/esp_now.html

http://wiki.ai-thinker.com/_media/esp8266/docs/30c_esp-now_user_guide_en.pdf

Linux Wireless Networking:

https://www.linux.com/blog/linux-wireless-networking-short-walk

https://stackoverflow.com/questions/48271119/how-to-send-both-802-11-management-frames-and-data-frames-using-raw-sockets-in-l

After swimming in a lot of information, I think I found the perfect starting point to send and receive raw 802.11 packet !

https://warmcat.com/git/packetspammer

Update 12/14/2018: Here is an other lib to send and receive raw wifi packet, It even has a python binding...To be tested soon..

https://github.com/kismetwireless/lorcon

Update 03/01/2019: The useful information was here

https://www.kernel.org/doc/Documentation/networking/mac80211-injection.txt

  • 1kHz closed loop control of up to 16 motors over WiFi !

    Thomas Flayols6 days ago 0 comments

    It's alive! Thanks to our ESP-NOW linux library, we are now able to do closed loop control of fast brushless motor at 1kHz wirelessly !!

    For that we designed a small PCB hosting an ESP-32 and exposing 8 SPI lines. With this setup we can control up to 8 custom made dual BL motor drivers. On the experiment below, position of 6 motors where sent via ESP-NOW to a RT-Preempt PC. The PC where 10m away from the motors, computing torque (current) to servo 5 motors on the 6th one position.

    Additionally, The board is including a Ethernet port. This provides a backup solution for non WiFi friendly environments.

    The board is open source and will be integrated to a VERY exciting robotic project. 

    More on that latter...

    I think this board could also serve as a base for many other remote controlled robotic project. With the Ethernet cable, it makes a flexible solution for fast and real time communication with hardware.

  • More testing (with RT-PREEMPT)

    Etienne04/24/2019 at 08:06 0 comments

    I installed (and patched) a real-time Linux on a computer, to do some 'more serious' testing. I chose Ubuntu 16.04 (with 4.14.109 kernel). And I patched it using RT-PREEMPT.

    DISCLAIMER: I am no expert! And I probably did a lot of mistakes/inappropriate things. The following paper does not explain a universal method to install a real-time Linux on your machine. It just traces what worked for me.

    Here is what I did :

    https://github.com/thomasfla/Linux-ESPNOW/blob/master/Misc/RT_Preempt_patch_Linux.pdf

    The computer I used was a Dell Precision (with an Intel Xeon).

    I tried with 2 different PCIe Wireless Network Interface Cards :

    • TP-Link TL-WN881ND (Seems to work really bad in this current setup).
    • ASUS PCE-AC51 (The one that I used in the end because it seems to work the best in his current setup).

    Finally, after tuning these parameters, and modifying a little my ESPNOW code on the computer (to take better advantage of the real-time OS), here is what I got from the testing :

    Histos
    Histograms of the round trip time. Tested on 'almost-empty' WiFi channel, with 24MBps data rate.

    These results show more consistency in the round trip time. In fact, the histogram shows only 1 sharp peak. Moreover, I did this test several times, and each time the average round trip time was very close to the ones I measured earlier (ranging from 1100µs to 1400µs).

    Whereas, in my previous post, the average round trip time could vary from 1000µs  to 2100 (or more) within one test...

    HistosPcktLoss
    Histogram representing the size of the group of packet lost.

    I don't notice any big changes, concerning the packet loss, other than it is more consistent over the test (same as for the round trip time).

  • Some results !

    Etienne03/29/2019 at 10:13 0 comments

    All the results presented here have been obtained following this procedure (using the code mentioned in the previous Log) :

    • Every 1ms, the ESP32 sends a packet to the computer with a payload of 127 bytes. This payload contains the time of emission.
    • A C++ code running on the computer (with the highest priority) receives this packet, copies the time of emission in a new packet (with a payload of 127 bytes), and sends it back to the ESP32
    • The ESP32 receives this response and measures the round trip time.

    Effect of the data rate on round trip time and packet loss :

    The figure 2 shows the round trip time and the number of packet loss depending on the data rate. Each point of the graph was obtained by averaging the values of 3 batches of 10,000 packets. These tests have been done both when the surrounding WiFi activity was 'very low' and 'normal'.

    It seems that the data rate does not impact much the packet loss. A much more significant factor for the packet loss is the surrounding WiFi activity.

    Moreover, as expected, the trend is that the higher is the data rate the shorter is the round trip. However, this relationship is not linear.

    Further analysis on packet loss :

    Since we choose to disable ACK to speed up the connection, we need to check that the packet loss is reasonable. However, the most important is that the loss is scattered. In fact, it would be very harmful in our robotic application to have big batches of packets being lost. It would mean that there is no feedback for 'long' periods of time.

    The figure 4 shows that the size of the 'groups' of lost packets seems to follow a normal law centred on 0. The variance seems to be very low, so it is very rare (less than 0.1% in this example) to lose 3 or more packets in a row. Therefore the loss of packets is more acceptable.


    Future improvements :

    The next steps are :

    • Try different Wireless Network Interface Cards on the computer to see how it impacts the performances
    • Patch Linux on the computer to be real-time.

  • Implementation

    Etienne03/29/2019 at 07:57 0 comments

    On the computer

    Library :

    A C++ library has been created to handle ESPNOW packets with Linux. You can find it here : ESPNOW_lib.

    The code also contains a main() that send back every ESPNOW packet received (it acts like an echo).
    If you want to use this code don't forget to turn on the monitor mode on your wireless interface and also tune it to the right channel (because the library can be set-up to use any channel). As a reminder, this is how you do it :

    sudo ifconfig wlan0 down
    sudo iwconfig wlan0 mode monitor
    sudo ifconfig wlan0 up
    sudo iwconfig wlan0 channel 1 
    

    Berkeley Packet Filter (BP Filter) :

    The BP Filter that has been attached to the socket was generated using the tcpdump command :

    sudo tcpdump -i wlp5s0 'type 0 subtype 0xd0
    and wlan[24:4]=0x7f18fe34
    and wlan[32]=221
    and wlan[33:4]&0xffffff = 0x18fe34
    and wlan[37]=0x4
    and wlan dst 11:22:33:44:55:66
    and wlan src 77:88:99:aa:bb:cc' -dd

    It filters packets according to :

    • Type and subtype : Action frames (0xd)
    • Category code of the action frame : Vendor specific (0x7f)
    • OUI : Espressif (0x18fe34)
    • Type of the vendor specific action frame : ESPNOW (0x4)
    • Source and Destination addresses


    The Assembly code generated has been modified a little bit afterward to fit any source & destination addresses. You can see it at line struct sock_filter temp_code[this->bpf.len]  in ESPNOW_manager.cpp.

    On the ESP-Module

    Acknowledgment (ACK) :

    The aim of this project is to have a fast wireless connection between 2 devices, with the least delay.

    However, the ESP-NOW protocol is built to wait for an ACK frame after every packet sent. It ensures the reception. And, while the ACK is not received, the ESP-NOW library tries to send the same packet over and over again.

    This slows down the whole process !!

    Moreover, in our robotic case, if a packet is lost there is no point on trying to send it again. In fact, this packet is now outdated and it is possible to send a newer one (which is more interesting for our system).

    The easy way found to avoid ACKs, is to send the packet over the broadcast address (and not directly to the other device address). So the ESP will neither send or wait for ACKs.

    In the code, you simply need to declare the peer with the FF:FF:FF:FF:FF:FF mac address :

    esp_now_peer_info_t peer;
    
    for (int i = 0; i < 6; ++i ) {
        peer.peer_addr[i] = 255;
      } 

    Remark: Setting the ESP to send over broadcast to avoid ACKs is only available on ESP32. That is what motivated the choice to change from ESP8266 to ESP32.

    Data rate :

    The modification of the data rate was decisive to decrease of the round trip time. The default rate for ESP-NOW on the ESP32 is 1 MBps. At this rate, physically send a packet (header + 127 Byte of payload) takes approximately 1.5ms. So the round trip cannot be less than 3 ms!

    By disabling some default 'safeties' in the WiFi library, it is possible to change the data rate directly at the WiFi level (and bypass the default 1 MBps in the ESPNOW library). Now the data rate can be set up to 54 MBps which correspond to a physical sending time of 0.05 ms. To change it, here is what you need to do :

    #include <WiFi.h>
    #include <esp_wifi_internal.h>
    #include...
    Read more »

  • Berkeley Packet Filter (BPF)

    Thomas Flayols03/05/2019 at 13:01 0 comments

    Project status: The code is now able te send arbitrary ESPNOW data to a chosen MAC address.

    What we need now is an efficient way to filter incoming packet. This is possible by software in the user space via a simple function that check the destination MAC, the type frame and some ESP now specific filed. But there is a better solution called Berkeley Packet Filter (BPF).

    BPF can be attached to a socket. A filter is a program that will analyse any incoming packets and return True or False whether the packet should be accepted or rejected. This program is executed in kernel space in a virtual machine (after some analysis of the code). How cool is that?

    To program a BPF, we have to write the filter with assembly, given a specific instruction set.

    An exemple can be found here https://gist.github.com/oro350/8269805

    In this example, we can see how a filter looks like:

    static struct sock_filter bpfcode[6] = {
    	{ OP_LDH, 0, 0, 12          },	// ldh [12]
    	{ OP_JEQ, 0, 2, ETH_P_IP    },	// jeq #0x800, L2, L5
    	{ OP_LDB, 0, 0, 23          },	// ldb [23]
    	{ OP_JEQ, 0, 1, IPPROTO_TCP },	// jeq #0x6, L4, L5
    	{ OP_RET, 0, 0, 0           },	// ret #0x0
    	{ OP_RET, 0, 0, -1,         },	// ret #0xffffffff
    };

    Writing this program from scratch can be a bit tricky, if we need to adapt it to non fixed length header, compare a full MAC, etc.. 

    But tcpdump can generate this for us with the -dd option.

    Going further...

    So now we have a hint for filtering packet in the kernel. But we can do better! The BPF could be done directly at the reception of incoming packet in the MAC80211, before the radiotap header is calculated etc..

    This is not officially implemented in the current MAC80211 driver yet, but here is a patch and discussion about it

    https://patchwork.kernel.org/patch/9679455/ 

    I must admit that I never thought I would have to go so deep into linux for this project, but it's quite fun.

  • A working example !

    Thomas Flayols02/28/2019 at 10:12 0 comments

    After a lot of testing with different lib and radio driver, together with @Florenc Caminade, we managed to capture and send ESP now packet! And it turns out to be quite simple, we just use linux sockets to send and receive packet. The tricky part was to generate a legit packet the card would accept and transmit as a vendor specific action frame. We had to include a radiotap header...

    At the moment, packets are hard coded raw data, so it will only work if your ESP has the same MAC address, and it will send always the same data.. So not very useful but a good proof of concept

    The code is on GitHub: https://github.com/thomasfla/Linux-ESPNOW/ and need a bit of cleaning.

    To test it, you have to use this MAC address:

    ESP: 84:F3:EB:73:55:0D
    PC:  F8:1A:67:B7:EB:0B

    you also have to set your interface in monitor mode, and it has to support packet injection.

    compile and run using:

    cd wifiRawSender/
    make
    sudo ./bin/sender wlan0

     where wlan0 is your interface up in monitoring mode (same applies for receiver example).

    What's next?

    • Generate or parse the packets (Host MAC, Destination MAC, Payload and payload size, ...)
    • Simplify the radiotap header
    • Find an efficient way to implement a MAC filter in monitoring mode
    • Write a clean library
    • Measure the round-trip time

    Notes:

    About the ACK behavior, I switch to an other wifi interface that does send ACK in monitoring mode !
    I did not find if and where this is documented. For my application I don't mind to use a specific card.

  • How ESP-NOW really works ?

    Thomas Flayols10/18/2018 at 13:29 0 comments

    To implement the ESP-NOW Protocol, we first need to understand how it works in details.

    The datasheet doesn't say much on the really low level, especially on the acknowledgment behavior.

    To understand what ESP-NOW does, I use Wireshark with my PC Wifi card in monitoring mode:

    > sudo ifconfig wlan0 down
    > sudo iwconfig wlan0 mode monitor
    > sudo ifconfig wlan0 up
    > sudo wireshark

    With one ESP8266 sending ESP-NOW to a non existing MAC adress, I got the following packets:

    The same packet is sent multiple times, with a decreasing datarate :

    • 2 tries at 6Mb/s
    • 2 tries at 2Mb/s
    • 7 tries at 1Mb/s (maybe this last number of tries depends on a timeout...)

    First bad news: ESP-NOW continues to send the packet again and again for 20ms !

    I hope their is a way to disable this resend behavior... Indeed, for a robotic control, we prefer to wait for the next up-to-date packet than getting an outdated one, and jamming the channel...

    Let's look at the packet itself:

    0000   00 00 12 00 2e 48 00 00 10 0c 6c 09 c0 00 d5 03   .....H....l.À.Õ.
    0010   00 00 d0 00 3c 00 84 f3 eb 73 55 0d 86 f3 eb 73   ..Ð.<..óësU..óës
    0020   ca 61 84 f3 eb 73 55 0d a0 09 7f 18 fe 34 17 71   Êa.óësU. ...þ4.q
    0030   47 8c dd ff 18 fe 34 04 01 62 12 12 12 12 12 12   G.Ýÿ.þ4..b......
    0040   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
    0050   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
    0060   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
    0070   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
    0080   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
    0090   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
    00a0   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
    00b0   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
    00c0   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
    00d0   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
    00e0   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
    00f0   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
    0100   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
    0110   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
    0120   12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12   ................
    0130   12 12 12 1d 2c ba 98                              ....,º.
    

     The payload is "0x04 0x12 0x12 ... 0x12" (on my test I choose to begin with a packet counter to distinguish different packets.)

    The frame format seems to follow the documentation from expressif, and the header should be easy to generate:

    https://docs.espressif.com/projects/esp-idf/en/latest/api-reference/wifi/esp_now.html

    Now, if I turn on the destination ESP8266, the packets are sent only once, and a quick acknowledgement frame is sent back in (about 60us):

    I guess this acknowledgment frame will be generated by the low level MAC layer, so we shouldn't take care of managing it... 

    The next test is then to send ESP-NOW packets to my laptop MAC address, and check that ACK are sent automatically even in monitoring mode... 

    Hum... After a quick test, when I Sending a packet to my laptop's MAC address, it does't send an ACK back. Maybe the monitor mode disable this in the MAC layer ?

View all 7 project logs

Enjoy this project?

Share

Discussions

electrobob wrote 5 days ago point

This is very good work, please keep it up! 

The ESP-Now can be a good alternative for low cost home nodes. I am not ready to drop my RFM69 radios yet, but soon it might be possible. 

I keep seeing this 20 devices limitation (or 10 with encryption). Do you know where it comes from? could a more powerful GW handle more ?

  Are you sure? yes | no

Thomas Flayols wrote 5 days ago point

Thanks!

If you use ESP-NOW directly, I think you can communicate with as many devices as you want. You need to manage the list of your peer MAC addresses yourself, and filter the incoming packets yourself too.

For this project, we use such direct access, you will find all you need in the github project.

  Are you sure? yes | no

electrobob wrote 16 hours ago point

Thanks for the info!

  Are you sure? yes | no

Germán Martín wrote 08/21/2019 at 06:37 point

Great project!. I'm doing some work with ESP-NOW to connect sensors with a gateway based on ESP8266. Would be great to use a RaspberryPi as a gateway instead.

You can have a look to it: https://github.com/gmag11/EnigmaIOT

Latest ESP8266 SDK allows broadcast comms too. You simply need to use ff:ff:ff:ff:ff:ff as destination address.

  Are you sure? yes | no

[deleted]

[this comment has been deleted]

morgan wrote 04/22/2019 at 20:01 point

Don't like Windows? That's fine, don't use it.

But not only does calling community members losers for >their< choice to use it contribute nothing, it's likely against the community guidelines. Don't do this.

  Are you sure? yes | no

reginaldparker0 wrote 02/16/2019 at 14:58 point

I just started with this and want to use a Raspberry to collect all the data. So far, all I found out is how to put a wireless card into monitoring mode in Windows...

  Are you sure? yes | no

Marko wrote 04/08/2019 at 15:40 point

Im having trouble putting my wireless card into monitor mode in Windows, can u help me? :(

  Are you sure? yes | no

reginaldparker0 wrote 04/19/2019 at 18:20 point

You need to install Microsoft Network Monitor 3.4

  Are you sure? yes | no

Thomas Flayols wrote 02/09/2019 at 14:53 point

I haven't had time to do any development on this yet, but the project is definitely not dead.

I intend to work on this next month. 

Feel free to join the project if you think you can help in any way. 

  Are you sure? yes | no

reginaldparker0 wrote 02/09/2019 at 11:20 point

any new developments?

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates