close-circle
Close
0%
0%

USB Packet Snooping

A low cost hardware dongle for capturing and analyzing Full Speed (12Mbps) traffic using ARM microcontroller

Similar projects worth following
close
While working on my "Low Cost KVM Switch", I need a simple way of snooping keyboard for detecting hotkey for switching to a different computer.

While working on a bit-bang USB for the STM8, I also see a need for a hardware device to capture raw and may be malformed USB packet that software only solution cannot capture.

I am working on a software only USB device for the STM8 as part of my work on HID Multimedia Dial. Windows fails to recognize the device as there are some low level bugs.

I tried a software based USB logger, but the software cannot capture the packet unless it is recognize by the operating system. These software based solutions relies on install additional low level drivers onto the target for capturing packets. What if the target system is a non-PC? What if the packet is malformed and rejected by the PC USB controller? A logic anaylzer is the closest solution right now, but it has limited support for protocols or exports. Traffic analyzer tools exists but are priced outside of hobbyist's budget.

One of the hurdles of open source software based solution under Windows - code signing for low level driver. This is the fate of USBPcap that ran into a BSOD bug and requires code signing certificate for a bug fix.

A transparent hardware device does not relies on additional software installed on the test system nor does it interfere. I am thinking of dumping raw USB packet onto a PC and exporting traffic log to wireshark files for additional analysis.

3D modelling of dongle:

Logging software on Windows: (work in progress)

The problem with USB is that it is a half duplex protocol and there are no receive only USB SIE (serial interface engine) for snooping the traffic. One could probably use a high end microcontroller with both Host and Device ports for transparently passing through packet while snooping. It is also possible to implement a custom SIE using hardware or software. e.g. V-USB is a pure software implementation, but is limited to Low Speed (1.5Mbps).

This project will focus on Low or Full speed USB and use a microcontroller with minimum of additional components. This project relaxes some of the hard real time requirements as it does not interact with the traffic. My initial thought was to take advantage of the SPI for sampling the serial stream and introduce a phase shift to sample the closer to the center of the data window. As I let my mind ponder the question while running some errant, I realized that I could simply sample the signal at 2X the data rate and worry about recovering the data afterwards.

While the datasheet specifies an upper limit of 18Mbps data rate, I have successfully overclock the SPI MOSI of the STM32F030 to 25Mbps in "Low Cost VGA Terminal Module". I hope the same would be true of the MISO input. DMA is require to capture data at 24Mbps (3M bytes/s).

USB waveforms looks something like this:
Source: AN57294 - USB 101: An Introduction to Universal Serial Bus 2.0

The signals for 1.5Mbps/12Mbps are full swing 3.3V logic level, so they can be interfaced to GPIO. A brief Single Ended Zero (SE0) state is used to indicate End Of Packet (EOP).

To detect this using an external interrupt, we would need to add an external logic gate. (I have also considered making a missing pulse detector out of a timer.) The interrupt would then terminate the DMA transfer. We have no way of knowing what packet type and length will be received until the PID field is decoded.

I have decided to use MOSI as the other data pin. It can be used for figuring out the connection speed.

For capturing, we have up to 32 clock cycles during the SYNC. According to ARM, the interrupt latency is 16 cycles which already includes pushing a few registers onto the stack.

This is an optimistic number as:

  • The ARM core will be running from FLASH with 1 wait state.
  • This latency applies to the core. There are additional cycles for external signals - synchronization crossing clock boundary, delays inside peripherals

Reference:

AN57294 - USB 101: An Introduction to Universal Serial Bus 2.0
USB in a NutShell
USB Made Simple
USB

From: Lakeview Research - USB Development Tools

Windows serial port reference:
https://www.codeproject.com/Articles/43073/Win-SDK-Serial-Comm-Made-Easy
https://msdn.microsoft.com/en-us/library/ms810467.aspx...

Read more »

  • PCB Assembly

    K.C. Lee05/01/2017 at 20:35 0 comments

    I made one mistake so far for the USB Serial PCB I sent off to OSH Park. The yellow segment for the USB D- signal was missing. This was caught when I submit the files to github over the weekend. The file on OSH Park has now been updated.

    Front:

    The crystal is probably sitting in the warehouse next to Canada Custom right now. Probably going to be there for another month or so. For the mean while, I soldered in a much larger one I have on hand.

    Back side:

    I had to rework (i.e. desolder/reflow) the chip 3 times because I had to solder in a wire under it. SMT is pretty easy to rework if you have the right tools.


    Found some discolouration in the ENIG finish on one of the PCB. Circled in red. Mounting hole at ear upper right corner, tow row SMT pad 2nd from right, right solder pad 2nd row from bottom right corner. Not a big deal for me as the PCB is still fresh.

    This is the top side of the Snoop PCB. USB memory stick for scale.

    Still waiting for the crystal. For now, I'll use the internal RC and PLL to generate 48MHz. It would be interesting to see the drifts between the sampling frequency and the USB signals anyways.

    The back side:

    There is a 4 pin header on the left side. I made a press fit debug connector that fits there. A 90 degrees header would also fit, but needs to be solder in before the USB socket.

    This is how things fit with the Serial daughter card on top. It is a very tight fit. :)

    Update: The rest of the tiny parts arrived finally. - crystal, screw, nut. I replaced the 2mm header with a low profile one.

    I hand rolled a standoff from a transformer core. I hand filed the 12mm screw. Bottom left corner is my press-fit SWD connector.

    The SWD interface is working. Now have to sit down and code some more.

    The mechanical clearance is tight but okay, but one should consider using an extension cable.

    I plugged in a memory stick. I won't be able to debug it, but the extra circuits doesn't interfere with the 480Mbps (USB 2.0 High Speed) signals.

    I programmed in the USB serial number using the vendor's EE writer tool so that Windows won't assign a random one when I move it to a different USB ports.


    What it means is that the hardware init made it all the way without crashing after the code I cut and pasted from old projects.

    That's the LED running at 211uA - driven from a 3.3V GPIO with a 4.7K series resistor. It is 1/10 current of what I used to drive them years ago It is closed to 1/100 of the traditional 20mA.

  • PCB tracking

    K.C. Lee04/27/2017 at 18:12 0 comments

    This time around, UPS is pretty fast like passing a hot potato to the next carrier in the chain. In the past, the PCB tends to show up 2-3 working days as they are handled as XpressPost.

    8 days is a very long time crossing the border. May be Canada Post now treats them as regular packages now? I hope not.

    The only way to find out is to use Canada Post tracking number which is kinda pointless as it is not available in the UPS tracking. I can only find out from the additional barcode sticker on the package after it is delivered.


    I wasn't expecting this soon after passing customs as they don't work on weekends. It probably made a direct flight to Ottawa.

  • Enumerating COM ports in Windows

    K.C. Lee04/27/2017 at 17:27 0 comments

    I have done zilch in serial port programming since DOS days. The plug and play nature of USB devices added a few winkles to the operating system to keep track of the device and naming them. The serial ports are no longer restricted to COM1 to COM4 and sometimes it leaves a big gap in between ports. Instead of sequentially trying COM1 to COM99(or 999?), there is a much better way of figuring where the current active COM ports are.

    I modified the source code from David MacDermot's serial port tutorial:

    void Init_ComNames(HWND hwndDlg, int DlgItem)
    {
      DWORD lpCount, myType = REG_SZ;
      HKEY hKey;
      TCHAR regValue[MAX_PATH],portName[MAX_PATH];
      int port, lenValue, lenName;
    
      SendDlgItemMessage(hwndDlg,DlgItem,CB_RESETCONTENT,0,0);
    
      //open the regkey where the serial port data is stored
      if (RegOpenKeyEx(HKEY_LOCAL_MACHINE,_T("HARDWARE\\DEVICEMAP\\SERIALCOMM"),
                       0,KEY_READ,&hKey) == ERROR_SUCCESS)
      {
        if (RegQueryInfoKey(hKey,NULL,NULL,NULL,NULL,NULL,NULL&lpCount,
                            NULL,NULL,NULL,NULL) == ERROR_SUCCESS)
        {
          for (port = 0; port < (int)lpCount; port++)
          {
            lenValue = MAX_PATH;
            lenName = MAX_PATH;
    
            //cycle through reg values to get port names 
            if (RegEnumValue(hKey,port,regValue,(LPDWORD)&lenValue,NULL,
                             &myType,(LPBYTE)portName,(LPDWORD)&lenName)== ERROR_SUCCESS)
            {
              // add COM port name into combobox
              SendDlgItemMessage(hwndDlg,DlgItem,CB_ADDSTRING,0,(LPARAM)portName);
            }
          }
        }
        RegCloseKey(hKey);
      }
    }

    Windows uses the registry to keep tracks of the hardware device. It is not as scary as it seems.

    • SendDlgItemMessage(..CB_RESETCONTENT..) clears the ComboBox.
    • RegOpenKeyEx() opens "HARDWARE\\DEVICEMAP\\SERIALCOMM" under the registry KEY_LOCAL_MACHINE (See picture below)
    • RegQueryInfoKey() is used to find out the number of keys (2 in this case)
    • RegEnumValue() is used to query the individual keys (0,1) and their values. i.e. COM1 and COM5
    • SendDlgItemMessage(..CB_ADDSTRING..) adds them to a ComboBox.

    A bit of C code later, I get to this point:

    I have also added a serial number to the PL2303. Windows can use it to keep track a device and assign the same COM port even if it was plugged into a different USB port.

  • Data packets, compression

    K.C. Lee04/19/2017 at 02:58 0 comments

    I have been doing a bit of thinking on how to move the collected raw data to the PC via serial port for analysis.

    • USB data can take on any 8-bit values and 4 additional specialized USB symbols. i.e. 260 symbols that cannot be fitted into 8-bit.
    • need to tag specialized packets for serial protocol e.g. time stamp, non-data
    • a lot of small packets from 8 bytes to 64 bytes
    • a way of synchronization or re-synchronization in case of missing data due to over-run.
    • Streaming protocol with limited buffers and not enough bandwidth and memory for ACK/NACK.
    • optional data compression?

    One way of doing this is to make a packet with a header, a size field and data with ACK/NACK. The serial link has less bandwidth than USB, so it is done on a best effort. Error recovery would consume more bandwidth and may make the problem worse.

    I am thinking of trying something else. This is a simple way of tagging at the byte level with 1 or 2-bit Huffman encoding header.

    • If the MSB starts with a '0', then the rest of the byte carries 7 bits of raw USB data.
    • If the MSB starts with '10', then the remaining 6 bits is RLE (Run Length Encoding) counter
    • If the MSB starts with '11', then it is a special tag (with a byte count) which could be one of the following
      • Special USB symbol e.g. Single Ended Zero (SE0), Start of Frame (SOF), End of Packet (EOP), Reset
      • Initial value of RLE
      • Header for time stamp, text (e.g. configuration, command return code), corrupted packets, data over run and other errors etc

    Data compression

    If the data cannot be compressed easily, then the efficiency is 87.5% as 7 bits is sent every byte. If there are very long strings of '0' or '1', then the RLE could reduce the amount of data. (RLE data compression is a bonus as there are left over bits after special tag.)

    Serial data Re-synchronization

    Bad USB raw packets i.e. ones with bad CRC16 can be tagged with special tag. So if there is a mismatched CRC16, it means that the serial protocol layer have missing/corrupted data. The PC can use the next time stamp tag for synchronization.

  • Mechanical design

    K.C. Lee04/10/2017 at 22:41 6 comments

    It might be a simple thing that you don't think about, but designing the mechanical takes a lot of time.

    I am trying to make the dongle as small as reasonable as I don't want to block off neighboring ports or add too much length. I have decided on a daughter card for the USB to serial converter which can also be swapped out for different chip.

    I have offset the plug side to make room for a screw (in the lower right corner). The Micro USB connector is off the side and can exert a lot of leverage against the stacking connector. I have decided against soldering, hot gluing, duct tape and settled on using a screw.

    A lot of small adjustments to the layout have to be made. It took a long time for me to settle on a M1.6 x 12mm screw as it is big enough to be available in smaller quantities. 49 leftover screws is a lot better than 999 that I would never use. The crazy thing is that these small orders cost more than the rest of the parts.

    The screw encroachs the pad of the USB connector. I might look into changing the pad into a slotted pad or just live with what I have. I'll have to make the standoff as I can't find something this small. Going to do more clean up etc.

    Looks like I am in luck. This transformer core is the right height I'll need for the standoff.

    A bit of trial and error cutting it to the right length, I rolled the piece of core into a tube. At this scale the camera is very unforgiving. Just need to polish it up. The O.D. is 3mm which fit within the PCB.

    Total length of screw = 1.3mm + 1.57mm + 6.477mm + 1.57mm = 10.917mm

    11mm would have been the perfect size, but they only sell the even number. i.e. 10mm or 12mm. A bit of filing should be fine.


    I have decided to make the boards available separately. I have placed an order today and hopefully will show up in 3-4 weeks time frame, but things can happen.

    These designs are not tested yet.

    USB Serial.brd


    USB Snoop.brd

  • Dongle design

    K.C. Lee04/10/2017 at 01:59 0 comments

    This is what I have so far:

    Pretty simple design. USB D+ goes into MISO which is sampled by the SPI using DMA at 2X sampling rate. D- can be polled from the GPIO pin. A NOR gate is used to detect SE0 (Single Ended 0) which cause an interrupt to stop the DMA. The data collected is transfer to 3.3V TTL serial port using DMA and hardware RTS/CTS handshake for data rate of up to 6Mbps.

    Here is a quick survey of various popular USB serial chips:

    The USB serial design is based on this project log: STM8 breakout board with USB Serial

    I have also made provision for using the UART as a second SPI for logging data by connecting the clock pin to one of the handshake lines.

    This is what the preliminary assembly looks like with the USB serial board on top. (I have to find models for the 2mm connectors.)

    I ordered the crystals from China today, but probably won't be seeing them for about 2 months. :( I do have larger crystals.

  • Initial look at raw USB signals

    K.C. Lee03/23/2017 at 04:57 0 comments

    I soldered on a 3-pin header onto an old pass through dongle that I have made for powering an amplifier. This comes in handy for snooping the signals and for powering the dongle.

    To get an idea of what the USB signals looks like, I set the sampling rate of my logic analyzer to 24Mbps. This is also the highest speed that the SPI in master mode can operate. The following is the capture for a Full Speed device (ST Link).

    This looks promising to the human eyes. The firmware will be looking at a sequence of packed binary coming from MISO pin of the SPI and have to process the data in bit level one by one.

    While the signal is differential, one of the pairs can be used for recording the data stream. Both of the signals are needed to determinate the side band signalling. e.g. a single ended zero (SE0) when both D+,D at '0' can be used to signify a device reset if held for more than 10mS.

    The initial pulses is part of the synchronization. It is supposed to a series of 8 bits of alternating '1' and '0'. Not sure why I am seeing 7 transitions.

    Interrupt latency might insert delays jitters when the sampling starts. There are 4 bits in the PID field which follows the SYNC. It is used to identify packet type. The bits are packed with their complements, so it is useful for figuring where the packet starts.

    PID0 PID1 PID2 PID3/PID0/PID1/PID2/PID3

    http://www.beyondlogic.org/usbnutshell/usb3.shtml#USBProtocols explains the basics of the packet format at this level.

View all 7 project logs

Enjoy this project?

Share

Discussions

Chris Cox wrote 10/15/2017 at 04:19 point

How is the project progressing? We haven't seen updates in a while.  Is anyone shipping these devices yet?  And where is the source for the host software? (which probably still needs to be ported to Linux and MacOS)

  Are you sure? yes | no

Nadhem86 wrote 09/27/2017 at 22:28 point

Hi, good projec, any progressing so far.

  Are you sure? yes | no

Paul Stoffregen wrote 04/18/2017 at 16:03 point

Wow, impressive work so far.  If you can pull this off and publish as open source, I'm sure you'll make a lot of people very happy.  Well, except maybe a few.  When you write "logic anaylzer is the closest solution right now", perhaps you've not seen the Beagle Protocol Analyzer from Total Phase?  I'd imagine a very low cost open source hardware design on the market won't make them really happy.  But I'm sure everyone else will love it!

  Are you sure? yes | no

K.C. Lee wrote 04/18/2017 at 16:51 point

Thanks.  I am aware of protocol analyzer, but not doing enough USB development to justify one.  

  Are you sure? yes | no

Similar Projects

Does this project spark your interest?

Become a member to follow this project and never miss any updates