Close

DC26 Retrospective

A project log for AND!XOR DC26 Badge

The Wild West of IoT

ZappZapp 09/01/2018 at 04:190 Comments

Wow! Another fantastic year. We can’t believe how far we’ve come since our very first badge at DEF CON 24.  Watching the other badge makers come up with new and interesting designs has been equally fun.

This year we set out to improve our overall manufacturing process, build a badge with a kickass design (including add-ons), create something folks can learn with, and make people say “WTF?” when they see it.

Looking back, we achieved those goals.


Design

Our design for DEF CON 25 was very popular, using a Fear and Loathing in Las Vegas theme is an easy win. People loved it. We had to up our game. But we struggled to top it. Bender and Hyr0n spent weeks bouncing competing western themes off each other but nothing really clicked. Finally we called in professional help, Doc from the Packet Hacking Village immediately came through and knocked it out of the park. We can’t say enough how grateful we are for his help and his amazing skills.

Zapp worked with him over several weeks to refine the design and shading to optimize the PCB manufacturing process.

Hardware

From a hardware perspective our DEF CON 25 design was relatively simple. We used a BLE module, SD card, screen, and LEDs. At the time it was a complex design for us, just barely testing our skills. For DC26 we felt the need to step it up a notch or three in complexity.

Screen

We upgraded from a 128x128 color LCD to a 220x176 color LCD. The additional pixels required a lot more memory and faster SPI bus to push more pixels to the display. The ESP32 afforded us a 40mhz SPI bus, 4-bit SD card with 40mhz clock. In testing we were able to push the display at 30 FPS or 22Mbps - that’s a lot of Rick Astley. However, due to the notorious SD card, we ended up reducing that to 12 FPS or 9Mbps.

We’re still not sure exactly the part number the screen is. Zapp found one seller on Taobao and the only way he can confirm the part is by looking at the ribbon cable. Still we were able to source enough screens, plus spares, to get the job done.

ESP32

Zapp spent a lot of time this year looking for a new module. While we loved the BMD-300 from Rigado, we needed more RAM and a faster SPI bus. ESP32s seemed popular and the SDK had matured so we bought some dev boards and got to work. There is an impressive ecosystem around the ESP32, lots of great examples (other than the Arduino ones), and Espressif is very actively building the SDK. We ended up with the WROVER32 modules due to the 4MB of RAM critical for LULZCODE and a double buffered display.

From a spec perspective, the ESP32 modules are impressive. Similar ARM modules simply don’t match them on paper. But there’s no such thing as a free lunch and it turns out it’s difficult to make efficient use of all that power.

4MB sounds like a lot but it turns out, the most critical RAM in the ESP32 is the DMA memory. Not only that but the largest users of the RAM are in the SDK. We continually found ourselves competing for DMA memory with WiFi, BLE, FreeRTOS, and ourselves. When we ran out of DMA memory, suddenly our SPI bus stopped working. Wifi seemingly ignored settings to use the 4MB of RAM for its buffers and quickly filled DMA memory causing bling to crash.

On top of that we found the architectural documentation of ESP32 to be lacking. The SDK API is well documented but typical app notes and reference designs/pointers were not available. This caused us to learn lessons like DMA memory the hard way.

Did you know you can’t access NVS from an ISR? We do now.

Did you try extracting our firmware from the chip or anyone else’s using an ESP32? You most likely can. While read back protection is a common practice in the security community, Espressif chooses a scheme called encrypted flash. This prevents read back and you can only do it...3 times before a write fuse is permanently blown. W...T…F…. So imagine if our badge was a real consumer IoT device being sold on shelves? You have to choose between protecting your firmware with readout protection and only 3 security patches in its lifecycle...or letting anyone just grab a copy of your firmware. Not cool.

Also, while the ESP logger is generally good and color codes items, imagine our surprise why (up until a few weeks before the CON with SDK 3.0) that the WiFi logging was grey...and couldn’t be controlled or turned off?... Because they f-ing did not use the logging API. Their closed source WiFi module is tied into the serial bus and does a printf of EVERYTHING to UART. W...T…F…. There was a bash script we found to modify this where it would dump all messages to null, but then how does one debug? Or have to modify the SDK every time they update? So go out and find any consumer IoT devices using Espressif which are older than a few months, you can tap right into the UART and see everything going on with the WiFi stack that they decided to log.  

The power draw by the ESP32 to do basic BLE is ridiculously high. Idling our badges pull 100mA at 2.8V. We will give Espressif credit, they released BLE modem sleep in May and we were able to make it work reliably. Prior to modem sleep we were burning 170mA at 2.8V. How we long for an efficient ARM Cortex MCU again… Espressif system in packages are great for prototyping, but not really efficient enough to be using as a wearable on battery power if you expect it to last a few days.

Also a “small” ESP32 binary is 1MB. Yikes.

SD Card

Never again.

LEDs

After our manufacturing experience with WS2812B (AKA Neopixel) LEDs, we intentionally switched to common anode LEDs. These things cost us about 1 cent each but require a driver. Having been inspired by the Queercon and Mr Robot Badges from DC25 who used similar drivers we selected the IS31FL3736. This ended up driving 31 RGB LEDs and the 3 LEDs in the screen. It also formed the centerpiece of our design, the eye with bloodshot traces.

USB to Serial

To support the B.E.N.D.E.R. game, the serial console had to be accessible. To accomplish this we decided to include a micro USB port. Typically we see folks using FTDI. However, we found the CP2102N from Silabs met our requirements and at a much better price. Not only that, but its drivers come with Linux so users should just be able to plug and go. Windows and OSX require an install (we put the drivers on the SD card and linked to the SiLabs website), but it’s super quick and painless.

Power

A big complaint within the team was the weight of our DC25 badge. We intentionally reduced our badge weight by dropping from 3 AA batteries down to 2 AA batteries. This meant a new voltage regulator but overall weight was reduced by over 20%.

Additionally, our DC25 badges had issues with the VREG letting out the magic smoke, we suspected this was due to excessive current draw when shorted with other badges so a reliable VREG was required.

Following in the footsteps of Queercon we selected the Skyworks AAT1217 3.3v boost regulator These things worked perfectly. As of this time, no badges that we know of failed due to the VREG.

Shitty Add On’s

We designed a handful of shitty add ons, but more importantly did an homage Chris Gammell tutorial of KiCad and the SAO standard: AND!XOR Getting To Shitty.


B.E.N.D.E.R

The Badge Enabled Non Directive Enigma Routine…. B.E.N.D.E.R…. Initially we think the coolest part of this was crowdsourcing the name. Public surveys went out to submit candidate names, then a second round of a downselect was available for the peoples to choose what they liked the best. For those not familiar, B.E.N.D.E.R. is essentially a text based adventure game played over serial. However, the actions one takes in the game affects the operation of the badge hardware itself (e.g. WiFi packets, file manipulation, LED blinks, Screen animations, etc…). The reason for this is that we didn’t want a simple text based adventure game for the lulz, but a way to add some story driven flare and context to our badge hacking challenges. Some may think badges are purely for blinky light eye candy bling and that is simply not the case for AND!XOR. In fact, a big component of what we try to put out there is teaching others some of the foundational hacking principles used in security. We want people to learn something from our puzzles that they can re-use IRL and hopefully be entertained in the process.

Without giving away any big spoilers (one can look into the source code once it is posted or buy us a beer and we’ll talk about it in person…), here are the basics of the game. Once a player logs in they are presented with the situation that they wake up in the middle of the desert, no idea how they got there, at the intersection of two trails. This gives them the option of heading to the North, South, East, or West in order to explore and attempt to beat the challenges of each area. To the North, a hardware hacking puzzle. To the South, a Static Binary Analysis Reverse Engineering puzzle. To the East, a Cryptography puzzle (ya know, Crypto, where you mine a bunch of dogecoin cuz crypto means cryptocurrency). To The West, a Wireless Hacking and Packet Capture puzzle. Once each challenge has been completed, the player would obtain an item useful in the game to assist them in defeating another challenge (or unlocking easter eggs), in addition to a special craft beers the team had been drinking throughout the year while working on the badge. Finally, once all challenges had been completed, the player must go back to the starting Home area and drink from all beers (given their weight and gender/non-gender association) to obtain a BAC of 0.1337.

The response was better than we had expected. Between tweets, DM’s, and talking to people in person at the conference we counted over four dozen people actively hacking away at the challenges. Moreso, they told us how it gave them a reason to try something outside their normal comfort zone and learn something new. Hopefully there were more folks playing than the people who talked to us about it :) However, there is one area where they all failed… because no one beat the game. We were clear before the badge came out, that the overall challenge required a variety of hacking skill sets, which usually one person would not have if they were new to this kind of thing and even when folks are experienced hackers they tend to focus in a single domain. Everyone was heavily encouraged to make friends, team up, or even visit the appropriate villages who are experts of their own domains. Alas, many of the participants did not or we saw requests for collaboration go out when we tweeted the “challenge ends in 48 hours” message. We also know this, because there were many folks so adamant about giving us feedback (and sometimes were also phishing for hints) ;) Collectively they had solved the badge challenge, but individually were on average ¾ of the way there in different areas. We couldn't help but laugh and think “We’ll if you just talked to this person over there, or just tweeted out looking for people to team up, you would win…”  In fact, a large group of folks were complimentary in that they either had the East Village or the South Village left to complete the game. A instruction described in the menu of the badge hinted that there was a social challenge of the game, that we expected people would need to make new friends, talk to people, or interact with and learn from the villages. There was even a Reddit thread we found were people suggested teaming up, but it seemed to go stale and die off right before DEF CON started. There was a very low likelihood that anyone would win without socializing and teaming up with other people. We love, absolutely love, how so many people were in to the game and hacking away at it on their own.

Drumroll…. THIS IS WHY YOU ALL FAILED <8

Now there is a prize box sitting back at AND!XOR headquarters (i.e. a van down by the river behind a burning dumpster fire), which no one gets and sits on the shelf until DC27. There are no participation awards in the world of B.E.N.D.E.R. either. We heart you all.

Also, we would like to put an additional thanks to our hacker family out there whom appeared in the game. With permission, we used their digital likeness for lulz and context. In one way or another, they all have been friends, mentors, teachers, accomplices, or just great people who share their knowledge with the community that we always look forward to seeing when we meet up at various cons. So a heartwarming thanks to Viss, Joe Grand, April Wright, Darkmatter, and ArdJect. Our only regret is that we can’t include everyone, but thats why there is always another year with another badge, and another chance for B.E.N.D.E.R. to make a return with some lulzy cameos.

LULZCODE

This was our big “WTF” feature. Based on our lessons learned from TCL from DC25, we set out to develop a full toolchain to make LULZCODE development easy and a core part of the badge. In the end most of the user-facing interaction was handled in LULZCODE. It was amazing to see all the new bling people made in it.

For full documentation see our Geocities page: http://geocities.ws/andnxor/lulzcode/

We have tried to link this through twitter, but apparently it filters geocities as a malicious site… all the better a place to document this part of the project.

We also create an Atom plugin that support code highlighting and completion https://atom.io/packages/atom-lulzcode

A tip for the LULZCODE newbies, read it out loud. It helps.

Additionally we did some live tutorials on writing in lulzcode. It got broken up due to Youtube botz thinking we were violating copyright protection… (lesson learned don't use Youtube to play music in one tab while live streaming Youtube...SMH).

AND!XOR Flappy Bird in LULZCODE Part 1

AND!XOR Flappy Bird in LULZCODE Part 2

Failures

Failures coming out of Macrofab were greatly reduced from last year. Less than 1% are complete failures at this time. Zapp is still going through the badges that need fixing from DEF CON. But by and large the most common failure are SD cards which are easy to fix. We lost a couple screens due to the postal service, and a few LEDs. Ignoring the SD card issue we are satisfied with the badge performance this year.

Miscellany…

Sophie Kravitz at Hackaday put together a DEF CON badge life documentary interviewing some of the groups who create electronic badges (both official for the conference as well as independent groups like us).

If only Hyr0n didn’t set the world on fire by reminding folks that “gif” is pronounced JIFF (with a soft g)...

Also, it wouldn't’ be DEF CON if we didn’t ask you spend your nights doing some really stupid shit for a blinky trophy.  Assisting us this year was F4nci3, our lovable suited director of lulz-gistics. Off the top of our head we can kinda remember…

“What is a plastic cup, other than a giant straw?”

Charity

It's not just about raising funds to get us to conferences, making badges, teaching folks, and donating booze to parties, but giving back to the community as well. This year we were happy to support three non profits with donations:

Next Year

We will be back for our fourth badge next year. Things will change. Our three designs incremented on each other. Expect something different. We do this to learn so we need to find new challenges.

Discussions