Close

Building the Central Scrutinizer, one mistake at a time

A project log for Central Scrutinizer: a serial adapter for M1/M2/M3

Open source serial adapter / reboot controller for Apple Silicon Macs

mazmaz • 10/08/2023 at 11:43•0 Comments

What is this?

I've been building this serial adapter for M1/M2 Macs for some time now, and I thought that I'd document how it all came to life. There is nothing amazing in it. Only that it demonstrates the power of open source software and hardware. I didn't invent anything. I just put two and two together, and end-up with something that people found useful. It doesn't get better than that.

February 2021

M1 mini has landed. Screw macOS, I have other plans.

The Asahi project already has some basics in place (the m1n1 firmware), and thankfully describes a rough method to get a serial port out of the machine: https://github.com/AsahiLinux/vdmtool . That's where everything starts.

I'm rather adverse to soldering, so aiming for a Lego-like setup by using a bunch of things sourced from Tindie/Amazon/Whatever as well as making liberal use of my junk box:

A couple of hours of  "work" assembling everything, tons of flying leads all over the place (and probably the reliability of a Morris Minor), and I end-up with this:


Of course, nothing works, and while the Arduino seems to correctly talk to the FUSB302, that sucker keeps connecting/disconnecting. After some debugging (and Marcan's help), I realise that both CC lines are connected all the way, which isn't quite what USB-C expects... RTFM!

Multiple solutions:

It turns out that the pass-through board has a number of vias to route all the signals across. Which is great news, as I have a very small screwdriver that I can apply to one of these vias (yes, this is terrible -- I don't care):

... and that's one CC line gone for good. Hopefully I won't regret this. And guess what, it all starts working. We have lift off! Both serial and reboot are working, and I can hack on the M1 remotely, without having to hear the stupid "Dong" each time the machine reboots (about 20 times an hour). I narrowly avoided taking the same screwdriver to the speaker...


Now that's what I call engineering! Note the unused micro-USB connector on the FUSB302 board. Eventually, I used that connection to upload payloads to the Mac. To this day, this machine still boots entirely from USB.

And frankly, this should have been it. End of the story.

November 2022

Almost two years later. The M1 is still going strong. I wrote a PCIe driver for it, KVM/arm64 is up and running (although I had to work around some of ugly deviations from the architecture as well as some silly HW bugs), and it is my main KVM development machine.

Only problem is that I now have a *second* M1. And I need another serial contraption. Yes, first world problem. Frantically trying to buy the ReclaimerLabs board again, but it is sold out, and the seller doesn't reply to my anxious messages. At no point it occurs to me that I could just upload the design to a PCB shop website and get it done. No.

Instead, I decide to reinvent the wheel. Just because I can. And my square wheel looks like this:

The big square in the middle is a bare FUSB302 soldered on a DIP adapter, the USB-C breakout board is a much cheaper version that exposes the SBU signals (just chop these pesky resistors off the CC lines), and dangling is the same FTDI adapter. And it works! Well, as long as you don't try to move the ball of wires around. Or breathe next to it.

OK, I can't really leave it like this. Surely I can come up with something better integrated. Which involves overcoming my hatred for soldering. Hey ho, let's see what we can do with a veroboard and some wires.

What an upgrade from the first build! My job here is done! Except that once you've done something twice, you start seeing the warts and getting annoyed with them. I definitely do. So what's wrong with this?

So what's the next move? Well, the two first points are the most annoying ones. Of course, I could integrate a USB hub. But I really hate the idea of adding hardware. If anything, I want less of it. And it occurs to me that if I could use the Arduino UART to capture the mac's serial traffic and multiplex it with the reboot control, I could save a USB connection. Except that the Arduino doesn't have any sort of native USB (hence the CH340).

OK, time to think of a replacement for the Arduino. What are the requirements:

After a couple of days of looking around only to find that STM32 boards are nowhere to be found and that the various clones don't necessary have what I need, the penny finally drops. What is being designed next door and available by the bucketload? The infamous RPi Pico. It has more features than I'll ever need for this project, and is cheaper than the damn Arduino clones I've been using. Only drawback is that it uses 3.3v logic, while I have at least one signal that requires 5v. Never mind, that's still tempting.

Another few days, and I'm the proud owner of two Picos. Why two? Dunno. I guess I fully expect to destroy one. Also, I've gained some extra Lego bricks:

Of course, all of that is totally oversized. I only need one MOSFET, and I only have two lines to shift around, not eight. But if I can get things to work, I'll look at being more frugal. If. Maybe.

But real work gets in the way, and then it's the holiday.

January 2023

Happy new year! I'm still being nagged by this silly feeling, so time to see if I can do something about it. When you spend the whole week hacking the Linux kernel and talking to hardware people, what do you do to relax during the weekend? Of course, you hack some hardware and write some software!

So let's draw some basic schematics on paper, get a feel for a basic board layout, and start soldering! I'm still using the DIP adapter for the FUSB302, and start cramming all these small boards on my 5x7cm piece of veroboard. And the result looks like this:

And what about the software? It didn't take too long to port the original vdmtools source code by hacking it to death, coupled to some very basic bridging between UART and USB-CDC emulation. The pico-sdk is easy enough to work with, and the stdio over USB feature makes it incredibly easy to bring something up. Also, plenty of helpful documentation and code snippets to look at. Within hours, I had the basics going on, which means that I was debugging deadlocks when dealing with simultaneous USB and UART traffic. Thankfully, that's the kind of stuff I can deal with.

And I really get what I wanted from it:

OK, project done. I now have everything I want. Except that...

You now see where this is going...

February 2023

For the past few weeks, I've used my trips to London to play with KiCad. The trains are usually delayed, and the network coverage is laughable. So reviewing kernel patches doesn't really work, and I need a distraction from listening to the depressing news.

So what have I done with it? I've converted my paper sketch into a "proper" schematic, traded the TXS0108 for a pair of 74AVCH1T45, a single BSS108 for VBUS switching, but otherwise it's all very similar. And then I've started putting together a PCB layout. Last time I designed a PCB, it was a 4MB memory extension for my Atari ST, some 35 years ago. Needless to say, things have changed!

One major design decision is that I didn't want to deal with the layout complexity of the RP2040. Yes, I'm a fsckin' chicken. But in keeping with my Lego approach, I aimed for a (tasty) board sandwich and used the Pico as is. Everything else being low speed, low complexity, I could get away with a two layer board and a number of routing horrors. And routing tracks while on the shitty Cambridge/Kings Cross train ended up being incredibly therapeutic!

Also, given that a board only requires one I2C bus, a UART and a couple of GPIOs, it is clear that a Pico could deal with two of them. The layout thus accommodates that by providing two distinct configurations, controlled by a set of 0 Ohm resistors.

Anyway, after much pondering and hesitation, I decided to put my money where my mouth is, and ordered 5 assembled boards from JLCPCB. That process was greatly helped by the JLCPCB Fabrication Toolkit plugin (https://github.com/bennymeg/JLC-Plugin-for-KiCad), allowing me to place the LCSC references directly into the schematic and (mostly) automate the whole thing.

But the best part of it is that the board now has a name. It started as "m1-ubmc", which was both inaccurate and utterly boring. That morning, I was listening to Joe's Garage, by Frank Zappa. And "The Central Scrutinizer" struck me as the obvious name for the board. Central Scrutinizer thou shall be.

v0

Fast forward by a week, DHL delivers a blue box with my five assembled boards, which I call v0. They look super cool. And they are DOA.

I spot the first problem by connecting the board to a 3.3v power supply and measuring the supply on the serial level shifters. It comes out as 2.8v instead of 1.2v. Errr. Good thing I didn't connect the mac to it! One of the resistors on the voltage divider that creates the 1.2v reference is measured at 100R instead of 470R. Check the schematics: all good. Check the LCSC reference: bingo. I somehow copy-pasted the wrong reference in the KiCad design, and the mistake propagated all the way to the assembled board. Lesson learned: check everything at design upload time...

Let's fix this the ugly way. Remove R6, and install a full 1/4W monster instead. With the correct value. Apply power, measure, 1.2v appears, and we're in business. Or so I think.

Let's be brave. I slap a pair of 20 pin female headers on the board, stick my pre-flashed Pico on it and connect it to the host. USB comes up, I2C communication with the FUSB302 is established. A small miracle! The moment of truth: I connect the USB-C cable to the mac, and I get a connection, the serial port gets negotiated, and I claim victory over the hardware!

Except that there is absolutely nothing on the serial lines. No amount of hitting the keyboard leads to any output. I frantically switch back to the non-PCB version, which still works, indicating that I haven't fried anything. Yet.  What could be the simplest explanation? I've already checked the power supply to the shifters, and I know the UART on the Pico works. After that it is nothing but wires between the USB-C connector and the shifters... Check the schematics: RX goes to RX, TX goes to TX, and.... And I instantly nominate myself for the title of moron of the month. I blindly followed the labels when drawing the schematics, ignoring the obvious that RX on the mac must be TX on the Pico, and vice-versa.

After a brief moment of despair at the wasted effort, I start evaluating my options. Because I knew I would make mistakes, I've made a point in baking some "get out of jail" options on the board. Specifically, any signal that I would need to extract from a tightly packed connector (such as USB) is exposed on a set of through-hole pads. So the SBU signals are readily available there. All I need to do is to disconnect the level shifters from these lines, and reconnect them swapped. But lifting a pin from a SOT-23-6 package is out of my league. Thankfully (and mostly due to my routing inexperience), I've used vias to connect the SBU lines to the 74AVCH1T45. And I can drill those out without impacting the rest of the design, giving me the option to reconnect the pins to the correct signals using magnet wires.

An hour later, I'm looking at this:

Bodged v0

If you think this looks awful, you'll be absolutely right. But it bloody works, and therefore I win.

I now have a fully operational Central Scrutinizer board, and I couldn't be more pleased. On the same day, I amend the design to fix the bugs and order another set of boards. Beer time!
v1 arrives shortly after, and it works out of the box. I push the code and the design out to k.org, and send a couple of boards to fellow hackers for testing.

April 2023

Yes, I'm still working on this crap.  No idea why. Actually, I exactly know why. The current design still has a bunch of the original Arduino design limitations:

I really shouldn't care, but it ends up bugging me. And when something bugs me, I sleep badly. You don't want to be around me when I haven't slept. Given the choice between pharmaceutical-grade chemistry and hacking, I chose the latter. YMMV.

Solving the first issue itself requires three fixes:

Let's start small. I fish out the first Pico prototype (Lego-style) from the junk box and wire CC2 and VCONN. Plug it, witness that nothing works. The FUSB302 can make up its mind what orientation is the correct one, keeps going into reset mode and generally makes no sense. Which is odd, as this is how things should have been wired the first place. This is however reminiscent of the issue I had when I initially started with the first Arduino setup.
One thing is pretty odd in the logs though: with only the CC1 line connected, the measurements that the FUS302 makes to detect the orientation are pretty clear cut: either CC1=2 and CC2=0, or CC1=CC2=0, where 2 means connected to a CC line, and 0 means disconnected. With CC1 and CC2 connected, I get things like CC1=CC2=1, which indicates that both are connected to some sort of e-marker device. This smells of some software bug... What would happen with a stupid USB2.0 cable, without e-marker? Lo and behold, I get a reliable CC1=2, CC2=0 or CC1=0, CC2=2. So something is amiss with the full-fledged cables I use.

I start spending some quality time with the FUSB302 spec, and realise that measuring the the current on the CC lines is driven with a set of pull-ups that can be set to CC1 or CC2. Or both. Then I looks at the measurement code. It is absolutely fine, and even documents that we clear the CC PUPs before doing anything.

Actually, it says it, but doesn't do it. Oh well:

        /* Clear pull-up register settings and measure bits */
-       reg &= ~(TCPC_REG_SWITCHES0_MEAS_CC1 | TCPC_REG_SWITCHES0_MEAS_CC2);
+       reg &= ~(TCPC_REG_SWITCHES0_MEAS_CC1 | TCPC_REG_SWITCHES0_MEAS_CC2 |
+                TCPC_REG_SWITCHES0_CC1_PU_EN | TCPC_REG_SWITCHES0_CC2_PU_EN);

VCONN also suffer from the same "sticky PUP" issue:

        if (state[port].vconn_enabled) {
                /* set VCONN switch to be non-CC line */
-               if (polarity)
+               if (polarity) {
                        reg |= TCPC_REG_SWITCHES0_VCONN_CC1;
-               else
+                       reg &= ~TCPC_REG_SWITCHES0_CC1_PU_EN;
+               } else {
                        reg |= TCPC_REG_SWITCHES0_VCONN_CC2;
+                       reg &= ~TCPC_REG_SWITCHES0_CC2_PU_EN;
+               }
        }

Ah, we're getting somewhere. The orientation looks properly detected now. But nothing in the code actually applies VCONN to the opposite pin, and it looks like the relevant code has been killed from the copy of the FUSB302 library I'm using. Which isn't totally surprising, given the above bugs. After some digging, I restore fusb302_tcpm_set_vconn() to its full glory and wire it into the CC probing routine. 

I thus declare success, but of course this is pointless, as nothing swaps the SBU signals. What I need is a pair of switches. I could try and hack something on the Lego board, or go straight to an updated board... You know what? I'm done playing with Legos.

May 2023

So here is v2! Not a lot has changed, but U5/U6 are two SPDT switches (NC7SB3157), controlled by a single GPIO, and used to swap SBU1/SBU2 between the level shifters and the USB-C connector.

The other change is that I gave up on the 0R resistors as the configuration selection. Using PCB jumper pads is much easier to work with (just cut the trace and solder the opposite pad). The whole thing works really well, and I can flip the USB-C connector all night long. But of course, I have only solved one of my pet problems. The other problem is still there (USB cables without SBU lines).

At this point I realise that designing this sort of hardware is very different from designing software. Adding incremental features is a huge mistake. Iterative development is a significant cost in time (design) and material (prototype production), not to mention the impact of this stuff going around the planet. What I should have done is to cram all the possible improvements in one single pass, with options to disable or rewire the new bits. Another valuable lesson, I guess. As a result, this version hasn't seen much use (only two people have it, and the rest of the batch is in my junk box).

June 2023

Yes, I'm on a roll. Having found out that I could replace my two NC7SB3157 with a single USB switch, I decide to go big by picking something very small, a PI3USB102EZLEX. This little guy can flip *two* high speed analog signals such as USB2.0, and takes absolutely no space on the board. Which means I can use one for my SBU routing, and another to route the USB signals between the USB connector (if using SBU for the serial) and the level shifters (if using the D+/D- lines for serial). Additional bonus, these switches have an "enable" pin that can be used to totally disconnect the inputs. We can use this to fully disconnect the SBU lines when D+/D- are used for serial communication.

These things look tiny even in KiCad, but surely that's not a problem. At all.

And a week later (the usual JLCPCB round-trip latency), I get this:

And it is yet another disaster. Out of 5 boards, only one works correctly. An 80% defect rate. Why, oh why? The fact that one board works correctly means that I'm not solely at fault. So what went wrong? Looking closely at the board, I start seeing some of the problems:

On all the defective boards, I see solder bridges on these tiny switches. On one of the boards, the component is even off-spot by about a millimeter. No wonder nothing works. I guess I went over the limit of what JLCPCB can assemble. It also shows that their QC has "some room for improvement". Understatement of the day! Having moaned at JLCPCB, I get a voucher covering about 30% of the costs, which is better than a kick up the arse. 

Anyway, I have a working v3 board with lets me use dumb USB2.0 cables. The software changes for that are positively tiny, and actually result in a cleanup. The other change is that I now expose pads for the secondary Pico serial port. I may use this later...

On the positive side, I have improved my testing infrastructure. Having to solder connectors each time I need to test a board was proving both expensive and time consuming. What do people in cases like mine? They build a jig. Unsurprisingly, mine is a quick hack, but it seems to do the trick:

On the other side of this board is the expected Pico. I can simply screw the board to be tested, and the pogo pins are providing temporary contacts. One day, I'll make that a proper PCB, should the need arise.

July 2023

Another month, another version. I have decided to take the v3 lesson on board and reduce my ambitions. Which here means using larger components. I've elected to replace the Diode switches with a pair of RS2227XN. The MSOP-10 package should not cause any issue on the fabrication side, and they are half the price of the previous ones.

Nothing else has changed, and both v3 and v3.1 are functionally the same. Even the firmware behave the same way, which is very reassuring. This is the first version I have started distributing on a "larger" scale. Which is something like 10 boards. Success!

But what is really amazing about this version is that people are starting to build their own! I get reports of runs of 5 to 15 boards being successfully assembled by people who have just checked out the git tree and ordered their own stuff. It makes me really glad I did all of this in the open.

Temporary epilogue

That's it for now. Although I have since built a v3.2 version which is only a very minor improvement on v3.1, I don't plan to make much more changes to it. You have to know where to stop (famous last words...). On the other hand, I have started looking at a more "data-centre friendly" version... Stay tuned!
Next log

Serial to serial

10/17/2023 at 09:54 • 0 comments

Discussions

Yes, delete it Cancel

Going up?

About Us Contact Hackaday.io Give Feedback Terms of Use Privacy Policy Hackaday API

© 2024 Hackaday