Project Logs

Collapse

• Of PCB assembly quality control...

  maz • 04/05/2024 at 15:06 • 3 comments

  It now has been about 18 months since I have started designing and producing the Central Scrutinizer. It has been a fun trip into building something physical, as opposed to just writing software.

  I have learned a lot in the process (see my first log entry), and I'm really glad that I managed to build something that people do find useful.

  However, a constant source of frustration has been the PCB assembly process. I only own a basic soldering iron, and my eyesight is not what it once was, so while I'm happy to assemble the through-hole stuff, I leave the SMD work to the PCB shop, aka JLCPCB. The whole project is configured to make it super easy to upload the design to their website and get working boards by mostly accepting the proposed default options.
  

  I'm also not a huge operation, far from it. I produce boards in small batches (about 10 to 20 at a time), giving most of them for free to fellow open-source developers, and selling whatever is left on Tindie. This makes the buyers indirect sponsors of the open-source projects that use the boards, something I quite like. I don't make any money out of it, other than the odd beer with fellow hackers.
  

  So it's all good, Sort of. What really gets me is the poor level of quality control that gets applied to the produced PCB. And poor doesn't quite describe it.

  Here's the latest example:
  

  U5 is off. Not a bit off due to the component having drifted. It is exactly off by a whole pin. Something that any automated checking should spot immediately. And that's the issue. I'm pretty sure that given the rock-bottom price for the PCBA  process, there isn't any margin left for actual quality control. So whatever crap that has been produced ships.

  I've seen other similar issues, such as micro-USB connectors soldered at a funny angle, solder bridges and the odd dead component. For each of those, that's another board on the scrap pile. The planet thanks you. Not. Maybe I should invest in rework equipment...
  

  Frankly, I'd happily pay significantly more for a reliable service with  an ordering procedure as good as JLCPCB's. If you know of any, please let me know! I wonder how other people with my level of experience (close to zero) and equipment (similarly null) cope with this...

  On the plus side, I've grown a solid enough test procedure, so I'm able to spot a bad board within a few minutes, and only ship the good'uns. I guess that's as good as it gets for an outcome.
  

• M3, baby!

  maz • 11/29/2023 at 20:59 • 0 comments

  As I found out today, the Central Scrutinizer is working out of the box on a M3-Max machine (and no, that's not mine):

  https://oftc.irclog.whitequark.org/asahi-dev/2023-11-29#32702519;

  I guess the good folks in Cupertino had no reason to change something that was working just fine!
  

• Shape of things to come?

  maz • 11/19/2023 at 12:27 • 0 comments

  People who know me know that I don't really care about non-ARM architectures. I have been involved with it for a long time, and I'm yet to be excited by what has been coming from the other side of the fence (on the other hand, if you know of a new CPU architecture that doesn't repeat all the combined  mistakes of x86+ARM+Power, please let me know).
  

  That was one of the reasons to pick the RPi Pico as the uC for the Central Scrutinizer: an architecture I was comfortable enough with and can debug easily.

  The same people also know that my OS of choice is Linux. I have been doing kernel stuff for a few decades, and this is the only OS I have used since 1993. Yeah, it's been that long.

  Of course, picking the Pico meant that running Linux (*real* Linux, not uC/Linux) was out of the question. And for something that is essentially a set of state machines, it really doesn't matter much.

  But then I saw this. The form factor is of course a direct nod to the Pico, but the characteristics are on a different scale: a 64bit CPU that is capable of running Linux, plenty of peripherals (including Ethernet), and a price that is close enough to that of the Pico that I start thinking of new possibilities... What if, instead of plugging the CS board into a console server, the CS was becoming *the* console server? And make use of the MIPI interface to do HDMI capture? Does anyone else think "full BMC"?
  

  Of course, people paying attention will say "but this is a RISC-V CPU, what's wrong with you?". And they'd be right to ask. But maybe that's the sort of incentive I need to start looking at other architectures... ;-). And if you know of an interesting arm64 board with the same form factor, please shout!
  

  Now, this thing is as far as you can imagine from being usable. There is a v5.10 BSP for it, which I am not going to touch with a barge pole (it mandates fetching unknown x86 binaries and running them locally, which isn't going to work anyway on my arm64 box). So I've started looking at the upstream support for it, and as of v6.7-rc1 we have interrupts, timer and UART working. Nothing else. Not going to ship anything with it any time soon.

  But I definitely like the idea of it, and I plan to spend some time improving the upstream support. Maybe in a year or so, I'll be able to get to the point where the firmware for the CS is only a userspace process, and nothing else. That'd be cool... and would open the door to more interesting experiments!
  

  Stay tuned!
  

• I didn't have this in mind, but...

  maz • 11/17/2023 at 14:32 • 0 comments

  It happened a while ago, but I thought I'd mention this hack on top of my own hack:

  Note that I'm not affiliated with this in any shape or form, other than being pleased to see someone making an unexpected use of it!
  

• Everything looks better in gold

  maz • 11/01/2023 at 18:06 • 0 comments

  A couple of  weeks ago, I ran out of Central Scrutinizer boards. Between those that went on Tindie and the ones that I handed to fellow developers, that's about 50 boards that I distributed. Not bad for a pretty niche project!
  

  I hesitated making another run, as I'd prefer to see people building their own. But since I kept getting requests, I ended up producing another small run of v3.2 boards. This time using the ENIG process instead of the standard HASL that I used so far. So instead of the silvery and slightly bumpy finish, I get these gold-colored and super-flat pads.
  It doesn't seem much, but it makes quite a difference at assembly time. Soldering them becomes almost an enjoyable experience!
  

  Other than that, they are still the same! I took this opportunity to retire my last v0 and v1 boards, and treat myself to some quality gold stuff! I'll still maintain the firmware for these early revisions though.
  

• Behold the new test jig monster...

  maz • 10/26/2023 at 19:49 • 0 comments

  If  you have read my first (and way too long) log entry, you will be familiar with my first attempt at building a test jig, which looked like this:

  Not exactly pretty, but functional enough: slap a board to be tested on top, secure it with the bolts, and let the Pico that lives underneath do the work. I was able to test a number of boards this way without having to solder anything.

  However, this is a one off, and I thought I'd build something a bit less hackish. Also, I would like to be able to test the dual-board setup, which requires to be more inventive.

  So I settled on a PCB design that would give me some flexibility: a jig PCB can either hold a board or a Pico, but not both. If you want to connect it to anything else, I have an additional connector for that in the middle:

  Of course, this results in twice the number of PCBs for the same thing as my original contraption:
  

  The (modest) advantage of this version is that I can swap the top interposer for one that has a different configuration. And this is not restricted to a Central Scrutinizer board either.
  

  Now, back to my earlier plan of being able to test two boards at once. I found this carrier board (https://thepihut.com/products/pico-omnibus-dual-expander) that allows me to plug two boards. With two test jigs plugged in, and two CS boards to boot, that's quite a setup! I have since replaced the metal screws and bolts with nylon equivalents, which is much safer.
  

  Of course, the PCB design is pushed out in the repo. Feel free to reuse it for something else!
  

• Serial to serial

  maz • 10/17/2023 at 09:54 • 0 comments

  These days, a serial adapter is widely understood as a way to convert a TTL serial stream to a more "modern" (read: more pointlessly complicated) protocol. And that's what the Central Scrutinizer does in its most basic incarnation by converting the serial stream to USB CDC. 

  However, the good old RS232 is still alive and kicking, specially outside of the embedded environments My test farm, for example, is largely composed of machines that have RS232 interfaces which I connect to my (home made) console server.

  There is also a trust issue: It is easier to integrate a pure serial device conforming to the RS323 spec than a USB device running untrusted firmware and potentially capable of tripping interesting bugs on the host side. BadUSB anyone? Of course, *I* trust my code. Who else does? Rhetorical question!
  

  For all these reasons, it is sometimes desirable to trade the convenience of a USB serial port for the robustness and trust of a RS232 connection.

  Exposing a Mac serial port over RS232 shouldn't be a big deal: a couple of level shifters, and that'd be it. However, if you want the current behaviour of multiplexing control (such as the reset capability) and data planes over the same channel, you need some on-board processing. Which is fine, as the Pico has two UARTs, and we can use the second one for the upstream connection.

  I was looking at fitting a MAX3232 on the Central Scrutinizer when I discovered this add-on board that seems designed for it: https://www.waveshare.com/wiki/Pico-2CH-RS232 A MAX3232, all the required passives, and a bunch of visually annoying LEDs. What's not to like? And it is available almost everywhere (eBay, Pi Hut...).
  

  A couple of software hacks later (all pushed out onto the git repo), we have a full serial to serial bridge. Look Ma, no USB! Well, you still need to apply some 5v somewhere, so you actually need a USB connection. But it can be to a dumb (and trusted) power supply, without any data involved. Oh, and the SW automatically guesses that the RS232 board is connected, so the same build works in all conditions.
  

  A slightly different hack would be to use both RS232 channels and split the control plane from the data stream. Not sure if there is an interest for that though.

  In any case, it is pretty amusing to see a modern Mac being accessible over an RS232 connection!
  

• Building the Central Scrutinizer, one mistake at a time

  maz • 10/08/2023 at 11:43 • 0 comments

  What is this?

  I've been building this serial adapter for M1/M2 Macs for some time now, and I thought that I'd document how it all came to life. There is nothing amazing in it. Only that it demonstrates the power of open source software and hardware. I didn't invent anything. I just put two and two together, and end-up with something that people found useful. It doesn't get better than that.

  ---

  February 2021

  M1 mini has landed. Screw macOS, I have other plans.

  The Asahi project already has some basics in place (the m1n1 firmware), and thankfully describes a rough method to get a serial port out of the machine: https://github.com/AsahiLinux/vdmtool . That's where everything starts.

  I'm rather adverse to soldering, so aiming for a Lego-like setup by using a bunch of things sourced from Tindie/Amazon/Whatever as well as making liberal use of my junk box:

  • a FUSB302 breakout board (https://www.tindie.com/products/reclaimerlabs/usb-c-power-delivery-phy-breakout-board/)
  • a USB-C breakout/pass-through board (http://www.elabguy.com/USB3_1_PT.php) exposing the full collection of USB-C signals
  • a 1.2v-capable FTDI adapter (https://www.tindie.com/products/saimon/debug-board-usb-to-uart-with-voltage-translator/)
  • an Arduino Nano clone
  • a piece of Veroboard with a hack together 3.3v to 1.2v voltage divider
  • A funky USB-C cable extender (https://www.amazon.co.uk/gp/product/B08BBXMPWZ -- something that shouldn't exist the first place...)
  • A cardboard box to house the whole thing

  A couple of hours of  "work" assembling everything, tons of flying leads all over the place (and probably the reliability of a Morris Minor), and I end-up with this:

  
  Of course, nothing works, and while the Arduino seems to correctly talk to the FUSB302, that sucker keeps connecting/disconnecting. After some debugging (and Marcan's help), I realise that both CC lines are connected all the way, which isn't quite what USB-C expects... RTFM!

  Multiple solutions:

  • hack the FUSB302 board
  • hack the cable
  • hack the pass-through board

  It turns out that the pass-through board has a number of vias to route all the signals across. Which is great news, as I have a very small screwdriver that I can apply to one of these vias (yes, this is terrible -- I don't care):

  ... and that's one CC line gone for good. Hopefully I won't regret this. And guess what, it all starts working. We have lift off! Both serial and reboot are working, and I can hack on the M1 remotely, without having to hear the stupid "Dong" each time the machine reboots (about 20 times an hour). I narrowly avoided taking the same screwdriver to the speaker...
  

  
  Now that's what I call engineering! Note the unused micro-USB connector on the FUSB302 board. Eventually, I used that connection to upload payloads to the Mac. To this day, this machine still boots entirely from USB.

  And frankly, this should have been it. End of the story.

  ---

  November 2022

  Almost two years later. The M1 is still going strong. I wrote a PCIe driver for it, KVM/arm64 is up and running (although I had to work around some of ugly deviations from the architecture as well as some silly HW bugs), and it is my main KVM development machine.

  Only problem is that I now have a *second* M1. And I need another serial contraption. Yes, first world problem. Frantically trying to buy the ReclaimerLabs board again, but it is sold out, and the seller doesn't reply to my anxious messages. At no point it occurs to me that I could just upload the design to a PCB shop website and get it done. No.

  Instead, I decide to reinvent the wheel. Just because I can. And my square wheel looks like this:

  The big square in the middle is a bare FUSB302 soldered on a DIP adapter, the USB-C breakout board is a much cheaper version that exposes the SBU signals (just chop these pesky resistors off the CC lines), and dangling is the same FTDI adapter. And it works! Well, as long as you don't try to move the ball of wires around. Or breathe next to it.
  

  OK, I...

  Read more »

View all 8 project logs