PolyKybd (Displays In Your Keycaps)

Finishing the second hardware prototype turned out to be more involved than expected! I had some troubles (with noise I think) on the signal lines, which went away when I removed the pull-up resistors which I had on the SPI pins (I shouldn't have used them in the first place, as it is not I2C but at some point these solved a problem with floating pins when I was using the STM407F MCU and they just stayed - well, until now).

Once a while, there is still a glitch, which I need to investigate. The strange thing is that also without refresh, I can see some pixel light up one a while. My best guess so far is that I unintentionally introduce some capacitance or resistance due to my soldered together flex-cables (the pitch is only 0.5mm so it is possible that two neighbor lines almost touch each other) and these cause some troubles. That problem would go away if I could get the customized displays with the longer flex cables... but I cannot prove it for now.

Needed Software Changes: Since I have two identical boards for my "test keyboard setup", I started using the eeprom to store which side is the actual left and right side(with `#define EE_HANDS` and make handwired/polykb/wave:default:left` `...right`), but that is not needed as the is a master detection with ChibiOS and it just sets the master to the side with the connected USB cable (just connect the able to the 'correct' side).

Additionally, I had to double the key matrix and num of RGB LEDs as if everything runs on one side (turns out that actually happens - the master just receives the keypress from the other side and runs the logic!). So in the end this is what I changed:

rules.mk:

SPLIT_KEYBOARD = yes

config.h:

#define RGBLED_NUM 72
#define DRIVER_LED_TOTAL RGBLED_NUM
#define RGB_MATRIX_SPLIT { 36, 36 }
#define SERIAL_USART_TX_PIN GP5
#define SPLIT_TRANSPORT_MIRROR
#define SPLIT_LAYER_STATE_ENABLE
#define SPLIT_LED_STATE_ENABLE
#define SPLIT_MODS_ENABLE
#define SPLIT_OLED_ENABLE
//and also define a new LAYOUT to have both sides in one:
#define SPLIT_LAYOUT

( ... My first few connection tests failed, however after changing a couple of things back and forth I got some basic input working, however, just for one the master (left side) and !some! key on the slave (right side):

Again, it took me some time to think about it, but after the realization that only the master really executes the logic it was clear that all my static helpers were never updates on the other side (and some initial mistakes when adding the second side to the layout).

The solution to this was to use the housekeeping task and check for changes (as we already configured QMK to sync status changes like modifiers and status LEDs) and then perform the needed updates. As I can do the same on the master the display update now moved from the matrix processing to the housekeeping as well, which should be an improvement for the scan rate!

To also get the language change propagated (which are no QMK layers), I had to add a rpc callback which is triggered by the master to send that information together with the display brightness:

The `user_sync_poly_data_handler` task is executed on the the `slave` side, which just sets the static variables and then asks for a refresh of the displays which will be done in the housekeeping task.

There are still open software issues I need to address at some point (RGB matrix lighting not working correctly at the moment, fade in and out after timeout has some side effects, code cleanup - yes it's messy at the moment, the OLED display update could be faster, ...) but for now you can find the work-in-progress code here: https://github.com/thpoll83/qmk_firmware/tree/PolyKeyboard

And here a small video I posted on Twitter: https://twitter.com/thpoll2/status/1591026041434226688?s=20&t=j8DVR1XFWFR__iBi13svhw

For most people the typing experience itself is the most important part of the keyboard which it is hard to quantify.

So it shouldn't be a surprise that my thoughts circle around that aspect as well when thinking about the PolyKeyboard.

What I cannot influence for now, is the keycap profile from the relegendable keycaps I use. It's maybe close to XDA, however the swale is not that profound, the corners are a bit more distinct and the overall height is only 8mm.

And what about the key-stroke?

How much does the flex cable influence the behavior of the key switches?

To figure that out, I made a little test station and measured how much weight is needed before and after adding the flex cable to the key cap:

The LED on the board will light up when the key switch conducts.

The keycap with the 3D printed stem and the display come with a weight of 2.2g:

My circular tray to hold the weights add another 7g.

First, I did a test without the flex cable, so let's quickly deduct its weight from about 0,5g:

When the LED finally lights up, the following weights rest on the key switch:

Stem with keycap (minus display for the first test): 1.7g

Tray to hold the weights: 7g

Weights: 20g + 20g +1g

That makes a total weight of 49.7g when the switch activates, which matches very good with manufacturer's data for the yellow Gateron switch of 50g for the operating force. So now, how much does that change with the plugged in flex cable? The same thing again, I'm adding weight until the red light turns on:

Stem with keycap and display: 2.2g

Tray to hold the weights: 7g

Weights: 50g + 2g + 1g

Here we have a total of 62.2g, which is 12.5g more. So this is definitely something to consider when choosing switches for the PolyKeyboard!

There is maybe a bit room to get that weight a little bit more down when customizing the flex cable length and maybe also the cable width, but I think the current value of 12.5g added on top of your key switches actuation force is a good orientation.

As you may know this is not the first keyboard with displays in the key caps and I'd like to use this post to draw some comparisons with my project. LC Board / E3 Keys

Maybe the great grandfather of all these keyboards is the LC Board, you can see the commercial version live here: https://www.youtube.com/watch?v=3U20m6KjVrc The first prototype was created in 1984 in Germany and got produced as commercial version from 1987 on by "Hohe Elektronik". It featured 34 programmable keys with monochrome LCD dot matrices:

(https://deskthority.net/download/file.php?id=7771)

(https://deskthority.net/download/file.php?id=7772)

According to https://deskthority.net/viewtopic.php?f=62&t=5726 it was invented by Reinhard Engstler who still owns a company that produces and sells keys with LCD displays (even RGB versions)!

However these switches are not the usual keyboard switches, but a custom version with a size of 24.5 mm by 23.5 mm and seem to be used in control panels for industrial / mixing applications:

(https://www.e3-keys.com/products.html)

You can find out more on https://www.e3-keys.com, they even have some more extensive history post there: https://www.e3-keys.com/news.html#news1

Art.Lebedev Optimus Maximus keyboard

Maybe the best known keyboard with displays:

The 'Optimus Maximus', still advertised on the manufacturer's website https://www.artlebedev.com/optimus/maximus/ However, it is out of stock since almost a decade and the initial price tag was around 1600 USD according to https://en.wikipedia.org/wiki/Optimus_Maximus_keyboard

Most critics you can find online are around the typing experience: People used to get tired after 30 min of typing as the keys are so stiff. Other people mentioned a high pitched sound coming from the keyboard or the separate power supply you needed to operate it. Independent of it's flaws I would say this one was very ambitious the most influential keyboard with displays.

There were variation like the 'Optimus Popularis' which used a single big screen under the keys and the key caps are just transparent to let you see through the underlying segment of the display, so the key caps are suspended on the side.

(https://www.artlebedev.com/optimus/popularis/)

According to this review, the typing experience is better than expected, however it still had a price tag of 1000 USD and the layout is maybe not a fit for everyone.

Additionally there was a 6 key macro pad called 'Mini Six' for around 400 USD. IMHO a bit too much for 6 keys!

Nemeio

One of the more recent attempts to create a keyboard with programmable displays, it uses the same approach like the 'Optimus Popularis' with a singe screen at the bottom and transparent key caps on top, but it uses e-Ink instead of a full RGB display: https://www.nemeio.com/

Funded via a campaign on kickstarter and indiegogo, this keyboard was initially planned to be released by the end of 2019. The current schedule says 'End of 2022'. As it is not yet release we have no review yet and I do wonder about the typing experience of those keys. Of course the fit of the layout is another open question.

Private Project by James Brown

This is a project very similar to the PolyKeyboard by fellow Twitterer James Brown who uses the same displays but with shorter flex cables that are soldered on a flex PCB which is then once again soldered to the main PCB of the keyboard. I was surprised how fast (compared to my project) James came up with working solutions.

The main difference to the PolyKeyboard is that the displays are controlled via I2C instead of SPI and the flex cable doesn't go through the RGB slit of the key switch but just through the PCB via a slot in front of the key switch:

https://twitter.com/ancient_james/status/1525645446923702273?s=20&t=AAAGqh2W81XA7f4Jm9mOxg

I have to admit that this design leaves much freedom of choice for the key switches. 

https://twitter.com/ancient_james/status/1538808294118424578?s=20&t=nmTf1NkwTr6LqAmxBh_jOQ

Also the resin cast keycaps look pretty nice.

Sonder Design

https://en.wikipedia.org/wiki/Sonder_Design

Another single screen e-ink keyboard that according to Wikipedia got bought by Foxconn and the disappeared. The Guardian also covered that keyboard including a YouTube video of the prototype where you can see the e-ink screen in the background refreshing:

E-inkey Dynamic Keyboard

Not sure if this project ever got beyond the design phase as there are only renderings and the indiegogo campaign which wasn't a success. However, it was the only e-ink keyboard that considered to have the displays in the key caps:

There are some more, but only in form stream decks:

Infinitton

Once again the famous technique to use a single RGB screen in the back, this time for a smaller format, founded via kickstarter: https://www.kickstarter.com/projects/2086416348/infinitton-a-smart-touch-screen-keyboard?ref=discovery&term=keyboard%20screen

Elgato Stream Deck XL

I'm not sure how it works under the hood but it also looks like a single RGB display with transparent keycaps. One of the few 'Display Keyboards' that are still produced: https://www.elgato.com/en/stream-deck-xl

Finally the PolyKeyboard! (now PolyKybd)

What can I say about the PolyKeyboard? Well, one day, hopefully, it will be a kit! Which can be assembled by anyone ;)

From the beginning, I stayed away from solutions that would not work for kits.

• For instance resin cast key caps:  These look beautiful, but I cannot even do casts for 10 keyboards. That would be too labor intense. Instead I'm using relegendable key caps which are commercially available in different sizes, the only customized part here is a 3D-printed stem which is doable with my printers (or with some on demand print farms).
• Also soldering the flex cable of the displays to the PCB would be a maintenance burden if a display ever stops working. Instead I'm using a socket to plug the flex cable in.
• No custom key switches! Use the key switches of your choice: At least to some degree, as the flex cable needs to fit though the RGB slit of the key switch. I published a compatibility chart in an earlier post and think we can maybe get even smaller flex cables so that there are more compatible switches out of the box.

It is open source / open hardware, already by today (okay, I know, I have to organize it a bit more) so your are welcome to join in! The firmware is made with QMK, which means that there is no limit to the customization possibilities. The current dev-kit, which I try to put together, is a split keyboard, but I'm also thinking about a 70%ish staggered layout. The only customized component I need, is the display with an extended flex cable, as the current length is not sufficient to plug it into the socket (and I don't want you to extend the flex cable yourself, it is possible and I do that for my prototypes, but it is very painful!!). I just need to reach the minimum order quantity so that my Chinese manufacturer is willing to customize the attached flex cable. For that I need some budget upfront... currently thinking about my possibilities.

I hope you enjoyed my short overview. Maybe you are aware of another project I didn't mention? Please let me know.

Edit: People informed me that there was also such a keyboard by Microsoft, the so called "Adaptive Keyboard" and there is still some information left on Microsoft's website: https://www.microsoft.com/applied-sciences/projects/displaycover

And also here: https://www.engadget.com/2010-08-12-microsoft-adaptive-keyboard-prototype-debuts-at-center-of-uist-s.html

It looks like there were multiple iterations, some with e-ink displays, others with RGB displays and a touchscreen above the F key row:

And now there is also the Flux Keyboard:

https://fluxkeyboard.com/

A split keyboard needs two sides.

Before ordering another PCB (for the right side), I want to make sure that the communication works. And luckily you always get two PCBs when ordering an assembled board, so I have two left sides. Enough to test (so stay tuned for another update with some results)!

Why do it like that? It gives me the chance to correct mistakes from the left side (see bodge wires posted earlier) and test an improved right side :)

From the beginning I prepared to cross Rx and Tx via solder jumpers so that I can run that test:

Maybe a bit over-engineered and in the meanwhile I found out that QMK supports half-duplex serial communication via a single wire, but I will give it a try. Unfortunately, I put these solder jumpers on the front side, which is inaccessible if I put the keyboard plate on.

Right next to the solder jumpers you can see the SMD footprint of the RJ12 connector. It is not that common for split keyboards, but I do like the fact that it doesn't expose blank pins like TRRS (which is indeed compact). I've seen a lot of people using USB-C but imho that's also wrong as it's not a standard conforming USB-C. Happy to hear suggestions!

Btw, did you recognize that the picture of this post already shows some 1.25U? I do like them a lot. Let me know your opinion about it~

Let me talk a little bit about software, since this also takes time - apart from my hardware challenges!

Okay, so next to my QMK keyboard firmware, I need some glyph rendering for my OLED screens. To truly support multiple languages, I have handle characters beyond ASCII-7/8.

Initially, I started to use the Adafruit-GFX Library as base, which provides the rendering of basic ASCII characters (a font has to be converted to pixel data from a .tft or .oft file of choice).

To support characters for Japanese, Korean, Arabic, etc. I had to first extend all std::sting / const char* to std::u16string (I know there are 4 byte characters, but for my use cases, 2 bytes are enough for now), which wasn't a big deal.

Next, I needed to extend my own interface of the keyboard firmware to provide other fonts (pixmaps) for the new character ranges. For instance Japanese Hiragana characters are in the range from 12353 to 12447 (instead of the printable ASCII range from `space` 32 to `tilde` 126). To do that, my font rendering accepts an array of fonts and tries to find the matching character in the range of the first provided font and looks up in the next font if that fails.

To prove that this approach works, I started manually patching the generated font files (.h files), but over time it became a burden and so I decided to update the fontconverter as well.

Also here, the first step was to extend indices from 1 byte to 2 bytes and remove checks that narrowed the output to the ASCII range. For more convenience my fontconverter version also allows multiple ranges, eg from 32-126 and from 138-142 at once. However, due to some restrictions in the font rendering library the entries of characters in-between the ranges will be filled up with zeros. So, if the gap between two ranges is big it might be still better to split that up into two separate ranges (a lot of zero also need some space).

You can find that fonconvert version here: https://github.com/thpoll83/Adafruit-GFX-Library

Let me show you a small example to generate all isolated Korean vowels (I'm using Google's Noto Korean font - you can use any font that has Korean vowels), which I can specify with 3 ranges 0x1161 to 0x1169, 0x116d to 0x116e and 0x1172 to 0x1175:

In the output you can see some zero-ed entries in-between the ranges as described - annotated with 'skip'. You might have noticed the slightly changed parameters, so here is the `usage` output:

I usually evaluate the result right away with this awesome online tool: https://tchapi.github.io/Adafruit-GFX-Font-Customiser/ You can just copy & paste the output and it will visualize the generated pixel font. In addition allows you to modify the font as well. A really useful tool, which thankfully also works despite my changes to the output format:

If everything fits, the generated text goes into a .h file, which is than used by my QMK firmware, which you can find here: https://github.com/thpoll83/qmk_firmware

There wasn't much progress on the firmware recently as I was on a holiday break with my family ;) I hope you enjoyed this software post and see you next time!

I finally got a set of test key switches to check which are compatible with my PolyKeyboard.

What's needed? Actually only a LED slit that is 8.5mm wide. This is usually not mentioned in any datasheet, so I decided it's best to try out!

The result table:

From that you can see that only a few Gateron switches and the Kailh Sabre & Cream switches work without modification.

Luckily you get much more choice when you cut away the little centerpiece in the LED slit. This can be done with either a sharp knife or a needle file. With that you can use all switches with a compatible stem (the cross) and all entries that say "needle file" for the 8.5mm slit. I added an exclamation mark in case you also have to trim away a bit plastic from the side (so in fact a bit smaller than 8.5mm).

If you are willing to modify your switches a bit more, you will get a much wider selection. For example, there are a couple of switches that have a slit, but at the bottom (base) there are only 4 holes for the LED pins. With a Dremel or similar you could easliy cut though the base (but it will be a lot of work... for each and every switch).

Of course this list is not complete, not at all. It's only what I was able to test. Let's hope that this list expands at some point!

I really like the low profile key switches from the second set of test switches and maybe I can use such switches for the lower 3 thumb switches (or even the whole bottom row?):

Unfortunately those have an incompatible pin layout, so I'm thinking about getting some Cherry MX low profile switches and just cut a slit into them...

I had to wait a little bit longer to assemble the left side as I was waiting for the Mill-Max 0305 sockets to arrive. Of course, I could just solder the key switches to the PCB, but that would make eventual debugging or applying bodge wires much harder. Better wait a bit!

At least I could use the time to prototype a fitting case with the 3D printer:

I swear, I measured it before printing, but it doesn't always work right away. The current case also provides some hohls to mount tenting stands, but I haven't tried those yet.

At the time the sockets arrived, I found out that these sockets make the switches stand out a little more than before so I had to print some underpads:

And with that the sandwich got a bit thicker and the displays flex cables got some more clearance, so actually, I could print the case again, reducing the height by 2 more millimeters. But that's for another day! For now I was just happy to finally see the left side come together:

Here is a very short video I posted on twitter: https://twitter.com/thpoll2/status/1534847633612845058

Right now I'm working on the firmware as there is still plenty of stuff to do to really show what I try to achieve and of course there is also the right side which needs to be revised and assembled, so it will not get boring any time soon.

An exciting moment as I worked really long time to get to that point.

My designed keyboard PCB and aluminium plate arrived! As usual for JLCPCB you get a minimum of 5 pieces. If you chose to assemble, you have to assemble at least 2! However, that is fine for me, as it gives my the chance to connect two boards of the split keyboard and prototype a bit further. Sure, two left sides, but it doesn't matter for testing the software communication.

First, I tested if plate and PCB fits together:

I really like how the silk screen turned out and the alignment of PCB and plate is great. Unfortunately, the key switches sit a bit lose as I needed another notch at the top of every switch to let the LED shine through (the actual LED slit will be occupied by the flex cable of the OLED displays). So, next time, I will try to make them a bit tighter.

Testing the power supply next:

Glad that those work and also give the right voltages. I was a bit worried about that as I don't have much experience here. In the end I just followed the data sheet of the ICs and hoped it works out (on my previous macro pad I used boost/buck converters from AliExpress but wasn't too happy with them).

As JLCPCB does only one side PCBA, I had manually solder on the front side LEDs and one of the shift registers, which I decided to put under the keyboard status display. In the end not the best decision since the space under the display is much more limited than expected and I had to move SMT parts on the backside of the display:

Luckily, that all worked out and a first test shows all LEDs and the status display come to live. This was still without the RP Pico directly soldered to the backside.

Before doing that, I had to apply some fixes:

Not too bad, but definitely have to fix that for the right side. I somehow managed to exchange the SPI clock and data line and also one pin needs to stay unconnected as SPI requires to setup a MISO pin despite the fact that I have no device talking back to the MCU. With that, some pins shifted around and caused some additional bodge wires as well.

Nevertheless, with these modifications, the board was ready to drive the small displays:

At the beginning of my keyboard journey I was sure to make a standard 100% layout, now, as I got more into that topic, I figured out that I don't want that any more and maybe people who are more into mechanical keyboard can understand.

So ultimately, I decided to go for a split keyboard with 36 keys plus one encoder on one side. That is about the maximum I can operate with a RP2040 without the need of a port extender.

As I already made my first steps in KiCad with the PolyKB Atom, I felt confident enough that this works out.

Maybe I should have done some more studies on ergonomics, but we all have to start somewhere, so this how the left side of my keyboard will look like:

As you might be able to see, the RP Pico board can be soldered to the PCB on the backside (unfortunately it only has a Micro USB as we know :/ ) and I decided to use an RJ12 jack to connect the two boards.

I know a TRRS jack would be sufficient, but it feels a bit... "strange" to power the second board with that, as the power "pins" would be totally unprotected when disconnecting in operation. Also, I was not sure if I need another pin for bi-directional communication (maybe not, but we will see)

For the keyboard plate I'm using aluminium as I can order it together with the PCB from JLCPCB (as they are making aluminium PCBs as well) and I can design it in KiCad as well, no need for anything else.

Since it is possible to add a silk screen to the aluminium plate I wanted to have an appropriate design for my project: True to my goal to make a keyboard that speaks multiple languages I found the perfect fit, the oldest piece of multilingual history - the Rosetta Stone.

As I wanted to have the shiny side on top, I designed the plate with the silk screen directly on the front aluminium and the solder mask on the back (where you could apply another silk screen).

JLCPCB doesn't make guarantees about the quality of the silk screen on that side but I will find out :)

Chip shortage. Any more explanation needed? When I started, the STM32F407 was cheap, fast and had plenty of IO pins and therefore, an ideal choice I thought :)

Now you have to pay 40 bucks for the dev board I used and as I'm trying to make a dev kit out of this project I need some more accessible components.

The RP Pico (with an RP2040) comes at a price of 4 Euro, so I ordered a few boards to get started.

Next, I removed my old mainboard:

Goodbye and thank you for your service!

Here comes the new generation (and there are significantly less IO pins, actually, I need them all and some more):

Right, so I had to hard-wire the enable pin of the displays power supply to +3V3, there was just no pin left.

And it took some time with the logic analyzer to figure out the little differences in the SPI setup to finally get something showing up:

Exactly one key gave me something and that something was also distorted. But after getting the timing, reset, d/c etc. signals and default low/hight states right, ALL displays were willing to work!

Another step forward and in parallel I started with my efforts in designing a more complete keyboard PCB.... still based on my Atoms.