The rotary encoder is quite straightforward to read out, the internet is full of examples. After some tryouts the pins were found to be connected to P3.0 and P3.1, and the switch that can be activated by pressing the knob is on P3.3. The two outputs of the encoder just go high one after another, depending on which direction the knob is turned:

#include <Serial.h>

uint8_t OUTA = 30;
uint8_t OUTB = 31;
uint8_t SW = 33;

uint8_t sw_prev = 0;
enum {RSTAT_SETTLED=0, RSTAT_A, RSTAT_AB, RSTAT_B, RSTAT_FULL};
uint8_t stat_prev = RSTAT_SETTLED;
uint8_t stat_seen[4] = {0};


void setup() {
  pinMode(OUTA, INPUT_PULLUP);
  pinMode(OUTB, INPUT_PULLUP);
  pinMode(SW, INPUT_PULLUP);
}

void loop() {
  uint8_t a = digitalRead(OUTA) ? 0 : 1;
  uint8_t b = digitalRead(OUTB) ? 0 : 1;
  uint8_t sw = digitalRead(SW) ? 0 : 1;
  uint8_t status = (a && b) ? RSTAT_AB:
                          a ? RSTAT_A:
                          b ? RSTAT_B:
                  stat_prev ? RSTAT_FULL:
                              RSTAT_SETTLED;

  if(a && b) {
    stat_seen[RSTAT_AB] = 1;
  }
  else if(a) {
    stat_seen[RSTAT_A] = 1;
  }
  else if(b) {
    stat_seen[RSTAT_B] = 1;
  }

  if(sw_prev != sw) {
    if(sw) {
      USBSerial_println("Rotary encoder pressed");
    }
    else {
      USBSerial_println("Rotary encoder released");
    }
  }

  if(status == RSTAT_FULL && stat_prev == RSTAT_A && stat_seen[RSTAT_B] && stat_seen[RSTAT_AB]) {
    USBSerial_println("Rotary encoder finished clockwise move");
    memset(stat_seen, 0, sizeof(stat_seen));
    status = RSTAT_SETTLED;
  }
  else if(status == RSTAT_FULL && stat_prev == RSTAT_B && stat_seen[RSTAT_A] && stat_seen[RSTAT_AB]) {
    USBSerial_println("Rotary encoder finished counter clockwise move");
    memset(stat_seen, 0, sizeof(stat_seen));
    status = RSTAT_SETTLED;
  }
  else if(status == RSTAT_FULL) {
    USBSerial_println("Rotary encoder got back to starting position");
    memset(stat_seen, 0, sizeof(stat_seen));
    status = RSTAT_SETTLED;
  }

  stat_prev = status;
  sw_prev = sw;
}

There is no debouncing in the code above, so you might get some extra readings when pressing the knob.

This means the whole pad is figured out now (in record time, at least for me), next steps will be to write the USB keyboard code. And some nice way to configure the keys, LEDs and encoder without having to reprogram the ch552 with the Arduino IDE. Onward!