My components have arrived and my boards have shipped... so I'm continuing to bang my head against the encoder problem in the meantime.
As an aside, this process of poking around has been greatly helped by the configuration tools at start.atmel.com, which give you a way to configure all the libraries and peripherals you need for your AVR or Atmel ARM project and export the setup code as an Atmel Studio project ready to go. Since the '816 is significantly different than earlier ATTinys, this has been a huge help in getting me up and running and futzing around with peripherals.
I can certainly capture every other transition easily enough -- configure LUT0 as an XOR on the two phases and feed it into one of the event lines and drive the capture input of one of the timers from there. But I could do twice as well so I'm spinning my wheels a bit.
If I feed the output of LUT0 and the output of the optional D flip-flop into LUT1 and configure it as another XOR, I should get a one clock cycle pulse on the output of LUT1 -- enough to clock the flip-flop and trip the counter. I tried doing this with the outputs of LUT1 and the DFF fed out to two I/O pins, but I did not see any evidence of output using my logic analyzer.
My next trick is to hook up an interrupt to the counter capture so I can toggle a pin every time the counter capture fires... and this isn't firing at all. That troubles me more than almost anything else because if I can't get that hookup to work it means that I'm SOL on getting any help from the CCL on making the encoder work better than I get from just hanging pin change interrupts on the two input lines.
The encoder I've been using for testing (a super cheap thing designed for panel controls) has some mechanical bounce to it, which means I have a way to judge the minimum time to sample a single edge.
The first time I tried this, the results were awful -- 3-4 us of delay from a transition to the pin toggle I attached to the interrupt. Then I realized I'd left the clock prescaler turned on. Once I turned it off, I'm clocking a very consistent 720 ns from incoming edge to toggle. Since I anticipate a minimum time between transitions of ~33 us, this is good enough for a passable velocity controller.
So enough about that problem and on to the next one.