It really took a long time to get the damn MAX chip working. Although I made it set MUX line high or low, I couldn't make it talk to me via SPI. On the scope, this line looked like nonsense:
In the process of investigating, I managed to burn 3 Teensy boards and 2 MAX chips. Don't ask how. I consulted tech support of Maxim Integrated as well. They were really friendly and prompt reacting (thanks Mohammed). But, before I got the answer from them, I decided to try one thing.
MUX bits are distributed in two registers. MUX is in reg 5 while MUX[2:0] in reg 2. My initial code was writing reg2 first and reg5 later. For no clever reason, I decided to switch the order of writing. Guess what? Problem solved.
I'm using SPI read to get some data out of MAX2871. These are for chip temperature, VCO tuning voltage, and autoselected VCO number. During the development I found out that ADC that reads VCO voltage and temperature must be set and reset after each readout. Otherwise, it won't read properly.