I spent a little quality time at the bench this morning, trying to diagnose the issues with the first prototype board. The first find took much longer than it should have. I was poking around hopelessly at the front end with the oscilloscope when I realized I had a spectrum analyzer in front of me!
To capture these traces, I plugged a 10:1 oscilloscope probe into the input of the SA. This is always a little scary since the SA input is a bit delicate, but I was very careful. The vertical is set to linear mode to show the problems more clearly. The yellow trace is the signal right at the input jack, while the magenta is after the transformer (across R1), shown here for reference:
The difference in levels between the two is in part due to the transformer boosting the voltage. But, the shape of the traces shows the real problem. Although the input level drops off slightly up to 90 MHz, it picks back up afterwards, becoming very large between 140 and 160 MHz. I think this is due to reflection from the transformer performing poorly at these frequencies. The magenta trace tells a similar story. The magnitude drops off precipitously between 90 and 100 MHz, right where the input stage stops working.
I think the first thing to try is a different input transformer. I have a box of various cores I can try. Transformers on this core have worked very well for me in the past, but that has all been in the HF range or 50 MHz. It's interesting to see the limits.
I also suspect the protection diodes. They add a decent amount of capacitance, which could be an issue. Now that I think about it, removing the diodes (a single package) is easier than swapping the transformer, so I'll try that first. I'm not even sure I need the diodes with the 560-ohm termination there.
I also have a bunch of MSA0486 MMICs I could use for a preamp stage on a next PCB spin. They're almost perfectly suited to this application: DC - 3 GHz, with 8dB nominal gain and a +12 dBm P1dB. The only problem is the power supply. They need 5V at 50 mA, which has to be regulated with a resistor from a higher supply (at least a few volts higher). The easiest way to do this would be to bring 12V onto the board from somewhere, but it's kind of a big change.
While I'm thinking about next spins, there are a bunch of ECL flip-flops that could easily generate the I/Q outputs up to the maximum 1.5 GHz output of the tracking generator. ECL is a little bit of a pain to work with, but it might be interesting to make a version of this board that would go that high. I'd probably learn a whole bunch of stuff.
Output Source Terminations
I didn't have any 43-Ohm 1206 resistors, but I had some 39's. I swapped those out for the 27's I had originally to try to make a better match to 50-Ohms. The output swing went from 2.0V to 1.8V into a 50-ohm termination. A little more algebra shows that the output impedance is now 41.6 Ohms, so I should use a 47-ohm resistor instead. The 39s I had are only 5% tolerance, so this isn't exact, but It looks like it's converging. I do have some 47-Ohm 5%'s I can try on there next. They're easy enough to swap out.
I did a quick search on DigiKey this morning, and there a number of 3.3V regulators in the SOT89 package with current limits up to 800 mA (although 200 seems more common). Either would do for the approximately 150 mA maximum I'm seeing on the board. I need to re-check the power dissipation and make sure heat won't be an issue, though.