I know that the weakest part is currently my 1st LO so I tried to check what will happen if I replace it with a better synthesizer. I used temporarily STuW81300 which is single chip synthesizer with integrated VCO. It improved results noticeably. All random spikes disappeared from displayed spectrum and it was possible to achieve resolution down to about 1kHz.
STuW81300 on 1st picture (resolution down to 1kHz), my synthesizer on 2nd picture (no comment):
I also discovered that there are some unexpected frequencies about 30kHz away from the measured frequency. I couldn't figure out where they come from. I tried many changes: different power supply (linear instead of switching), changing LCD clock, slightly tuning main oscillator, tests at different input frequencies, different source of signal, different 2nd LO, different 1st LO (I couldn't sweep from microcontroller so I was sweeping it manually while observing level), removed input attenuator module different 5V for 1st IF amplifier. All the time there was some 30kHz modulation. When I provided 2nd IF (228MHz) signal from professional synthesizer to input of IF board and swept manually I didn't notice anything 30kHz away from center frequency so it seems that the 30kHz adds somewhere in 1st and 2nd mixers board. UPDATE: I think I found that - my signal generator (nanoVNA v2) generated the modulation or jitter and and also 2nd LO generated similar modulation or jitter. So as long as I tried to change one thing every time I saw the same effect but from different sources. Conclusion: use different source instead of nanoVNA v2 and improve 2nd LO.
I did some testing to see where noise on IF board comes from. It seems that IF amplifier makes the noise (of course, 1st stage is critical). When the board is shielded, there's not much difference if I completely disconnect input of the amplifier. Maybe some changes of amplifier circuit will help but I don't expect that big improvement is possible. Improving IF filters to loose less signal may be more effective.
The design is most likely overcomplicated, considering availability of off-the-shelf components which can replace most of the work. My goal is to learn play with RF, practical usability of the project is not a major concern.
The architecture in a few words is as follows: - the input signal 0-3GHz is upconverted to first 4GHz IF - the signal is downconverted in a few steps: 228MHz, 28MHz, 3MHz - the final analog 3MHz is converted by ADC and further processed by FPGA - the FPGA mixes the signal with sine and cosine and moves the spectrum down to baseband complex signal, then filters and downsamples the signal (one or more times, depending of selected resolution bandwidth), finally RMS is calculated from the filtered signal
A few words about why it doesn't work:
- The biggest issue is really bad 1st LO. It's a wideband 4-7GHz synthesizer, quite hard to make. It has VCOs up to 3.5GHz so the frequency must be doubled. This creates need for adding switched filter banks. The filters and switches are quite bad and introduce a lot of loss, which is then compensated by a lot of gain. The result is that the small signal is spoiled by noises from ambient and also the amplifier with huge gain tends to be unstable. And the whole board is not shielded yet. Bad quality of the 1st LO creates a lot of random spurs in the measurement.
- IF part which converts signal from 228MHz to 3MHz contains filters and mixers which adds some losses, then IF amplifier introduces noise. The overall SNR seems too high. Filters should be higher quality (maybe still with air coils but larger), amplifiers may need to be somehow optimized, also some filtering at the end may help to cut off more low frequency noises. So the IF part works but it's not great.
- First and second mixers - this part is probably not great but it's probably not the biggest problem (may need more testing but possibly it's good enough for now).
- Digital part - it works but may need some optimization, improvements, debugging.