While testing the temperature measurement, I noticed that the values were rather unstable. I quickly noticed this was caused by an unstable supply voltage. In the STM32 the ADC is referenced to VDDA, which is more or less VCC with filtering. I probably should have added more filtering / larger caps, but first I tried to address that issue in software.
Compensate against VREF
The STM32 have an internal 1.2V reference. This reference can be measured by the ADC and the result can be used used to compensate other ADC values for varying operating voltages. The ST_VREFINT_CAL value is a constant for the ADC readout of the internal reference at VDDA=3.3V. The lower the operating voltage, the higher the reference reading will get.
vin = ((3300ULL * ST_VREFINT_CAL) * <ADC_IN>) / <ADC_REF> / 4096;
This compensation pretty much solved the noise issue, but from time to time a few outlier caused the controller to look a bit unstable.
Since the signal is now pretty stable over all, we only have to address a few outliers. This is done by calculating the difference of the average difference to the average of the last 16 readings.
If err is below a threshold, the value is accepted and used for the controller. If the values is rejected, the controller re-uses the last reading. This is no problem, since the controller cycles are rather short, typically 10ms. The new value is added to the window in any case, so a sudden jump in the values will not stop the controller form working.
What causes the noise / how to avoid it?
The two methods eliminate the noise issue, but it would be better to figure out, why we have noise in the first place. I've been measuring the VCC with an oscilloscope and can confirm, there is noise on the VCC line. Its not just minor noise, there are some larger drops in the supply voltage. My suspect is the display, but since I soldered it onto the board, there is no easy way to test without it. I tried to add some caps across its voltage pins, but it didn't improve much. So for now I leave it like that since its working and I'm kind of happy with the software solution.