Example of such potentiometer is the RV112FF from Taiwan Alpha, and basically consist of two potentiometers with no end-stops and with a 90 degree phase difference.
Finally I calculate the change in value based on the direction, difference in ADC measurement (to make sure a fast turn increase value more than a slow turn) and a constant representing the sensitivity desired from the potentiometer, i.e. how many turns is needed to go from min to max value.
For my application I want to adjust the value based on relative position, so the absolute position is of no interest. If you knew what direction the shaft is being turned you could use the change in ADC value to caluclate a relative position change. But one challenge is that when the value reaches the top it goes back down again, which it also will do if you change direction exactly at the top. So you need to use the phase difference to determine if the value decrease due to change in shaft direction, or due to passing the top point. And same for the bottom point.
Additional practical challenges is that the resistance value is not linear at the top and bottom points, but have a small section of flat value, and also the noise on the signals and inaccuracies in the ADC will have the signal jump back and forth.
The algorithm I deviced first decides for each tap on the pot, lets call them tap A and tap B, what the direction of signal is (up = 1, down = -1 or unchanged = 0), and then look at the combination of the two taps' directions and the values/phase to determine actual direction for the shaft rotation. The following diagram show the relationship between direction of the signal of tap A and tap B for shaft direction Up/Increase and Down/Decrease. It also shows the relationship for the phase, i.e. if signal A is higher or lower than signal B, and if signal A/B both are is at higher end or lower end. With all this information combined I can correctly determine the rotation direction of the shaft.