I've used the median filter in IrfanView for years to remove noise from high-res text scans, but never thought about how it works. As it turns out, a median filter works more or less like a blur filter.

A raster image consists of pixels that encode a color for a specific coordinate in 2D space / 2D array. A new / filtered image is created by taking pixels from the old image, doing some math with it to calculate a new color and setting the pixel in a new image to that color. Repeat this with all pixels in an image and a whole, filtered image appears. For simplicity, only greyscale images are used for now, simplified pseudocode is used.

A simple blur filter takes the color values of the source pixel and its surroundings, calculate the average (maybe weighted by their distance to the source pixel for a larger blur (or std. deviation / Gauss...) or just the surrounding top/bottom/right/left pixels for a start) and set the target pixel value.

Such an formula may look like...

target[x][y] = (source[x][y]+source[x+1][y]+source[x-1][y]+source[x][y+1]+source[x][y-1])/5

Original image	image with blur

Noise gets removed, but edges are blurred away, too. What about another function, similar to average... median function?

If the same surrounding source pixels are taken, but instead of just calculating the average, they are sorted and the middle one (=median) is used? This gets rid of "unusual" values in this area. As a side effect, think about what happens if this is done on one or the other side of a sharp edge - yes, the edge is preserved - great!

unsorted = list (source[x][y], source[x+1][y], source[x-1][y], source[x][y+1], source[x][y-1] )

sorted = sort(unsorted)

target[x][y] = sorted[ length(sorted) / 2 ]

Original image	median filter

The noise is gone and edges are still there. Ok, corners are a bit rounded - could be worse, ok for my planned use case.

As there is already a sorted list to get the middle/median one, why not take the difference of the last and the first one and thread this value as "contrast in this area"?

unsorted = list (source[x][y], source[x+1][y], source[x-1][y], source[x][y+1], source[x][y-1] )

sorted = sort(unsorted)

target[x][y] = sorted[last] - sorted[first]

Borders detected. Note that the sharp borders are bright while the gradient in the background shows up grey-ish - so there is a contrast but a much lower one. On the upper left and lower right there is only one brightness in the source image, so the resulting "contrast" is low/black. The brightness correction of your screen (view from top/bottom) might come in handy here.

Things get complex for different corner cases (namely left,right,upper and lower edges ;) as source pixel coordinates outside the source pixel have to be dealt with, or pixels along the border are not processed for the target image. Wrap around? Ignore? Multiple ways to go. In general, color versions of the algorithms above are not much difficult, just calculate the three R/G/B color channels for their own.

Demo with bigger image - Input:

Median filter:

Median-by-product border filter: