I was getting these huge output voltage spikes in the op amp output. I tried using different values for the decoupling capacitors but that didn't help. It turned out to be a dodgy connection on my breadboard which I had used to connect my 1000 micro-farad decoupling capacitor. I used another socket and the problem was gone.

While I was considering so methods to compensate for the spikes in software(because I was literally at my wits end and couldn't trace the problem) I came up with the idea of using exponential smoothing on the final signal. This would reduce the noise that we are left with after the oversampling and decimation. This technique would reduce response time but that is not a concern for us in our application.