I spent a good part of the afternoon watching videos from this udacity course ( https://www.udacity.com/course/ud617 ). From the examples that they showed for using hadoop for sorting through logs I think that this might be a good tool to sort through my apache logs, and my xbmc/kodi logs and provide updates, as well as the inputs from various devices. There is going to be some learning curve, it looks like I will have to use hadoop, mapreduce, and I am not sure which, hive or impalla or both, and sqoop.

If I get it to work I am hoping that I can use old computers as we generate them to break the throwaway cycle.