For the high voltage, there are two main ideas:
1) Flyback transformer stolen from a CRT TV set:
One could perhaps make use of the tremendous amount of e-waste in third world countries. On second thought, having to disassemble a CRT does not exactly make it easy to assemble. Its still an idea though.
2) High Voltage Module: http://www.amazon.com/Ardokit-3-6V-6V-Module-Voltage-Generator/dp/B00K5OWR2U
I really like this module for several reasons. First, it is very affordable, and very easy to use. I recently received one in the mail, and it is very powerful for its size. Its the perfect HV supply for a supercapacitor, because of its low voltage, high current input.
My only concern is the lifetime of the device. I suspect that proper care of the device will result in a very long lifetime.