The main criteria for picking components was cost. Fortunately I had recently discovered Tayda Electronics, which seems to specialize in cheap hobby level electronics. I bought all my components from either them or Digi-Key.

Mechanical Parts

These are usually the most expensive parts since they have moving pieces. These include the driver, pots, and switch. It was easy to find the cheapest slide switch on Digi-Key, but their selection of pots and drivers was actually pretty small. Fortunately Tayda  has a selection of these kind of parts. I was surprised at how cheap their pots were, since I'm used to seeing them cost multiple dollars/part and Tayda had them for 67 cents. They also had small mylar speakers for under a dollar.

Linear Regulator

There weren't a lot of requirements for the regulator. 8Note is powered by batteries, so the regulator doesn't need a high power supply rejection ratio. The current draw of the circuit is relatively low, so the max current output can be low .

I bought the absolute cheapest one I could find, LP2950-33LPRE3 from TI for about 60 cents/unit on Digi-Key. Generally I try to stay away from through hole components because they take more time to clean after soldering, but cost was more important here.

Battery Holder

I wanted some sort of surface mount battery holder, but apparently it's cheaper to add actual wires instead of just making the wire terminal contacts extend past the plastic. I guess most applications don't involve mounting the holder directly to the PCB. I bought ones with wire, and found a way to solder them that was pretty clean looking.

PCB

I haven't found a PCB house that can beat JLCPCB for small quantity, simple orders. I think OSHpark has higher quality, but for basic boards you don't need it. A basic board from JLC "costs $2" for 5 pieces, their advertised price. You can usually get 10 for a few dollars extra. They tack on $25 for shipping and there's an import fee that's around $15, but it's still cheaper than OSHpark for anything larger than a few square inches.