The main criteria for picking components was cost. Fortunately I had recently discovered Tayda Electronics, which seems to specialize in cheap hobby level electronics. I bought all my components from either them or Digi-Key.
These are usually the most expensive parts since they have moving pieces. These include the driver, pots, and switch. It was easy to find the cheapest slide switch on Digi-Key, but their selection of pots and drivers was actually pretty small. Fortunately Tayda has a selection of these kind of parts. I was surprised at how cheap their pots were, since I'm used to seeing them cost multiple dollars/part and Tayda had them for 67 cents. They also had small mylar speakers for under a dollar.
There weren't a lot of requirements for the regulator. 8Note is powered by batteries, so the regulator doesn't need a high power supply rejection ratio. The current draw of the circuit is relatively low, so the max current output can be low .
I bought the absolute cheapest one I could find, LP2950-33LPRE3 from TI for about 60 cents/unit on Digi-Key. Generally I try to stay away from through hole components because they take more time to clean after soldering, but cost was more important here.
I wanted some sort of surface mount battery holder, but apparently it's cheaper to add actual wires instead of just making the wire terminal contacts extend past the plastic. I guess most applications don't involve mounting the holder directly to the PCB. I bought ones with wire, and found a way to solder them that was pretty clean looking.
I haven't found a PCB house that can beat JLCPCB for small quantity, simple orders. I think OSHpark has higher quality, but for basic boards you don't need it. A basic board from JLC "costs $2" for 5 pieces, their advertised price. You can usually get 10 for a few dollars extra. They tack on $25 for shipping and there's an import fee that's around $15, but it's still cheaper than OSHpark for anything larger than a few square inches.