Today I had some time to run some pulse shape tests. I revised the bit generation function of my script several times to generate 3 waveforms: triangle, saw, and half-period sine. I tested each of these waveforms at 300 bits/sec. The top waveform was computer generated, the bottom was recorded to tape and replayed to the computer.
I noticed the deck changed the test waveforms with regards to DC offset. The way I'm encoding information generates a waveform that "rides" one side of the ground reference voltage for more than one half-period of the wave. This waveform is ambiguous on tape since information is encoded using changes in flux (rather than around a fixed reference). The resulting waveform "straddles" 0v where the source waveform is only on one side.
My fundamental misunderstanding of how tape works has been a good opportunity to learn the hard way. I can currently think of a couple ways around the limitations I've found. The first is to use a carrier tone to cancel out DC offset and somehow cancel out this carrier (perhaps using both stereo channels). The second is to modify the encoding scheme not rely on pulses on either side of 0v, but on cusps like the ones seen in the saw test.