Eric HertzApparently the answer is "Just don't".
I'm too tired of the whole scenario to go into it. But let's just say that some once-standards seem to have been brushed-aside in really awful ways that make me lose faith in far too many things.
I commented on it in a previous log... Basically having come across several linux mailing list archives of folk submitting patches to support Xon/Xoff for various USB-Serial dongles (whose chips actually support it at the hardware level!) and yet those patches being abandoned for essentially "most USB adapters don't support it" despite the fact the drivers claim to. AND, the fact that the stty-default, when you plug those dongles in, is to claim that it's enabled. Worse than that, there were even folk submitting patches to at least give a *warning* to the user that xon/xoff isn't supported, and even *those* patches were seemingly driven out of the kernel.
I also found this: https://hackaday.io/project/16097-eforth-for-cheap-stm8s-gadgets/log/49010-a-serial-terminal-for-linux-with-working-xonxoff
Wherein Thomas went into a lot of digging to figure out (and share) where the hurdle exists...
...
This is now the /third/ such thing I've run into that was basically once such a standard as to be in nearly all the great references that were de-facto reading for generations of folk using RS-232.
The first was long-enough ago that it's a bit muddled in my mind. As I recall it had to do with counting edges on handshaking lines. The great example I recall of its disappearance (and yet claiming to still exist at the driver level) was a GPS standard which is often used to synchronize systems' realtime clocks, e.g. data-logging systems which aren't able to connect to the internet... Like what? Think trailcams if you can't imagine scientific research. Isn't that pretty much *exactly* what linux was once so great for? It blew my mind how, frankly, rude folk were toward this guy, and indirectly toward the entire scientific community, for not using things "the one way" "everyone" does. Goes ENTIRELY against everything that I thought made linux great.
....
The second was custom baudrates.
The Standard, for generations, was to assign your custom baudrate to 38400baud. The Idea being that nearly every terminal application, or serial application, supports selecting that baudrate from a list, whether supplied by the OS, or in a list in the program itself. Thus nearly *every* serial program could make use of a custom baudrate, as long as you configured it before loading the program.
Yes, at the driver-level that might mean running a custom program to actually set the appropriate registers, but even that had become commonplace enough that linux has provided the appropriate and common tools to do-so, for decades; one for countless different serial chips. EXCEPT. USB-Serial dongles. Why? Searches of mailing lists result in pretty much the exact same sentiment, over and over... "Most USB serial chips don't support it" which, frankly, wasn't even true, in my experience, decades ago, and far less today. AND, again, the drivers seem to allude to the support being there, and configuration-programs give no warning it isn't.
Again, this isn't just about buying cheap hardware, we now live in an era where USB is darn near the only reasonable option. This is downright absurd. Their arrogance is affecting everyone from kids learning Arduinos to government-sponsored research endeavors. Nevermind the folk who put tremendous effort into making great software that stood the test of time.
And, nevermind the folk who wrote excellent resources/reference-manuals that we still are referred to as "The Best" now in an era where those things we learned are now just flat-out lied-about still existing in our configuration utilities and drivers.
...
The third is XON/XOFF, and again, frankly, I'm still seeing red that I spent a week or more implementing that in an embedded project *because* stty reports Xon/Xoff is enabled As Default when I plug in my USB-Serial dongle from a very well-respected chip manufacturer whose documentation is incredible for this era, AND, has apparently gone out of their way to support Xon/Xoff *at the hardware level* for exactly the reasons of today's huge buffers and USB packets meaning that a CPU might not receive the XOFF until plausibly hundreds of bytes (even at 9600) or more have already been queued. WHICH would be FAR too much to expect for most of the very hardware that these things exist/ed/ for. The resounding sentiment: "Most USB dongles don't support it". RIGHT. But: Linux has darn-near always been about supporting *many* types of equipment, terminals, even teletypes. Nevermind specialized stuff that was never cheap, nor abundant, and is hard to replace like plotters and test equipment... If your equipment needs something like that, *linux* is "The Guy!" Until the USB-era, it seems.
Sure, your $20,000 piece of test equipment *could* run on even those serial dongles, if someone bothered to write the software for it (send one byte, wait for a little bit, make sure there's no XOFF, heck, do it right and the driver could get a good measure of the device's buffer size, send a burst of 1/4 as many bytes, wait, look for xoff...) There ARE solutions.
But, worse than those solutions' not existing is the fact that the ability to make our own solutions is downright hobbled by that recurring "most don't support it" mentality. You can write a chip-specific /driver/ patch to enable these things, but connecting that to the kernel, in exactly the way the kernel's serial documentation explains, means going through the generic "USB-Serial" driver, which doesn't, though certainly could, support it. What The What?! Since When is the friggin' kernel source full of such laziness and inconsistencies (and frankly rude thoughtlessness) as to not try to make every option available?! I mean, seriously, I can connect some friggin' random I2C port-expander device to the I2C bus dedicated to my DIMMs, and access its pins through /dev/gpio, but I can't friggin' use the *default* options pretty much in every manual on RS-232?!
The worst part is the flat-out lying involved. The driver says it's supported. In fact, it's default. Can you imagine the number of man/woman-hours *wasted* world-wide trying to hunt something like this down, Over and Over again?!
Wait, is that the worst part, or is the worst part that SO MANY /have/ hunted it down as to there being NUMEROUS folk submitting patches, and their hard work, given freely to everyone, being snub-nosed?
And did I mention the horrendous backward-step in rendering-useless multi-generationally-tested/fine-tuned/bugfixed software and reference manuals?
I CAN'T BELIEVE what I've seen.
Discussions
Become a Hackaday.io Member
Create an account to leave a comment. Already have an account? Log In.
it never occurred to me that the USB adapter itself might implement flow control independent of the host application.
Anyway, I would think that in this case that you'll find hardware a bit more straightforward because:
1) software flow control is great when you only have RX and TX but no status, but in this case you do
2) hardware flow control obviates the need to escape or otherwise limit the data transmitted -- binary is fine
3) hardware tends to be more immediate, whereas I think you've noticed some issues with potential buffering and the need to send an 'expedited' XOFF and have incurred some extra complexity in doing so
But hey, it's a voyage of discovery.
Are you sure? yes | no
Ugh. I spent two friggin hours in responding. Then my phone died while it was plugged in.
Long story short: "RTS" can mean *either* the sender of the RTS signal is requesting to send data to the other device OR the sender of the RTS is requesting that the other device send data to it... completely contradictory, yet equally-valid interpretations of the same wording.
The "standard" is ambiguous, and many an intelligent and experienced graybeard have written highly-respected references on the matter.
Which, I guess, really brings me to the point of using standardish three-wire hacks, like enabling a half-second pause after the host sends a line-end...
If high speed transmission is really important, then it seems I'd have to write the host-side software, too... But, at least for the ihex-flasher, I'd rather it be compatible with standard tools, even if it means non-default settings and some slowdown :/
Are you sure? yes | no
"The standard is ambiguous" in *many* such ways, not limited to RTS which was just an e.g.
And I really should go find and link (and print) those graybeards' reference-manuals. I've visited them countless times over the past two+ decades.
Are you sure? yes | no
Ooh, it's giving me more time!
The thing about hardware flow control, at least over USB, is that it [just like Xon/off] really depends on a lot of factors that many chip designers and driver writers can have interpreted things differently than many others... If I "get it working" with my USB-serial chip, that doesn't necessarily mean it will work with another. Not due to signal-purpose interpretations, but due to e.g. the size of the chip's transmit-buffer, the driver's implementation of interrupt-polling regularity, whether a hardware "stop" signal is expected to be handled by the CPU vs. by the chip itself, etc... a 128byte receive buffer is really not much... one line of text leaves very little space for catching the bytes transmitted after a "stop". OTOH, maybe I'm just too stubborn, but I feel like a buffer just a tiny bit larger than a line of text *should* be plenty, considering how many devices out there deserve to have continued life in this new era... So, I guess, I'm trying to find some sort of "standard" solution that fits within the absurd limitations the USB-Serial team have thrust on us.
Are you sure? yes | no
Some ideas include host-side hacks... E.G. maybe somehow tricking the host *not* to fill the transmit buffer. If USB 1.0 can handle 960 packets per second, the buffer would only ever be one character deep... and surely a stop of either sort could be downloaded via USB, responded-to, uploaded again via USB, and responded-to by the chip within a few 960ths of a second. 4 bytes slipping through is nothing compared to the 128 in a chip's tx buffer if that chip, e.g., uses the same queue for Tx as it does for processing USB commands (which seems stupid, but is *entirely* plausible from some manufacturers)
Are you sure? yes | no
Yes, I'm afraid my UART experience predates USB adapters, and as an 'adapter' things can be a little different than a true UART. We just crossed RTS->CTS in both directions and called it macaroni.
The notion of allocating a buffer you think is big enough and hoping for the best gives me the heebie-jeebies.
Are you sure? yes | no
I agree about those heebie jeebies, whole-heartedly, which is why I've put so much time into this... Seems like there *must* be some common-denominator.
I've looked a bit at stty and termios's docs... It seems many of the POSSIX features/options are, actually, not even implemented in linux, period. Not merely limited to xon/xoff, nor to USB dongles.
This, actually, is rather disheartening to me. How many times I've had to parse that whole manual, I'd gleaned plenty of insight into capabilities I might've planned to use in future endeavors that just flat-out aren't implemented, without its saying so. E.G. cr# inserts a delay after every carriage return. Looking into Oracle's manpage gives more detail about what that means. cr3 makes the delay 150ms. For, logically, allowing a teletype time to return its carriage! Brilliant, POSIX-standard... And useful here, where each carriage-return in the ihex document ends a line of data (16 bytes in hex with a header is about 3/4ths of an 80col line, which is the most zmac seems to output per line). 150ms should be just about perfect for the worst-case where 16 of those lines have been buffered, so then get programmed into their flash sector, which is a slow process (tens of ms, as I recall).
BUT, realistically, I guess its lacking isn't so bad toward our goals: It would've been a heebie-jeebie hack, only considered since xon/off is also lacking, and cts/rts (believe it or not) is actually mentioned in the manpage as "not in linux"... (which *really* boggles my mind, because, like you, I'd thought that, over all methods, would've been the single most likely to be considered standard! I almost don't believe it. How else would linux have talked to modems?!).
Also, realistically, and sorta brought to the forefront by all this: my goals kinda relied on another heebiejeebie-factor, which was: no ability for the device to indicate to the host if the checksum failed, and to request a resend.
For this, though, I'd already had an idea in mind... It'd just send a "[not] OK" at the end. Retransmission of the whole shebang would've been fine. I'd already planned to only write a sector if its previous contents differed. The whole process is maybe 30sec, so no biggy.
I guess it's not a big deal to work around these lackings. In some ways it makes things easier. But avoiding those heebiejeebies is now much harder, so I fear this progression may continue, wherein each step of the way new-folk become increasingly-inclined (unaware of alternatives!) to not even consider our heebiejeebies :/ Which, then, could mean hardware designers will get away with increasingly heebie-jeebie-inducing designs, and a spiral into chaos. Heh.
Also, it really bugs me that long-established softwares and references are now rendered somewhat worthless, and even misleading. It took great amounts of knowledge, experience, and probably revision after revision to get those things as trusty as they were for decades. And now to just throw them under the bus, with basically things like "stack-exchange" as the only reference... Don't get me wrong, their info is often reliable-enough, especially when it comes to current trends, but unlike a reference manual, wherein folk are exposed to thinks they hadn't thunk, with things like stack-exchange as one's only remaining reference-material, one has to know what they're looking for, as they re-implement, from scratch, the trusty tools they're replacing.
I think it's a "very disheartening" trend :/ (which is to say I've spent hours yelling at my walls about it, daily, for nearly a week).
You, my friend, are spot-on. RTS/CTS *should* be defacto. And I'm willing to bet you gained that knowledge through years of experience, turning that knowledge into wisdom. Again, I think it's truly a shame how little regard there seems to be for that, these days.
Taken further, the learning-curve is tremendously higher... (imma save this and come back)
Are you sure? yes | no
The learning curve *for doing things reliably* is much higher, now. E.G. if it takes three USB packets to respond to a NOT-CTS, then the huge buffer at the chip could have been refilled at least twice after the !CTS signal. (and, of course, 115200baud is increasingly-common as a 9600 replacement) So, now, the device has to have a *huge* buffer, (heebie-jeebie-assumption on the host's part), or, better, a way to request retransmission. The latter, basically, requiring the embedded-device programmer to *also* program a host-side application. It means somehow recognizing *where* the good data ends... Now we're talking CRCs. This, *all*, now, is basically being *required* of newbs just wading into the pool!
It's insanity, frankly. En masse.
I remember hooking up a stepper motor to transistors on my parallel port. Programming in BASIC. I quickly learned of steppers' weird speed/torque curves. The entire project was probably two days from a book that talked about doing night-ranger LEDs. I didn't have any idea of "poke" before I started.
Things don't *have* to be as hard as they are, now!
As yet another example: Arduinos are great, but there are *many* more layers to learn before getting that stepper spinning. Many more intermediate pieces of hardware, just to turn on a single LED. And, frankly, it *encourages* not understanding things that are core. (what friggin' language is that, anyhow?!) Relocating so much of the learning-experience from the low-level universalities to instead learning the environment/hardware-specific API, itself... The list goes on. Then, to the libraries, which again are great, but if you think about the user-to-contributor ratio, we're again talking about a very select few (often very experienced) who even understand SPI (nevermind registers) enough to program a well-documented SPI LCD with onboard framebuffer and drawing routines. Again, I'm glad graybeards are making such things available, but it really doesn't encourage *understanding.* And again, when those Arduiners become graybeards, themselves, it's no wonder we can't have CTS/RTS anymore. No wonder my nieces can't just poke 0x3F8 to turn a stepper motor. And so on. :(
ARGH!
Are you sure? yes | no
This all, really, has quite a bit to do with *why* this project (and the TI-86, and the AVR-8088 substitutuion, and so many future ideas) intrigues me so much.
It's also why I was quite pleased, earlier, in discovering that C could be used, at a *very* low level... *long* before some sort of "BIOS," serial, or even RAM is implemented. Making way for, maybe, a bit of a "universal" approach in learning/"bringing up" whatever CPU/microcontroller one might have...
Maybe inspiring a little bit of looking at the assembly-output, in the process of hand-modifying the output for the device-specific reset-address, without having to learn a whole bunch of chip-specific assembly just to blink an LED... or quickly thereafter drive a stepper.
I dunno, I kinda dig it. Maybe it's crazy. Still have to think about things like how to load that code into the ROM (and whattabout microcontrollers?). A device-specific USB programmer, or even the need for a generic external chip-programmer, seems kinda against my intent. OTOH, when I started getting into AVRs, the opensource (albeit purchased) USB-Tiny-ISP quickly replaced the bitbanged parallel-port dongle I made from others' shared work. Bitbanging parallel is definitely, now, an even bigger ordeal for most than the chicken-egg problem of needing a programmed microcontroller to program a microcontroller.
(Interestingly, I recall someone here figured out how to "bitbang" USB, at *very* slow speeds; directly wiring an SPI device to the USB pins, as I recall. The trick involved sending "resets" to the USB controller chip... If the chicken can be fabricated from nothing more than an old USB cable and maybe an hour's upload to a firmwareless homebrew USB-Tiny-ISP-alike, there may be something there... hmmm)
I dunno if any of this will pan-out. It may be one of those off-n-on many-year endeavors.
In the meantime, for the firmware uploader, I think I've settled on the three-wire approach and using a four-liner bash script to insert a pause between sending each line of ihex. Hokey, but enough to "bootstrap" in-system firmware-programming
Are you sure? yes | no
A few days later... well... Yeah, I've basically settled on host-side "custom" software to resolve many of these matters.
Think of a teletype needing time to move the carriage to the far-left after receiving a "carriage return" signal. The POSIX standard is one of three options, the longest of which is 150ms. What, then, does one do if the particular machine needs two seconds?
The answer isn't really that complicated. A simple bash script could parse the text line-by-line and insert whatever delay is necessary.
Sure it not only works, but indpires a bit of "out of the box thinking," (aka "hacking")...
But...
I fell asleep writing this and can't parse it now.
Are you sure? yes | no