TL;DR: I'm about to go on a lengthy rant, because I too have spent over a decade obsessing over this subject. I feel like I've been starting to actually get somewhere, but there's still a lot up in the air. Also my meds are wearing off, so all eighdy HD's are present. Read at your own risk...

-----

It's a game of compromises. The first problem is to decide what goals you value most.

The current compromises are decided by the goal of backwards compatibility. In that sense, we are "steeped in tradition". It's a mess, but at least decisions have been made; and we are recognizing - right here, right now - just how significant that fact is on its own.

> For "TUI" apps: something like a GUI toolkit, but should remain reasonably performant over a poor connection

That sounds a lot like what web browsers are; which reminds me of the sheer magnitude of options we have available. It's easy to get stuck with decision fatigue. We aren't making a universe simulator here, so we need to break this project down to a smaller domain. The constraints of the problem domain are what define a project, so pinning them down is 90% of the work.

I'm thinking back at what my motivation was in the first place: frustration with the status quo. Having a simple text UI interface is evidently useful. The shell is the first and last thing I use: it's my home, and that's why I care so much in the first place. Something with generally the same constraints of a terminal emulator & shell is what I want. My frustration lies with the complexity in both the terminal emulator itself, and the way a shell (and every other terminal-based program) interfaces with that. So what's the bare minimum that these two things need to function?

First, the terminal needs input and output. But these needs are really just an extension of the text-based program's needs, right? So what does a shell or a text editor need to function? Well, it needs input and output. Hold on now, do we really need the terminal emulator? Why not have the shell do that?

Well, there are some reasons: Shells can be run without any IO. Inside shells, we can pipe programs into each other. Those pipes are made of text. Is that the domain we want? Well, using text as the interface is one of my main frustrations with terminal emulators: even the ASCII table itself shoves key-presses into the domain of text, and that's an ugly hack I don't want to keep. There are functional needs for dropping that, too: "ctrl-m" is the same ASCII code as "enter", and keyboard layouts aren't always very compatible with ASCII in the first place. There are too many assumptions baked in here.

And while we are on the topic of ASCII, escape characters are the ugliest hack in here. On the terminal, we use them for input - to extend the ASCII table to accommodate a few more keys - and for output - for decoration like text color, boldness, and even flashing background. It's like a markup language, but made out of non-drawable characters that were supposed to be typed with the escape key. There are so many ugly edge cases here, you could write a novel about them!

But shells aren't always about plain text. You can pipe gzip to dd or a file, and it will totally work. But if you pipe it to your terminal output, weird things can happen. That's because some raw binary looks like escape characters, and we are already solving that problem by making user input a separate thing. Progress!

So what do we do with keystrokes? Let the user decide! But what data will end up getting sent to the program? The status quo is to be a REPL by default. That's the foundation for a lot of the ugly assumptions we just dropped. It's the reason that the OP has two whole chapters just on disabling that behavior. But most programs we invoke from a shell are either printing, or running a REPL of their own. And a lot of them are bad at it, which is why GNU readline is a thing. There's even rlwrap, which wraps a program in readline! That's very helpful, but I would love to never need it again. Plus, readline makes some UI decisions of its own, like its emacs-like key bindings, and I would rather it not.

So the big question here is about how UI should flow. That question gets me all kinds of excited, and all kinds of overwhelmed. This is where I start shifting my focus away from shells and back to text editors.

---

Emacs has its own shell. But Emacs isn't a dumb terminal emulator. It's built out of lisp, which represents data as text in a much cleaner way than ASCII escape sequences could ever dream of. And the way its interface flows is definitely more interesting than a REPL. In fact, some of the crazy shit you can do in eshell is one of the main things that get me thinking about shells and terminals and the ways they hold me back.

And if you really want a good example of how text-based UI can flow, look no farther than Vim. The modular flow is elegant and efficient in ways that the primitive REPL could only dream of.

Yet both of these wonderful programs bring me back to the very same frustrations I have with terminal emulators and shells: they are steeped in tradition!

Vim was my first. I was happy to sin against the Church of Emacs, the stubborn atheist I am, and vi is a seductive mistress. But while vi could do most anything I wanted, and it could bend and extend in many exciting ways, I still managed to find its limits. When I started typing with a non-standard keyboard layout (workman), I tried remapping Vim's keys to compensate. First you remap a key, then you remap the one it replaced, then the next, etc... and eventually you hit a circular dependency. It can't be done. So I decided to take the opportunity to learn me an Emacs.

Emacs is incredibly customizable. If you can navigate your way into its elisp structures, you can mold it into anything you want. But what Emacs already is by default may be easy to decorate and add to, it's not so easily unmade. I had big ideas about making my own vi-style modular editing keybindings once, but I never could quite find my way. Every move I made, I found myself tripping over what was already there; what I had come to replace. The whole time, in the back of my mind, my conscience screamed, "We just need a clean slate! Emacs without the defaults. VoidMACS."