Here's what to learn, leaving how to learn it up to you.

---- PROGRAMMING TERMINALS THEMSELVES ----

The standards that you need to have to hand are ECMA-35, ECMA-48, and ITU T.416 .

* Most of the parts of ECMA-35 won't apply if you target UTF-8 terminals. The standard in the most part deals with switching 7-bit character sets in and out, the "ISO 2022" mechanism that you may have heard about that makes terminal output illegible. But ECMA-35 is still the explanation of escape sequences as a general concept, even though character set switching using some specific escape sequences is becoming a thing of the past.

* ECMA-48 deals mainly in control sequences, which are not escape sequences, and do not necessarily even begin with escape sequences (since in this almost wholly 8-bit-clean world C1 control characters can be sent as-is). The structure of control sequences is quite regular, and explicitly allows for vendor-defined "private" control sequences.

* The relevant parts of ITU T.416 deal in the SGR control sequences for colour changes beyond the 8-colour model of ECMA-48. These deal in "indexed" and "direct" colour, i.e. 256-colour and 24-bit colour in practice. Note that there's a very common misrepresentation of these control sequences, that fails to have colon-separated sub-parameters and fails to then get the sub-parameters right; so don't consult WWW samizdat sources on these control sequences. Terminal emulators have been moving to get this right over the past few years, coming to understand the correct control sequences, but that hasn't stamped out or fixed the reams of incorrect doco floating around the WWW and elsewhere for almost three decades.

The extensions to the standards that you need to be aware of are:

* The DEC VT extensions, most of which are private control sequences in the form that ECMA-48 designs in. These range from area attribute fills (DECCARA) to cursor shapes (DECSCUSR). The programmers' reference for the DEC VT525 is a good reference for this, but note that no terminal emulator actually fully emulates a DEC VT525.

* The AIXterm and DTterm extensions. AIXterm extended SGR control sequences from the ECMA-48 8 colours to 16 colours. DTterm repurposed (at the time) almost never used parameter values in a private DEC VT control sequence (DECSLPP with 0 to 24 lines per page) to mean entirely different things, related to DTterm's graphical user interface.

* The Xterm extensions, relating to mouse reports that aren't in the style of the DEC Locator, and other things. Note that the XTerm mouse report formats evolved, as problems were discovered. The one enabled/disabled by Private Mode 1006 is the most sensible, albeit that some mouse button up events confuse VIM, and DEC Locator reporting still works and is almost as good.

Also note that it is ECMA-48 in both directions. Most softwares, including ncurses, get this wrong and do pattern matching, but keyboard input is in fact encoded as ECMA-48 control sequences. In particular, note that the DEC VT conventions, laid out the VT52x and VT42x programmers' references (You have to consult both for the full picture.), encode modifier keys as control sequence parameters. DECFNK is a control sequence to be decoded in the ECMA-48 way, not a pattern to be matched; and furthermore no terminal in practice uses the FNK control sequence from ECMA-48. Also note that (U)RXVT has a totally botched input encoding system, best addressed by fixing the terminal emulator.

---- UNIX TERMINAL I/O ABSTRACTION ----

On the Unix side you need to learn about the line discipline, how to temporarily turn almost all of its mechanisms off whilst your TUI is active, and how to properly handle SIGTSTP, SIGTTIN, SIGTTOU, and SIGCONT signals. You also need to learn about the TERM environment variable, and fallback conventions to turn complex user interface features off like TERM=ansi and TERM=dumb .

You moreover need to learn something pointed out at length in the XTerm FAQ: where terminal emulator doco says that it is XTerm-compatible, it is always an outright falsehood; and people who say that TERM=xterm is a universal are very wrong.

---- UI LIBRARIES ----

ncurses is a well-known and well-trodden library, and you could indeed uses ncurses for a player, as someone in this very discussion already has.

But it has a whole bunch of warts, a lot stemming from it being built upon the terminfo database, whose way of encoding terminal type differences does not match how real terminals and terminal emulators (since the 1980s, and surviving into the 21st century) in fact work. There isn't a "cursor addressing mode" that can be switched on and off: actual (non-block) terminals either always do or always do not understand control sequences to move the cursor. There's no notion of a numeric keypad that sometimes does not transmit: rather there are numeric/calculator and cursor keypads that are switchable between two sets of control sequences, application and normal. Keyboard input isn't fixed strings to be matched: non-glyph keys are encoded as escape sequences, or as control sequences with modifiers as one of the (variable, not fixed) parameters.

Then there are the dissimilarities of ncurses and terminfo abstractions with the modern world. Modern terminal emulators are gaining the ability to have multiple underline styles (originating with Goyal kitty but spread to other terminal emulators) and to support the ECMA-48 framed and encircled attributes. Boldface and faint are generally not mutually exclusive, in the ECMA-48 style, and this is not ambiguous any more; nor (with one prominent exception) are these attributes conflated with colour changes. Blinking has gone out of fashion, thanks to devices enjoying low power sleep modes. Paper terminals with half-lines, NLQ/draft modes, and whatnot are almost never targets for interactive application user interfaces.

In addition to people avoiding ncurses because of its terminfo basis, people who program in languages other than C and C++ also often use other libraries, written for those languages. There are non-ncurses TUI libraries for languages ranging from C# to Ruby. And there are quite a few of them. Sadly, some do better than others at handling TERM=dumb and suchlike, and some still employ some of the same mis-matched abstractions.

---- PSEUDO-GRAPHICS ----

For how to do pseudo-graphics, learn:

* Block characters, Box drawing characters, Braille characters, and the (recently added) "Symbols for Legacy Computing" in Unicode. The last, on the assumption that Unicode version 13 support becomes widespread, in particular provides the way to have some GUI elements like titlebars and scrollbars in text mode in the way that the Apple IIc did (i.e. MouseText).

* Sixel graphics, which comes from real DEC VTs and which is understood by a fairly broad range of GUI terminal emulators (e.g. genuine XTerm, MinTTY (per MinTTY issue #572), Aminal (https://github.com/liamg/aminal), iTerm2 (per iTerm2 issue #3240)); but not by all and not, of course, by TUI terminal emulators.

* ReGIS graphics, similarly from DEC VTs but far less widely understood by emulators.

* The Tektronix 4014 graphics system. In the 1970s and 1980s, people actually did split (or alternating) graphics and text on real terminals, for CAD and other things. The 4014 system is what survives. This is emulated by genuine XTerm, by MinTTY, by TeraTerm, and by VersaTerm. TeraTerm and VersaTerm are full, serial-line-capable, emulators of remote terminals; not merely local terminal emulator programs for Unixes and Linux-based operating systems, which do not use serial terminal devices but only pseudo-terminal devices. Most local terminal emulators don't emulate Tektronix 4014. You'll find a lot of doco saying that in fact XTerm is the only local terminal emulator that supports Tektronix 4014 graphics. For a long time (decades) this was true, but (per MinTTY issue #896) MinTTY gained support for it in June 2020.

Whilst there are several video players that use various libraries to transcode video into coloured Unicode, and some that transcode into Sixel, a video player that uses Tektronix 4014 graphics seems to be a niche that has not been filled yet. I couldn't find one, at least. A video player that used both XTerm windows, DEC VT and Tektronix, would be an achievement. An audio player is a more modest first goal, though, yes. (-: