I think we haven't completely figured out software development yet, and its a bit of a mess :)

Yes, this is typical of today's software development process. The codebase may be good in terms of code quality, but it sounds like it doesn't have the best architecture.

Unfortunately, good architecture is extremely hard. The architect has to really think about the problem and come up with abstractions that are simple, intuitive, orthogonal, self-consistent, universal, not overwhelming, programmable (not configurable) and stand the test of time

Simple -> means that abstractions should only represent one thing. A rule-of-thumb test is: if you ask yourself what does this abstraction do or define, it should be a single item (not a list)

Intuitive -> abstractions that resemble existing abstractions that are in common use, as well as things in the real world. (or in the problem domain) For example, its possible to reuse a File abstraction for many things. Another example, UI classes should mirror things that are shown on the screen; in simulations classes should mirror the things being simulated, and so on.

Orthogonal -> abstractions should either introduce new functionality, or combine multiple functionalities into one - never both at the same time. For example, a function that reads settings from environment variables then calls another function with those settings is often likely to benefit from split into 2 functions: one to read the variables and another to glue together readVariables and callOtherFunction.

Consistent -> an abstraction should have laws that are never contradicted (they're factual, never changing), and users should only rely on those laws and no other properties. The best examples of this can be found in Haskell's standard type class tree.

Universal -> an abstraction should apply to all cases where possible. E.g. all mappable things should be mappable in the same way (Arrays, Tasks, Observables, etc) and all parts of the program should be able to take advantage of that

Not overwhelming -> The human brain can keep track of up to 10 concepts at the same time. The architect should keep this in mind and properly compartmentalise abstractions in a way that doesn't overwhelm. The fewer things you need to keep track of at the same time, the better.

Programmable, not configurable -> Whenever possible, good code is programmable, not configurable. For example, a chart plotting function could take a detailed configuration object that contains a description on how to format the labels, e.g. { decimals: 1, units: {1: 'm', 1000: 'km'}}, could allow the user to pass a formatting function that takes the number and generates the label text. The second one is always more flexible and extendable.

Some of these properties may sometimes be at odds at eachother.

A note about microservices: they're popular because they enforce some of these properties somewhat: most small services are simple, the API (interface) is given some thought to make it intuitive and consistent, they nudge towards orthogonality as inter-service dependencies have a much higher cost and are therefore avoided... However they have some downsides (e.g. they favour configurable instead of programmable, as its somewhat hard to pass code to be called by the microservice)