Rich Hickey (creator of the Clojure language) has a good talk "Maybe Not" that touches on these issues, with a nice way of explaining it

https://www.youtube.com/watch?v=YR5WdGrpoug

The capnproto link explains it concretely in terms of a message bus example, which is useful.

But more abstractly you can think of the shape of data (aka schema, names and types) and field presence (optional/required) as separate things

https://lobste.rs/s/zdvg9y/maybe_not_rich_hickey

First, being valid or invalid with respect to a static type system is a GLOBAL property of program -- writing a type checker will convince you of that. And big distributed systems don't have such global properties: https://news.ycombinator.com/item?id=36590799

If they did, they'd be small :) Namely you could just reboot the whole thing at once. You can't reboot say the entire Internet at once, and this also holds for smaller systems, like the ones at say Google (and I'm sure Cloudflare, etc.).

So the idea is that the shape/schema is a GLOBAL property -- you never want two messages called foo.RequestX or two fields called "num_bar" with different types -- ever, anywhere.

But optional/required is LOCAL property. It depends on what version of a schema is deployed in a particular binary. Inherently, you need to be able to handle a mix of inconsistent versions running simultaneously.

---

To be pedantic, I woulds say "making illegal states unrepresentable" DOES apply, but you can't do it in a STATIC type system. [1] Your Maybe<T> type is not useful for data that crosses process boundaries.

A distributed system isn't a state machine.

1. Lamport showed us one important reason why: the relative order of messages means that there is no globally coherent state. You need something like Paxos to turn a distributed system back into a state machine (and this is very expensive in general)

2. The second reason is probably a consequence of the first. You can think of deploying a binary to a node as a message to that node. So you don't have a completely consistent state -- you always have an in-between state, a mix of versions. And presumably you want your system to keep working during this time period :)

And that coarse-grained problem (code versioning and deployment) implies the fine-grained problem (whether a specific message in a field is present). This is because protobufs generate parsers with validation for you -- or they used to!

---

tl;dr Think of the shape of data (aka schema) and field presence (optional/required) as different dimensions of data modeling. Maybe<T> mixes those up, which is fine in a single process, but doesn't work across processes.

---

[1] A very specific example of making illegal states unrepresentable without static types - my Oils project uses a DSL for algebraic data types, borrowed from CPython. The funny thing is that in CPython, it generates C code, which doesn't have any static notion of Maybe<T>. It has tagged unions.

And in Oils we first generated dynamically typed Python at first. Somewhat surprisingly, algebraic data types are STILL useful there.

Now the generated code is statically typed with MyPy (and with C++), and we do pleasant type-driven refactorings. But using algebraic data types were still extremely useful before static typing. They made illegal states unrepresentable -- but you would get the error at runtime.