They didn't explicitly decide to break compatibility. They actually put a lot of effort into maintaining it; the breaks in compatibility are forced by v4-only software/hardware/protocols. Unless you have some way to deal with those, you can't improve compatibility over what v6 already does.

> There is nothing impossible about designing compatible system, if the compatibility is your goal. Let's think about "simple address expansion" - there is a "IP-long" ethertype which is used with IP addresses longer than 32 bits; and all code is structured such that if IP address fits in 32 bits, then old IPv4 packets are used. All apps are hardcoded to use a single address family (IP-long), but as if you are running in IPv4-compat mode, only last 32 bits of address would be nonzero.

This is basically IPv6. It has its own ethertype, and software can be hardcoded to use v6 sockets only, with v4 addresses being represented by e.g. "::ffff:8.8.8.8". Using addresses with that prefix generates v4 packets, and IPs outside of that prefix generate v6 packets.

But how do you handle software that doesn't support it? That's where the compatibility gets broken, and you didn't propose anything to help with that.

> We could have IP-long capable devices operating in IPv4 compat mode for years.. until one day the assigned address changes to be longer than 32 bits and now the new protocol is in effect. Since majority of kernel code would be the same for IPv4 and IP-long packets, there was a good chance it would "just work".

Yes, this is IPv6 again, you can absolutely have v6-capable devices working in v4 compatibility mode for years, and then you can stop assigning v4 on the network and start assigning v6 on it to cut over to the new protocol.

But again... how do you handle devices or software that don't support it? Addresses longer than 32 bits will never "just work" with v4-only stacks. It's just fundamentally not something they can do. And most kernels already merge most of the code in their v4 and v6 stacks, so that's not a new suggestion.

> ARP

You're right that ARP could have been used for v6, but doing so wouldn't have made v6 any more compatible with v4 devices because those devices couldn't handle the new PTYPE or the addresses in it. (It would have saved people from needing to write NDP implementations, but how many people do you know who are blocked on deploying v6 because they need to implement NDP?)

The compatibility problem was imposed by v4 devices, so it wasn't an explicit choice in v6. It did present an opportunity to use something other than ARP though, which we took not because somebody didn't like ARP but because it fixed a problem we were having: ARP doesn't scale to network segments larger than a few thousand hosts due to all the broadcast traffic, which was a limiting factor in network size. You might think "nobody runs VLANs with more than a few thousand hosts", but... this is the reason they don't.

> DHCP

Fun fact: DHCP was released at around the time v6 was in development, and it wasn't ubiquitous on v4 networks until the late 90s. DHCP seems like the obvious choice to you now, but at the time v6 was being developed it wasn't popular.

v6's main autoconf was based on things like IPX (which was considered much easier to configure than IP), so they weren't starting from scratch. This wasn't breaking compatibility so much as not knowing what to be compatible with.

Using DHCPv4 options to configure v6 machines wouldn't have made v4 and v6 any more compatible though, because you still have the issue of v4 machines not being able to handle v6 addresses. You can already mix and match clients on a network with v6, but how can you expect the v4-only ones to talk v6? Mashing the config protocols together wouldn't change any of this.