Way back in the day, HTML was implemented as an application of SGML. SGML was a quite complex markup format, that had lots of features that were kind of complex to implement, and so web browsers didn't actually implement all of SGML, just that which was necessary for HTML and the HTML found in the wild. However, the HTML found in the wild was frequently invalid, so browsers had to implement some clever rules to do something reasonable with invalid markup.

Eventually, people thought it would be nice to have a simpler, easier to parse markup format, with a proper specification, and developed XML. Of course, once you have a shiny new thing, people will come and start bolting things on to it, and so they added features like namespaces, and they included some of the worse features from SGML like the DTD (it turns out, you don't generally want a document to reference its own schema and entities; an application should know what schemas it can accept, and having a URL for a schema in a centralized location means lots of poor implementers would actually download that, and entity expansion is just a nightmare).

However, XML wasn't compatible with HTML, and it certainly wasn't compatible with HTML in the wild. XML parsers are required to reject any invalid markup. The W3C developed XHTML as a way to have a subset that could work in either HTML mode or in XML mode; the idea was that people could start moving to XHTML in HTML mode, and then once everything was cleaned up they could switch to XML mode with strict parsing.

The problem was, strict parsing was never a benefit for either publishers or users. One little bug somewhere in a template substitution which allowed an unquoted < sign could cause the whole page not to parse, and users just be left with an error message. Without completely changing how the majority of web apps worked, there was no way to ensure that all of your content would be strictly compliant XML without the chance of breaking.

In the browser world, XHTML was implemented, with strict parsing in XML mode, but almost no one used it.

At this time, the browser landscape was pretty bleak. Netscape had died. Mozilla was in the process of building their new browser on a completely new engine, but early versions were fairly bloated and slow; it required a rogue group of developers within Mozilla to fork just the browser portion without a lot of the other functionality to produce what is now Firefox. Opera existed but had a tiny sliver of the market share; it didn't help that the free version came with ads, or you could pay for it without ads, while all of the other browsers were just free. IE was dominant, and eventually captured a huge percentage of market share, and then Microsoft just rested on their laurels and pretty much stopped development.

Soon enough, Apple came around and forked KHTML to build WebKit and Safari. In doing so, they did a huge amount of reverse-engineering effort to make their browser compatible with all of the parsing and layout quirks in IE and Gecko; the standards are not at all sufficient for compatibility. This gave them a browser that could actually be used on the majority of web content. With the introduction of Firefox instead of the bloated Mozilla browser, there was actually competition in the browser landscape again.

Meanwhile, the W3C decided that the failure of XHTML to be adopted because it met no ones needs was not sufficient; they went on to start working on XHTML 2.0, a backwards incompatible change with maybe a few nice features for document publishing but which didn't address any of the needs that web developers wanted, like rich interactive web apps.

So the WebKit, Mozilla, and Opera developers decided to get together and actually start providing a spec for what would actually work in the real world on real web content. They called this group the WHATWG. The W3C was not interested in working on this; the W3C group insisted on continuing work on the backwards-incompatible XHTML 2.0. Inspired by the clever algorithm the WebKit developers had come up with for parsing invalid content, they actually wrote up something a lot like that into a spec (with some improvements), and this spec eventually was implemented by all of the major browsers, providing much more robustness and consistency between browsers in how content was handled.

Other features that people wanted, like the ability to draw on a canvas, were prototyped in browsers and specced out in the WHATWG group. Some features were adapted from browsers that had already implemented them; for instance, Microsoft had implemented XMLHttpRequest, which turned out to be hugely useful for interactive web apps, so the WHATWG wrote up a spec for this and other browsers implemented it.

Google eventually wound up forking WebKit for Chrome, then merging back in, and eventually forking again into Blink, and joined this group as well.

Finally, the W3C realized that the work is was doing was irrelevant. No one was interested in implementing XHTML 2.0. What everyone wanted were the new features the WHATWG HTML specification, new features like the canvas, and so on. So they agreed to take the current WHATWG spec and edit it into HTML5.

However, it didn't take long for this to break down again. There were people in the W3C who objected to some of the changes made in the spec for the purposes of matching up with the real world. For instance, there had been some accessibility features like the "longdesc" attribute which were specified as containing a URL pointing to a page with a longer, more detailed description of the item in question for accessibility purposes (something that could, say, contain markup, when "alt" wouldn't be sufficient for a description). However, no browsers had ever actually implemented any reasonable way to get to this, and a survey of web content found that even if you did try to implement it according to the spec, very little content actually used it, and a large amount of the content that used it used it incorrectly, pointing to broken URLs or just including plain text like the "alt" attribute. So the WHATWG spec dropped this, and recommended other ways to link to descriptions which would show up even without special accessibility tools. One of the problems with specialized accessibility attributes is that people who aren't using screen readers can't easily test them out, so it's easy to bit-rot, but providing a normal link and annotating that so that screen readers can link it to the image make it possible to see in a normal browser.

Anyhow, some people in the W3C objected to this, and so rather than just providing an edited version of the WHATWG spec, they started tinkering with the spec themselves, adding things back in, removing some things that had been in the WHATWG spec. The W3C structure is very bureaucratic, and it has all kinds of members who are only peripherally involved with any actual tooling for web development, so it makes it very easy for various people with big egos but no real skin in the game to get involved in the process, while the actual browser developers who would be implementing the features can be shut out of the process.

So, eventually the cooperation between the WHATWG and W3C died down again, with the W3C publishing what it wanted, and the actual browser vendors continuing to work on their living standard document that is a much closer representation of the real world.

And this seems to be another case of yet another attempt to reconciliation and break down. The W3C decided agreed to start with the WHATWG DOM specification, then decided to make some incompatible changes without very much justification, and is now trying to publish a new version.

I think that in a lot of ways, there's an ego thing going on here. The W3C was originally started precisely to specify HTML, CSS, and things like that. While their CSS working groups have managed to stay pretty reasonable and are willing to work with those doing the actual implementation, their HTML and DOM groups keep on being hijacked by people with particular agendas, people who won't work in good faith to try to resolve differences reasonably, and people who think that because they're the W3C, they "own" the spec and so think that the WHATWG is just a rogue group, as opposed to a group of the people who actually have the most skin in the game because they have to actually implement the browsers that billions of people use without breaking a hugely complex and diverse amount of content out there.