I'm a scientist working on satellite data products at one of the two four letter agencies you named. I have published papers on these data products and myself work on processing of level 2 and 3 data products. A key point that needs to follow your comments is that various data products that are often purpose made for the goals of the people that use them... and that interpreting said data requires understanding the choices made, although this can be found in the papers that document the products. What this means though, in the long term, is that there's often more than one data-set for each parameter and they're all horribly documented.

[most data of this kind] .. is unfortunately mired in a sea of complexity, filled with shifting departments

NOAA works directly with NASA for much of this kind of data, and stewards it online

You seem to have a good grasp of the current state of "climate data" in the US (and Europe). It's hard for scientists to simply get enough money to launch a satellite and that money usually doesn't include a budget for "packaging and organizing data to be published". For the most part, NASA provides "official" data products for its satellites and then enters into partnerships with other agencies like NOAA (and other scientists) to use and host these data products. Often this means refining the NASA products into something more useful. Every grant I've seen that's creating/taking its own have had one line that says "will put data up on ftp" and nothing more.

Also, as you're likely quickly finding out... the storage and processing requirements are often "enterprise class". For quick reference, the level 3 product I work on (EASE-gridded daily means) is roughly 30TB. This is one product from one satellite for one type of observation. The level 1 data that's at native time/spacial resolution is something like 3TB per day. There are several agencies who work on a similar product with different goals, from the same level 1 data.

I can say in no uncertain terms that you're unlikely to see anyone organize these data sets any time soon. I admire you for attempting to tackle it. I'm not sure what your mission is but if I were to advise you myself (having seen 30 different agency sponsored hosting sites come and go) your time would be most wisely used in two ways:

1) If you're really determined to host the data yourself... find and choose what you think are the most important products from each satellite, archive those products and deeply document the sources (i.e. papers published and up-to-date links to related products that you aren't hosting).

2)Or what I think needs to be done: Create a web portal... like... a really good web portal. This doesn't exist, which is obviously the motivation for what you're doing. Something that cleverly organizes and documents the different types of climate data and the places that you can find them and the places/papers they're documented in. In my humble opinion, this is what needs done most. NASA/NOAA/everyone else will gladly host the data but they're never going to create a "climate data portal" that documents and catalogs data from the other agencies and sources.

Well, not never... they've tried. It's just really difficult, these are the ones I know:

https://climatedataguide.ucar.edu/ (NCAR Boulder, CO, USA)

https://rda.ucar.edu/ (NCAR "research data archive")

http://nsidc.org/data/search/ (NSIDC Boulder, CO, USA)

https://data.nasa.gov/ (NASA, USA)

http://cci.esa.int/data (European Space Agency)

http://www.geoportal.org/ (ESA portal)

http://www.ipcc-data.org/ (IPCC focused data sources)

http://www.ciesin.columbia.edu/ (CIESIN Columbia University)

https://iridl.ldeo.columbia.edu/index.html (IRIDL Columbia)

https://help.ceda.ac.uk/article/99-download-data-from-ceda-a... (CEDA, London UK)

the right hand doesn't know what the left is doing here

It's more so that the right hand has nothing to do with the left hand. Each agency, each satellite, each group of scientists, is totally independent of the other. While sometimes there's noble leaps taken to work together... there's not much incentive financially or otherwise. Not to mention that each team is so specialized that they often know nothing about the other 98% of the data. Take me for example, I've helped work on a very popular data set for a specific part of the earth system... I don't think I've ever once looked at ocean data... I don't even know what's out there or how good it is or where you find it.